Are people angered when they see Muslims in and/or near Ground Zero?

I hear that a Mosque in Ground Zero is a slap to the face of America and the victims of 9/11... but its just a building. I know for a fact that many Christians believe that their church is not the building itself, but the people that are comprised. The building is just used to facilitate. So that being said, wouldn't a Muslim in Ground Zero be a slap to the face??? After all, they represent Islam more than a Mosque ever could. In other words, if you're going to block a Mosque from being in ground zero, you might as well block Muslims as well. Right???