It's a community center, which contains prayer rooms, where Muslims pray. So really, what it comes down to is certain people* don't want Muslims to pray too close to the World Trade Center. That's insane.
Think of the contrast between the way conservatives high-fived each other over the keyboard courage displayed during "Everybody Draw Mohammed Day," and their response to Rauf, whose own patriotism and intellectual challenge to those who would do terrible things in the name of Islam was ignored in favor of using his criticisms of U.S. foreign policy to smear him as a terrorist sympathizer. This whole incident has been an absolute disgrace, an indictment of American tribalism, craven political manipulation, and outright bigotry. Rauf's detractors had little actual evidence on which to base their accusations, but they didn't need much, because we let our fearful imaginations fill in the blanks. — Adam Serwer
* It's pretty bad when Pat Buchanan gets to play the Godwin's law-yer.