This is actually very important. Most people are not racist, and do not object to having an Asian guy, or a preppie Black dude, or a proud Orthodox Jew, as neighbors. 

The people pointing this out are not the people "misreading the room." Ranting about "curry monkeys" - or "bitches and sluts," etc - is stupid and offensive to most citizens. It brands YOU as an enemy.