Well, sometimes they're offended that they can't understand what people are saying, or they're offended at the inconveniences that sometimes arise when you're forced to interact with someone who doesn't speak English all that well.

Or they're a little racist.