Would it bother you if you did, indeed, look Jewish?
And, of course there is a "look" that certain Jews have that identifies them as Jewish. Could any intelligent person look at Allen Dershowitz and not know that he's a Jew? There's also a Polish look and an Italian look and a French look, etc. To deny it is simply "political correctness", IMHO.