You're absolutely right it has nothing to do with his race, but people will accuse you of being a racist if you try to impeach him.
You make a great point about Elena Kagan btw, and that point is further illustrated by comments made recently by Woody Allen. He said something along the lines that it would be good if Obama could be a dictator for a few years. The liberal elite have zero regard for the democratic principles that America was founded on. If they could, they would set up a lite dictatorship.
I really wish America would move towards a parliamentary system like Britain, Canada or Israel. I find it keeps the governments a lot more honest because they can't rule with impunity for 4-8 years and do whatever they want. Yeah, it's rough at times in that when a good party takes power they sometimes get held back by other parties, but it also eliminates the chance of a crazy party like the dems destroying the country in 1 term.