I'm inclined to agree.  I really can't think of any instance of racism rooted absolutely in ideology.  It inevitably has something to do with the subjected minority perceived as being in the way, or a threat to the ruling class's pursuit of wealth, land, power, and their general "way of life", etc.  

Which is not to say that people don't also hold that metaphysical belief.  A lot of people truly believed also that Native Americans and Africans were a disgusting class of subhumans.  Not to mention accounts of Nazis taking Jewish women as secret lovers.  Anyway, ideology is usually what you feed the masses when you're trying to rally support for your plan to acquire as much personal power as possible.  It is a means to an end, not the end itself.

Those who can make you believe absurdities can make you commit atrocities. -Voltaire