Well Racism was originally the belief that people of certain colours could be grouped together by specific abilities according to said colour. It is now more appropriate to say racism is believing that because of a race, a certain person is defined by that.
I wouldn't label it a double slandered. I think it it's fair that a white person shouldn't say the N word for example.