"if a bunch of black people turn up at an alleged white supremacist rally, is it really a white supremacist rally?"

Maybe. It depends on whether the rally exists to promote white supremacy.

Poor people showed up at Tea Party rallies to support plutocrats. That didn't change the fact that those rallies were promoting a capitalist ideology at odds with the material interests of many of the people who participated in them.