While the sensation of color is well defined, the perception of color is indeed shrouded in ambiguity (Brainard, 2001). There are no colors, just electromagnetic radiations that oscillate in different frequencies in the frequency domain. Our brains interpret those various wavelengths as color.
Now on to your experiment: if you find a statistical difference and you later find out that a group of people perceive the color blue as red and vice versa, the first thing you want to do is control for that population and take them out from the data set. Then you can compare the 'true seers' with the 'reversed seers' and see if they are statistically different in their emotional response. If yes, you'd better split them and analyze them separately. If not, a pooled analysis is fine, perhaps you can add the normal or inverse perception as a co-factor, dependent on the exact analysis.
But then, perhaps, the most important question here is - how would you find out what a person actually perceives?
Reference
- Brainard, Int. Encyclopedia Social & Behavioral
Sciences, Smelser & Baltes (eds.), Pergamon Press, Amsterdam