As @Gray mentioned, the philosophical problem you are interested in is known as the inverted spectrum. Unfortunately, @Gray's claim about no empirical difference is not exactly true. As @ChuckSherrington pointed out, we can have differences in color perception due to brain lesions, but this is cheating in way. We don't have to go this far, we already have differences in color perception between neurotypical people based on their culture/language. More dramatically, we can observe this difference within a single individual!
As I explained in a previous answer: physics does not have colour, it just has a continuous spectrum of wavelengths. Even when you look at the sensitivity of the 3 types of cones in the retina it is not discrete, but continuous. The categories of colours (i.e. "that's red", "that's blue") are produced by perception and these discrete-ish categories form the basis of colour qualia. Scientists can study these categories by asking participants if various stimuli feel like the same colour. The arbitrary boundaries of the categories people draw between colours is language dependent (Regier & Kay, 2009)! In other words, we have support for the Whorf hypothesis: language effects your subjective conscious experience.
But the buck doesn't stop there. Gilbert et al. (2006) showed that the Whorf hypothesis is supported in the right visual field but not the left. In other words, when I present colours in one part of your visual field, you experience them one way and when I present them to the other then you experience them in a fundamentally different way.
Thus different colours can be experienced differently by different neurotypical people, and in fact they can be experienced differently by the same person based on which visual field the stimuli is presented in. Further, this difference is empirically measurable! Of course this doesn't resolve the inverted spectrum problem completely, but that is to be expected since philosophy always has a way to run away from science.