What if someone perceives a color as 'red' when it is actually 'green'?
Since different people have preferences for different colors, and colors are perhaps constructed in the mind, is it possible that the entire space of colors can be experienced quite differently for some people?
Yes, this scenario is possible, occurring with certain cases of brain lesions in specific areas of the visual cortex, the fusiform, lingual and posterior parahippocampal gyri. These areas are analogous to what is referred to in primates as V4, or the 4th visual cortex, and are known to be involved (at least partly) in the perception of color (though see this commentary).
Courtesy wikipedia
In this study, subjects with brain lesions were asked to identify colors in a blue ambient light (meant to mimic the outdoors) and under a reddish light (meant to mimic indoor lighting).
... the three patients...with abnormal matches on the blue, green and red chips made different matches than control subjects for pink, violet, turquoise and green targets... Their matches for these chips were shifted towards orange-red for the pink and violet targets and towards green for the turqoise target. One patient displayed a shift toward turquois[e] for the green target.
So, the level at which this "signal" is getting crossed can only be inferred from the level of the lesion, but presumably, the transduction of color by the cones and the processing of the colors in the P and K cells of the Lateral Geniculate is all still intact.
Clarke, S., Walsh, V.,Schoppig, A.,Assal, G., Cowey, A. (1998) Colour constancy impairments in patients with lesions of the prestriate cortex. Exp Brain Res 123:154-158.
As @Gray mentioned, the philosophical problem you are interested in is known as the inverted spectrum. Unfortunately, @Gray's claim about no empirical difference is not exactly true. As @ChuckSherrington pointed out, we can have differences in color perception due to brain lesions, but this is cheating in way. We don't have to go this far, we already have differences in color perception between neurotypical people based on their culture/language. More dramatically, we can observe this difference within a single individual!
As I explained in a previous answer: physics does not have colour, it just has a continuous spectrum of wavelengths. Even when you look at the sensitivity of the 3 types of cones in the retina it is not discrete, but continuous. The categories of colours (i.e. "that's red", "that's blue") are produced by perception and these discrete-ish categories form the basis of colour qualia. Scientists can study these categories by asking participants if various stimuli feel like the same colour. The arbitrary boundaries of the categories people draw between colours is language dependent (Regier & Kay, 2009)! In other words, we have support for the Whorf hypothesis: language effects your subjective conscious experience.
But the buck doesn't stop there. Gilbert et al. (2006) showed that the Whorf hypothesis is supported in the right visual field but not the left. In other words, when I present colours in one part of your visual field, you experience them one way and when I present them to the other then you experience them in a fundamentally different way.
Thus different colours can be experienced differently by different neurotypical people, and in fact they can be experienced differently by the same person based on which visual field the stimuli is presented in. Further, this difference is empirically measurable! Of course this doesn't resolve the inverted spectrum problem completely, but that is to be expected since philosophy always has a way to run away from science.
I would point you towards the debate on qualia in cognitive science. It has been argued by some philosophers such as David Chalmers that there are internal qualitative states separate from their physical realization. http://en.wikipedia.org/wiki/Inverted_spectrum
With the exception of color-blindness and other differences in visual circuitry such as Tetrachromacy there would be no empirical difference in the response of such a person with differences in their qualitative experience of color. So its an interesting question whether such differences in qualia exist or are even meaningful to talk about.
