Are there any standard psychometric scales that are only applicable to within-subject designs?

Question

A while ago, I asked the question whether Presence questionnaires are valid in between-subject designs (see the question for the background to this assumption).

While the question garnered some encouraging feedback, no one seems to know the answer as of yet.

Example: By way of a contrived example, I was suggesting that the situation would be similar were we to perform an experiment in which half the participants are told that they are to receive 1 litre of free beer, whereas the other half are told that they are given 10 litres of free beer.

If you asked all participants to rate their level of happiness at receiving the news that they are given said quantity of free beer, I think you would not see a difference (because free beer is free beer!).

However, if you were to repeat the experiment a second time with the same participants and now swapped the amount of beer between the two groups, I would expect people to lower their happiness ratings if their amount of beer got lowered, and up their ratings if their free beer levels went up. (NB: Participants do not actually receive free beer but are simply asked how happy the news of receiving free beer would make them, so they don't actually receive 1 litre instead of 10 or get 11 litres)

Question: So I'm wondering: Are there any standard tests or phenomena that are, in a sense, meaningless when applied to between-subjects but work as a relative scale when applied to within-subjects experiments?

Answer 1

I think your intuitions are correct, and the reason is that Likert scales often suffer from poor construct validity.

One example that comes to mind is Alter et al. (2010), who use a mixed design, and note that participants in both conditions gravitate toward the middle of the scale on their first rating. They suggest that participants do not know how to use the scale at first; however, their second rating is informative because it can be compared to the first.

Several other studies (e.g. Dittrich et al., 2007; Olson et al. 2007) suggest that Likert scales are best used for relative rather than absolute judgments. Within-subject designs may abide by this rule because participants will implicitly use their previous ratings as anchors.

Although within-subject designs are often preferred, many times they are not feasible. (For instance, an experimental manipulation may make it impossible for a subject to be in both conditions without jeopardizing the effect of the manipulation). However, a properly designed scale may be usable in a between-subjects design. The key here is wording the question such that subjects will make judgements relative to some other criterion. So you may say something like:

"Compared to the last time you received one free beer, how happy were you upon receiving
 ten free beers?"

-5       -4       -3       -2       -1       0       1       2       3       4       5          
less happy than last time      neither more or less happy     much happier than last time

This example is a little contrived--in fact it's not that great since the question is leading--but hopefully you get the idea.

Since you mention happiness, I will note that this issue has arisen in the happiness literature. Happiness is often assessed by a Likert scale, but the wording of the question can evoke different comparative standards. For instance, Steffel & Oppenehimer (2009) report that people respond differently whether they think about intra- vs inter- personal happiness (i.e. "How happy are you compared to others?" vs "How happy are you compared to how happy you would like to be?") When left to their own accord, they typically adopt an inter-personal comparison. However it is best to make this comparison explicit whenever possible to avoid these issues.

Alter, A. L., Oppenheimer, D. M., & Zemla, J. C. (2010). Missing the trees for the forest: A construal level account of the illusion of explanatory depth. Journal of personality and social psychology, 99(3), 436. PDF

Dittrich, R., Francis, B., Hatzinger, R., & Katzenbeisser, W. (2007). A paired comparison approach for the analysis of sets of Likert-scale responses. Statistical Modelling, 7(1), 3-28. PDF

Olson, J. M., Goffin, R. D., & Haynes, G. A. (2007). Relative versus absolute measures of explicit attitudes: Implications for predicting diverse attitude-relevant criteria. Journal of Personality and Social Psychology; Journal of Personality and Social Psychology, 93(6), 907.

Steffel, M., & Oppenheimer, D. M. (2009). Happy by what standard? The role of interpersonal and intrapersonal comparisons in ratings of happiness. Social indicators research, 92(1), 69-79.