Experiments that collect subjective ratings (e.g. on a Likert 1-7 scale) necessarily have to account for between-subjects differences in how the range of the rating scale is used by different people, i.e. the fact that, for various reasons, some subjects are quite restrictive in their usage of the scale (e.g. they rarely rate lower than 3 or higher than 5), while others use up the entire scale quite liberally. I know that for this problem it suffices to simply z-score all the raw ratings within each subject.

However, another problem that often occurs with such data sets is when it turns out (as per post-study debriefings) that subjects change their scale-using strategy during the course of the experiment, such as for instance if they realise that an item they've previously rated 7 was in fact not really worthy of a 7 rating, since the current item is in fact "worthy of a 7", and so they wish they could go back and correct the previous rating to, say, a 5 or a 6 - which the experimental paradigm might not allow them to do.

My question: Is there any statistical way to correct for this second sort of problem, i.e. within-subject variability in the rating scale usage? Clearly, z-scoring the ratings within-subject does nothing to fix this sort of mid-experiment strategy change in how the scale ratings are given; and in fact only works to eliminate between-subjects variability if the within-subjects variability described above is assumed to be negligible.

Note that, to a previous question I had on the topic of rating scales, someone helpfully suggested that the problem of between-subject variability can be reduced by using "anchor points" during the instructions, i.e. providing examples to subjects of what a "1" stimulus, as well as what a "7" stimulus, are respectively like. This would of course solve both problems described above, but does assume that the right instructions were given during data collection, whereas my question concerns correction of an existing data set, one that presumably did not inocculate against this beforehand.

  • $\begingroup$ Note that I will now ask this on the Stats site as well, as perhaps the lack of answers here suggests the question is not a good fit for the Cogsci site. $\endgroup$ – z8080 Dec 5 '16 at 11:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.