Group differences reflecting different use of a rating scale vs actual difference in perception

Question

I was reading a paper that reports average ratings of chords by two different groups of subjects (trained musicians, and non-musicians).

The paper makes the argument that "These group differences may reflect either the manner in which the two groups of subjects used the rating scale or in their perceptions of the chords."

Obviously, one would hope that the latter is the case rather than the former, although from this graph one can't be sure since error bars, which would be very informative in this case, are not shown. My question is: is it correct to say that the answer to this question can be given if one compares the variances across groups, e.g.with Levine's test? And that, if found to be unequal, one can argue that the group difference reflects an artefact, i.e. a difference in the way the two groups of subjects used the rating scale?

Answer 1

I indeed think that it is not possible to conclude that the scales are used differently, based on variance alone. Normally one would assume the scale is used properly and the effect you see is indeed the result of (not) having a trained ear.

There do exist ways to investigate how one uses a subjective scale. In a paper I presented, we looked at how people interpreted a subjective workload scale. This scale contained verbal descriptors that conveyed a particular amount of workload, and this amount was estimated by the participants. This scale was designed to be linear and cover workload from "no workload" to "work too demanding". Therefore, we would expect a linear line that starts at 0 (no workload) to 150 (complete overload; we used 150 instead of 100). The results indeed confirmed this and the groups perceived the items on the scale similarly.

So far, the method is kind of similar to your paper. The difference is that we gave the participants anchor-points to help them rate the items: The zero-point (no workload) and the 150-point (complete overload). This way the participants had the same reference points, which should enable them to use the entire range, regardless of their experience or something.

You could also use these anchor points to see how consonance ratings differ. Let them listen to a perfect dissonant and a perfect consonant tone and say they are rated zero and ten respectively (or something like that). If the two groupsc rate the sounds similarly now, they have probably interpreted the scale differently before. However, if you still find the same difference as before, it may suggests that untrained musicians can indeed not distinguish the sound very well (and the scale was interpreted properly before).

Answer 2

No, I don't think so . Greater variance could reflect variance in perception just as much as variance in the way the rating scale is used.