How to efficiently get a sense of the meaning of a score on a non-aptitude test that you encounter in a journal article?

Question

Context: When reading research articles in psychology, you often encounter new non-aptitude self-report psychological tests (e.g., measures of personality, well-being, psychopathology, learning style, stress, goal orientation, etc.). Most of these tests involve obtaining the sum or the mean of a set of items for one or more scales or subscales. Each item typically involves item text and a consistent set of response options (e.g., Likert-type response scales, yes-no response scales, etc.). There are thousands of such scales, some which are frequently used; others are used only once.

A lot has been written about how to assess test validity. However, here I wanted to focus particularly on how to assess what a particular individual or group score means. Specifically, I'm thinking of the situation where you are reading a journal article and you encounter a scale that you have not seen before, such as:

• A 10-item measure of self-esteem on 1 to 5 scale; group mean is 3.2. Is the sample generally high, low, or moderate in self-esteem?
• A 30-item check-list of good eating habits with a mean of 20.7. Does the sample have good eating habits?

For well developed tests, there is a test manual. Other times there is a website or key journal article with information. Finally, in some cases, very little information is readily available. In some cases useful information is given in the methods section, but typically, on its own, this is inadequate. Thus, the process of trying to locate information such as item content and normative comparisons begin.

A lot has been said about normative and criterion referenced test interpretation. If you are using a test in clinical practice or you are using a test in your own research, you should invest time in understanding what different scores on the test actually mean. However, when casually reading a journal article you can encounter many tests. In such cases, the article is one of many, and you might just want to get a quick sense of what a given score means.

Question

• What is a good way to get both a normative and absolute understanding of the meaning of a group or individual non-aptitude test score?
• What is an efficient way of obtaining the information required to make such a judgement?

Note the use of the term "non-aptitude". I'd like to exclude discussion of intelligence testing, ability testing, and knowledge testing, and so on.

Update: To clarify, I didn't mean to imply in the question that item information was not available anywhere. I guess the question is referring to the general situation where there probably are previous uses of the measure in the literature, and there might be a key article that describes the measure somewhere. I guess I'm thinking of a good general strategy for securing relevant information and then using it to form an understanding of the scale properties.

Answer 1

Tough question.

I'm going to assume that the question pertains to measures which lack all the information you've outlined: test manual, validity studies, website and key articles, item content and normative comparisons. If these assumptions are correct, then what you are a describing is an insufficiently explained measure. I will argue that we necessarily cannot know what such a scale means in any scientific sense.

A scale is a measure. A scientific measure is a process which generates evidence, i.e., observations that discriminate between different possible answers to a specific question. (Data may constitute evidence with respect to one question and not another.) The outcome of this discrimination, or analysis, is what the scale "means", i.e., the logically sound interpretation we may make on the basis of analyzing the evidence.

To determine whether a method does, in fact, generate evidence, we need to be able to replicate it independently. But the premises imply that we don't have enough information to do so. Because we cannot independently verify which hypotheses the evidence generated by the method discriminates between, if any, there is no way to know what it means.