I have to design my own psychometric test to measure a construct. I have a construct, but part of the requirement is to include a rationale for the number of items in the instrument, which I don't know how to justify. Also, using a Likert scale, I have to justify why I am using a 5-point Likert scale. It would be great to have some literature to refer to.


Cross Validated has plenty of info on this topic; arguably it's a duplicate of some question over there. Check out "Under what conditions should Likert scales be used as ordinal or interval data?" and my synopsis of Bollen and Barb (1981) in this answer. If you have more to say about why this is a question specific to Cognitive Sciences (beyond the presumable involvement of subjective judgments and self-reporting on attitudes), you should say it; I might be able to add unique substance to this answer. (BTW, there is plenty of literature out there on this question. Frankly, I'd guess you haven't tried very hard in looking for yourself. If you have, you could also improve answers by noting what you know so far.)

As for non-unique substance that mostly approaches the problem from a statistical perspective, consider what you'll probably want to do once you have the data. If your theory of your latent construct is typical, you probably want to estimate individuals' scores on a continuous dimension, not just rank people. I.e., you probably want to be able to say how different one person is from another, not just who ranks higher and how many people are between them.

The basic problem with Likert scales is that they don't really allow you to do this inherently. E.g., the difference between ratings 3 and 4 isn't necessarily the same as the difference between 2 and 3 even if you use exactly opposite anchors like 2/4$=$dis/agree and 3 is given a completely neutral anchor. Therefore you can't really assume the sum of a few items has this property (interval-level measurement, basically) if you just treat these ratings as numbers.

Another halfway decent place to start reading is Wikipedia's classical test theory (CTT) page. This might not be the gentlest introduction to the underlying issues, but it's probably nicer than sending you straight to IRT. IMO, that's where one ultimately has to go to understand latent dimension estimation in terms of thresholds between Likert scale points...but if you can satisfy CTT assumptions, you can avoid a lot of complexity, which I assume you want to do, for better or worse.

Bollen and Barb (1981) demonstrate that Likert scales with $\ge5$ rating options can be treated like interval measures without weakening correlations too terribly much, but more is diminishingly better. Fewer simplifies the rating task somewhat in general, but can sometimes make it harder for respondents with nuanced opinions that can't be reflected accurately with fewer options...and either way, it costs you info. Conversely, having tons of options might start to elicit arbitrary noise rather than meaningfully distinct responses...so consider whether additional options actually mean something different from existing ones. If they do, you're probably not going to do any harm by adding them.

Another technical issue for which I have no better reference than my former advisor (he never told me where he read it): more scale options tends to increase the size of the general factor (in a factor-analytic sense). I suspect this has something to do with response biases like acquiescence bias and extreme response style. If you know what you're doing, you can control for "nuisance factors" like these, but it's advanced stats. See "Factor analysis of questionnaires composed of Likert items" for more.

As for the number of items, it depends somewhat on similar concerns: your abilities to approximate and estimate a continuous dimension, and your ability to differentiate items meaningfully. A minimum of three items are necessary for statistically meaningful estimation of latent factor scores (this may be true of latent score $(\theta)$ estimation using IRT as well; I'm not sure, but I think it should be). Two doesn't permit identification of a unique solution in factor analysis – there's no indication of how to weigh your two items as measures of any common factor(s). If you're trying to measure more than one latent construct (or trying to separate out nuisance factors from your construct of interest), you'll need at least one item for each that only really measures that construct, and three that measure each. That's only four items for two correlated constructs, but six for two unrelated constructs.

Again, more items is generally better for the sake of approximating and estimating a continuous latent dimension...but if you're just going to sum or average Likert ratings, you're implicitly assuming that all items measure the latent construct equally well, or that their inequalities at least balance out overall. Therefore you don't want to just add more items if they're going to be substantially worse, unless you're willing to do some nontrivial statistical work to sort them out properly.

I'd also hesitate to add overly duplicative items, because that could encourage inattentive responding and thus engender both arbitrary noise and (more) systematic bias. Using semantic opposites might be defensible: e.g., 1) Tumblers are better than pumpers; 2) Pumpers are better than tumblers...? This can help manage acquiescence bias if it's done properly, but it can also irk respondents and introduce nuisance multidimensionality.

Multidimensionality is yet another problem that can arise with too many items...but again, it's not necessarily a problem if you know what you're doing statistically. There are ways of separating factors of interest from others that might influence responses if you have enough items. If you or someone else might actually figure out how to do this someday (it's not THAT hard), it's better to have the items from the outset than to have to add or change them later, because you probably want to maintain a consistent stimulus across all data you collect over the long-term life of the measure.

However, if it's important to keep it simple for the sake of avoiding complex calculation or for the sake of reducing respondent fatigue (yet another source of inattentive responding, noise, and bias), you might want to limit yourself to 10–20 short, clearly-phrased, single-idea items. Some questionnaires have 100–500 items, but that requires some serious compensation for a participant's time and concentration; people do not like to think that hard for that long. Consider the concerns of your target population and any distracting elements in your response collection scenario, and try piloting your first draft of the test with a few people to see how it comes across and whether it takes as long to complete as you think it should. Several stages of revision aren't uncommon when this is done properly, but improper practice isn't uncommon either, nor is it much easier to fail completely than it is to pull it off perfectly.

| improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.