Rationale behind an “all-or-nothing” design for a test for the purposes of diagnosis and epidemology

Question

In the Autism-Spectrum Quotient test, the subject is confronted with fifty statements such as:

I don’t particularly enjoy reading fiction.

I prefer to do things the same way over and over again.

and is asked whether they

• definitely agree,
• slightly agree,
• slightly disagree, or
• definitely disagree

with them. There is no neutral option.

The test is scored as follows: If the subject agrees with an “autistic” statement, they score a point. The same applies, if they disagree with a “non-autistic” statement. There is no distinction between slightly and definitely agreeing (or disagreeing).

I would expect that this test design (no neutral option, no distinction between slight and strong tendencies) leads to an amplification of small effects, such as interpretation of the question, slight tendencies, priming, chance, and so on. While I acknowledge that this effect may be desired in customer surveys and similar, it seems problematic to me in a test for the purposes of diagnosis and epidemology (such as this test). For example, I would expect that people who are already diagnosed with autism (and know this) or people with clichéically autistic interests like mathematics are more likely to tend for the more autistic option – even if their actual stance on the question in neutral.

Now, I am a layman and I acknowledge that designing such a test is not easy, as one has to account for several effects. Hence I wonder: What are the justifications for such a test design, in particular in light of its applications (diagnosis and epidemology) and my above criticism? The paper introducing the test does not provide any citations or reasons for this, at least not in those sections where I would expect it.

Answer 1

Many construct scales are developed with relatively little attention to the content. Researchers may start with a large set of candidate questions, versions, formats, and phrasings, and then through a validation process, narrow them down to a subset that correlates well with the construct that they are trying to measure. The goal is a balance between keeping the test short while also retaining its predictive value. In Baron-Cohen et al's (2001) introduction, they note:

The AQ as shown in the Appendix is the outcome of piloting multiple versions, over several years.

In many cases, researchers care less about why the questions work, and more just that they work.

Having said that, the decision to go with a forced-choice format vs. say, a Likert-type rating scale with more granularity and a neutral option, can be political:

Ipsative measures may be more useful for evaluating traits within an individual, whereas Likert-type scales are more useful for evaluating traits across individuals.

This can make forced-choice measures desirable for stigmatized labels to discourage comparing individuals to one another:

Encouraging pupils to beat their previous scores can take peer pressure out of situations and eliminates the competitive element associated with norm-based referencing.

Note also that the original study confirms that mathematicians score significantly higher on this scale, as is to be expected:

... scientists score higher than nonscientists; and within the sciences, mathematics, physical scientists, computer scientists, and engineers score higher than the more human or life-centered sciences of medicine (including veterinary science) and biology. This latter finding replicates our earlier studies finding a link between autism spectrum conditions and occupations/skills in maths, physics, and engineering.

This suggests that the authors thought it perfectly valid for mathematicians to score higher on their autism scale (ie, this scale is not designed to differentiate the autistic tendencies of mathematicians from those of individuals diagnosed with autism).