4
$\begingroup$

In the Autism-Spectrum Quotient test, the subject is confronted with fifty statements such as:

I don’t particularly enjoy reading fiction.

I prefer to do things the same way over and over again.

and is asked whether they

  • definitely agree,
  • slightly agree,
  • slightly disagree, or
  • definitely disagree

with them. There is no neutral option.

The test is scored as follows: If the subject agrees with an “autistic” statement, they score a point. The same applies, if they disagree with a “non-autistic” statement. There is no distinction between slightly and definitely agreeing (or disagreeing).

I would expect that this test design (no neutral option, no distinction between slight and strong tendencies) leads to an amplification of small effects, such as interpretation of the question, slight tendencies, priming, chance, and so on. While I acknowledge that this effect may be desired in customer surveys and similar, it seems problematic to me in a test for the purposes of diagnosis and epidemology (such as this test). For example, I would expect that people who are already diagnosed with autism (and know this) or people with clichéically autistic interests like mathematics are more likely to tend for the more autistic option – even if their actual stance on the question in neutral.

Now, I am a layman and I acknowledge that designing such a test is not easy, as one has to account for several effects. Hence I wonder: What are the justifications for such a test design, in particular in light of its applications (diagnosis and epidemology) and my above criticism? The paper introducing the test does not provide any citations or reasons for this, at least not in those sections where I would expect it.

$\endgroup$
  • 1
    $\begingroup$ This is a totally valid, but super-broad question. There are 4 sub-questions to this one question. First, was is the history of the Autism-Spectrum Quotient test and how has it been validated/contested over the years. Second, how much are the types of tests affected by priming and stereotypes. Third, is this test undermined by a lack of neutral option. Fourth, are surveys without methods of comparison still valid. Please split this question into sub-questions and I will upvote them all. $\endgroup$ – Seanny123 Dec 21 '16 at 8:51
  • $\begingroup$ @Seanny123: I understand your point and given that I cannot even remotely estimate how long potential answers would be, you may very well be correct. However, before I follow your suggestion, please consider the following: 1) I am not much interested in the test’s history or validation per se, unless it provides answers to my main question(s). 2) I am not that much concerned about the issues addressed in your last three questions on their own (e.g., I fully acknowledge that the influence of priming and stereotypes is inevitable). Instead I am concerned in their interplay. $\endgroup$ – Wrzlprmft Dec 21 '16 at 9:13
  • $\begingroup$ I understand that you're (legitimately) only interested in their interplay, but I guess what I'm claiming is that I personally think we have to first establish their individual validity before considering their interplay. $\endgroup$ – Seanny123 Dec 21 '16 at 9:15
  • $\begingroup$ @Seanny123: I narrowed down the question, though not as far as you suggested. I still hope that this is sufficiently narrow now. $\endgroup$ – Wrzlprmft Dec 21 '16 at 13:30
  • $\begingroup$ I believe the question has been reduced in such a degree that it became answerable. Simply, why do the developers of the Autism-Spectrum Quotient test give four options, if they scale it down to just two (agree or disagree)? $\endgroup$ – Robin Kramer Feb 8 '17 at 7:56
3
$\begingroup$

Many construct scales are developed with relatively little attention to the content. Researchers may start with a large set of candidate questions, versions, formats, and phrasings, and then through a validation process, narrow them down to a subset that correlates well with the construct that they are trying to measure. The goal is a balance between keeping the test short while also retaining its predictive value. In Baron-Cohen et al's (2001) introduction, they note:

The AQ as shown in the Appendix is the outcome of piloting multiple versions, over several years.

In many cases, researchers care less about why the questions work, and more just that they work.

Having said that, the decision to go with a forced-choice format vs. say, a Likert-type rating scale with more granularity and a neutral option, can be political:

Ipsative measures may be more useful for evaluating traits within an individual, whereas Likert-type scales are more useful for evaluating traits across individuals.

This can make forced-choice measures desirable for stigmatized labels to discourage comparing individuals to one another:

Encouraging pupils to beat their previous scores can take peer pressure out of situations and eliminates the competitive element associated with norm-based referencing.

Note also that the original study confirms that mathematicians score significantly higher on this scale, as is to be expected:

... scientists score higher than nonscientists; and within the sciences, mathematics, physical scientists, computer scientists, and engineers score higher than the more human or life-centered sciences of medicine (including veterinary science) and biology. This latter finding replicates our earlier studies finding a link between autism spectrum conditions and occupations/skills in maths, physics, and engineering.

This suggests that the authors thought it perfectly valid for mathematicians to score higher on their autism scale (ie, this scale is not designed to differentiate the autistic tendencies of mathematicians from those of individuals diagnosed with autism).

$\endgroup$
  • $\begingroup$ Unless I completely misunderstood the terminology, the AS test is not an ipsative test. Rather it is a Likert-scale test without a neutral option. I also fail to see what assessing pupils and beating previous scores has to do with the AS test. $\endgroup$ – Wrzlprmft Feb 20 '17 at 11:25
  • $\begingroup$ the original study confirms that mathematicians score significantly higher on this scale, as is to be expected – Sure they do, but that does not address my doubts, which (somewhat exaggerated) are: The way the test is designed, small tendencies are amplified. For example, a mathematician would will slightly agree with an “autistic” option, because they are primed that they are somewhat more autistic than average. Therefore a high AS score for mathematicians is expected from the test design and does not tell us anything about reality. $\endgroup$ – Wrzlprmft Feb 20 '17 at 11:35
  • $\begingroup$ @Wrzlprmft Yes, it is a forced-choice (ipsative) test, the authors themselves state this in the paper. A Likert-scale would have different scoring for slightly vs strongly (dis)agree. I was merely pointing out that scoring in such tests discourages comparisons between patients (compared to say, IQ tests), which may make them preferable for diagnostic tests that have potentially stigmatizing labels. The authors do not explain their decision one way or the other. $\endgroup$ – Arnon Weinberg Feb 20 '17 at 17:00
  • $\begingroup$ Re telling us about reality, this is the purpose of validation (a topic far too large for this post, but see cogsci.stackexchange.com/questions/9610/… for more detail). The validation process confirmed that the amplification is both expected and realized, and does in fact tell us what we want to know from this scale about reality - ie, that mathematicians do have higher than average autistic tendencies. It's possible that you think this scale should measure something different than what the authors think it should measure. $\endgroup$ – Arnon Weinberg Feb 20 '17 at 17:05
  • $\begingroup$ Regarding the subject of validation: I think I understand this. My problem is rather that the test has properties that will make it succeed validation irrespective of the desired qualities. As a simple and blatant example, consider a world where 80 % of all mathematicians are autistic and vice versa and consider a test that simply asks the subject whether they are mathematicians. This test is obviously flawed, yet it will have pass a basic validation. $\endgroup$ – Wrzlprmft Feb 20 '17 at 19:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.