1
$\begingroup$

What is the optimal mean correct on a multiple choice test item in order to maximise the measurement of individual differences?

$\endgroup$
1
$\begingroup$

My understanding is that in general a percentage correct of 50% is optimal where there is no scope for guessing. However, on multiple choice tests there is a certain number of response options. Thus, the probability of getting the correct answer if you know nothing, is one over the number of response options.

In this case I believe that the optimum is:

$$1/k + \frac{1 - 1/k}{2}$$

where k is the number of response options.

So the optimum for different values of k would be:

  • k = 2: optimum proportion correct = 0.75
  • k = 3: optimum proportion correct = 0.67
  • k = 4: optimum proportion correct = 0.625
  • k is infinite: optimum proportion correct = 0.50

That said, I'd still like to find the classic references where I obtained these ideas from. And I imagine item-response theory might have a little more to say about this, particularly regarding the common recommendation to vary the proportion correct over a range.

Furthermore, none of the above recommendations address the issue that setting the difficulty at the above level for most items can be intimidating for some test takers.

$\endgroup$
  • $\begingroup$ If you believe that "the probability of getting the correct answer if you know nothing, is one over the number of response options", then would you not just expect 0.5 instead of 0.75 for two options? Could you explain a little more why you would add the second part of the formula? $\endgroup$ – Robin Kramer Jul 5 '16 at 6:30
  • $\begingroup$ If responses to a test are purely random (i.e., the mean correct is 50% on a TRUE/FALSE question) then you are most likely learning nothing about the test takers. It is only when some people are able to answer the question and some are not that you learn about individual differences. Or more precisely, people will differ on a distribution of knowledge and placement on that distribution should influence the probability of answering an item correctly. So an informative difficulty level is half way between random responding and everyone answering the item correctly. $\endgroup$ – Jeromy Anglim Jul 5 '16 at 6:58
  • $\begingroup$ Ah now I see. So this value would correct for between-test, within subject differences, such that a score of 75 on a two-choice test would correspond to a 67 on a three-choice test? $\endgroup$ – Robin Kramer Jul 5 '16 at 9:01
  • $\begingroup$ It's not so much about test differences. It's more a question of how difficult a test designer should make multiple choice items in order to maximise reliability and validity. $\endgroup$ – Jeromy Anglim Jul 6 '16 at 1:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.