What is the optimal mean correct on a multiple choice test item in order to maximise the measurement of individual differences?
$\begingroup$
$\endgroup$
asked Jul 5, 2016 at 3:03
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
4
My understanding is that in general a percentage correct of 50% is optimal where there is no scope for guessing. However, on multiple choice tests there is a certain number of response options. Thus, the probability of getting the correct answer if you know nothing, is one over the number of response options.
In this case I believe that the optimum is:
$$1/k + \frac{1 - 1/k}{2}$$
where k is the number of response options.
So the optimum for different values of k would be:
- k = 2: optimum proportion correct = 0.75
- k = 3: optimum proportion correct = 0.67
- k = 4: optimum proportion correct = 0.625
- k is infinite: optimum proportion correct = 0.50
That said, I'd still like to find the classic references where I obtained these ideas from. And I imagine item-response theory might have a little more to say about this, particularly regarding the common recommendation to vary the proportion correct over a range.
Furthermore, none of the above recommendations address the issue that setting the difficulty at the above level for most items can be intimidating for some test takers.
answered Jul 5, 2016 at 3:13
-
$\begingroup$ If you believe that "the probability of getting the correct answer if you know nothing, is one over the number of response options", then would you not just expect 0.5 instead of 0.75 for two options? Could you explain a little more why you would add the second part of the formula? $\endgroup$ Commented Jul 5, 2016 at 6:30
-
$\begingroup$ If responses to a test are purely random (i.e., the mean correct is 50% on a TRUE/FALSE question) then you are most likely learning nothing about the test takers. It is only when some people are able to answer the question and some are not that you learn about individual differences. Or more precisely, people will differ on a distribution of knowledge and placement on that distribution should influence the probability of answering an item correctly. So an informative difficulty level is half way between random responding and everyone answering the item correctly. $\endgroup$ Commented Jul 5, 2016 at 6:58
-
$\begingroup$ Ah now I see. So this value would correct for between-test, within subject differences, such that a score of 75 on a two-choice test would correspond to a 67 on a three-choice test? $\endgroup$ Commented Jul 5, 2016 at 9:01
-
$\begingroup$ It's not so much about test differences. It's more a question of how difficult a test designer should make multiple choice items in order to maximise reliability and validity. $\endgroup$ Commented Jul 6, 2016 at 1:40