10
$\begingroup$

I have built several apps in Norwegian about hunting, philosophy etc. Students use the app to learn a subject and test their knowledge before doing their exam. In current state i have one question with three alternatives, where one is correct. The student will be presented with the solution to the question after they have chosen the alternative, and not on the end of the test.

For example:

What is the the capital city of Australia?

  1. Canberra
  2. Sydney
  3. Melbourne

I was wondering if anyone know if there is any research on if students learn more/less/equal by being presented with 4 alternatives instead of three.

For example:

What is the the capital city of Australia?

  1. Canberra
  2. Sydney
  3. Melbourne
  4. Edmunton

Does this have any effect for learning, since it's less probable to just check the correct by chance. Maybe students have to do more active thinking since it's more alternatives to think about, and therefore learn more?

$\endgroup$
  • 1
    $\begingroup$ Hi Hakonbogen. An app I'm using called Semper, a vocabulary learning app, uses multiple choice testing, among others. The more often you have studied an item, the more options you are given, as to increase difficulty. They are supported by a few universities so I bet there is some research behind it. Maybe that is a nice starting point for your search. getsemper.com $\endgroup$ – Robin Kramer May 23 '16 at 14:27
  • $\begingroup$ Only in the sense that easier exams discourage "learning", iff the "learning" is only motivated by passing the exam. That's why stuff like this is unlikely to get studied directly. Too many external variables. Also, for non-trivia knowledge, an opinion is that multiple choice-question are just bad ncbi.nlm.nih.gov/pmc/articles/PMC3433302 $\endgroup$ – Fizz Aug 5 '18 at 18:58
3
+50
$\begingroup$

I doubt this has been studies as you ask, i.e. effect on learning. Most of the literature seems to hammer/contrast the effects on learning of multiple-choice vs constructed-response exams, e.g. https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3433302/

There have been more limited investigations of 3 vs 4/5 choice in tests. One meta-analysis from 2005 titled "Three Options Are Optimal for Multiple-Choice Items: A Meta-Analysis of 80 Years of Research"

Multiple‐choice items are a mainstay of achievement testing. The need to adequately cover the content domain to certify achievement proficiency by producing meaningful precise scores requires many high‐quality items. More 3‐option items can be administered than 4‐ or 5‐option items per testing time while improving content coverage, without detrimental effects on psychometric quality of test scores. Researchers have endorsed 3‐option items for over 80 years with empirical evidence—the results of which have been synthesized in an effort to unify this endorsement and encourage its adoption.

This meta-analysis has about 300 citations in Google Scholar, so at least people took notice. I don't know how well received its conclusions were. Wikipedia's page Multiple choice doesn't even mention this paper, but the page is rather sparse in citations anyway.

I did find one newer study 2010 (citing the meta-analysis), which confirmed its conclusions:

Overall, three-option items perform equally as well as four-option items. Since three-option items require less time to develop and administer and additional options provide no psychometric advantage, teachers are encouraged to adopt three-option items as the standard on multiple-choice tests.

And another one from 2009, roughly confirming it:

The low frequency of items with three functioning distractors in the four-option items in this study suggests that teachers have difficulty developing plausible distractors for most MCQs. Test items should consist of as many options as is feasible given the item content and the number of plausible distractors; in most cases this would be three. Item analysis results can be used to identify and remove non-functioning distractors from MCQs that have been used in previous tests.

In these papers, the reason why 3-item MC tests are often just as good has mostly to do with who makes the exam (and the time spend developing it) rather than than who takes it.

On the other hand, there's a 2012 paper on listening tests (evaluated with MCs), which somewhat disagrees:

Mean test scores of the three-option tests were significantly higher than those of four- and five-option tests. While no difference was found in mean item discriminations across the three different test formats, reliability coefficients showed inconsistent patterns depending on the number of options and test versions. One possible interpretation of the low correlations among the scores of three test formats is that items with different numbers of options tap into skills other than listening. The findings suggest that statistically, three options may or may not be optimal depending on the point of view taken – from that of the test score users, or from that of the test stakeholders. Test developers must consider multiple statistical, affective, and contextual factors in determining the optimal number of options.

This was supposedly a high-stakes exam, according to the paper title, namely the CSAT. Of the post-2005 papers, this last one has the most substantial literature and arguments review, so it's worth reading its first 4-5 pages.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.