5
$\begingroup$

Assume a teacher constructs a four-choice multiple choice test. Each item has only one correct response. The test is scored from 0 to 100 representing the percentage of items answered correctly.

I want to have some rules of thumb that could be informative regarding how many items are required to achieve a given standard error of measurement. For example, it would be nice to be able to advice teachers who write their own multiple choice exams "if you have 100 items that are reasonably well worded, you can expect a standard error of measurement of 2.5".

The standard error of measurement is often defined as:

$$s_e=s_x \sqrt{1-r_{xx}}$$

where $s_x$ is the standard deviation and $r_{xx}$ is the reliability.

Furthermore internal consistency reliability can be calculated from the number of items $k$ and the mean inter-item correlation $\bar{r}_{ij}$ (i.e., average correlation between item $i$ and item $j$ for all $k$ items where $i\neq j$):

$$r_{xx}=\frac{k(\bar{r}_{ij})} {1 + (k -1) \bar{r}_{ij}}$$

However, I'm wanting to translate the above information into meaningful recommendations for teachers. Thus, this assumes that I have some empirical estimate of typical values of $\bar{r}_{ij}$ and that I have an estimate of $s_x$. It then requires application of the formulas to calculate standard errors of measurement for likely numbers of items $k$. In particular, I was thinking about numbers of items equal to: 10, 20, 50, 80, 100, 120, 150, and 200.

Thus, I was wondering whether there are any published estimates of the standard error of measurement teacher constructed multiple choice tests.

|improve this question
$\endgroup$
  • $\begingroup$ I'm only just learning statistics, so my question stems from my ignorance: What does the standard error in a multiple choice test mean? You are not drawing a random sample from a population, but testing the complete population (the population being the class taught by the teacher). I always assumed the standard error referred to the difference between the sample and the population that it is supposed to represent. $\endgroup$ – user1196 Mar 10 '13 at 17:19
  • 1
    $\begingroup$ Standard error of measurement is the standard deviation that would be obtained if you were able to repeatedly obtain a measure for a particular individual under hypothetical identical circumstances. I.e., it's a measure of the uncertainty you have about a measure you have obtained on a person. $\endgroup$ – Jeromy Anglim♦ Mar 10 '13 at 23:31
  • 1
    $\begingroup$ This is awesome! It'd be great if I received all my exams and assignments back with error bars. $\endgroup$ – Artem Kaznatcheev Mar 14 '13 at 12:48
  • $\begingroup$ A major issue here is how related are the items? In particular, I would worry that the item similarity increases as the number increases --- a 20 item test may be rather different, but a 200 item test may have substantial redundancy. $\endgroup$ – Joshua Mar 15 '13 at 18:39
3
$\begingroup$

Harvill mentions an estimate by Lord (1959). Lord (1959) presents some data for the standard error of measurement for some moderately difficult cognitive measures. While there are many caveats (e.g., the estimate of the standard error is most accurate for scores around 50% and the estimates are based on tests that are neither particularly easy or particularly difficult with means in the .35 to .75 range), Lord provides a simple formula that can be used as a rule of thumb for predicting standard error of measurement in his sample of cognitive measures which performed quite well.

$$\hat{s}_e = .432 \sqrt{k}$$

where $k$ is the number of items. Alternatively, if you are interested in the mean correct on a 0 to 100 scale rather than the total correct, you can divide by $k$ and multiply by 100.

$$\hat{s}_e = \frac{.432 \sqrt{k}}{k} \times 100$$

When I plugged this into R for some sample values I obtained:

> lord_approximation <- function(k) 0.432 * sqrt(k) /k * 100
> k <- c(10, 20, 50, 80, 100, 120, 150, 200)
> cbind(k, sem=round(lord_approximation(k), 2))
       k   sem
[1,]  10 13.66
[2,]  20  9.66
[3,]  50  6.11
[4,]  80  4.83
[5,] 100  4.32
[6,] 120  3.94
[7,] 150  3.53
[8,] 200  3.05

Of course, not all of this reduction in standard error of measurement is due to greater accuracy. Some of it comes from the smaller standard deviation in true scores that occurs when you take the mean of more items. Furthermore, these estimates are based on relatively well designed cognitive measures. Teacher designed tests may have slightly lower reliability and thus larger SEM.

References

  • Harvill, L. M. (1991). Standard error of measurement. Educational Measurement: Issues and Practice, 10(2), 33-41. PDF
  • Lord, F. M. (1959). Tests of the same length do have the same standard error of measurement. Educational and Psychological Measurement, 19, 233-239.
|improve this answer
$\endgroup$
  • 1
    $\begingroup$ interesting, but this seems to assume that each question is independent of the others. However, in an actual test there is extremely high correlations between questions. In particular, a valid measure should at least account for questions coming from some fixed number of relatively-independent units (i.e. chapters or topics) with high question-question correlation within the units. You could also give teachers a software tool to infer typical question-question correlations by doing statistics on student results on their previous exams. $\endgroup$ – Artem Kaznatcheev Mar 14 '13 at 12:52
2
$\begingroup$

To me the most natural solution is to just use item response theory (IRT). IRT has been around for a few decades, so it is well established, implemented in a variety of software packages and provides a sensible, extensible framework for this type of problem.

Essentially, one assumes an underlying latent construct of interest, values of which should drive responses on the test. For multiple choice where the answer is "right" or "wrong", you can use a series of logit (canonically) or probit models. Then for each student, you can estimate the score on the underlying latent variable and that will naturally come with some estimate of its quality/variability.

Issues that are automatically handled:

  • If everyone (or nearly everyone) gets an answer correct, it contributes very little information
  • Corollary to #1, if almost no one gets an answer correct and someone does, it should be weighted more heavily. Essentially, item difficulty is automatically handled.
  • Inter-dependence among items is accounted for. Asking the same item 10 times in a row will not artificially decrease your measurement error.

If you take a Bayesian view, for each student, you could use the model and their test responses to calculate a posterior distribution for the latent construct of interest, which would allow both a point estimate (e.g., posterior mean, median, or mode), as well as estimates of variability (e.g., standard deviation; 95% high posterior density region).

This sort of thing is essentially what big nation wide tests and testing services do. It is actually not too hard to do, but probably enough effort most teachers who already feel overworked do not adopt them.

|improve this answer
$\endgroup$
  • $\begingroup$ Solid answer, maybe teachers would use this if you made a plug-and-play version of this that directly takes the results from scantron sheets (are those still in use?) and does the statistics for them? $\endgroup$ – Artem Kaznatcheev Mar 15 '13 at 18:50
  • $\begingroup$ +1 I agree IRT (bayesian IRT in particular) provides a great framework for modelling multiple choice tests. In my particular case, I'm looking for heuristics that teachers might use to guide their decision about how many items to include in a test. I'd like to be able to give teachers a rough sense of how much more accurate their test could be if they for example increased their test from 80 to 100 items. $\endgroup$ – Jeromy Anglim♦ Mar 17 '13 at 23:55
  • $\begingroup$ In addition to difficulties in implementation, I think IRT is not applied in teacher constructed test settings because there is often an implicit contract with the students. That is, a student's mark is directly related to the proportion of items answered correctly. Thus, in IRT there is still the question of how thetas will be mapped on to exam grades. Of course this can be overcome, but I think it presents another obstacle to uptake, particularly where the test is used to make judgements about absolute performance and not just normative performance. $\endgroup$ – Jeromy Anglim♦ Mar 18 '13 at 0:54
  • $\begingroup$ @JeromyAnglim I agree with that, with respect to absolute performance. However, the implicit assumption there is that teachers actually have a valid measure of absolute performance. I think that is a much stricter assumption than that they have a valid measure of relative performance (I am open to being argued out of that). Regarding the accuracy of the test, I am not sure how reasonable any heuristic could be. That is dependent on the relative (in)dependence of items, right? In the extreme, can a teacher ask the same question another 20 times to get to 100 and still be more accurate? $\endgroup$ – Joshua Mar 24 '13 at 4:29
  • $\begingroup$ With regards to heuristics: Any heuristic would be based on the assumption that the teacher would be writing a set of additional items of roughly comparable quality, difficulty, and variety as the existing items. I acknowledge that some teachers might differ in their skill in doing this, but at the same time, a heuristic could be frame in terms of a ball park or a typical range. $\endgroup$ – Jeromy Anglim♦ Mar 24 '13 at 12:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.