What is the standard error of measurement for teacher made multiple choice tests?

Question

Assume a teacher constructs a four-choice multiple choice test. Each item has only one correct response. The test is scored from 0 to 100 representing the percentage of items answered correctly.

I want to have some rules of thumb that could be informative regarding how many items are required to achieve a given standard error of measurement. For example, it would be nice to be able to advice teachers who write their own multiple choice exams "if you have 100 items that are reasonably well worded, you can expect a standard error of measurement of 2.5".

The standard error of measurement is often defined as:

$$s_e=s_x \sqrt{1-r_{xx}}$$

where $s_x$ is the standard deviation and $r_{xx}$ is the reliability.

Furthermore internal consistency reliability can be calculated from the number of items $k$ and the mean inter-item correlation $\bar{r}_{ij}$ (i.e., average correlation between item $i$ and item $j$ for all $k$ items where $i\neq j$):

$$r_{xx}=\frac{k(\bar{r}_{ij})} {1 + (k -1) \bar{r}_{ij}}$$

However, I'm wanting to translate the above information into meaningful recommendations for teachers. Thus, this assumes that I have some empirical estimate of typical values of $\bar{r}_{ij}$ and that I have an estimate of $s_x$. It then requires application of the formulas to calculate standard errors of measurement for likely numbers of items $k$. In particular, I was thinking about numbers of items equal to: 10, 20, 50, 80, 100, 120, 150, and 200.

Thus, I was wondering whether there are any published estimates of the standard error of measurement teacher constructed multiple choice tests.

Answer 1

Harvill mentions an estimate by Lord (1959). Lord (1959) presents some data for the standard error of measurement for some moderately difficult cognitive measures. While there are many caveats (e.g., the estimate of the standard error is most accurate for scores around 50% and the estimates are based on tests that are neither particularly easy or particularly difficult with means in the .35 to .75 range), Lord provides a simple formula that can be used as a rule of thumb for predicting standard error of measurement in his sample of cognitive measures which performed quite well.

$$\hat{s}_e = .432 \sqrt{k}$$

where $k$ is the number of items. Alternatively, if you are interested in the mean correct on a 0 to 100 scale rather than the total correct, you can divide by $k$ and multiply by 100.

$$\hat{s}_e = \frac{.432 \sqrt{k}}{k} \times 100$$

When I plugged this into R for some sample values I obtained:

> lord_approximation <- function(k) 0.432 * sqrt(k) /k * 100
> k <- c(10, 20, 50, 80, 100, 120, 150, 200)
> cbind(k, sem=round(lord_approximation(k), 2))
       k   sem
[1,]  10 13.66
[2,]  20  9.66
[3,]  50  6.11
[4,]  80  4.83
[5,] 100  4.32
[6,] 120  3.94
[7,] 150  3.53
[8,] 200  3.05

Of course, not all of this reduction in standard error of measurement is due to greater accuracy. Some of it comes from the smaller standard deviation in true scores that occurs when you take the mean of more items. Furthermore, these estimates are based on relatively well designed cognitive measures. Teacher designed tests may have slightly lower reliability and thus larger SEM.

References

• Harvill, L. M. (1991). Standard error of measurement. Educational Measurement: Issues and Practice, 10(2), 33-41. PDF
• Lord, F. M. (1959). Tests of the same length do have the same standard error of measurement. Educational and Psychological Measurement, 19, 233-239.

Answer 2

To me the most natural solution is to just use item response theory (IRT). IRT has been around for a few decades, so it is well established, implemented in a variety of software packages and provides a sensible, extensible framework for this type of problem.

Essentially, one assumes an underlying latent construct of interest, values of which should drive responses on the test. For multiple choice where the answer is "right" or "wrong", you can use a series of logit (canonically) or probit models. Then for each student, you can estimate the score on the underlying latent variable and that will naturally come with some estimate of its quality/variability.

Issues that are automatically handled:

• If everyone (or nearly everyone) gets an answer correct, it contributes very little information
• Corollary to #1, if almost no one gets an answer correct and someone does, it should be weighted more heavily. Essentially, item difficulty is automatically handled.
• Inter-dependence among items is accounted for. Asking the same item 10 times in a row will not artificially decrease your measurement error.

If you take a Bayesian view, for each student, you could use the model and their test responses to calculate a posterior distribution for the latent construct of interest, which would allow both a point estimate (e.g., posterior mean, median, or mode), as well as estimates of variability (e.g., standard deviation; 95% high posterior density region).

This sort of thing is essentially what big nation wide tests and testing services do. It is actually not too hard to do, but probably enough effort most teachers who already feel overworked do not adopt them.