What is the standard error of measurement for teacher made multiple choice tests? Assume a teacher constructs a four-choice multiple choice test. Each item has only one correct response. The test is scored from 0 to 100 representing the percentage of items answered correctly. 

I want to have some rules of thumb that could be informative regarding how many items are required to achieve a given standard error of measurement. For example, it would be nice to be able to advice teachers who write their own multiple choice exams "if you have 100 items that are reasonably well worded, you can expect a standard error of measurement of 2.5". 

The standard error of measurement is often defined as:

$$s_e=s_x \sqrt{1-r_{xx}}$$

where $s_x$ is the standard deviation and $r_{xx}$ is the reliability.

Furthermore internal consistency reliability can be calculated from the number of items $k$ and the mean inter-item correlation $\bar{r}_{ij}$ (i.e., average correlation between item $i$ and item $j$ for all $k$ items where $i\neq j$):

$$r_{xx}=\frac{k(\bar{r}_{ij})}
{1 + (k -1) \bar{r}_{ij}}$$

However, I'm wanting to translate the above information into meaningful recommendations for teachers. Thus, this assumes that I have some empirical estimate of typical values of $\bar{r}_{ij}$ and that I have an estimate of $s_x$. It then requires application of the formulas to calculate standard errors of measurement for likely numbers of items $k$. In particular, I was thinking about numbers of items equal to: 10, 20, 50, 80, 100, 120, 150, and 200.

Thus, I was wondering whether there are any published estimates of the standard error of measurement teacher constructed multiple choice tests.
Harvill mentions an estimate by Lord (1959). Lord (1959) presents some data for the standard error of measurement for some moderately difficult cognitive measures. While there are many caveats (e.g., the estimate of the standard error is most accurate for scores around 50% and the estimates are based on tests that are neither particularly easy or particularly difficult with means in the .35 to .75 range), Lord provides a simple formula that can be used as a rule of thumb for predicting standard error of measurement in his sample of cognitive measures which performed quite well. 

$$\hat{s}_e = .432 \sqrt{k}$$

where $k$ is the number of items.
Alternatively, if you are interested in the mean correct on a 0 to 100 scale rather than the total correct, you can divide by $k$ and multiply by 100.

$$\hat{s}_e = \frac{.432 \sqrt{k}}{k} \times 100$$

When I plugged this into R for some sample values I obtained:

&gt; lord_approximation &lt;- function(k) 0.432 * sqrt(k) /k * 100
&gt; k &lt;- c(10, 20, 50, 80, 100, 120, 150, 200)
&gt; cbind(k, sem=round(lord_approximation(k), 2))
       k   sem
[1,]  10 13.66
[2,]  20  9.66
[3,]  50  6.11
[4,]  80  4.83
[5,] 100  4.32
[6,] 120  3.94
[7,] 150  3.53
[8,] 200  3.05


Of course, not all of this reduction in standard error of measurement is due to greater accuracy. Some of it comes from the smaller standard deviation in true scores that occurs when you take the mean of more items. Furthermore, these estimates are based on relatively well designed cognitive measures. Teacher designed tests may have slightly lower reliability and thus larger SEM.

References


Harvill, L. M. (1991). Standard error of measurement. Educational Measurement: Issues and Practice, 10(2), 33-41. PDF
Lord, F. M. (1959). Tests of the same length do have the same standard 
error of measurement. Educational and Psychological Measurement, 19, 233-239.