4
$\begingroup$

What methods or approaches exist or might be developed to measure precision of measurement in a classical test theory framework? Standards for Educational and Psychological Testing (American Psychological Association, 2014) talks about precision of measurement in several places, but does not seem to provide any references.

Also, to what extent is precision of measurement in IRT theory similar to or different from absolute agreement in score (as opposed to inter-rater reliability or test reliability as measured by alpha) in CTT?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
3
$\begingroup$

'Precision' in classical test theory

Most accounts of classical test theory do not have a notion of precision as such, but occasionally, reliability may be called precision instead. The relationship is probably most concisely illustrated with the standard dartboards. This is also explained on the Wikipedia Item Response Theory page, but as you can see, in CTT, precision is to reliability what accuracy is to validity.

enter image description here Tuft U

(Wikipedia Reliability article juxtaposed with a Tufts University guide.

Origin of 'precision' in classical test theory

Cronbach (1951) suggested Coombs (1950) as the origin of the reliability/precision confusion.

Coombs (6) offers the somewhat more satisfactory name "coefficient of precision" for this index which reports the absolute minimum error to be found if the same instrument is applied twice independently to the same subject. A coefficient of stability can be obtained by making the two observations with any desired interval between. A rigorous definition of the coefficient of precision, then, is that it is the limit of the coefficient of stability, as the time between testings becomes infinitesimal.

I'm not entirely sure if I'm interpreting the secondary question right, but IRT precision is a measure of precision under IRT, and ICC is a measure of reliability under CTT. The main difference is that CTT expresses reliability as a single value, while IRT expresses precision for different values of the underlying trait. This isn't specific to absolute agreement, though, so maybe I'm misunderstanding.

References

Improve this answer
$\endgroup$
2
  • $\begingroup$ I do not understand your vocabulary: "accuracy is to validity." Are accuracy and validity synonyms? Are reliability and precision synonyms under CTT? $\endgroup$
    – Joel W
    Commented May 8, 2015 at 0:00
  • $\begingroup$ It's more of a parallel. Put shortly, there is no "precision" in CTT, only "reliability." $\endgroup$
    – Christian Hummeluhr
    Commented May 8, 2015 at 6:27

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.