I was sent an email by a researcher with the following question:

You wrote that difference scores are okay to use if measurement error is minimised.

  • Does that imply that biological data are more reliable for statistically analysing change than say a self-report personality scale?
  • For example, I've measured fear via cortisol and also self-report scale, does this mean that I can look at hormone measurements in a difference score but not on the self-report scales?
  • $\begingroup$ The following paper might be interesting: Tomarken, A.J. (1995). A psychometric perspective on psychophysiological measures. Psychological Assessment, 7 (3), 387-395. $\endgroup$
    – Gala
    Jun 30, 2013 at 18:59
  • $\begingroup$ Cortisol measures would not tell what kind of stressor has triggered the release - fear, sorrow, rage, allergies, physical trauma, etc. $\endgroup$ Apr 8, 2016 at 10:45

1 Answer 1


I think the relevant reliability would be the expected test-retest reliability of the measure at the two time points which are otherwise being used to examine difference scores. Thus, if you were looking at difference scores derived from an experiment where measures are taken one day apart where an intervention is applied on one of those days, then you would want to know the correlation between those two days in the absence of any intervention.

In general, I would question your assumption that hormone measures are more reliable than self-report personality measures. I'm not experienced with hormone reliability data, but a good personality scale should have test-retest reliability around .80 or 90. My intuition is that hormone measures have much lower reliability than this, but I'd be very interested in anyone who can cite test-retest reliability measures for various hormone measures.

I found this discussion of cortisol test-retest reliability (Nicolson):

Test-retest reliability of acute stress response measures appears to be low, probably because of both the underestimated noise introduced by spontaneous pulsatile activity (Young, Abelson, & Lightman, 2004) and the tendency of the cortisol response to habituate following repeated exposures. Low reliability of cortisol outcome measures means that laboratory stress experiments are particularly vulnerable to Type 2 error. This issue is important in all studies, but especially needs to be taken into account in intervention studies, where stress reactivity is compared pre- and postintervention. More analyses are needed to determine how many cortisol measures per session and how many repeated sessions are necessary to obtain reliable measures of stress reactivity for different subject populations and stressors (Gunnar & Talge, 2007; Hruschka et al., 2005).



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.