Are cortisol measures more reliable than self-reported fear measures for looking at difference scores? I was sent an email by a researcher with the following question:


  You wrote that difference scores are okay to use if measurement error
  is
  minimised.
  
  
  Does that imply that biological data are more reliable for statistically analysing change than say a self-report personality
  scale?  
  For example, I've measured fear via cortisol and also self-report scale, does this mean that I can look at hormone measurements in a
  difference score but not on the self-report scales?
I think the relevant reliability would be the expected test-retest reliability of the measure at the two time points which are otherwise being used to examine difference scores. Thus, if you were looking at difference scores derived from an experiment where measures are taken one day apart where an intervention is applied on one of those days, then you would want to know the correlation between those two days in the absence of any intervention.

In general, I would question your assumption that hormone measures are more reliable than self-report personality measures. I'm not experienced with hormone reliability data, but a good personality scale should have test-retest reliability around .80 or 90. My intuition is that hormone measures have much lower reliability than this, but I'd be very interested in anyone who can cite test-retest reliability measures for various hormone measures.

I found this discussion of cortisol test-retest reliability (Nicolson):


  Test-retest reliability of acute stress response measures appears to
  be low, probably because of both the underestimated noise introduced
  by spontaneous pulsatile activity (Young, Abelson, &amp; Lightman, 2004)
  and the tendency of the cortisol response to habituate following
  repeated exposures. Low reliability of cortisol outcome measures means
  that laboratory stress experiments are particularly vulnerable to Type
  2 error. This issue is important in all studies, but especially needs
  to be taken into account in intervention studies, where stress
  reactivity is compared pre- and postintervention. More analyses are
  needed to determine how many cortisol measures per session and how
  many repeated sessions are necessary to obtain reliable measures of
  stress reactivity for different subject populations and stressors
  (Gunnar &amp; Talge, 2007; Hruschka et al., 2005).


References


Nicolson, N. A. Measurement of Cortisol. http://www.sagepub.com/upm-data/17430_Chapter_3.pdf