1
$\begingroup$

Any science has to be built on the scientific method. While in all fields there are novel phenomena that are not yet understood, an essential step to doing robust science is understanding how one's instruments work, before using them to measure something more complicated.

For example, in physics, we might not yet know what is a Higgs boson, but we understand exactly how magnets and light detectors work. When we make an experiment and gather data, we can select the simplest model which explains all of the observed data, except for the error introduced by those magnets and light detectors, which can be precisely estimated.

Or, for example, in neuroscience, we still have very limited understanding how hippocampal ripples give rise to memory consolidation and what that even means, but the electrophysiological and optogenetic instruments are quite precise, we already sort of understand how single neurons work, and are continuing to iteratively improve instrumentation until we can record a sufficient amount of information from a brain area to fully be able to model what is happening inside of it.

I am trying to understand, where does psychometrics land in this regard. From what I have read, psychometricians use questionnaires as their instruments. I would like to know, how do psychometricians learn what exactly their instrument measure, and how do they estimate the accuracy of the instruments. Here are some naive questions from the top of my head

  1. Let's say I want to construct a question that measures whether a person is depressive. Sure, questions like "are you sad often" or "do you find it hard to get out of bed in the morning" sound plausible to me, but how do I test that they measure what I want? How do I even decide what exactly I want to measure? The definition of depression in DCM5 is a bit imprecise...
  2. Let's say I know that my question measures depressiveness, but it also is influenced by a whole bunch of other stuff: Is the person honest; Are they generally an optimistic or pessimistic person; What mood are they in today; Do they understand the question; etc. How can I control for these confounds? How can I estimate the uncertainty in my instrument, in order to have an informed decision about the performance of a model I want to fit?
  3. Let's say I have gathered a few questions that all measure depressiveness, and have some known uncertainty. I would proceed to fit some model in order to estimate depressiveness. I have heard that common models in modern psychometrics are Rasch models and their extensions. As far as I understand, these models assume that the uncertainty in the questions is caused by random noise, which is independent across questions. I am interested to know whether and how this assumption is typically justified. Naively, I would say that there is no randomness in this process - if asked the same question twice, most people would likely answer the same, if they have the minimal incentive to be honest and precise. Thus, any uncertainty is due to uncontrolled confounding factors. It is certainly plausible that a single confounding factor would affect multiple questions simultaneously. For example, a pessimist would likely write lower numbers for most questions even if they are perfectly healthy. Such confounding would result in correlated noise, and could significantly bias inference. How is this controlled for?
$\endgroup$
9
  • $\begingroup$ These are excellent questions. I am not a psychometrician so am not best placed to answer but I recommend having a look at the work of Eiko Fried, starting with this paper - eiko-fried.com/wp-content/uploads/… $\endgroup$ Jan 12 at 6:08
  • $\begingroup$ @MalcolmForbes Thanks :). Btw, I picked depression just as an example, it is not my ultimate goal. Still, a very interesting topic $\endgroup$ Jan 12 at 8:29
  • 1
    $\begingroup$ @MalcolmForbes The publication by Eiko Fried is an interesting read. Indeed, they arrive at the main conclusion that having a solid mechanistic theory of the construct (in this case, biochemical/medical state of depression) is integral to understanding what exactly is being measured, and thus essential in constructing a valid and reliable metric. $\endgroup$ Jan 12 at 9:24
  • 1
    $\begingroup$ @BryanKrause thanks, I will have a look. Let's see if their discussion sections are more detailed than what I have encountered so far $\endgroup$ Jan 12 at 20:52
  • 1
    $\begingroup$ These particular examples may not be the best cases for finding arguments about other measures. But I do think the original BDI paper explains their validation procedure clearly. $\endgroup$
    – Bryan Krause
    Jan 12 at 20:55

1 Answer 1

1
$\begingroup$

I'm not a psychometrician but I use these tools and have some familiarity with the topic so I thought I'd offer some brief thoughts.

How does one validate a psychometric instrument?

There are a few steps and this list isn't exhaustive:

  • Conceptualise and clearly define a construct (not done particularly well in depression, which has changed definitions over time)
  • Develop questions/statements that can adequately identify the construct (see Eiko Fried's paper in my comment on the poor overlap between seven commonly used depression rating scales)
  • Establish norms and appropriate cut-offs (for depression, ensuring your criteria do not inappropriately diagnose normal sadness or neurotic/pessimistic dimensions of personality as depression)
  • Assess reliability between the individual items/questions (are the questions internally consistent?)
  • Assess construct validity (does your instrument correspond to similar instruments that have already been validated?)
  • Assess criterion validity (does your instrument correspond to the 'gold standard' for diagnosis?)
  • Assess test-retest reliability (does it measure the same thing over time?)
  • Use factor analysis to understand if there may be subtypes or dimensions to the construct

Depression, as it's defined in the DSM-5-TR and the ICD-11 (the two most commonly used diagnostic manuals) has modest inter-rater reliability and fairly poor validity, in part because we don't have a robust theoretic model of depression, have no objective tests to identify the presence or absence of depression, and have significant methodological limitations affecting the instruments we use to measure depression (highlighted in your question). Currently, the 'gold standard' by which questionnaires are compared to is the opinion of a psychiatrist, which is subjective and prone to substantial noise.

I don't have good answers to many of the important questions you ask. I hope someone with specific psychometric experience will. I also hope that over time, with respect to depression symptom measurement, we'll move from discrete time point assessment of symptoms (e.g. you visit your doctor and see them again one month later) to ecological momentary assessment, involving multiple repeated measures of mood and behaviour from the same person in real-time to obtain a richer understanding of the emergence of trends and establishment of each individual's baseline.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.