3
$\begingroup$

Consider for exemple the stereotype threat, of which a recent meta-analysis can be found here. People try to assess how can a stereotype might explain a difference between two groups.

However, they almost never control for IQ or Big Five which seems to me to be latent variables with much more explanatory power.

My questions are :

  1. Why don't they ever control for these two important factors ?
  2. Does it mean that the conclusion of these studies are flawed in some way ?
Improve this question
$\endgroup$
3
  • $\begingroup$ Genetics are another major explanatory factor for many psychological constructs, including IQ and personality. Just think practically what it would take in terms of power, to control for such factors - studies would need orders of magnitude more participants. $\endgroup$
    – Arnon Weinberg ♦
    Nov 17, 2020 at 20:26
  • 1
    $\begingroup$ Yes I understand that it would greatly reduce the power of the studies but wouldn't it be better to accept null hypothesis because of a too small sample rather than derive conclusions from flawed premisses ? $\endgroup$
    – outofthegreen
    Nov 18, 2020 at 13:35
  • $\begingroup$ I always wondered about the opposite. Why would researchers control for all sorts of variables although their theories and hypotheses do not say anything about these variables. Including variables without logic has little to do with the scientific method. $\endgroup$
    – Johannes Titz
    Aug 23, 2021 at 11:51

2 Answers 2

Reset to default
4
$\begingroup$

A quick look at your reference focussed on a meta-analysis of an intervention. Evaluating interventions is mostly about estimating the mean effect of an intervention. So, even if personality or intelligence influences the outcome, this is not directly relevant to evaluating whether on average the intervention has an effect.

In other contexts, controlling for personality and intelligence maybe quite relevant. In general, controlling for personality or intelligence is more relevant (a) where the predictors are conceptually or empirically related to personality or intelligence, and (b) where personality or intelligence is a strong predictor of the outcome of interest.

There are a whole range of psychological constructs that overlap substantially with either personality or intelligence. And in those cases, you do have a body of literature that examines the degree of overlap with personality or intelligence. And there's also literature that examines the incremental prediction of the variable of interest relative to personality.

For example:

  • Grit: Is it different from conscientiousness?
  • Assessment centers: Do they incrementally predict over and above intelligence?
  • Type D personality: Is it more than a composite of neurotcism and extraversion
  • Trait emotional intelligence: Is it just a weighted composite of the Big 5
  • Ability emotional intelligence: does it predict workplace outcomes over and above intelligence?

Researchers are always confronted with issues of which variables to measures. It is often not practical to measure everything. In particular, intelligence takes quite a bit of time to measure well. So you often get research that doesn't control for these factors. And then later on, other studies come along and fill the gap. There is also a bit of a process "construct proliferation" in psychology. A researcher comes up with a new pet construct. And other researchers point out later on how it's not that different to personality or how it doesn't incrementally predict. A lot of heated debates arise over these things.

Researchers also differ in theoretical orientation and practical concerns. For instance, in industrial/organisational psychology, intelligence and Big 5 personality assessment are very well embedded. So, incremental prediction over and above these factors is a common question for a researchers to ask. In other fields, they may be less grounded in personality or intelligence frameworks.

Improve this answer
$\endgroup$
5
  • 1
    $\begingroup$ Ideally, in a randomized experiment there should be no differences in your groups in other factors - that's why you randomize. You could still investigate variables that affect the strength of the intervention, but to evaluate the intervention in general it's not necessary. Not sure if the interventions OP referred to involved randomized experiments. $\endgroup$
    – Bryan Krause ♦
    Nov 18, 2020 at 1:47
  • 2
    $\begingroup$ Hi Bryan. I agree with all your points. $\endgroup$
    – Jeromy Anglim
    Nov 18, 2020 at 2:27
  • $\begingroup$ @BryanKrause this is the point I'm struggling with, as a student in statistics, I expect some difference between the groups even if they are sampled from the same population, just because of randomness so I wonder if controlling for iq between the measuring and the control group might not improve our interpretations of the psychological litterature. $\endgroup$
    – outofthegreen
    Nov 21, 2020 at 11:30
  • $\begingroup$ @JeandeLéry What about all the other possible confounders you fail to measure? In a small trial, you may have imbalances that affect the result, but not enough data to effectively correct for it. In a larger trial it becomes exceedingly unlikely to find large imbalances on any one variable. Same for meta analyses. Unless there is a bias where a confounder makes people drop out of one arm but not the other, randomization solves the issue. $\endgroup$
    – Bryan Krause ♦
    Nov 21, 2020 at 13:11
  • $\begingroup$ I'm no psychology graduate, let alone expert so I thought that IQ and big five were standards like weight and height in biometrics and thus it would make sense to control for these two as when I am doing a linear model in biology I always control for weight & height. But then, if actually there are much more significant cofounders of a size comparable to big 5 and IQ, then my idea was flawed and I am glad you corrected it $\endgroup$
    – outofthegreen
    Nov 21, 2020 at 14:32
0
$\begingroup$

You only control for factors that have been proven to matter. IQ has not been shown to be significant. So why measure or control IQ variation if there is no evidence of difference.

In fact there is evidence that the activation of stereotype threat confounds cognitive performance and therefore IQ. If a person has their stereotype threat activated, and they perform poorly in an IQ test, is that a fair representation of their cognitive abilities? Many articles and studies prove the confounding ability of stereotype threat and the need to minimise this factor in IQ testing, it seems complex and unfair to apply a measure that is significantly influenced by the stereotype threat factor that you are studying.

- Stereotypes impair intellectual performance blog

Improve this answer
$\endgroup$
15
  • 1
    $\begingroup$ Well I don't know, in terms of math performance which is the most famous example of stereotype threat I can think of, IQ seems to me the best predictor available so I don't really understand what you're saying by "You only control for factors that have been proven to matter. IQ has not been shown to be significant". $\endgroup$
    – outofthegreen
    Nov 18, 2020 at 13:37
  • 1
    $\begingroup$ @JeandeLéry The current question is fairly general, and I think it's a good general question. That said, I think you might have an interesting question that you could ask separately regarding stereotype threat. $\endgroup$
    – Jeromy Anglim
    Nov 19, 2020 at 5:04
  • 1
    $\begingroup$ The assumption prejudice can be controlled and managed still remains highly problematic. The research literature is on implicit and explicit bias is far more contentious and disappointing than what you realise. Happy to discuss the literature rather than insisting on broad principles that are not based on usual nor proven practice. Happy to discuss specific studies were IQ has been used to mitigate and manage its confounding effect - thelancet.com/journals/lancet/article/PIIS0140-6736(18)32267-0/… $\endgroup$
    – Poidah
    Nov 23, 2020 at 2:12
  • 1
    $\begingroup$ Otherwise prejudice remains quite a difficult concept to measure for while the research in the field seems highly contested hotly debated - sciencemag.org/news/2020/03/… $\endgroup$
    – Poidah
    Nov 23, 2020 at 2:14
  • 1
    $\begingroup$ There is also evidence and debate that IQ tests are racially biased - sites.google.com/a/haverford.edu/the-psychology-of/contact/… $\endgroup$
    – Poidah
    Nov 24, 2020 at 9:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.