Which Cognitive Sciences are the most affected by the Replication Crisis?

Question

As many here are probably aware, a replication crisis seems to have been occurring in many scientific fields for some time now. Allegedly, Psychology is at the center of this crisis, with many of its high-profile studies turning out to be impossible to replicate.

As someone who tries to regularly consume academic literature in order to educate oneself, this makes me wonder what areas within the broader umbrella of 'Cognitive Sciences' have had the most replication attempts fail and/or have the most information available about replication failures.

Answer 1

One finding from the Reproducibility Project was the following:

Van Bavel et al wrote that

Inbar (2) notes that the reproducibility rate of social psychology (28%) is lower than cognitive psychology (53%).

In general, one might expect that subdisciplines with the following features to have greater reproducibility issues:

• particularly interested in novel effects
• many researcher degrees of freedom (e.g., analytic choices, many possible outcomes variables, many possible predictors, etc.)
• between subjects designs rather than repeated measures designs
• lower statistical power (typically because getting a sufficient sample size requires substantial resources)
• those embracing fewer open science practices (e.g., pre-registrration, open data, etc.)

So, for example, cognitive psychology often uses repeated measures designs to examine fairly intuitive cognitive effects.

Personality and individual differences research is often concerned with quantifying the degree of relationships and often has very large sample sizes.

A lot of neuroscience research traditionally used techniques like fmri that often involves many researcher degrees of freedom (e.g., many brain sites), and the research is expensive per participant so sample sizes are often too small.

Social psychology has a tradition of being interested in novel experimental effects. Many of these studies have traditionally been underpowered (e.g., 20 or 30 participants per group, when perhaps 100 or 200 per group would be required for adequate power), and they have often had some degree of flexibility (e.g., choice of outcome variable; case exclusion rules; whether to transform; whether to covary for something; etc.).

These effects also combine with publication bias (preferences to publish significant effects) and file draw effects to reduce replication (although the same bias reduces the awareness of these issues).

That said, in general, a lot is happening in the open science area to try to both increase understanding of replication issues, as well as change research practices.

Reference

• Van Bavel, J. J., Mende-Siedlecki, P., Brady, W. J., & Reinero, D. A. (2016). Contextual Sensitivity Helps Explain the Reproducibility Gap between Social and Cognitive Psychology.