6
$\begingroup$

'So, replicability [of all studies in this book] is somewhere between 12% and 46%. Even if half of the results are replicable, we do not know which results are replicable and which one’s are not.'

'Readers of “Thinking: Fast and Slow” should read the book as a subjective account by an eminent psychologists, rather than an objective summary of scientific evidence.'

A Meta-Scientific Perspective on “Thinking: Fast and Slow

I have no background in social sciences or statistics so I don't know if claims and math in this article are correct. Could somebody with more knowledge comment on this? This is HUGE if true.

$\endgroup$
3
  • 1
    $\begingroup$ Why "huge if true"? Pop-science books aren't really how science works and the "replication crisis" in psychology is quite well-known. $\endgroup$ Jun 29, 2021 at 22:49
  • 2
    $\begingroup$ Well the book informed countless sci popularizers and countless people. I've seen this book praised by people with psych backgrounds as 'One of the very few pop-psych books that largely got the scientific literature right'. reddit.com/r/AskSocialScience/comments/5t17ry/… $\endgroup$ Jun 30, 2021 at 5:44
  • 5
    $\begingroup$ I have expertise in social psychology and statistics. A lot of the work mentioned hasn't replicated: there is not a consensus about exactly how much. Not replicating doesn't mean not true: interpreting replications is not trivial. $\endgroup$ Jun 30, 2021 at 10:25

1 Answer 1

8
$\begingroup$

This is complicated. There's no easy answer, but the outlook for replicability/reproducibility of a lot of the empirical evidence is not great.

The R-index (that the blog authors use to rank the chapters) is itself a heuristic, which shouldn't be taken as statistical evidence that a study will necessarily fail to replicate. That being said, it could potentially be considered a useful heuristic, as is the idea that people process information over time (i.e., some mental processes happen more quickly than others).

An issue of Psychological Inquiry might be helpful to people who want to learn about the most recent conversation among a select group of psychological scientists. This journal publishes a "target article" and allows other people in the field to comment, to which the original authors reply. The whole conversation is published as an issue - this one is from April 2021. Without going into detail, many priming effects do not replicate, and social priming might be a terrible term to use, in part because it is associated with questionable scientific literature (target article here, one cherry-picked commentary here, the authors of the target article reply to the commentaries here).

Kahneman himself warned of problems with the literature in 2012 (see a scientific news report here (4) and his email here).

  1. Sherman, J. W., & Rivers, A. M. (2021). There’s nothing social about social priming: derailing the “train wreck”. Psychological Inquiry, 32(1), 1-11.

  2. Harris, C., Rohrer, D., & Pashler, H. (2021). A train wreck by any other name. Psychological Inquiry, 32(1), 17-23.

  3. Sherman, J. W., & Rivers, A. M. (2021). A final word on train wrecks. Psychological Inquiry, 32(1), 49-52.

  4. Yong, E. (2012). Nobel laureate challenges psychologists to clean up their act. Nature News.

Hope this helps.

$\endgroup$
2
  • $\begingroup$ Thanks, yes, it is helpful. I've read that the priming research was a mess, but priming studies are only like 15% of the studies used in the book. This means some 40+% of studies on effects other than priming are expected not to replicate as per this statistical tool. I was wandering about those other effects too. $\endgroup$ Jul 1, 2021 at 7:13
  • $\begingroup$ True, there was a lot in that book. Is there a specific topic you were curious about? They list numerous chapters that received a "failing grade" in the R-index blog post you mentioned (low R-index means that replications are heuristically less likely to produce similar results). $\endgroup$
    – pep
    Jul 5, 2021 at 2:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.