Why doesn't psychology impose higher standards to solve its replication crisis?

Question

This question is inspired by a previous one on Physics.SE. If we believe the answer to that question, particle physics doesn't suffer nearly as much from the reproducibility crisis because it has much higher standards than the social sciences.

The 'replication crisis' is that many effects in social sciences (and, although to a lesser extent, other scientific fields) couldn't be reproduced.

...

If anything, there is a suspicion that the practices at the LHC might even result in the opposite of the 'replication crisis;' analyses that find effects that are somewhat significant might be examined and tweaked until they decrease.

But if that answer is true, that suggests a simple way to solve the reproducibility crisis in fields like psychology: just impose higher standards.

Question: why hasn't psychology (and other fields as a whole) done that? Or is that answer misrepresenting what psychologists already do?

Answer 1

High standards are being proposed, but not taken up very quickly.

Psychology is currently undergoing a transition in methods between the traditional approach of hypothesizing after results are known (e.g., Bem, 1980) and new statistical and design methods that have fewer false positives, such as focusing on effect sizes and confidence intervals over significance, pre-registration, Registered Reports, sharing of code and data, replication, meta-analysis, multi-site replication such as the Psychological Accelerator and Many Labs, error detection tools, etc. Note that even if these techniques are demonstrably superior for inference quality, their uptake within the discipline and coverage in undergraduate and graduate coursework depends on current faculty changing their beliefs and practices. Historically in science, this is a slow and fraught process (Kuhn, 1962); sometimes fields have had to wait for faculty to die before new methods can take full root. It doesn't look like it will take that long, but I mention this to highlight the difference between superior methods existing and whether they are implemented within a field. Psychology is currently undergoing something of a struggle between reformers and supporters of the status quo.

@Chris Rogers makes another good point that measurement of psychological constructs such as motivation is necessarily indirect, and people are exceptionally variable by context, making the design of measures more contextual than in physics. The lack of stable evidence is not an indication of standards alone.

Answer 2

The basis of your question

..particle physics doesn't suffer nearly as much from the reproducibility crisis because it has much higher standards than the social sciences

You said this in response to innisfree (2019)

The 'replication crisis' is that many effects in social sciences (and, although to a lesser extent, other scientific fields) couldn't be reproduced. There are many factors leading to this phenomenon, including

• Weak standards of evidence, e.g., $2\sigma$ evidence required to demonstrate an effect
• Researchers (subconsciously or otherwise) conducting bad scientific practice by selectively reporting and publishing significant results. E.g. considering many different effects until they find a significant effect or collecting data until they find a significant effect.
• Poor training in statistical methods.

You need to bear in mind, that what innesfree said is only part of many different possible reasons, which also doesn't really go in depth into the reasons. Saying that social sciences has much lower standards required to demonstrate an effect can be seen as inflammatory remarks which has no real merit due to the complexity of the situation.

The short answer

The final part of your question:

why hasn't psychology (and other fields as a whole) done that? Or is that answer misrepresenting what psychologists already do?

Makes your question a great (unbiased) question, and in a sense, that answer is misrepresenting what psychologists already do ... in most cases ... but then, there are arguments which could be made for those other situations.

Psychology and the Replication Crisis

As I pointed out in the comments, there is a lot of debate, even within the field of social sciences, on what is pseudoscience within the social sciences and as you can see within the meta site, whilst I am a defender of Freudian and Neo-Freudian psychology, these areas of psychology, along with a few others are considered by some here to be pseudoscientific.

You can read those links to the meta site for more on that, but it is the replication crisis which underpins the debate and what is considered to be scientific within research and results, and what is pseudoscience.

The basis of the replication crisis is that:

for anything to be considered scientific (not pseudoscience), experimental results should be reproducible and verified by other researchers

Henriques (2016) points out that

psychology was defined by the application of scientific method(s) and psychologists conduct valuable research and have developed some key insights into animal behavior, cognition, consciousness, and the human condition.

The demarcation between what is science and what is pseudoscience within social sciences has been debated in many books and an attempt to create a solid demarcation has been put forward by Sven Ove Hansson (1996; 2008; 2013)

Where does the problem lie?

As @StevenJeuris pointed out in the comments,

The more 'high-level' observations are performed, the more room there is for noise, variance in data, and the need for subjective interpretations in the process.

'High level' cognitive functions (Driver, et al. 2007) (e.g. memory, awareness, thoughts, emotions, attention, executive function...) in humans cannot be physically recorded and measured, and therefore they can only be measured and recorded subjectively.

How does this pose a problem?

Where there is no point of reference, either you need a very large group of people who fit a specified criteria in order to assess statistically how predominant certain traits are evident, or you start by assessing a few people (or maybe even one person) with a view to encouraging more research.

This could be seen when looking at Erikson’s Life Stages of Development (Erikson & Erikson, 1998) — also discussed in my answer to What happens to someone missing teenage years of development?.

Erik Erikson’s 8 life stages of man, originally put together in 1950 in his book Childhood and Society, was delevoped based on his own experiences of life. A 9th stage was added in 1998 as the Eriksons came to believe that,

the role of old age needs to be reobserved, rethought (Erikson & Erikson, 1998: p. 62).

You could say that this theory on the stages of development is pseudoscientific, but without a better frame of reference, Erikson provided the best he could for the time being.

Where subjective measurement can also pose major problems is highlighted in Forde (2017). When you are trying to profile and treat for recidivism amongst sexual offenders, you can only rely on subjective observations. What Forde pointed out is that in this area of psychology within the prison system, psychologists are relying on questionnaires along with the psychologist's opinion on what is required, and he highlighted areas in which this can pose big problems, and why.

How are standards kept as high as possible?

The debate as it stands is the attempt by all social scientists to ensure the best methods of observation can be maintained. Where deficiencies are highlighted, it is the social scientists' duty to ensure that those deficiencies do not affect their work, and where more research is needed, they attempt to conduct the necessary research to correct those deficiencies where possible.

The thing is that you need to remember that what fits one person, may not fit all people, as everyone is different in their own right. Each and every person will have been subjected to different internal and external influences. Without objective measurement techniques, there is no other way of scientifically coming to an agreement on what occurs and how within the high level cognitive functions.

So going back to the crux of your question,

Does particle physics have much higher standards than the social sciences?

That is open to opinion, but based on the evidence I have put forward, the standards are as high as possible to date within the social sciences, and they are under constant review.

References

Driver, J., Haggard, P., & Shallice, T. (2007). Introduction. Mental processes in the human brain. Philosophical Transactions of the Royal Society B: Biological Sciences, 362(1481), 757-760. doi:10.1098/rstb.2007.2085 pubmed: PMC2042528

Erikson, E. H. & Erikson, J. M. (1998). The Life Cycle Completed. New York, NY: W.W. Norton.

Forde, R. A. (2017). Bad Psychology: How Forensic Psychology Left Science Behind. London, UK: Jessica Kingsley Publishers

Hansson, S. O. (1996). Defining Pseudoscience. Philosophia Naturalis, 33(1): pp. 169—176
Info. at: https://philpapers.org/rec/HANDP

With the publication of the second issue of the 2013 vintage, Philosophia Naturalis has ceased publication.
Please order the single issue of the printed issue directly from the publisher via e-mail.

(translated from https://www.klostermann.de/epages/63574303.sf/de_DE/?ObjectPath=/Shops/63574303/Categories/Zeitschriften/Phil-nat)

Hansson, S. O. (2008) Science and Pseudo-Science. The Stanford Encyclopedia of Philosophy (Summer 2017 Edition)
Retrieved from: https://plato.stanford.edu/archives/sum2017/entries/pseudo-science

Hansson, S. O. (2013). Defining Pseudoscience and Science. In M. Pigliucci & M. Boudry (Eds.) Philosophy of Pseudoscience: Reconsidering the Demarcation Problem. Chicago, IL: University of Chicago Press.

Henriques, G. (2016). The “Is Psychology a Science?” Debate. Psychology Today
Retrieved from: https://www.psychologytoday.com/blog/theory-knowledge/201601/the-is-psychology-science-debate

innesfree (2019). Answer to: How do we know the LHC results are robust? Physics StackExchange
Retrieved from: https://physics.stackexchange.com/a/468900