In Richard Feynman's classic Cargo Cult Science speech. He talks about a type of scientific error made, where a person takes a previous experimental result for granted when designing an experiment.

Other kinds of errors are more characteristic of poor science. When I was at Cornell, I often talked to the people in the psychology department. One of the students told me she wanted to do an experiment that went something like this--it had been found by others that under certain circumstances, X, rats did something, A. She was curious as to whether, if she changed the circumstances to Y, they would still do A.

So her proposal was to do the experiment under circumstances Y and see if they still did A. I explained to her that it was necessary first to repeat in her laboratory the experiment of the other person--to do it under condition X to see if she could also get result A, and then change to Y and see if A changed. Then she would know the the real difference was the thing she thought she had under control.

I'm not sure I understand what sort of fallacy is being committed in this instance. What exactly is the problem?


  • $\begingroup$ Also see pathological science $\endgroup$
    – Dan D.
    Dec 12 '13 at 11:43

I think it's a basic concept of experimental design that you have control groups. So for example, if you want to study the effect of X (e.g., a drug) on Y (e.g., performance), you could manipulate X (e.g., give half the drug and the other half no drug) and randomly assign the participants to either group. Random assignment is a strategy for making the groups on average equivalent. Thus, it is an attempt to control for confounding factors that might cause Y.

Of course, you could say that other researchers have conducted the equivalent of the control group condition, and just do a one sample study with just the drug condition. However, studies differ in all sorts of ways (participants differ, your exact measures might differ, the context might differ, etc.). Thus, you couldn't unambiguously attribute any changes to the effect of the drug.

That said, we often do compare the experimental outcome of different studies that differ in some respect. For example, meta-analyses often examine moderators which compare effect sizes across study features. While this is useful, you'll often get a cleaner answer to your research question by incorporating the moderator into a specific study where there are fewer alternative sources of explanation.


I'm a student of psychology. Every week we read several papers and discuss them in class. I'm in my seventh semester now, but as yet I have not read a single psychological journal article that we did not find fault with! Often the basic premises are at best tentative. Very often the statistical methodology used is not actually applicable to the case at hand. Almost always the effects are extremely small. Rarely is the sample representative. And so on.

For myself, I have come to the conclusion, that for any experiment what has been found is that maybe some things are somehow a bit similar to what this experiment may appear to have shown. And maybe not.

The proven facts of psychology can be counted on the fingers of one hand: human beings perceive; they have emotions; they act. That's about it.

If you do research, you do it on the basis that you know nothing, and the truth might actually be the opposite of what other experiments have shown. Scientists are sceptics, that is the basic frame of mind that you do research in. Scientists don't believe, they question.

Even in physics, no-one in their right minds would claim that they know that there are electrons. Electrons are simply a good model to explain what happens when you turn on the light. As long as that model explains what we observe and helps predict results, we use it. As soon as it stops being useful, we abandon it. In the case of electrons we are already abandoning that model. And it won't be long when Bosons and Higgs particles will no longer prove useful, too. There is no such thing as scientific truth. Especially not in psychology.


I haven't heard Fineman's speech, but from your description it seem that the problem he is getting at is that just because a certain finding has been made in one experiment, it is not a universal truth. It is quite possible that, when you try to replicate the findings, you will not be able to obtain them again. Jeromy already said something along these lines when he mentioned meta-analysis.

Fineman goes so far as to say, that the only way really to be sure is to conduct the experiment yourself. In other words, when you want to know the difference between treatment X and treatment Y, you have to design an experiment with both of these treatments yourself rather than rely on findings from someone else.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.