9
$\begingroup$

Thanks to this website, I've seen a number of papers and scholarly articles that deal with cognition. I'm interested in how repeatable the findings are that are discovered as a result of experiments.

For example, when a mechanical system is tested, it is tested to work the same way every single time:

  • A lock opens and closes when a key is turned.
  • A car door opens up to the maximum degree allowed by design.

How much is a cognitive system like a mechanical system? Will it work the same way every time?

For example, I'm thinking about the design of an experiment I've been reading about: The Ikea Effect Experiment. Volunteers are asked to assemble Ikea boxes and bid on the box after assembly. The experiment shows that people value things they built themselves more than things that are just offered to them.

Now if the machine-like definition of repeatability is applied to this experiment: Would test subjects value boxes they built themselves next time they do an experiment? How about next month? What about in 3 years? Would people in the 1940's have the same response?

Have there been any studies that deal with this "repeatability" question? For example, studies that used the same test subjects doing the same experiment multiple times?

Improve this question
$\endgroup$
  • 3
    $\begingroup$ When dealing with humans, you always have to take into account learning effects in experimental design. The main difference with the type of machines you are referring to is those machines don't have a memory. Humans do, and subsequent tests will always be influenced (in predictable and unpredictable ways) by previous experiments. $\endgroup$ – Steven Jeuris♦ Nov 18 '12 at 1:10
  • 2
    $\begingroup$ I think that this question is very broad and could be improved if you specified which area you are interested in. As Steven pointed out running the same experiment twice in human subjects is done when effects of learning are investigated. If no learning effects are expected and performance is hypothesized to be stable, researchers often investigate 'retest-reliability'. So if you google/pubmed the area you are intetested in (e.g. memory) plus 'learning' or 'retest-reliability' you will find lots of studies. $\endgroup$ – jokel Nov 18 '12 at 9:24
  • $\begingroup$ The area that I was originally interested in is the "stimulus-response" or "sequence of tasks performed - result". I don't know what the correct word for this, but it's not memory. Is it behavior? $\endgroup$ – Alex Stone Nov 18 '12 at 19:30
5
$\begingroup$

Humans are, as you would expect, much noisier than physical systems. As such, replication of experiments in psychology is very important. Replication provides evidence that the original results were not spurious.

As pointed out in the comments, replication with the same subject pool is not always the way to go. Many tasks have learning effects, and so you would not expect to achieve the same results by re-administering the experiment.

There is a vast statistical literature on reliability testing. A good place to start would be at the Wikipedia article for Reliability (psychometrics)

Improve this answer
$\endgroup$
  • $\begingroup$ If I understand this correctly, there are two types of experiments then: the one shot ones, and those focused on learning, right? $\endgroup$ – Alex Stone Nov 19 '12 at 4:01
  • $\begingroup$ I suppose you could say that, but no one actually does. It's just as arbitrary as saying "There are two types of experiments: those that study the effects of aging on memory, and those that don't." Sure it's true, but it's meaningless. $\endgroup$ – Jeff Nov 19 '12 at 4:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.