5
$\begingroup$

Can someone please explain this to me because it's driving me bonkers. In cognitive research when one group of participants are exposed to a number of crossed factors manipulated by the researcher, is it a quasi-experimental design or a true experiment?

All the sources I read always mention random assignment and random sampling when talking about true experiments. So do cognitive science experiments such as the repeated measures factorial design a lot of studies use have the right to claim that they establish causal links between the outcome and the manipulations?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
5
$\begingroup$

If the researcher randomly samples individuals from the population of interest and randomly assigns them to different experimental treatments, then it is a true experiment. Quasi-experimental design occurs when there is no random assignment to treatments.

In general, true experiments can make claims about causality. Since all subjects in the experiment are randomly assigned to a particular treatment, any systematic differences between the groups after the experiment are likely to be due to the treatment.

However, this is all a game of probabilities. There's always a chance that any systematic difference between groups is from the random assignment. This is why inferential statistics are such an important aspect of evaluating the outcomes of experiments. We can estimate the probability of a particular difference between groups occurring due to random variation. If that probability is low (the typical convention is below 5%), then we can make a causal claim about the treatment.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Hi Josh. But what happens if there are no groups per se and everyone is exposed to all manipulations? Group differences or random assignment are completely irrelevant then, but does a lack of a control group mean it's not a true experiment? $\endgroup$
    – Rooirokbokkie
    Commented Jun 4, 2015 at 15:58
  • 1
    $\begingroup$ You're describing a repeated-measures design. In this kind of design, each subject is tested under different experimental conditions, and the performance in one condition is compared to another. It's still a true experiment, because we can look at performance in a particular condition and compare it to performance in another condition, with the only thing that differed between conditions being the experimental manipulation. $\endgroup$
    – Josh
    Commented Jun 4, 2015 at 16:49
  • $\begingroup$ Thqt is my reasoning as well. But influential texts such as Shadish, Cook and Campbell (2002) seem to argue that without randomization you have no experiment as there is no baseline to compare the absence of treatment to for example. Though participants technically serve as their own baseline any interaction between the factor and the participant may be mediated by a particular trait "within" the participant that the researcher did not control for $\endgroup$
    – Rooirokbokkie
    Commented Jun 4, 2015 at 19:14
  • 2
    $\begingroup$ There is still randomization in a repeated-measures design. If I want to compare how fast people are at finding an object with different numbers of distractors, then I can test the person with different numbers of distractors. The number of distractors on any given trial is 'randomized' and any systematic difference in performance that is based on the number of distractors can be interpreted as a causal effect of the number of distractors. $\endgroup$
    – Josh
    Commented Jun 4, 2015 at 19:31
  • $\begingroup$ Thanks for your clarifications Josh. Do you by any chance know about good sources that deal with designs and methodologies in cognitive science specifically? The majority of generic texts I've read always frame this question in such a way that they fail to deal with pure repeated measures experiments where all factors are crossed and each level contain numerous replications; a design that is a cornerstone of cognitive experiments on attention. $\endgroup$
    – Rooirokbokkie
    Commented Jun 5, 2015 at 7:28
0
$\begingroup$

The real issue is equivalence of groups. As Josh implies above, it is a game of probabilities when randomly assigning to groups, but it is considered very improbable that non-equivalence of groups will occur through random assignment when the number of assignments is relatively high (no less than 30 per group). So in an independent groups (between subjects) design, random assignment is used to balance out the nuisance variables such that all groups are about equally likely to have the advantages and disadvantages of whatever individual differences the participants bring to each group. Repeated measures (within subjects) designs, on the other hand, rarely get mentioned in this context because they create "groups" that are identical. Because the same participants are used in all conditions, equivalence is ensured (ignoring possible carryover effects that might result from exposure to earlier conditions). So, apart from carryover, the next concern is whether the sample is representative of the population to which the researcher wishes to generalize. In other words, equivalence of groups is an internal validity concern for between subjects designs, whereas the concern for repeated measures designs is external validity (is the sample representative of the population). Of course external validity is only a concern if generalization is important to the study (which isn't always the case).

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Welcome. Can you add resources to your answer to allow other users to background read on your answer? $\endgroup$
    – AliceD ♦
    Commented Apr 21, 2018 at 20:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.