2
$\begingroup$

One often sees researchers attributing a change in the dependent variable to their experimental manipulation, semingly without considering whether this change in DV occurs between or within groups.

For instance, in the classical Bargh et al. (1996) paper on behavioural priming, subjects in an experimental group were shown word primes recalling stereotypes of old age (e.g. "forgetful"), while subjects in a controlg group were shown neutral words. Everyone's walking speed as they left the experiment was measured, and because the first group's was slower, it was concluded that priming people with words relating to a certain concept (in this case: old age) makes them unconsciously take on characteristics of that concept (in this case: walking more slowly).

The question that comes to my mind is, however: how do we know that the mean difference in walking speed is not just due to pre-existing (baseline) differences between the groups? How can we be sure it is due to the IV manipulation (different word primes)?

To me, believable evidence for this claim would be if they had measured everyone's speed pre&post-experiment, and then found an interaction showing that, while controls' speed stayed the same pre-to-post, that of the old-age-primed group had decreased. Without such an interaction (involving the use of a mixed between-within model), is the authors' between-groups-only inference statistically correct?

Improve this question
$\endgroup$

2 Answers 2

Reset to default
3
$\begingroup$

There's a short straightforward answer, and a more nuanced answer.

The short answer is that people were randomly assigned to the two groups. Any baseline differences between people will be equally likely to affect both groups. Will there still be differences? Absolutely. This is why researchers use statistical inference to rule out the possibility that baseline differences are responsible for the observed difference.

The more nuanced answer is that we can't be certain that the observed differences aren't due to these random baseline differences. In any experiment, there will always be noise that causes differences in the groups. When researchers use statistical inference, they aren't able to say with certainty that the effect was due to the manipulation. They are merely assessing the probability that random noise could account for the data. If the probability that random noise can explain the data is very low, then researchers claim that the manipulation had an effect. However, this is a probabilistic inference and sometimes the researchers will be wrong and the difference really is due to noise. Within-subject designs help remove some of this noise, but not all of it, because people change over time, even over the short timescale of a social priming experiment.

Improve this answer
$\endgroup$
4
  • $\begingroup$ I understand your answer but I'm not sure I agree with it. Random assignment of subjects to groups does not guarantee the lack of a baseline difference, which would have to be checked beforehand as a 'pre' measure, effectively transforming the design to a mixed-model that can prove a priming x timepoint interaction, as I was suggesting in my initial post. $\endgroup$
    – z8080
    Oct 1, 2015 at 16:06
  • $\begingroup$ I'd have expected that the defense for their between-subjects ANOVA lies in the between-subjects SS term taking the "hit" to account for the noise, and thus the p-value of the main effect of priming condition becoming, in effect, corrected for the uncertainty due to this noise. Even so, however, it seems to me the effect just comes across much better by way of an interaction with the pre-post-experiment factor. $\endgroup$
    – z8080
    Oct 1, 2015 at 16:10
  • 2
    $\begingroup$ You're right that random assignment doesn't guarantee a lack of a baseline difference. In fact, there will almost always be some baseline difference with random assignment. But inferential statistics is used to see if the differences between groups can be explained by random variation due to baseline differences, or if the effect of the manipulation is strong enough that it is unlikely that baseline differences can explain the result. $\endgroup$
    – Josh
    Oct 1, 2015 at 16:50
  • 2
    $\begingroup$ Another point, more specific to this experiment, is that there are drawbacks to within subject designs. One is that you might give the subjects more information than you'd like. It might have been difficult to measure walking speed in a controlled yet inconspicuous way before the test and after the test without raising suspicion. $\endgroup$
    – Josh
    Oct 1, 2015 at 16:52
2
$\begingroup$

Excuse my english, I'll try my best to explain. The key concept here is "randomization", which involves randomly allocating the subjects across the different groups. If randomization is well done, this ensure that your groups are "equivalent" on all other factors that have not been explicitly accounted for in the experimental design. Thus, in theory, there should not be any difference in your pre-test betweens groups, and the difference between groups in the post-test should only be accounted by the manipulation of your VI.

Improve this answer
$\endgroup$
2
  • $\begingroup$ but surely the only way to verify that randomisation - i.e. group balancing in this case - has been done correctly is to check the DV beforehand as well; which would end up being equivalent to the mixed-model i was describing, right? $\endgroup$
    – z8080
    Oct 1, 2015 at 16:03
  • $\begingroup$ In theory, if the randomization is done right, and the condition were exactly the same (or very very similar at least) for both group, except of course for the manipulation of the VI, then both group should be equal on every pre-test on every variable. Now, if you're not sure that the condition are the same or that your randomization as not been carried out right, then you could check if both group are equals. The major problem is that you should not only verfy if they are equal on your DV, but on all the counfound variables... This thakes time and effort. That's why you've to be very carefull $\endgroup$
    – mat
    Oct 1, 2015 at 18:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.