3
$\begingroup$

How is doing a so-called manipulation check (i.e. checking to see whether the different groups actually differ for different levels of the independent variable) different from doing - as a main analysis - e.g., an ANOVA to see if the independent variable has had an effect on the dependent variable? From reading papers, it seems people just do manipulation checks prior to the main analyses.

$\endgroup$
3
$\begingroup$

It depends on what you mean by manipulation check.

Manipulation check to filter participants: A manipulation check is usually done to verify that some experimental manipulation has had some preliminary effect. For example, in one of my studies I asked people to answer a personality test (a) honestly, (b) as an ideal worker, and (c) as the worst possible worker. I asked each participant how they intended to answer the personality test. The manipulation check was there to verify that they had understood the instructions. In this case, the manipulation check was used to potentially exclude participants who failed to pass the manipulation check.

Manipulation check to verify group-level effect: Of course, you could perform such manipulation checks on an entire group using group comparison tools such as ANOVA for a numeric variable or chi-square for a categorical outcome. For example, in some psychological studies, the experimenter may attempt to manipulate mood as baseline manipulation. Thus, they may want to get check whether the group that has been made to experience negative affect does on average experience more negative affect. In that case an ANOVA makes sense.

Checking approximate group equivalence:: A third case is where you are merely checking whether the groups are the same at baseline and that has more to do with how participants were allocated to groups. It doesn't sound like you are talking about this case.

Using manipulation check outcome as covariate: In general, manipulation checks are typically reported prior to conducting main analyses. In this sense, they are a bit like assumption tests. They verify that some experimental manipulation has worked and that you can now proceed to examine the effects of substantive interest. That said, you could do a range of other analyses where the outcome measure in the manipulation check is included in your analysis of the substantive outcome. For example, if you were studying task performance in two groups, one with a positive mood induction and one with a negative mood induction, you could run an ANCOVA where condition was your independent variable, measured mood (i.e., the manipulation check variable) was your covariate, and task performance was your dependent variable.

$\endgroup$
  • $\begingroup$ Awesome answer, thank you very much $\endgroup$ – z8080 Aug 7 '15 at 13:18
3
$\begingroup$

Primarily, a manipulation check is an indicator of the internal validity of an experiment.

If the manipulation of your independent variable makes a statistically significant difference on the dependent variable, you have evidence for a causal effect of the manipulation. The manipulation check can give you more certainty that this effect is due to changes in the construct you are interested in.

For example, imagine you manipulated mood by showing a happy versus a sad movie clip. Subsequently, this affects the extent to which people are persuaded by weak versus strong arguments (argument strength doesn't matter in the happy movie condition). Showing that your manipulation has affected mood in the intended way with a manipulation check (better mood in the happy than in the sad movie) allows you to be more certain that this has something to do with mood. If your manipulation check would not show a mood difference, the difference on the dependent variable might be caused by some other aspect of the manipulation (for example differences in arousal).

The manipulation check also allows to do correlational analyses to strengthen this conclusion: Are changes in the dependent variable mediated by changes in the manipulation check? (Does persuasion change to the extent that mood has been altered?)

More pragmatically, if you have no effect on the dependent variable, the manipulation check may give you some insight. Is it because the manipulation failed? (No/to small effect on the manipulation check?) Or is it because is the problem on the side of the dependent variable? (Your measure doesn't work or, heaven forbid, your hypothesis is wrong).

$\endgroup$
  • $\begingroup$ Very helpful answer, thanks so much! $\endgroup$ – z8080 Aug 7 '15 at 13:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.