Do we need a pre-test to compare the effectiveness of two treatments?

Question

Assume, I want to evaluate how effective two teachers are in teaching English to German children. Both teachers have been teaching at the same high school for twenty years, and both use a distinctly different pedagogical methodology. In fact a small competition has arisen between them: they have published and discussed their ideas and practise in journals relevant to their profession, and they have now called in a data analyst (you) to conduct this evaluation which, so they hope, will decide their contest and reconcile the former friends.

The school, where they both work, is the only school for its small town. When pupils enter this school, they are randomly assigned classes: one half of the children are assigned to one class (and one maths teacher), the other half to the other class (and the other maths teacher).

The two teachers are tired of not knowing which method is best. In the interest of their pupils they want to finally decide on the better one and both use this from now on. They hope, that you don't need to test one cohort of children when they finish elementary school, have them taught for the 8 years from 5th grade until they graduate from high school, and then measure their mathematical ability again, to come to a conclusion. Therefore they ask you:

Is it enough to compare the levels of the dependent variable post-intervention? Or do you need to measure it pre-intervention as well? Why?

Answer 1

Answer based on your original depression example

Note that this answer was originally written based on your initial example, where you asked:

Assume, I have developed a new intervention for people with light depression. I want to compare the effectiveness of this intervention (E) with an existing intervention (C). For this, I recruit test subjects from the local psychotherapeutic ambulance and randomly assign them to either the experimental (E) or control group (C). The interesting dependent variable is of course depressivity.

In general, you don't need to measure pre-treatment for the dependent variable. In the limit random assignment ensures that the groups are equal. Or to put it another way, random assignment ensures that the groups are not biased to be higher or lower on the dependent variable at baseline. A typical between-subjects t-test comparing post-treatement scores would typically provide an unbiased test of whether the intervention had an effect on dependent variable (i.e., depression) relative to control intervention.

That said, there are many benefits to including a baseline measure:

• Including a baseline measure of depression will almost always give you more statistical power because you are able to control for much of the stable individual differences in the dependent variable (i.e., depression).
• If you participants drop out during the intervention, it can be helpful to see whether this is related to baseline levels.
• You can begin to assess individual differences in the effect of the intervention.
• Where there are questions about whether random assignment was performed correctly, you can test for baseline differences.

Note there are several options for analysing pre-post treatment-control designs including ANCOVA, difference scores, and interaction effects. See this discussion for further ideas.

Note also that there are good reasons to assess an intervention by measuring more than two time points. For example, you might obtain (a) multiple baseline measures to get a sense of the stability pre-intervention (b) multiple you might measures during the intervention to assess depression during the intervention, and (c) multiple follow-up measures particularly to see both the immediate and the longer term effect of the intervention.

Updated points based on the teaching example

• It is an empirical question whether first letter of last name is related to the effect of the teaching intervention or baseline differences. In general, it would be better to have a better form of randomisation of participants to groups.
• Where there are questions about the randomisation procedure, the presence of a pre-test measure can be beneficial to check this.
• There are several particular issues related to assessing the effectiveness of interventions relating to children in class rooms even when students have been randomly assigned to classrooms. (a) With just one classroom each, it is difficult to tease out what is the effect of the teacher and what is the effect of the curriculum; (b) there will typically be a lack of independent observations. Thus, for example, students within the classroom may influence each other. Thus, your effective sample size is not as great as it may seem.