Should I look at the data of an experiment before the dataset is complete?

Question

For a research internship I am running psychological experiments online. As it takes a while until the experiment is done (meaning that enough people participated, so that a sufficient sample size is reached), I could already look at the data and run a few analyses with that incomplete data set. This way I could already see a trend in what the final result might be.

Are there any methodological reasons that would speak against that? For example, am I biasing myself in this way? Or is there any other reason this could be considered bad research practice?

Answer 1

Short answer
From an ethical standpoint, not including interim evaluations may be bad practice.

Background
I will start off with a more extreme case than in your question example, just for illustrative purposes, namely that of a clinical intervention study. If it appears that the treatment group (say, experimental medicine Y, instead of the standard of care treatment X) features substantially, if not significantly, more cases of serious adverse advents, or even deaths, that may (or may not) be related to treatment Y, the ethical best thing to do is put the study on hold until things are sorted out. This, to prevent any possibility of further physical harm caused by the experimental treatment. This happens quite regularly and should be followed by a report paper stating the results and a discussion on the best way to proceed this research, if applicable.

In a more experimental setting it may also be ethically best practice to evaluate preliminary data, as possible experimental design flaws, unexpected results (weird outcomes or artifacts in, say, left-handed people) or confounding factors may become apparent, and timely adjustments to the experimental protocol can be made. Why is this ethically correct? Because you may be subjecting people to a flawed experimental paradigm and wasting many hours of otherwise more productive time.

The case provided by the other answerer, where the experiment is stopped, while prior statistical power analysis was done, that is malpractice. Conversely, adding more subjects post hoc based on 'near-significance' is also questionable practice. But this is more related to what you do with the experimental interim data. In my opinion, they should be critically evaluated, but not so much on effect size, but rather on feasibility, correctness and validity - basically to sanity-check the study proceedings.

Answer 2

gjacob is correct that optional stopping is a common research degree of freedom, and one that has a considerable and unfortunate intuitive basis. Yet, depending on the context of your research, AliceD's concerns are also important.

There is, however, a middle ground between not checking at all, and p-hacking: sequential analysis. There is a Bayesian version of sequential analysis, which I can update if that's your statistical paradigm, but I'm assuming you're wanting to conduct interim analyses using null-hypothesis significance testing, so that's what I'll focus on here. Lakens (2014) provides a nice overview of this practice. In essence, you take the level of $\alpha$ you want to maintain over your "peeks" (e.g., $\alpha$ = .05), and distribute that total $\alpha$ over the number of peeks you want to take along your total sampling process. Then, if $p$ is lower than this distributed $\alpha$ at any of your peeks, you can reject the null at $\alpha$ = .05, and you won't have inflated your Type I error rate like you would have with generic optional stopping.

It's a little more complicated than how I'm presenting here--and there are a number of methods for distributing your total $\alpha$--but not by much. If you can wrap your head around a bonferroni correction, it's a very similar technique.

Lakens, D. (2014). Performing high‐powered studies efficiently with sequential analyses. European Journal of Social Psychology, 44(7), 701-710.

Answer 3

This is an important question! This practice ("optional stopping" if you stop collecting data based on your early analyses, or "peeking" if you continue collecting data) is considered a bad idea nowadays. It's a "researcher degree of freedom"--a practice that, in the long run and averaged across the field, appears to (empirically) result in high false-positive rates. It's a form of exploratory analysis, and although EA isn't bad in and of itself, optional stopping/peeking can predispose researchers to chase significance for trends they see in their data, perhaps by selectively excluding certain observations, dropping their a priori hypotheses, ignoring their a priori power analyses, etc...

Instead, consider running a power analysis. (I recommend G*Power, which is freely downloadable). I advise to perform a power analysis before you start collecting data, determine the total N that you'll shoot for, and don't peek at your data until you've hit that. It's effectively "blinding" yourself, much in the way that medical researchers might use double-blind studies to ensure the reliability of their findings.

Check out this paper for a longer discussion of researcher degrees of freedom: http://journals.sagepub.com/doi/abs/10.1177/0956797611417632