5
$\begingroup$

For a research internship I am running psychological experiments online. As it takes a while until the experiment is done (meaning that enough people participated, so that a sufficient sample size is reached), I could already look at the data and run a few analyses with that incomplete data set. This way I could already see a trend in what the final result might be.

Are there any methodological reasons that would speak against that? For example, am I biasing myself in this way? Or is there any other reason this could be considered bad research practice?

$\endgroup$
5
$\begingroup$

Short answer
From an ethical standpoint, not including interim evaluations may be bad practice.

Background
I will start off with a more extreme case than in your question example, just for illustrative purposes, namely that of a clinical intervention study. If it appears that the treatment group (say, experimental medicine Y, instead of the standard of care treatment X) features substantially, if not significantly, more cases of serious adverse advents, or even deaths, that may (or may not) be related to treatment Y, the ethical best thing to do is put the study on hold until things are sorted out. This, to prevent any possibility of further physical harm caused by the experimental treatment. This happens quite regularly and should be followed by a report paper stating the results and a discussion on the best way to proceed this research, if applicable.

In a more experimental setting it may also be ethically best practice to evaluate preliminary data, as possible experimental design flaws, unexpected results (weird outcomes or artifacts in, say, left-handed people) or confounding factors may become apparent, and timely adjustments to the experimental protocol can be made. Why is this ethically correct? Because you may be subjecting people to a flawed experimental paradigm and wasting many hours of otherwise more productive time.

The case provided by the other answerer, where the experiment is stopped, while prior statistical power analysis was done, that is malpractice. Conversely, adding more subjects post hoc based on 'near-significance' is also questionable practice. But this is more related to what you do with the experimental interim data. In my opinion, they should be critically evaluated, but not so much on effect size, but rather on feasibility, correctness and validity - basically to sanity-check the study proceedings.

$\endgroup$
  • 1
    $\begingroup$ You and @qjacob provide some very good points. In fact, I did do a power analysis, which is what I meant when I wrote "until a sufficient sample size is reached", but I guess that could have been more clear. I am glad I am not completely on the wrong path then. Thanks a lot! $\endgroup$ – Mikkel Schöttner Mar 22 '18 at 9:28
  • 1
    $\begingroup$ This is a good answer, and I might also point out that a properly-designed experimental method is not subject to bias on the part of the person executing the method. It should not be possible to affect future results based on your knowledge of past results. If it is, you have a methodology issue. $\endgroup$ – theMayer Mar 23 '18 at 19:13
  • $\begingroup$ @rmayer06: to clarify, are you saying that optional stopping has no impact on Type I error rate (as gjacob suggests below) if an experiment is properly-designed? Don't want to mischaracterize your position before responding. $\endgroup$ – jsakaluk Mar 23 '18 at 19:47
  • $\begingroup$ @jsakaluk- yes, that's what I'm saying. All studies have time between runs, so there is stopping that takes place automatically. Obviously if you stop for a long period of time, you might have other issues and variability creep in. $\endgroup$ – theMayer Mar 24 '18 at 6:55
5
$\begingroup$

gjacob is correct that optional stopping is a common research degree of freedom, and one that has a considerable and unfortunate intuitive basis. Yet, depending on the context of your research, AliceD's concerns are also important.

There is, however, a middle ground between not checking at all, and p-hacking: sequential analysis. There is a Bayesian version of sequential analysis, which I can update if that's your statistical paradigm, but I'm assuming you're wanting to conduct interim analyses using null-hypothesis significance testing, so that's what I'll focus on here. Lakens (2014) provides a nice overview of this practice. In essence, you take the level of $\alpha$ you want to maintain over your "peeks" (e.g., $\alpha$ = .05), and distribute that total $\alpha$ over the number of peeks you want to take along your total sampling process. Then, if $p$ is lower than this distributed $\alpha$ at any of your peeks, you can reject the null at $\alpha$ = .05, and you won't have inflated your Type I error rate like you would have with generic optional stopping.

It's a little more complicated than how I'm presenting here--and there are a number of methods for distributing your total $\alpha$--but not by much. If you can wrap your head around a bonferroni correction, it's a very similar technique.

Lakens, D. (2014). Performing high‐powered studies efficiently with sequential analyses. European Journal of Social Psychology, 44(7), 701-710.

$\endgroup$
  • 2
    $\begingroup$ Welcome and thanks for the great answer.+1 $\endgroup$ – AliceD Mar 23 '18 at 19:41
  • 2
    $\begingroup$ Thanks! I've been lurking on the beta here for awhile :) Been hoping to encourage more replicability related questions being asked (and answered) here, so that we have searchable q&a's vs. endless repetitive social media questions. Glad to see these sorts of q's appearing here! $\endgroup$ – jsakaluk Mar 23 '18 at 19:44
  • 2
    $\begingroup$ This type of question might also be suited for CrossValidated and, perhaps, even Academia. So I think the number of questions like this posted on this stack will remain to be small. Nonetheless, I think it's an awesome question, as it can be targeted from many view points, as the diverse answers show. Good stuff here. $\endgroup$ – AliceD Mar 23 '18 at 20:50
3
$\begingroup$

This is an important question! This practice ("optional stopping" if you stop collecting data based on your early analyses, or "peeking" if you continue collecting data) is considered a bad idea nowadays. It's a "researcher degree of freedom"--a practice that, in the long run and averaged across the field, appears to (empirically) result in high false-positive rates. It's a form of exploratory analysis, and although EA isn't bad in and of itself, optional stopping/peeking can predispose researchers to chase significance for trends they see in their data, perhaps by selectively excluding certain observations, dropping their a priori hypotheses, ignoring their a priori power analyses, etc...

Instead, consider running a power analysis. (I recommend G*Power, which is freely downloadable). I advise to perform a power analysis before you start collecting data, determine the total N that you'll shoot for, and don't peek at your data until you've hit that. It's effectively "blinding" yourself, much in the way that medical researchers might use double-blind studies to ensure the reliability of their findings.

Check out this paper for a longer discussion of researcher degrees of freedom: http://journals.sagepub.com/doi/abs/10.1177/0956797611417632

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.