Skip to main content
added 119 characters in body
Source Link
AliceD
  • 20.8k
  • 8
  • 51
  • 142

This is an important question! This practice ("optional stopping" if you stop collecting data based on your early analyses, or "peeking" if you continue collecting data) is considered a bad idea nowadays. It's a "researcher degree of freedom"--a practice that, in the long run and averaged across the field, appears to (empirically) result in high false-positive rates. It's a form of exploratory analysis, and although EA isn't bad in and of itself, optional stopping/peeking can predispose researchers to chase significance for trends they see in their data, perhaps by selectively excluding certain observations, dropping their a priori hypotheses, ignoring their a priori power analyses, etc...

Instead, consider running a power analysis. (I recommend G*PowerG*Power, which is freely downloadable). I advise to perform a power analysis before you start collecting data, determine the total N that you'll shoot for, and don't peek at your data until you've hit that. It's effectively "blinding" yourself, much in the way that medical researchers might use double-blind studies to ensure the reliability of their findings.

Check out this paper for a longer discussion of researcher degrees of freedom: http://journals.sagepub.com/doi/abs/10.1177/0956797611417632

This is an important question! This practice ("optional stopping" if you stop collecting data based on your early analyses, or "peeking" if you continue collecting data) is considered a bad idea nowadays. It's a "researcher degree of freedom"--a practice that, in the long run and averaged across the field, appears to (empirically) result in high false-positive rates. It's a form of exploratory analysis, and although EA isn't bad in and of itself, optional stopping/peeking can predispose researchers to chase significance for trends they see in their data, perhaps by selectively excluding certain observations, dropping their a priori hypotheses, ignoring their a priori power analyses, etc...

Instead, consider running a power analysis (I recommend G*Power) before you start collecting data, determine the total N that you'll shoot for, and don't peek at your data until you've hit that. It's effectively "blinding" yourself, much in the way that medical researchers might use double-blind studies to ensure the reliability of their findings.

Check out this paper for a longer discussion of researcher degrees of freedom: http://journals.sagepub.com/doi/abs/10.1177/0956797611417632

This is an important question! This practice ("optional stopping" if you stop collecting data based on your early analyses, or "peeking" if you continue collecting data) is considered a bad idea nowadays. It's a "researcher degree of freedom"--a practice that, in the long run and averaged across the field, appears to (empirically) result in high false-positive rates. It's a form of exploratory analysis, and although EA isn't bad in and of itself, optional stopping/peeking can predispose researchers to chase significance for trends they see in their data, perhaps by selectively excluding certain observations, dropping their a priori hypotheses, ignoring their a priori power analyses, etc...

Instead, consider running a power analysis. (I recommend G*Power, which is freely downloadable). I advise to perform a power analysis before you start collecting data, determine the total N that you'll shoot for, and don't peek at your data until you've hit that. It's effectively "blinding" yourself, much in the way that medical researchers might use double-blind studies to ensure the reliability of their findings.

Check out this paper for a longer discussion of researcher degrees of freedom: http://journals.sagepub.com/doi/abs/10.1177/0956797611417632

Source Link
qjacob
  • 328
  • 2
  • 9

This is an important question! This practice ("optional stopping" if you stop collecting data based on your early analyses, or "peeking" if you continue collecting data) is considered a bad idea nowadays. It's a "researcher degree of freedom"--a practice that, in the long run and averaged across the field, appears to (empirically) result in high false-positive rates. It's a form of exploratory analysis, and although EA isn't bad in and of itself, optional stopping/peeking can predispose researchers to chase significance for trends they see in their data, perhaps by selectively excluding certain observations, dropping their a priori hypotheses, ignoring their a priori power analyses, etc...

Instead, consider running a power analysis (I recommend G*Power) before you start collecting data, determine the total N that you'll shoot for, and don't peek at your data until you've hit that. It's effectively "blinding" yourself, much in the way that medical researchers might use double-blind studies to ensure the reliability of their findings.

Check out this paper for a longer discussion of researcher degrees of freedom: http://journals.sagepub.com/doi/abs/10.1177/0956797611417632