I'm looking for examples of studies where the result is highly significant but the effect is so tiny that it is meaningless in a practical sense. These will be shared with undergraduate psychology students to illustrate that statistically significant doesn't always mean important or consequential.
$\begingroup$
$\endgroup$
2
-
3$\begingroup$ There's the recent Facebook emotion contagion study (e.g., "P < 0.001, Cohen’s d = 0.02" and "P = 0.007, d = 0.001"). $\endgroup$– mrtCommented Nov 11, 2015 at 19:00
-
$\begingroup$ Problem might get worse if you include CIs - as many CIs reported in studies with low p values include very low effect sizes. $\endgroup$– jonaCommented Nov 12, 2015 at 9:05
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
One example might be the effect of birth order on intelligence. Given a large enough sample there highly significant effects (p<.00001). However, these effects are very very small.
For a recent example, see Rohrer, Egloff, and Schmukle (2015): the effect of one birth order position was only one tenth of a standard deviation (1.5 IQ points)
To illustrate this small effect, in our between-family sample of sibships of two, a randomly picked firstborn had a 52% chance of having a higher IQ than a randomly picked secondborn; conversely, a secondborn had a 48% chance of having a higher IQ than a firstborn.
References
Rohrer, J. M., Egloff, B., & Schmukle, S. C. (2015). Examining the effects of birth order on personality. Proceedings of the National Academy of Sciences, 201506451. http://doi.org/10.1073/pnas.1506451112