I'm looking for references related to how researchers and laypeople understand concepts of statistical inference. In particular, I'm looking for articles in which subjects could demonstrate statistical competence by evaluating statistical claims. Does anyone have a relevant reference?
Even among researchers there is widespread misunderstanding of core statistics ideas. Look at the work by Geoff Cumming. Example paper title: 'Researchers misunderstand confidence intervals and standard error bars.'
To add to Tom's answer and expand on my comment that lay people and researchers both generally have an inadequate understanding of basic statistics, another study by Hoekstra, Morey, Rouder and Wagenmakers (2014) asked 120 researchers and grad students plus 442 first-year students in psychology to indicate whether the following six different interpretations of a 95% confidence interval were correct.
Questionnaire content (Hoekstra, Morey, Rouder and Wagenmakers, 2014)
- The probability that the true mean is greater than 0 is at least 95 %.
- The probability that the true mean equals 0 is smaller than 5 %.
- The “null hypothesis” that the true mean equals 0 is likely to be incorrect.
- There is a 95 % probability that the true mean lies between 0.1 and 0.4.
- We can be 95 % confident that the true mean lies between 0.1 and 0.4.
- If we were to repeat the experiment over and over, then 95 % of the time the true mean falls between 0.1 and 0.4.
(I highly recommend looking up the quite humorous questionnaire, featuring Professor Bumbledorf, in Appendix 2.
On average, participants endorsed more than three of these statements, and all three subgroups gave substantial endorsement for individual statements. All of the statements are wrong, however. Only eight first-year students and three researchers gave a flawless response, i.e., all false, despite the questionnaire explicitly raising this possibility!
The correct interpretation is, “If we were to repeat the experiment over and over, then the confidence intervals contain the true mean 95% of the time.”
(Mouse-over for the correct interpretation.)