6
$\begingroup$

The well known Dunning-Kruger effect is oft cited as a general explanation for why people with limited expertise overestimate their abilities. Other than the initial research that described this cognitive bias, there seems to be little actual data to support it outside of a sit-down questionnaire or IQ test in a university setting. Driving is a well-known example (i.e. '80% of drivers consider themselves above average') although I note from a previous post, the evidence for the Dunning-Kruger effect in driving is mixed.

Is there any data to support the Dunning-Kruger in outside of a sit-down test performed by university students?

$\endgroup$
0
3
$\begingroup$

Yes, there are research showing a relationship between skills and not only "sit down tests". A systematic review by Madmood (2016) found "self-reported and demonstrated information literacy skills" supported the Dunning-Kruger phenomena. Demonstrated information literacy was from derived sources such as "observations, field notes, participant reflexive journals, expert grading of searching tasks, analysis of students’ theses, quizzes, and information related assignments".

References:

Mahmood, K. (2016). Do People Overestimate Their Information Literacy Skills? A Systematic Review of Empirical Evidence on the Dunning-Kruger Effect. Communications in Information Literacy, 10 (2), 199-213. https://doi.org/10.15760/comminfolit.2016.10.2.24

$\endgroup$
1
  • $\begingroup$ Could you elaborate on the "self-reported and demonstrated information literacy (IL)"? As that is the crux of the question. Maybe a simple quote from the paper where these methods are explained. For example, isn't a sit-down questionnaire 'self-reported'? Just enough to understand it without having to read the full paper. Details of course can be deferred to the paper. $\endgroup$ – Steven Jeuris May 1 '20 at 10:43
2
$\begingroup$

Short answer: It's currently unclear.

In the 20+ years since the original paper was published, there have been many successful replications of the effect, including domains not involving written tests, such as debating, chess, bridge, bed-side manner, facial recognition, and yes, even driving. However, as Wikipedia summarizes:

... their conclusions are strongly challenged when subjected to mathematical analysis and comparisons across cultures.

That is, a significant component of the reported effect may be due to statistical artifacts that result from the method used.

This is not the first time that the study method used has been questioned: Theoretical criticisms appeared shortly after the original paper's publication (eg, Krueger & Mueller, 2002; Krajc & Ortmann, 2008; more here). However, subsequent replications of the effect correcting for these artifacts largely survived the criticisms with the main revision in results being that high performers are no longer thought to underestimate their ability. Nonetheless, some studies that successfully replicated the effect did notice significant artifacts remaining in their empirical data - for example, Bell & Volckmann (2011):

... students with the highest exam scores are just about as variable at gauging their course knowledge as students who have the lowest exam scores.

Empirical evidence that other artifacts could be substantial was finally demonstrated with the publication of a series of studies in Nuhfer et al (2016), Nuhfer et al (2017), who state:

Our results show that different numerical approaches employed in investigating and describing self-assessment accuracy are not equally valid.

Additional studies feature similar conclusions - for example:

Feld, Sauermann, & de Grip (2017):

Our results show that the unskilled are more overconfident than the skilled. However, as we predict in our methodological discussion, this relationship is significantly weaker than ordinary least squares estimates suggest.

McIntosh et al (2019):

... the pattern can be induced (and greatly inflated) by a host of other factors and biases, some psychologically interesting, and some ‘merely’ statistical.

Gignac & Zajenkowski (2020):

... although the phenomenon described by the Dunning-Kruger hypothesis may be to some degree plausible for some skills, the magnitude of the effect may be much smaller than reported previously.

The artifacts significantly diminish the Dunning-Kruger effect size, possibly eliminating it in some cases.

Additional notes:

  • The example in the question of people systematically overestimating their driving competency is not an example of the Dunning-Kruger effect. Rather, it is an example of a different bias called the above-average effect.
  • There is a large amount of literature on self-assessment - where subjects are asked to assess their own competence in some domain. Across many studied domains, people tend to be somewhere around "moderately good" at self-assessing their abilities, but certainly not perfect.
  • Contrary to the above-average effect, low-performers in Dunning-Kruger-type studies do not estimate their competence as higher than better-performing participants - rather, they rate their competence as higher than it actually is.
  • In their original paper, Dunning & Kruger established a non-standard protocol for measuring the self-assessment bias, in which they divide results into sorted quartiles, and compare average absolute differences between actual and estimated test scores. It is this protocol that is implicated in the statistical artifacts afflicting the results, as using more standard methods, such as adjusted scatter-plots, correlations, and error-bars, it does not appear as if there is much difference (if any) between low and high performers' precision.
  • Studies that adjust for statistical artifacts demonstrate that the non-standard protocol used by Dunning & Kruger causes data censoring (hiding the effects of noise), floor/ceiling effects (error ranges are bounded), and regression to the mean (due to comparison of 2 independent measures). Combined with some likely above-average effect, these statistical artifacts explain most (if not all) of the results. The artifacts are so significant that even random data can generate the effect!
$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.