Does the meta-analytic literature overestimate the relationship between conscientiousness and job performance?

Question

There are quite a few meta-analyses of the relationship between personality and job performance (for a review, see Barrick & Mount, 2012; and specific meta-analyses: Barrick & Mount, 1991; Murray R Barrick, Michael K Mount, & Timothy A Judge, 2001; Hurtz & Donovan, 2000; J. Salgado, 1997; Tett, Jackson, & Rothstein, 1991).

These often come up with raw meta-analytic mean correlations between conscientiousness and job performance of around r = .15.

However, there is a lot of discussion in general at the moment about replicability and publication bias. In particular, if papers that showed a correlation between personality and job performance, were more likely to be published, then this would inflate the meta-analytic estimate.

Alternatively, there have been quite a few studies of personality and performance with huge sample sizes (10,000+) and these would quite likely be published regardless of the exact results, and they also would have sufficient statistical power to detect even small effects. And for most meta-analyses, large sample size studies get a greater weighting in the estimation process.

I seem to remember reading a paper a few months back that examined this question. If I recall, the authors concluded that the meta-analytic estimate of relationship between conscientiousness and performance might be inflated slightly. However, I can't remember what was the reference.

Questions

• What research has been conducted on publication bias or other factors that might influence the accuracy of the meta-analytic estimate of the relationship between personality (and conscientiousness specifically) and job performance?
• What is the evidence for and against this?

Answer 1

I'm posting a summary here (thanks to @DJ Sims for directing me to the paper of interest). Kepes and McDaniel (2015) re-analysed data from an existing meta-analysis. The existing meta-analysis had been examining various moderators of the relationship between big 5 personality and job performance. Kepes and McDaniel performed a range of sensitivity analyses.

They summarise their results as follows:

Publication bias analyses demonstrated that the validity of conscientiousness is moderately overestimated (by around 30%; a correlation difference of about .06). The misestimation of the validity appears to be due primarily to suppression of small effects sizes in the journal literature.

As evidence for publication bias, they show that the average validity for conscientiousness is lower in unpublished samples:

The results for the sub-group distributions of samples from journal articles (k = 67) and non-journal sources (k = 46) [indicate that] samples published in journal articles reported larger average effect size estimates ( $\bar{r}_{oRE}$ = .19) than samples from non-journal sources ( $\bar{r}_{oRE}$ = .12.

A few critical comments:

When publishing results showing the relationship between big 5 personality and performance, there are five correlations presented. Assuming that publication bias sometimes led to a requirement to have at least one significant correlation, then the publication bias should be less.

The study does not cite or discuss other common meta-analyses on the Big 5 and job performance, which interestingly have often obtained meta-analytic estimates similar to what they say is the correct value. For example, the famous early meta-analysis by Barrick & Mount (1991) using 12,893 cases and 92 correlations obtained a mean raw correlation between conscientiousness and job proficiency of $\bar{r}$ = .13. In a meta-analysis by Hurtz and Donovan that exclusively used correlations for measures using the big 5, they obtained an overall validity for conscientiousness of $\bar{r}$ = .14. This also is very close to what might be expected.

These traditional estimates of validity from previous meta-analyses are sample weighted means. Thus, they naturally weight large samples more. So, for example, one study with 2000 participants would be weighted as much as 20 studies each with 100 participants.

There are good reasons to assume that publication bias would have less of an effect on large sample studies (e.g., n > 1000). First, when the sample size is sufficiently large, the results become intrinsically interesting. Reviewers can not argue that it was merely a lack of statistical power. Second, such studies generally have sufficient power to detect correlations in the r = .10 area. For example, assuming a population correlation .10, alpha of .05, and a two-tailed test, to have at least 90% power to detect a significant correlation you would need at least 1,046 participants.

My take away from the paper:

• There is probably publication bias in the literature in relation to small sample studies. I imagine this publication bias is relative. If I were to guess, I'd say strong bias at n < 100; moderate bias n < 200, and progressively less as you move to 500+.

References

• Kepes, S., & McDaniel, M. A. (2015). The Validity of Conscientiousness Is Overestimated in the Prediction of Job Performance. PloS one, 10(10), e0141468. http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0141468