The purpose of reviewing the literature for effect sizes is to form an estimate of what effect size you might expect in your present study.
Existing meta-analysis: The principles and techniques of meta-analysis provide a good starting point for generating a predicted effect size.
If a meta-analysis has already been conducted, then the estimated population mean and confidence intervals of effect sizes is a useful piece of information.
In some cases there will also be a moderator analysis reported which tests whether the effect size varies as a function of the moderator (e.g., adults versus children; lab versus field, etc.). If your study matches one level of an influential moderator, then it may be worth weighting estimates more by the estimated effect size for that moderator.
No existing meta-analysis: If no meta-analysis exists or if one exists but is out of date, then you may want to perform your own meta-analysis. There are several great books on meta-analysis. I've personally found Borenstein et al (2011) to be a good introduction and Hunter and Schmidt (2004) to be useful particularly if you are interested in correlations and various corrections. Here is a link to an overview of meta-analysis.
Conducting a rigorous meta-analysis for publication is a technical and resource intensive process. However, if all you want is a ball-park estimate of expected effect size for sample size and study design considerations (i.e., your not doing an explicit meta-analysis publication), then it may be more pragmatic to just do a quick informal meta-analysis. In some cases there might only be one or a few studies which are similar to your study. In this case, the meta-analysis can be very simple.
Generalisation: Another issue that comes up is generalisation. Students sometimes say to me that there have been no other studies like the one being conducted. This may be true, but at some level of generality this is almost always false. If a study is about a particular training program in the work place, then a meta-analysis of training in general may be informative. If the intervention has only been used with teenagers and is now being applied to adults, the meta-analysis of teenagers would be relevant. Broadly, theory should be applied to assess the likely generality of any broader research to the present study.
Theory: Theory is important in many respects. Theory may suggest that a particular modification to the design should increase or decrease the effect size relative to a previous studies.
Aggregating estimates: In general it is is important to consider how different effect size estimates should be aggregated. The more similar a previous study is to the present study, the more weight should be given to any effect size estimate. In this case, similarity requires theoretical judgement about what are the important factors that might moderate the effect. Similarly any aggregation needs to be mindful of sample size considerations where larger sample sizes will increase the precision of the estimate in any given study. Thus, at the ultra-specific, the present study may be an exact replication of a previous study. At another level, there may be meta-analytic evidence from similar studies. At an even broader level Jacob Cohen reluctantly proposed various rough rules of thumb of what to expect for small, medium, and large effects in behavioural science studies.
Importance of uncertainty: Another point is that you don't actually know the population effect size for your study. That's why you are doing the study. There will always be uncertainty about your effect size estimate. It can even be useful to quantify that uncertainty either using meta-analytic confidence intervals or something more ad hoc.
This is analogous to the concept of a subjective bayesian prior probability on a parameter of interest. See for example, this discussion of eliciting priors from experts. Meta-analysis provides some methods for quantifying uncertainty in an effect size estimate, but this may not be entirely appropriate given that you are not trying to generalise to the variation in the whole world of effect sizes. Rather you are trying to generalise a prediction for a specific study, and you may be pooling many different pieces of information, some quantitative, and some qualitative.
By way of example, I do research on personality faking. I am aware of a meta-analysis of the mean effect size of faking versus honest personality responses (e.g., perhaps around d = .6). However, I also know from previous research that the type of faking instructions makes a big difference to the expected effect size (e.g., if you tell people to fake you often get approximate d = 1.0 and if you just put people in a context with a motivation to fake you get approximate d = .3). Thus, If I were calculating expected effect size for any given study I would combine knowledge of the nature of my faking instructions with various meta-analytic estimates. I'd weight my own previous studies highly because they typically share more similar features (e.g., same personality test, similar participants, etc.) than the general literature. I'd also acknowledge the uncertainty in my estimate.
References
- Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2011). Introduction to meta-analysis. Wiley.
- Hunter, J. E., & Schmidt, F. L. (2004). Methods of meta-analysis: Correcting error and bias in research findings. SAGE Publications, Incorporated.
