3
$\begingroup$

This question is mostly quoted from A New Psychology of Women (2017) by Lips, page 63:

Finally, and perhaps most importantly, the studies included in most meta-analyses are based on a narrow range of participants (usually young, North American students) and are not reflective of the populations at large--although this limitation tends to be obscured by the combination of many studies into a single analysis (Halpern, 1995).

I mean, this claim is more than 20 years old. Is it true today? Is there a study that examines the demographics of meta-studies at large or something related? If it is true, is this the case for most psychology research, male-female differences or all studies in the social sciences?

$\endgroup$
  • 3
    $\begingroup$ Much (most?) research in psychology is conducted by psychology professors (probably in North America, or Europe where most funding is available), who naturally recruit undergraduate psychology students as subjects (for extra credit, or even mandatory in some cases). This practice is supported by many years of evidence that students are representative of the general population, as well as being convenient to recruit. Regarding gender differences specifically, see for example: cogsci.stackexchange.com/questions/8583/… $\endgroup$ – Arnon Weinberg Sep 20 '17 at 14:15
2
$\begingroup$

Hunter and Schmidt offer a a set of procedures to correct range restriction, these are included in many packages based on their meta-analytic methodology, here are some relevant references:

https://pdfs.semanticscholar.org/75e8/97a797dc876b785042dfd8ac4023a5fc85df.pdf http://onlinelibrary.wiley.com/doi/10.1111/peps.12122/abstract https://www.researchgate.net/publication/6644262_Correcting_for_indirect_range_restriction_in_meta-analysis_Testing_a_new_meta-analytic_procedure mypages.iit.edu/~morriss/siop2009/Le09.ppt mypages.iit.edu/~morriss/siop2005/Oswald05.doc https://www.corwin.com/sites/default/files/upm-binaries/5063_Hunter_Chapter_3.pdf https://utsa.influuent.utsystem.edu/en/publications/implications-of-direct-and-indirect-range-restriction-for-meta-an

hth,

larry

$\endgroup$
1
$\begingroup$

Meta-analysis is a tool that is applied across a wide range of disciplines, especially those involving human subjects. In general, meta-analyses aim to sample all relevant studies that have been conducted.

In some fields, the norm is to use undergraduate students (especially psychology students). Many psychology programs have a research participation pool; so for a lot of basic lab based research (e.g., cognitive psychology, social psychology), this represents a convenient sample. Note of course, that other disciplines will use other samples. Organisational psychology researchers often use samples of employees from organisations. Developmental psychologists study children. Social science researchers often rely on large scale panel surveys.

More recently, online samples are becoming particularly popular. This includes online convenience samples as well as paid samples. In particular, Mechanical Turk is particularly popular in the United States, but other platforms such as prolific exist.

In general, the choices that researchers make about sample are driven by many factors:

  • convenience/resources
  • discipline expectations
  • interests in generalisation (some researchers are mainly interested in generalising to their own culture)
  • where the researchers have traditionally come are from (e.g., most researchers in psychology come from the U.S., followed by UK/Europe/Australasia)
  • relevance of sample to research question (e.g., sometimes a convenience sample is fine; other times you really need a representative sample); in basic science, it's often fine, but if you're doing organisational or clinical research, you'll generally want a relevant org or clinical sample.

So in short, this phenomena is going to vary a lot across research questions. And in some cases it's probably not going to matter much; in other cases it will.

More generally, there is often a trade-off between convenience and rigour. For example, most researchers who measure job performance would love to have a rigorous objective measure of objective performance, but it's much easier to get self-report. So you'll often find that meta-analyses will be summarising studies that have opted for one point on the convenience/rigour trade-off.

Meta-analysts can and do perform moderator analyses if they think a particular factor might be relevant to the meta-analytic finding (e.g., whether the sample are undergraduate students or not).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.