What articles use structural equation modelling in a meta-analytic context to model mediation?

Question

I'm interested in structural equation modelling (SEM) meta-analysis and its application to modelling mediation relationships.

Mike W.-L. Cheung has written a few articles on the topic of SEM Meta-Analysis (e.g., Cheung, 2009). He's also written the metaSEM package in R.

One basic approach to SEM meta-analysis is a simple fixed-effect two-step approach. First, you extract a weighted average correlation matrix from the component studies. Second, you analyse that correlation matrix using standard SEM approaches.

However, in reality, things are more complicated.

• Correlations often vary between studies by more than just sampling error, suggesting that a fixed-effect approach may be invalid. If a random-effects model is adopted, this raises issues of whether to categorise the correlation matrices into groups of fixed-effects matrices or whether to somehow incorporate the random-effects into the overall SEM.
• Some studies may not have all the variables of interest, which introduces a form of missing data, which in turn raises issues of how to define sample size and the standard error.

So in general, I'm interested in reading examples of SEM meta-analyses applied in psychology to get a sense of how others have dealt with the various issues. In particular, I'm interested in meta-analysis SEM that incorporates the modelling of indirect effects (i.e., mediation).

Questions

• What studies have applied meta-analysis SEM in psychology to model indirect effects?
• How have they dealt with issues such as (a) fixed versus random effects and (b) missing data?

References

• Cheung, M.W.L. (2009). Meta-analysis: A Structural Equation Modeling Perspective. PDF

Answer 1

I'd like to address important issues that Jeromy Anglim raised in the "Personal thoughts" section of his answer, namely that correlation parameters (i.e., true, population, or infinite-sample correlations) often vary and covary among studies, and this between-studies\interstudy heterogeneity implies heterogeneity in studies' parameters for a structural equation model (SEM). I'll describe a method I've proposed to account for this heterogeneity when estimating and making statistical inferences about a SEM, such as a univariate or multivariate regression model, factor model, path model, or model for structural relations among latent variables, or specific quantities based on such a model (e.g., $R^2$, indirect\mediation effect, fit index). This approach's core idea is simple:

1. Use random-effects (RE) meta-analysis to estimate attributes of the between-studies distribution of correlation-matrix parameters (e.g., its mean and covariance matrix).

2. Transform those results to estimate attributes of the between-studies distribution of SEM parameters.

Many commonly used MASEM methods are based on fixed-effects (FE) models, and although meta-analyzing heterogeneous correlation matrices using certain FE methods performs reasonably well in some situations, it's probably not advisable for making unconditional inferences about a larger universe of studies (Hafdahl, 2008a).

Here I'll just give overviews of the problem and a three-step version of my proposed meta-analytic SEM (MASEM) method -- actually a collection of methods based on different choices for critical tasks at each step. To keep the exposition relatively accessible I'll gloss over several important technical details. Although this is an active area of my methodological research, the methods I'll sketch here are based largely on unpublished work (I'll cite references). Because some aspects of this approach would be prohibitively difficult for many applied researchers to implement, one aim of my work in this area is to develop user-friendly software. Until that's available, feel free to contact me for help with these techniques; such requests may motivate me to devote more time and other resources to this work.

Overview of Problem

In this section I'll describe my view of the problems for which MASEM is typically used. Suppose we have from each of $k$ independent studies a sample Pearson-r or Fisher-z correlation matrix among $p$ variables of interest (e.g., $X_1, X_2, \ldots, X_p$), and suppose we're interested in a particular SEM for those variables or a related quantity that can be expressed as a function of the correlation matrix (e.g., 1 or more [standardized] path coefficients, indirect or total effect, squared multiple correlation, fit index). It's useful to distinguish between each study's correlation-matrix parameter and that study's correlation-matrix estimate from a particular sample of subjects. Meta-analysts are typically interested in using several studies' estimates to understand their corresponding parameters or distributions thereof.

In this overview I'll mostly ignore several interesting but vexing complications that arise in practice. For instance, studies might use different versions of one or more variables (e.g., different measures similar to $X_1$), some studies might not contribute all $d = p( p - 1) / 2$ distinct correlation-matrix elements (e.g., due to missing variables or unreported elements), some of a study's correlations might be based on different subsets of its sample (e.g., if some subjects are missing some variables), a study might contribute two or more independent or dependent correlation matrices (e.g., from different groups or the same group in different conditions or at different times), studies' correlations might be influenced by so-called artifacts (e.g., [un]reliability, range restriction, dichotomization), and study-level covariates\moderators might partially account for between-studies heterogeneity.

Now, if we take the RE view that correlation-matrix parameters vary among studies -- that is, our studies are from a universe of studies whose correlation-matrix parameters follow some multivariate distribution -- then most functions of that correlation-matrix parameter will also vary among studies. In particular, an SEM's path-coefficient parameters and other quantities (e.g., indirect effects) will have a distribution over studies. To be clear, these distributions' between-studies (co)variation is not due to within-study sampling (co)variation caused by finite samples of subjects; instead, it may be viewed as due to studies' different constellations of features that produce different values for their correlation-matrix parameters and, consequently, most functions of those parameters. (As a simpler example, consider 1 Pearson-r correlation parameter from each of several studies: If it varies among studies, then so will various functions of it -- its square, coefficient of alienation, Fisher z-transform, common language effect size, etc.)

By analogy with typical tasks in conventional RE meta-analyses, we might be interested in the following for our SEM: estimates of the between-studies mean and variance of each SEM path coefficient or other related quantity (e.g., indirect effect) and inferences about each such quantity (e.g., confidence interval [CI], prediction interval [PrI], hypothesis test). When we're interested in two or more quantities from the SEM, we might wish to obtain bi- or multivariate generalizations of these estimates or inferences (e.g., confidence or prediction region\set); these might include quantities based on two or more distinct SEMs we'd like to compare, such as fit indices from two or more SEMs. We might also be interested in the entire between-studies distribution of one or more SEM quantities (i.e., not only its mean and [co]variance [matrix]), which we could in some situations depict graphically (e.g., density plot).

Most commonly used MASEM approaches neglect between-studies heterogeneity in the SEM parameters, which seems especially difficult to justify in the presence of heterogeneous correlation(-matrix) parameters. Certain aspects of the SEM parameters' between-studies (co)variance might be substantively important, such as if we're interested in how much particular path coefficients, indirect or total effects, squared multiple correlations, or fit indices (co)vary among studies. For instance, if we use RMSEA as a fit index, we might wish to know how much RMSEA varies among studies, a plausible range of RMSEA values (e.g., prediction interval), or what proportion of studies have RMSEA values within some "acceptable" interval (e.g., below .05).

Moreover, there's reason to treat heterogeneity cautiously even if we're interested in only the mean of SEM parameters: Because SEM parameters are typically nonlinear functions of the correlation matrix, it's unclear what's estimated by applying the SEM directly to a mean correlation matrix, as in most MASEM approaches; that might estimate the mean SEM parameter poorly. As a simpler example, suppose that for a heterogeneous effect-size parameter $Y$ we know its mean and variance over studies, $\mathrm{E}(Y)$ and $\mathrm{Var}(Y)$, but want to know the mean for $Y$'s square: Because for most distributions of $Y$ it's a fact that $\mathrm{E}(Y^2) = [\mathrm{E}(Y)]^2 + \mathrm{Var}(Y)$, simply squaring $\mathrm{E}(Y)$ gives a value lower than the desired $\mathrm{E}(Y^2)$, especially if $\mathrm{Var}(Y)$ is large. This basic problem with applying a nonlinear transformation to a mean of heterogeneous effect sizes was addressed for the case of a univariate correlation -- focusing on the z-to-r transformation -- by Hafdahl (2009) and Hafdahl and Williams (2009); the analogous situation was addressed for correlation matrices by Hafdahl (2008b, 2009b), for generic univariate effect sizes by Hafdahl (2011), and for generic multivariate effect sizes by Hafdahl (2009c).

Overview of Proposed Method

In the first paragraph above I mentioned the gist of my proposed two-stage MASEM method. To facilitate explanation, it's useful to divide the second stage -- transformation of results -- into separate steps for estimation and inference. For instance, if we start with $k$ studies' estimates of a Fisher-z correlation matrix, the first step might entail estimating the correlation-matrix parameters' between-studies mean and covariance matrix, and the second and third steps might entail transforming those results to obtain estimates of and inferences about the between-studies mean and covariance matrix of the SEM's path coefficients. Below I elaborate a bit on these three steps.

For convenience, let's use the following notation for correlation matrices from Study $i, i = 1, 2, \ldots, k$:

• $\theta_i$: vector of the $d$ distinct parameters in a correlation matrix for $p$ variables, in either the Pearson-r or Fisher-z metric

• $y_i$: vector of the $d$ distinct estimates in sample correlation matrix (i.e., $y_i$ is an estimate of $\theta_i$)

For example, if we're interested in the correlation matrix for $p = 5$ variables, then both $\theta_i$ and $y_i$ contain $d = 5(5 - 1) / 2 = 10$ correlations. In RE meta-analysis we typically assume that $y_i$ has a within-study (sampling\conditional) distribution whose mean is approximately $\theta_i$, and that $\theta_i$ (or just $\theta$) has the same between-studies distribution for all studies, with mean $\mu_\theta = \mathrm{E}(\theta)$ and covariance matrix $\Sigma_\theta = \mathrm{Cov}(\theta)$. (For simplicity I'll slightly abuse notation for random correlation parameters and estimates -- instead of using $\Theta_i$ and $Y_i$ -- and I'll ignore other quantities used in certain meta-analytic procedures, such as each study's sample size and conditional covariance matrix, whose inverse [i.e., precision matrix] is essentially used as a weight matrix.)

In terms of SEM notation, let's similarly denote the SEM parameters of interest in Study $i$ by $\gamma_i = g(\theta_i)$, where the function $g$ transforms a correlation-matrix parameter into SEM parameters. This $g$ might be a fairly complicated function, such as for the parameters of a SEM with respect to a particular objective\loss criterion (e.g., ML, WLS, ADF) or a fit index for that SEM. Also, $\gamma_i$ might be just one number (e.g., 1 path coefficient, indirect effect, fit index) or a vector (e.g., 2 or more path coefficients). At any rate, our MASEM goal might be to estimate $\gamma$'s between-studies mean or (co)variance (matrix), $\mu_\gamma = \mathrm{E}(\gamma)$ and $\Sigma_\gamma = \mathrm{Cov}(\gamma)$, and make inferences about either distributional attribute; we might also wish to estimate $\gamma$'s entire distribution.

Below are the three steps of my proposed method; Steps 2 and 3 presume we have in mind a specific SEM or related quantity that can be expressed as $\gamma_i = g(\theta_i)$. This is similar to Hafdahl's (2009a, 2011) univariate methods and Hafdahl's (2008b, 2009b, 2009c) multivariate methods but specific to MASEM.

1. Meta-Analysis for $\theta$: Apply multivariate RE meta-analysis to $y_i$ to obtain estimates of at least $\mu_\theta$ and $\Sigma_\theta$, which I'll denote $\hat\mu_\theta$ and $\hat\Sigma_\theta$, and perhaps $\theta$'s full between-studies distribution; perhaps also obtain a covariance matrix for only $\hat\mu_\theta$ or both $\hat\mu_\theta$ and $\hat\Sigma_\theta$, depending on how inference is handled in Step 3. Among several proposed methods for estimating $\mu_\theta$ and $\Sigma_\theta$, only a few handle incomplete correlation matrices -- nearly unavoidable in MASEM data sets -- in a principled way (e.g., Hafdahl & Wu, 2011; Kalaian, & Raudenbush, 1996; White, 2011). In particular, Hafdahl and Wu's extension of Becker and Schram's (1994) EM algorithm permits any one or more of $y_i$'s correlations to be missing, doesn't require imputing values for missing correlations, and yields a posterior distribution for each study's entire $\theta_i$ (given its possibly incomplete $y_i$); it also yields an estimate of $\theta$'s full between-studies distribution as a mixture of the studies' posterior distributions. Depending on the estimation method, a covariance matrix for $\hat\mu_\theta$ may be obtained by generalized least-squares (GLS) or other methods (e.g., based on Hessian matrix for maximum-likelihood estimators), some of which also provide a covariance matrix for $\hat\Sigma_\theta$. Hafdahl (2004) demonstrated substantial differences in performance among different techniques for multivariate RE meta-analysis applied to correlation matrices.

2. Estimation for $\gamma$: Use an appropriate transformation method -- based on the function $g$ -- to obtain estimates of at least $\mu_\gamma$ and $\Sigma_\gamma$, which I'll denote $\hat\mu_\gamma$ and $\hat\Sigma_\gamma$, and perhaps $\gamma$'s full between-studies distribution. One strategy is to use a first- or second-order Taylor series approximation, which essentially entails approximating $\gamma = g(\theta)$ by a simpler linear or quadratic function of $\theta$; estimates of this approximating function's mean and covariance can then be computed from Step 1's $\hat\mu_\theta$ and $\hat\Sigma_\theta$. Another strategy entails simulation: Sample values of $\theta$ from its distribution estimated in Step 1, transform these to values of $\gamma$, and estimate $\mu_\gamma$ and $\Sigma_\gamma$ from this simulated distribution; we might treat $\theta$'s distribution as multivariate normal -- such as $\theta \sim \mathcal{N}_d(\hat\mu_\theta, \hat\Sigma_\theta)$ -- or permit it to take some other form estimated from the data (e.g., mixture of posteriors from EM algorithm). Either strategy might also be used to estimate other attributes of $\gamma$'s distribution, such as tail or central areas (e.g., probability that 1 or more path coefficients or other quantities are near 0, positive, large, etc.) or $\gamma$ values bounding regions of interest (e.g., quartiles, middle 95%). (We could in principle transform $\hat\mu_\theta$ and $\hat\Sigma_\theta$ to $\hat\mu_\gamma$ and $\hat\Sigma_\gamma$ via integration, using the definitions of $\mu_\gamma$ and $\Sigma_\gamma$, but that'll often be intractable analytically and infeasible computationally.)

3. Inference for $\gamma$: Make inferences about $\mu_\gamma$ and $\Sigma_\gamma$, such as CIs, PrIs, or hypothesis tests for single-valued parameters or their multivariate generalizations for vector-valued parameters (e.g., confidence or prediction regions). One strategy is to use the (multivariate) delta method, which essentially entails using derivatives to transform the covariance matrix for $\hat\mu_\theta$ alone or both $\hat\mu_\theta$ and $\hat\Sigma_\theta$ to a covariance matrix for $\hat\mu_\gamma$ or $\hat\Sigma_\gamma$; the latter covariance matrix can be used to construct CIs or PrIs or test hypotheses. Another strategy, at least for CIs or confidence regions, is to use a bootstrap technique to essentially construct an empirical sampling distribution of $\hat\mu_\gamma$ or $\hat\Sigma_\gamma$; numerous bootstrap options are available, depending largely on how the sample of bootstrap replicates -- $\hat\mu_\gamma$ or $\hat\Sigma_\gamma$ for each resample from the data -- is constructed (e.g., parametric vs. nonparametric) and how it's used to construct confidence intervals or regions (e.g., standard deviation\error vs. percentile, bias correction or not).

Because this is already a fairly long overview, I'll close with a few remarks. First, despite its advantages over some other MASEM methods, my proposed method also has drawbacks and limitations; I won't elaborate here on these pros and cons, except to caution that my proposed method might perform unacceptably in certain circumstances. Second, my proposed method would benefit from considerably more work, such as refining aspects of each step and studying its performance in realistic situations defined by characteristics of MASEM studies (e.g., number of primary studies, distribution of sample sizes, distribution of correlation-matrix parameters, pattern and mechanism of missing data, choice of function g). To date there's been little evaluation of multivariate RE meta-analysis analytically or by simulation, for either correlation matrices (cf. Hafdahl, 2004, 2008b) or other multivariate effect sizes (cf. Riley, 2009; Riley, Abrams, Sutton, Lambert, & Thompson, 2007), and Hafdahl's (2009c) Monte Carlo studies of meta-analysis for functions of multivariate effect sizes did not include correlation matrices. Third, Bayesian approaches to meta-analysis, such as Prevost, Mason, Griffin, Kinmonth, Sutton, and Spiegelhalter's (2007) proposed method for correlation matrices, might be especially well-suited to MASEM due to their natural -- though computationally challenging -- strategies for constructing posterior distributions for functions of a study's parameters.

References

Becker, B. J., & Schram, C. M. (1994). Examining explanatory models through research synthesis. In H. Cooper & L. V. Hedges (Eds.), The handbook of research synthesis (pp. 357-381). New York: Russell Sage Foundation.

Hafdahl, A. R. (2004, June). Refinements for random-effects meta-analysis of correlation matrices. Paper presented at the meeting of the Psychometric Society, Monterey, CA.

Hafdahl, A. R. (2008a). Combining heterogeneous correlation matrices: Simulation analysis of fixed-effects methods. Journal of Educational and Behavioral Statistics, 33, 507-533. doi:10.3102/1076998607309472

Hafdahl, A. R. (2008b, July). Meta-analysis for functions of heterogeneous correlation matrices. Paper presented at the meeting of the Psychometric Society, Durham, NH.

Hafdahl, A. R. (2009a). Improved Fisher z estimators for univariate random-effects meta-analysis of correlations. British Journal of Mathematical and Statistical Psychology, 62, 233-261. doi:10.1348/000711008X281633

Hafdahl, A. R. (2009b, May). Meta-analysis for functions of dependent correlations. In A. R. Hafdahl (Chair), Advances in meta-analysis for multivariable linear models. Invited symposium presented at the meeting of the Association for Psychological Science, San Francisco, CA.

Hafdahl, A. R. (2009c). Meta-analysis for functions of heterogeneous multivariate effect sizes. Unpublished master's thesis, Washington University in St. Louis, St. Louis, Missouri. http://openscholarship.wustl.edu/etd/439/

Hafdahl, A. R. (2011). Translating meta-analytic results: Techniques to express random-effects estimates in other metrics. Manuscript in preparation, Washington University in St. Louis.

Hafdahl, A. R., & Williams, M. A. (2009). Meta-analysis of correlations revisited: Attempted replication and extension of Field’s (2001) simulation studies. Psychological Methods, 14, 24-42. doi:10.1037/a0014697

Hafdahl, A. R., & Wu, W. (2012, February). An EM algorithm for multivariate random-effects meta-analysis with incomplete effect estimates. Manuscript in preparation, ARCH Statistical Consulting, LLC.

Kalaian, H. A., & Raudenbush, S. W. (1996). A multivariate mixed linear model for meta-analysis. Psychological Methods, 1, 227-235. doi:10.1037/1082-989X.1.3.227

Prevost, A. T., Mason, D., Griffin, S., Kinmonth, A.-L., Sutton, S., & Spiegelhalter, D. (2007). Allowing for correlations between correlations in random-effects meta-analysis of correlation matrices. Psychological Methods, 12, 434-450. doi:10.1037/1082-989X.12.4.434

Riley, R. D. (2009). Multivariate meta-analysis: The effect of ignoring within-study correlation. Journal of the Royal Statistical Society—Series A, 172, 789–811. doi:10.1111/j.1467-985X.2008.00593.x

Riley, R. D., Abrams, K. R., Sutton, A. J., Lambert, P. C., & Thompson, J. R. (2007). Bivariate random-effects meta-analysis and the estimation of between-study correlation. BMC Medical Research Methodology, 7, 3. doi:10.1186/1471-2288-7-3

White, I. R. (2011). Multivariate random-effects meta-regression: Updates to mvmeta. Stata Journal, 11, 255-270.

Answer 2

The following reviews some of the articles that I found discussing and implementing SEM meta-analysis to examine mediation.

Cheung and Chan (2005)

The authors distinguish three approaches of meta-analytic structural equation modelling (MASEM).

• Univariate two-stage MASEM: This includes a collection of two step approaches (note that Cheung and Chan call it simply univariate, but other authors describe it as a two-stage approach). Stage 1 involves first generating a pooled correlation matrix and sample size. Stage 2 involves analysing this correlation matrix and sample size in standard SEM software.
• GLS (Generalised Least Squares for MASEM)
• Cheung and Chan's two-stage MASEM: This approaches use SEM to both synthesise correlation matrices and fit proposed models.

The authors state that if studies have heterogeneous correlation matrices "they cannot be aggregated legitimately" (p.46).

Shadish (1996)

Shadish (1996) provides an early review of combining meta-analysis and SEM approaches. He discusses a few issues and encourages further statistical work on the topic. He also summarises four early studies that had applied meta-analysis to elucidate causal relationships.

1. Harris and Rosenthal (1985): This early study reported used meta-analysis to obtain eight averaged correlations across a set of studies. Four of the correlations were between an independent variable and theorised mediators, and four were between the mediators and the dependent variable. The presence of correlations between all variables was used as support for mediation.
2. Premack and Hunter (1988): This study averaged correlations for a set of studies and performed path analysis on the resulting correlation matrix.
3. Shadish & Sweeney (1991): This study examined between study correlations.
4. Becker's (1992): This study used generalised least squares to model between and within study variation in correlations.

Other Studies

Stajkovic et al (2009) report a SEM meta-analysis which seemed to use the univariate approach. Pooled correlations were calculated and then entered into SEM software for analysis. The authors also examined invariance of parameters in the SEM mediation model across specified values of specified moderators.

Colquitt et al (2007) performed SEM meta-analysis looking at three predictors and four consequences of trust (i.e., the "mediator"). The authors appear to have used the univariate approach. The authors report examining whether correlations varied by the moderators:

Because the two moderators we examined, type of measure and trust referent, did not significantly impact the trust correlations, we used the overall correlation.

However, they also show that correlations vary between studies suggesting that even if moderators don't explain variation in correlations, variation in correlations presumably existed for other reasons.

Fried et al (2008) used the univariate approach to examine mediation effects between work stress, psychological mediators, and job performance. The authors used a random effects model to calculate pooled correlation coefficients and reported using the harmonic mean of cell sample sizes for the sample size in the SEM.

Dunst and Trivette (2009) used the univariate approach on studies related to family-centered care. The authors report pooling correlations based on a weighted average "giving more weight to studies with larger sample sizes and by taking into consideration other statistical artifacts." It is unclear what sample size was used in the subsequent SEM.

They addressed the issue of homogeneity of correlation matrices by writing:

3.1. Homogeneity of the Correlation Matrices. This is a test of whether the correlation matrices in the 15 different studies can be assumed to be derived from the same population. CFI was 0.91 and RMSEA was 0.09. The results indicate that the different correlation matrices were reasonably similar to produce a pooled correlation matrix.

However, an RMSEA of .09 actually sounds fairly large. I've seen recommendations for an RMSEA of less than .05 for a very good fit. I'm not questioning that the correlation matrices weren't similar. But in general we would expect some true score variance and to my mind such results seem consistent with some variation. The lack of straightforward analysis options for handling heterogeneity discourages exploration of how results would be different if the data was assumed to be heterogeneous.

Bamberg & Moser (2007) used the univariate approach to test a model of the theory of planned behaviour applied to environmental behaviour. Because correlations varied significantly between studies according to the Q-test, the authors used the random-effects estimate of the pooled true correlation (but they also report pooled estimates based on the fixed-effects model) and report the mean and 95% confidence intervals for the true correlations. The harmonic mean of cell sample sizes was purportedly used in calculating cell sample sizes.

Bauer et al (2007) used the univariate approach to test a model of newcomer adjustment. It was not clear to me whether the pooled correlations were based on fixed or random effects assumptions. Rationale for the sample size is not stated explicitly but looks broadly consistent with something like a harmonic mean of the cell sample sizes.

Joseph et al (2007) used the univariate approach to test a model of job turnover. Pooled correlations were formed by weighting sample correlations by sample size and correcting for measurement error. The harmonic mean of of cell sample sizes was used.

Chang et al (2009) used the univariate approach looking at organisational politics.

Haeussler-Keyton (2012) is a doctoral thesis using the univariate approach that performed meta-analytic path analysis on studies related to breast feeding success. The size used was calculated as such: the average sample size N in studies for each cell was calculated, and the smallest of these was used for analyses. Note that this results in a much smaller sample size than simply taking the average total cell sample size. The author fit models using pooled correlations based on both fixed and random effects calculations.

Zhang (2011) wrote a doctoral disertation reviewing the GLS approach and the multivariate two-stage approach of Cheung and Chan (2005).

Viswesvaran and Ones (1995)

Viswesvaran and Ones (1995) provide a tutorial-style guide for univariate MASEM . They cite several studies that have used the approach including Hunter (1983), Hom, Caranikas-Walker, Prussia, and Griffeth (1992), Peters, Hartke, and Pohlmann (1985), Brown and Peterson (1993), Ones (1993) and Viswesvaran (1993). The general approach is to compute true-score correlations using various standard meta-analytic techniques and use that correlation matrix as input to SEM.

Viswesvaran and Ones (1995) discuss the issue of dealing with missing cells in the correlation matrix. They mention several options for dealing with this:

(a) design a primary study to collect data with sufficiently large sample size, such that the effects of sampling error are reduced, to obtain stable estimates of the correlations not reported in the literature; (b) use the average (across all correlations) correlation in the empty cells; (c) look for patterns of correlations and impute values in the missing cells of the matrix; and (d) modify the test of the theory to include only the constructs for which a full matrix of estimated true score correlations is available in the liter- ature. A final option, as suggested by an anonymous reviewer, is to use Subject Matter Experts to estimate the missing correlation (e.g., Schmidt et al., 1983).

Viswesvaran and Ones (1995) also discuss the issue of varying sample size per cell in the correlation matrix. They mention several options: (a) use harmonic mean of the sample sizes across cells; (b) only include studies that include all variables; (c) assume sample is population and ignore standard error and confidence intervals.

Viswesvaran and Ones (1995) also acknowledge the issue that true correlations may systematically vary between studies. They suggest several approaches: (a) include moderators until true score variance is reduced to zero and include these moderators in the path analysis; (b) compute the three SEMs, one using lower 90% confidence intervals of meta-analysed correlations, another using upper 90% confidence intervals of meta-analysed correlations, and a third using the mean.

The above issue of variation in true correlations is one of the biggest issues that I'm concerned with. Most meta-analyses that I have read show variation in true correlations (hence various recommendations to prefer random-effects meta analysis). Furthermore, it is unlikely that available moderators would account for all true variation. In many contexts, I doubt that available moderators would even account for most of the true variation. Thus, while I can see that including meaningful moderators would be useful, I predict that in most applications, this would not solve the problem of variation in true correlations.

The second option of computing lower, mean, and upper bound correlations also does not seem to me to provide a solution to the problem. First of all, even if correlations vary between study, it is unlikely that such variation would lead to uniform increases and decreases in all correlations. For examples, in some studies the true correlation may be higher than the mean for one pair of variables and lower for another. That said, the idea of sampling the range of distributions does sound like it could have promise.

Cheung and Chan (2009)

Cheung and Chan (2009) provide a technical overview with appropriate formulas, a simulation, and an example on performing the two-stage approach to SEM meta-analysis. They acknowledge that heterogeneous correlation matrices are problematic. They suggest the following options: (a) clustering correlation matrices; (b) release some cross-group constraints (e.g., based on moderators) in the SEM.

Becker (2009)

Becker (2009) sets out how to perform model testing based on the GLS approach.

Personal thoughts

Pooling and random effects

In general, the random-effects model seems more reasonable. Unless the meta-analysis is composed of exact replications, studies typically vary in a wide range of ways. And this manifests in varying correlations. Using a random-effects model involves weighting sample correlations differently by incorporating between study variance. The consequence is that studies with smaller sample sizes are weighted more than they would otherwise be using a fixed-effects model.

Variation in correlations may not necessarily be normal. There may be outlier correlations in studies. Observed moderators may explain some of the between study variance in correlations. Similarly artefacts such as reliability, range restrictions, etc. may explain additional variance in effect sizes.

If there is no true-score variance then the two-step approach seems reasonable. Also, if after including moderators, true-score variance is accounted for, the two-step approach seems reasonable. Likewise, the idea of clustering correlation matrices to remove true-score variance might have promise.

Using a random-effects model to weight correlations is a reasonable way to get an estimate of the mean true correlation. However, using such pooled estimates in SEM raises several issues. First, such a procedure does not capture the true variation in effect sizes. Parameter estimates and SEM fit will vary across studies in systematic ways; only analysing the mean true correlations ignores this systematic between study variation. Second, using sample size estimates based on the average, minimum, or harmonic mean, of cell sample sizes seem to all assume a fixed-effects model of precision in estimating the true correlation coefficients.

Other Approaches

Another strategy would be to perform SEM on each sample and treat the parameters and fit statistics as values that vary between samples. The distribution (e.g.,mean and SD) of these SEM parameters and fit statistics could be summarised. This would be similar to how correlations and other effect sizes are typically modelled as random effects. So, for example the variation in the indirect effect across samples could be examined. The challenge would be to separate what is true score variation and what is due to random sampling.

This seems like a good idea, although often meaningful variance in effects will remain even after controlling for moderator and often the number of studies for a given moderator value can be minimal.

References

• Bamberg, S. & Moser, G. (2007). Twenty years after Hines, Hungerford, and Tomera: A new meta-analysis of psycho-social determinants of pro-environmental behaviour. Journal of environmental psychology, 27, 14-25.
• Bauer, T.N., Bodner, T., Erdogan, B., Truxillo, D.M. & Tucker, J.S. (2007). Newcomer adjustment during organizational socialization: A meta-analytic review of antecedents, outcomes, and methods.. Journal of applied psychology, 92, 707. PDF
• Becker, B. J. (1992). Models of science achievement: Forces affecting male and female performance in school science. In T. D. Cook, H. M. Cooper, D. S. Cordray, H. Hartmann, L. V. Hedges, R. J. Light, T. A. Louis, & F. Mosteller (Eds.), Meta-analysis for explanation: A casebook (pp. 209-281). New York: Russell Sage Foundation.

• Becker, B. J. (2009) Model-based meta-analysis, chapter 20, in The handbook of research synthesis and meta-analysis, editors: Cooper, H.M. and Hedges, L.V. and Valentine, J.C., Russell Sage Foundation Publications.

• Brown SP, Peterson RA. (1993). Antecedents and consequences of salesperson job satisfaction: Meta-analysis and assessment of causal effects. Journal of Marketing Research, 30,63-77.

• Cheung, M.W.L. & Chan, W. (2005). Meta-analytic structural equation modeling: a two-stage approach. Psychological Methods, 10, 40. PDF
• Cheung, M.W.L. & Chan, W. (2009). A two-stage approach to synthesizing covariance matrices in meta-analytic structural equation modeling. Structural Equation Modeling, 16, 28-53.
• Colquitt, J.A., Scott, B.A. & LePine, J.A. (2007). Trust, trustworthiness, and trust propensity: A meta-analytic test of their unique relationships with risk taking and job performance.. Journal of applied psychology, 92, 909.
• Dunst, C.J. & Trivette, C.M. (2009). Meta-analytic structural equation modeling of the influences of family-centered care on parent and child psychological health. International journal of pediatrics, 2009. PDF
• Fried, Y., Shirom, A., Gilboa, S. & Cooper, C.L. (2008). The mediating effects of job satisfaction and propensity to leave on role stress-job performance relationships: Combining meta-analysis and structural equation modeling.. International Journal of Stress Management, 15, 305.
• Joseph, D., Ng, K.Y., Koh, C. & Ang, S. (2007). Turnover of information technology professionals: a narrative review, meta-analytic structural equation modeling, and model development. MIS Quarterly, 31, 547-577.
• Harris, M. J., & Rosenthal, R. (1985). Mediation of interpersonal expectancy effects: 31 Meta-analysis. Psychological Bulletin, 97, 363-386. Haeussler-Keyton, K. (2012). Psychological and Social Factors Associated with Breastfeeding Success: A Meta-Analytic Path Analysis. Doctoral Thesis at Texas Tech University. PDF
• Hunter JE. (1983). A causal analysis of cognitive ability, job knowledge, job performance, and supervisor ratings. In Landy FJ, Zedeck S, Cleveland J (Eds.), Performance measurement and theory (pp. 257-266). New Jersey: Eribaum.
• Ones DS. (1993). The construct validity of integrity tests. Unpublished doctoral dissertation. University of Iowa, Iowa City.
• Peters LH, Hartke DD, Pohlmann JT. (1985). Fiedler's contingency theory of leadership: An application of the mieta-analysis procedures of Schmidt and Hunter. Psychological Bultetin, 97,274-285.
• Premack, S. L., & Hunter, J. E. (1988). Individual unionization decisions. Psychological Bulletin, 103, 223-234.
• Shadish, W.R. (1996). Meta-analysis and the exploration of causal mediating processes: A primer of examples, methods, and issues.. Psychological Methods, 1, 47.
• Shadish, W. R., & Sweeney, R. (1991). Mediators and moderators in meta-analysis: There's a reason we don't let dodo birds tell us which psychotherapies should have prizes. Journal of Consulting and Clini- cal Psychology, 59, 883-893.
• Stajkovic, A.D., Lee, D. & Nyberg, A.J. (2009). Collective efficacy, group potency, and group performance: Meta-analyses of their relationships, and test of a mediation model.. Journal of Applied Psychology, 94, 814.
• Viswesvaran, C. & Ones, D.S. (1995). THEORY TESTING: COMBINING PSYCHOMETRIC META-ANALYSIS AND STRUCTURAL EQUATIONS MODELING. Personnel Psychology, 48, 865-885.
• Viswesvaran C. (1993). Modeling job performance: Is there a general factor? Unpublished doctoral dissertation, LIniversity of Iowa, Iowa City.
• Zhang, Y. (2011). Meta-analytic Structural Equation Modeling (MASEM): Comparison of the multivariate methods. Doctoral Thesis at Florida State University. PDF