Using Exploratory Factor Analysis for Items That Belong to Different Scales

Question

I have a quick question about factor analysis from different scales. Is it advisable to use, conjunctly, items from various measures? For example, say that I want to evaluate factors that most influence the mental health of a certain subpopulation. I use a combination of items from four scales that measure different mental health domains, each of which having excellent construct validity and reliability estimates. I assume no a priori factor structure so I conduct a factor analysis to determine (a) which items cling together and (b) an interpretation per each clump.

I'm severely conflicted on this. One the one hand, these scales have been shown to do a good job of measuring their domain and thus it would make intuitive sense to include them in the analyses. But on the other, wouldn't we expect them to cling together?

If this technique isn't advisable, how do we actually begin gathering items for exploratory factor analysis? Do we just take a wild guess?

Answer 1

This is by no means a rigorous answer, but it seems that items will not always cling to their respective scales. It makes sense to use these items as they usually have psychometric support.

Still, it is probable that many (if not most) items will still cling together, especially if the reliability estimates of the original scales are relatively high. In this case, I wonder if there's a simple way to compare similar factors. Although confirmatory factor analysis (CFA) is the most rigorous approach, this technique requires MORE data and may not be practical.

Instead, it would be interesting to see if analyses on factor scores can help determine whether a set of factors are significantly different. If analyses show a lack of significant difference, it would be best via the principle of parsimony to use the scores from the original scale, as no new information seems to be established.

Still, I have no idea if this is a good approach. A comparison of means of factor scores depends on how the scores where calculated; for example, if these scores are just simply means of items, then testing mean differences may not be meaningful as the mean will depend on several arbitrary characteristics, including, but not limited to, number of items and scoring of items. If these scores are Bartlett Scores, then they are standardized and thus cannot be used in mean difference analyses. Perhaps a simple, directional ($r \geq .60$) correlational test can be used.