There are systematic ways to compare similar tests and determine which test best fits your needs.
Construct Definition
Identify the constructs (in this case, traits or abilities) you are interested in and clearly define the meaning of each construct. Cognitive performance can encompass many domains, including memory, processing speed, psychomotor functioning, and various aspects of intelligence (including social intelligence). Once you’ve identified which components you consider to be key to “peak cognitive processing,” use literature in your field to develop accepted or theory-driven operationalized definitions for each construct so you have a standard to which you can compare different tests.
Test Identification
Identify potential tests; generally their title or brief description will allow you to identify tests that are assessing constructs of interest. When comparing tests, however, there are several things to consider in order to 1) ensure the test is actually measuring your construct of interest, 2) compare similar types of tests, and 3) select the test which best suits your needs.
Validity
Assess various aspects of test validity for each tests. Construct validity assesses whether the test measures the construct of interest; a quick (though not thorough) way to assess construct validity is to compare the definition or theory the test’s developers use to the definition you have created. If you’ve defined “depression” in cognitive terms (difficulty thinking, distraction, ruminative thoughts) and the test you’re assessing has defined it in emotional or somatic terms (sad, hopeless, problems sleeping, loss of appetite), then that test would not match your definition of depression and may not capture the construct you are interested in. I advise using these definitions as a starting place, so that you feel confident the test is appropriate for your purposes. However, you should also consider other types of validity as well. Content validity assesses whether it captures part of a construct (like factual knowledge) or the whole construct (like factual knowledge and the application of that knowledge). Criterion/predictive validity is whether the test has shown to predict certain outcomes measured at a later time; an example is seeing if a reading test predicts year end grades in reading. Criterion validity is most often used in achievement or employment testing and may not be relevant to your work.
Other Psychometric Properties
Assess additional psychometric properties of each measure. Is there evidence of convergent (is it positively correlated with similar variables) and divergent (is there a negative or null correlation with dissimilar variables) validity? Has test-retest or inter-rater reliability been established? Do the scales have good internal consistency (Cronbach’s alpha)? Is the sensitivity (number of true positives identified) and specificity (number of true negatives) known?
Norms and Development
Think about what groups the test has been used with and developed in. Was it developed only based on samples of college students? Has it been validated in diverse groups? Some of the more well-known batteries may be “normed” tests, which allow you to compare scores to a well-defined reference sample. However, if that sample is drastically different from the population you work with, it may not be helpful to make that comparison.
Test Burden and Assessing Secondary Traits
You will also need to decide if there are certain factors that might complicate the use of a test, such as excessive length or a high reading level or difficulty. Finally, when you’ve settled on a valid, reliable, assessment of appropriate length that matches your construct of interest, look over the measure and be aware of any construct contamination issues. An example is a working memory test (perhaps making someone do solve oral story problems in his head) that is also timed. In this case, if the person is penalized for not working quickly enough, you might be capturing deficits in processing speed rather than working memory. Similarly, if you’re using the test to assess math skills, you might be capturing working memory problems rather than an accurate assessment of math ability; the person may be able to correctly complete the math problems on paper pencil if they are not required to remember the problem while solving it. You may be able to counter the effects of contamination by measuring the same skill (math) multiple ways (both orally and on paper, both timed and untimed).
Summary for identifying and comparing assessments:
- Ensure the test is assessing the construct of interest using the
operationalized definition you have in mind.
- Assess reliability and validity (including norms, sensitivity, and
specificity)
- Ensure length, language, etc. are practical for your purposes
- Be aware of possible construct contamination
References
Groth-Marnat, G. (2009). Handbook of psychological assessment. (5th Edition). Hoboken, NJ: John Wiley & Sons, Inc.
Furr, R. M., & Bacharach, V. R. (2008). Psychometrics: An introduction. Thousand Oaks, CA: Sage Publications, Inc.
