General points about practice effects
Intelligence tests differ in how much they are subject to practice effects.
Practice effects can also be distinguished:
- Time between taking the test: The shorter the timeframe the more likely you will see practice related improvement.
- General practice on similar tests and similar items versus practice on the same set of items. In general, practice effects will be greater where the items are identical between test sessions. Adaptive testing where items change each time should show fewer practice effects.
- Whether feedback is present. In general participants do not get feedback on what was or was not the right answer, but if they do receive such feedback, then practice effects will be much greater, especially for item specific transfer.
Test-retest variation
Another general point is that intelligence tests have a level of reliability such that a score on a test is only an estimate of an individual's true standing on the latent trait being measured. So obviously, some variation between administrations would be expected purely based on the standard error of measurement.
That said, the general model of test-retest reliability assumes reasonably controlled testing conditions and a participant that is applying reasonable effort.
When these conditions are not met, you could see much more variation:
- If a test taker applies minimal effort or is distracted, they may get a much lower score than their true level.
- If a test taker cheats in some way they may get a much higher score than their true level.
Issues of administration are more likely in an online environment particularly where there is either (a) no consequences for not applying full effort, or (b) an incentive to cheat.
Interpreting scores of someone who has taken a test multiple times
In general, if you are trying to measure intelligence, you should administer the test once and that first measure would be the measure of their intelligence.
Test administrations that follow immediately afterwards would be contaminated by practice effects. Unless you have a very good understanding of how practice effects operate and how they can be controlled for, then it would be difficult to incorporate information from these subsequent administrations.
That said, if you had additional information that said that on the first administration the participant was not concentrating or fundamentally failed to understand the test, then you may wish to tentatively rely on the score from a different administration.
In addition, a pattern of scores that differ by three standard deviations (e.g., 100, 115, 145) may suggest either that the test has reliability issues or that something strange is happening for that particular individual that you want to look into.
