Which statistical tests to use when analyzing NASA TLX?

Question

The NASA Task Load Index (NASA-TLX) is a subjective, multidimensional assessment tool that rates perceived workload. A set of different scales (Mental Demand, Physical Demand, Temporal Demand, Performance, Effort, Frustration) are combined into one measure ranging from 0 to 100, the task load index.

Which statistical tests can be used to compare these test results? I am particularly concerned with whether or not the task load index could be considered an interval variable, and thus whether, for example, a simple student t-test could be used. In particular, is the NASA-TLX scalar invariant? Can a difference between two means (e.g., 50 and 60) be interpreted as equal to a difference between two other means (e.g., 10 and 20)?

If not, what are the consequences of using a wrong test? What is a suitable test?

Answer 1

In Stevens' levels of measurement framework, the NASA-TLX is an ordinal scale, not an interval scale, because there is no way to know a priori how much "workload" each point corresponds to. In other words, we can only know how much workload a point corresponds to after observing the data from our sample. Indeed, Hart's own review of the NASA-TLX suggests that workload is "ill-defined" (Hart, 2006).

While levels of measurement often seems to be taught in behavioral and social science as accepted fact, however, the levels of measurement framework is quite controversial among methodologists. It is a useful enough heuristic taxonomy for didactic purposes, but it is not known to be inferentially meaningful. Analytic relatives of the t-test (i.e., least-squares/maximum likelihood regression) exist for all four types, so the type of scale should not necessarily concern you too much here. I have not encountered it outside the behavioral sciences.

Statistical inference is essentially a mathematical pathfinder that formally relates a specific question to a specific answer by way of the evidence. It is therefore impossible to say what a suitable test would be without knowing the specific question(s)―different analyses test different questions. For the same data set, one analysis may be meaningful (i.e., discriminates between conceivable answers) for one question, but not another; for another analysis, this may be reversed. The consequence of using a wrong test is that you effectively answer a question you did not ask, while not answering the question you did ask.

References

• Hart SG. NASA-Task Load Index (NASA-TLX): 20 years later. Paper presented at the HFES; San Francisco, CA. 2006.

Answer 2

I am particularly concerned with whether or not the task load index could be considered an interval variable,

Yup.

This is a fundamental assumption people make when constructing, administering and analyzing any measure under the classical test theory (CTT) paradigm i.e. count items and add 'em up.

That is to say, your variable is certainly treated as though it is linear, but are the scores actually linear? is it valid to combine scores on each factor to derive another score?

Even further, conceptually, let us presume that two individuals have the same task load index. Do they possess, quantitatively, the same amount of the latent variable? Not necessarily.

This is a fundamental assumption and inherent flaw in classical test theory.

Can a difference between two means (e.g., 50 and 60) be interpreted as equal to a difference between two other means (e.g., 10 and 20)?

They could be, but this is an assumption you are making by applying classical test theory. As noted above, you are also assuming that two individuals with the same score are equal. This is also an assumption that may or may not be correct.

If not, what are the consequences of using a wrong test?

This is a no-brainer and I’m sure you know the answer to this: the wrong inferences will be made regarding results, and how they are applied in a meaningful way.

Suppose perceived workload is non-linear and instead, say, exponential, and it is tightly correlated with the "likelihood of totally spazzing out." NOTE: despite earnest lobbying on my behalf, I was unsuccessful in convincing the APA to consider Low Spaz Threshold Disorder for inclusion in the DSM-V. Oh well, there is always DSM-VI. notwithstanding, incorrectly assuming workload is linear might lead you to erroneously think that someone with a score of 70 has only a slightly higher likelihood of spazzing out relative to someone with a score of 60.

What is a suitable test?

If this actually concerns you - and it should if you care about validly measuring the construct, look into item-response theory (IRT) which departs from the CTT that is commonly applied.

Briefly, IRT is a statistical model that specifies the probability of each response option to an item as a function of the target trait thought to be measured. Respondents are no longer scored by the number of items they answer correctly. Instead, they are assigned an estimate of their location on the underlying distribution of ability (ϴ). Although it's commonly applied for exams, where the term ability is meaningful, ability can also be taken to mean level of depression, stress, happiness, procrastination or any other latent trait of interest. In your case, perceived workload.

IRT can take into account the non-linear relationship between an individual's score and their level of ϴ. When estimating a person's ϴ, parameters such as item difficulty and discrimination can be taken into account.

For each item, one can derive an item characteristic curve (ICC), which maps the probability of endorsing an item according to an individual's ability. Think of ϴ as being scored in standard deviations. This figure shows the ICC for a dichotomous variable.

Of greater relevance to your problem, this can also be done for items that are polytomous. Note how the estimated ϴ shifts in a non-linear fashion as we go from endorsing 0, 1 ... 3 in step-wise manner. That is, within a single item, the level of ϴ can differ substantially between likelihood of endorsing "agree" versus "strongly agree."

A test characteristic curve can also be derived as shown below. The true score is the essentially the raw score. As you can see, about 15 points separate people with ϴ = -1 vs. 0, while about 10 points separates ϴ = 1 vs. 2. Once ability estimates are determined, appropriate statistical tests can be applied.

In theory, I think you could apply IRT to the NASA-TLX. You would probably need to treat each subscale as though it was actually an item, with the workload index as the summed score. There are also several assumptions that need to be considered when applying IRT. One important one in your context is that it assumes that single latent trait is being measured. As I know little about the NASA-TLX, it's unclear to me if this is the case. And you will need a large dataset to calibrate the curve functions.

A simple but informative primer can be found here:

A Simple Guide to the Item Response Theory (IRT) and Rasch Modeling

http://www.creative-wisdom.com/computer/sas/IRT.pdf

Note: the Rasch model is the most common model of IRT, but it deals with dichotomous data. What you're after is a polytomous model, but you would do well to first understand how the Rasch model is applied.