How to analyze reaction choice experiments?

Question

What are the state-of-the-art approaches to analyze reaction choice experiments?

Example of a reaction choice trial:

1. Fixation cross (randomized length between 500 and 1500ms)
2. Choice stimulus: Indicates to press left or right button (200ms)
3. Response: Which and if one of the two buttons is pressed (max 1000ms after stimulus onset)

This procedure produces the following types of data:

1. Response times, of
   1. correct responses
   2. incorrect responses
   3. both
2. Correctness of responses
3. Omitted responses

What are valid ways to analyze these interdependent yet disjunct types of results? Ideally including references to examples.

Answer 1

Omitted responses are not usually analyzed (what does that even mean, no response before a deadline?, a "does not want to respond" button?).

If you want to analyze jointly choices and RTs, you need a model. I would say the most admitted model right now is the drift-diffusion model (Ratcliff & Rouder, 1998), which models choice and RTs as the accumulation of noisy evidence toward a decision boundary. These models are actually quite old (the late 50s), but they regained a lot of interest in the past 10 years. Some toolbox (e.g. Vandekerckhove & Tuerlinckx, 2008) will do that for you. Note that these models originally predict equal RTs for correct and incorrect choices, which is typically not what is found. You'll have to add parameters to predict these aspects which will require more data do produce good fits. You can also tweak the model to predict opt-out choices (Kiani, Corthell & Shadlen, 2014). You can tweak it some more to make it predict trials that failed to produce a response before a deadline, but I doubt any toolbox will do it out of the box.

These models are especially usefull if you have various conditions on the same scale (for example 5 levels of luminance), because that will strongly constrain your fits. If you want to analyze differences between a small number of conditions, especially if they are not on the same scale, I would avoid using them. (1) Fitting a 10+ parameter model to compare just 2 conditions is highly dubious (despite that people do it all the time), and (2) your results become model dependent. What if the model is wrong, are your conclusions still valid? What you can do is simply compare z-transformed ratio correct responses and median of RTs (because the are not normally distributed, you can also take their inverse which make them more or less normal) in your favorite statistical test and make sure there is no speed-accuracy tradeoff.

Ratcliff, R., & Rouder, J. N. (1998). Modeling response times for two-choice decisions. Psychological Science, 9(5), 347-356.

Vandekerckhove, J., & Tuerlinckx, F. (2008). Diffusion model analysis with MATLAB: A DMAT primer. Behavior Research Methods, 40(1), 61-72.

Kiani, R., Corthell, L., & Shadlen, M. N. (2014). Choice certainty is informed by both evidence and decision time. Neuron, 84(6), 1329-1342.