What experiment in a simple decision task should we run to obtain one million trials?

Question

Context: Models of cognitive processes require very large datasets to be fitted. Sadly, it is difficult for a single laboratory to achieve this alone. I propose that we work collectively and collaboratively in realizing a 1 million RT experiment. This initiative is described on my ResearchGate page, www.researchgate.net/profile/Denis_Cousineau.

Beyond model fitting, such a dataset will be beneficial for other purposes. Examples: Are parameters describing performances normally distributed across subjects? Does training effect --resulting in faster RT-- imply that the parameters are all evolving in one direction? Does Bayesian estimation more apt than Maximum Likelihood at characterizing the parameters? How can we estimate non-decision times? What is the shape of the RT distributions and how training effect alter its shape? In Bayes again, after how many trials does priors have no longer an impact on the estimates? etc.

Task Requirements: The task must be behavioral only (response times and response choices) as it is not possible to move EEG or fMRI material all around the world. The precise task used is not a critical constraint because the current models of cognition (sampling models such as LBA or Diffusion) are very flexible; they aim at predicting performance in a wide variety of situations.

The tasks proposed will be evaluated based on the following criteria criteria:

• small number of response alternatives
• few conditions
• low error rates
• stimuli that can be characterized for modeling purposes
• possibility of extended training on that task.

Possible candidate tasks could include: A redundant target detection task (Miller, 1982) or a same-different task (e.g., Bamber, 1969). These tasks are simple and accuracy is typically high (above 90% correct); they are possibly performed using some form of parallel processing; response times are generally well below 700 ms.

So in summary:

What task and experimental design would be most suitable given the above constraints?

Please let's see what are the possible tasks, comment on others' proposals, and if you find the perfect task, upvote it. Have your colleagues make comments or propose new tasks.

Answer 1

I don't like the idea that the task should be low error, errors are needed for modeling choice. I don't like random dot motion because there are big individual difference making it hard to find a common calibration point over people. I prefer instead a numerosity judgement, e.g., are more of less than 50% of a square array of pixels (usually a large number) blue or orange? You can make the array big enough to get fine grained control of difficulty, and randomly choose which pixels to have in each color (or brightness) on each frame if you want to make particular strategies hard to use.

Answer 2

I like this idea! I think the task should be one where the data could be used by many analytic tools. Hence, the data would not only be beneficial for one particular question, for example parameter estimations, but also for other tools such as Systems Factorial Technology (see Townsend & Nozawa, 1995), which provides a deeper understanding of the underlying cognitive sub-processes architecture, and other interesting information such as the process' capacity.

I suggest that the task at least respect a double factorial paradigm. This design is very simple: you have four conditions in a 2 x 2 design, where you vary two variables. It could be any variable, the only restriction is that you have to vary them independently one from another. In other words, each variable should reflect a cognitive sub-process. In one condition, both variables are in scenario where the sub-process should work very effectively (or optimally). In an other condition, both variables are in a scenario where the sub-process should work less effectively (or sub-optimally). And in the two remaining conditions, one variable works optimally and the other sub-optimally, and vice-versa.

As for the very details of the task, such as the stimuli that will be used and the specific variables, I do not have preferences. Letters are very manipulable, but they do carry semantic (more or less). Abstract stimuli could work. We could play with color, shapes, complexity, etc. At Psychonomics this year, they were many experiments which have use this design.

Reference for citation: Townsend, J. T. & Nozawa, G. (1995). Spatio-temporal properties of elementation perception an investigation of parallel, serial, and coactive theories. Journal of Mathematical Psychology, 39(4), 321–359. doi: 10.1006/jmps.1995.1033

Answer 3

Just a few comments...

Since one of the advantages of a large data set is high statistical power, it might be a good idea to use a task where the key effect has not been found in previous studies. That would produce a guaranteed result: i.e., a high-power test of an important null result.

A pop-out condition in visual search would be one example (i.e., no effect of display size), but this is not ideal because it would limit the modelling end of the project. That is, it wouldn't be too helpful to model multiple conditions with identical performance (or nearly so).

A better example would be an additive-factor experiment (somewhat as suggested by Marc-André Goulet). I would suggest a simple 2x2 design for which previous studies had reported theoretically important factor additivity (i.e., non-significant interaction). A big study would provide a high-power test of the null interaction. And, as long as both main effects were present, there would be four distinct conditions to model. This design is also interesting because a lot is known about conditions that should be satisfied by the RT distributions under certain models (for examples, see the references below).

A final point: After selecting a task, it might be very useful to test out the planned analyses on simulated data, which are considerably easier to get than real data. This would help make the project's precise goals more explicit and also make sure the goals would be achievable using the proposed data set. It might also help to reveal what aspects of the planned task/conditions were necessary for getting informative results (e.g., minimum effect sizes).

References

Cortese, J. M. & Dzhafarov, E. N. Empirical recovery of response time decomposition rules II. Discriminability of serial and parallel architectures. Journal of Mathematical Psychology, 1996 , 40 , 203-218

Dzhafarov, E. N. & Cortese, J. M. Empirical recovery of response time decomposition rules I. Sample-level decomposition tests. Journal of Mathematical Psychology, 1996 , 40 , 185-202

Dzhafarov, E. N. & Schweickert, R. Decompositions of response times: An almost general theory. Journal of Mathematical Psychology, 1995 , 39 , 285-314

Roberts, S. & Sternberg, S. The meaning of additive reaction-time effects: Tests of three alternatives. In Meyer, D. E. & Kornblum, S. (Eds.) Attention and performance XIV. Synergies in experimental psychology, artificial intelligence, and cognitive neuroscience., MIT Press, 1992 , 611-653

Answer 4

I agree with Marc-André, the selected stimuli should have no semantic association (or as little as possible) which is why I would encourage the use of geometric shapes that vary in colour. Additionally, I believe that the task should be as simple as possible in order to encourage the overall cleanliness of the data.

As you suggested, one such task is the redundant attribute target detection task (Miller, 1982). In this task, participants are asked to keep in mind target dimensions (e.g. blue and square) and as soon as one of the dimensions is presented on screen, the participants respond as quickly and as accurately as possible. Results are typically fast, accurate, and slightly skewed – appropriate for this initiative! We could select a few colours and shapes (if we do go forward with this task) as well as what colour harmony should be selected.

Additionally, having colours that vary in “obviousness” can potentially give us a clearer understanding of the parameters within certain models. For example, an obvious blue should have quicker RT than a muted blue and the rate parameter of each individual channel should reflect that (assuming, of course, that the rate parameter is associated with "obviousness" at all). I am particularly interested in this aspect.

While a go/no would be great for this task, I also strongly believe that a 2AFC task would yield much more interesting results and the 1million RT initiative could help us really understand the “race” in “race models”. This could be as simple as modifying the task as one with “target present” or “target absent” decisions. With clearer trends in RT we could hopefully gather information on the independence of dimensions and whether there is crosstalk between detectors.

Additionally, having such data would not only benefit model fitting but also more solid architecture analyses. This would, in turn, help us get a more concrete understanding of coactive architectures (the architecture that Miller, 1982 suggested was at play in this type of task) which haven’t been as rigorously defined like other architectures (as noted by Houpt & Townsend, 2011).

EDIT:

Houpt, J.W., & Townsend, J.T. (2011). An extension of SIC predictions to the Wiener coactive model. Journal of Mathematical Psychology, 55, 267-270

Miller J. (1982). Divided attention: Evidence for coactivation with redundant signals. Cognitive Psychology, 14,247–279

Answer 5

I myself am interested in obtaining such a dataset on the Same-Different task (Bamber, 1969). The answer is binary (same or different), accuracy is very high and there is a learning effect (short term, at least). Concerning the number of conditions, we could simplify the original task to go from 14 conditions to 8 by either:

• limiting the number of letters to 4 (instead of 1, 2, 3 and 4)
• using 1, 2 or 4 letters stimuli, and having 1 or 2 mismatches for the 2 letters stimuli, and 2 or 4 mismatches for the 4 letters stimuli

The second option is closest to the original. The first, however, would be very informative on the 4 letters condition, as there is little data on them due to the experiment's design.

Answer 6

Given many individuals (mostly men) have anomalies of colour perception, you might avoid colour; you also want a simple task with 'easy' stimuli, to enable short trial durations. Visual search paradigms are good for this. I echo the call for a project in which the experimental design would address something of particular interest, and ideally that would be difficult to address with a 'standard' design (e.g., with 10,000 data points, rather than 1,000,000). Estimating the distribution of processing times for a process inferred from indirect measures may be an interesting problem domain.

PJ

Answer 7

I vote for a simple random dot motion task. This has the advantages that: 1. It's easy to program. It can be delivered over the web via javascript, which makes the "many labs" aspect easy. 2. It's very well studied and well understood. 3. It is easy to set 5-ish coherence levels that will sweep out almost the full range of accuracy, from chance level to 100%. This is important for distinguishing between the models.

Answer 8

May I ask, have you thought about using existing large datasets, such as those provided by Project Implicit? Dataset.

(sadly, it seems Project Implicit has not stored individual data points but rather a computed 'effect' per participant).

Answer 9

May I suggest collecting data online?

With the latest browser technologies, RTs are extremely accurate, in the order of 5 microseconds (discussion). Even with slightly older browser tech, collected RTs can be accurate (ref).

There are several sites where people freely volunteer to do research (discussion).

Collecting data over multiple sites is most certainly easier when there is setup time per location.

Answer 10

Please allow me to question your setup:

Your assertion that you need a million data points to distinguish between models raises the question whether your're still doing valid science, I think. Are you sure that the differences between the models are scientifically sufficiently interesting to warrant a monster project to try and distinguish between them? Especially, if this is all based on a single task. I am sure, your quest will result in a "winning model". But I am not sure that it will result a generalizable scientific insight.

Answer 11

A million trials isn't really that many. There are lots of tasks where you can collect 1000 trials an hour. With a ten person subject panel where each person works 2 hours a day (10 hours a week) you could collect a million trials in 10 weeks. Assuming you are paying 15 USD an hour you can collect all the data for 15,000 USD which is a small grant. To run a ten person panel you would ideally want 3 testing stations and an RA to run things.