# What experiment in a simple decision task should we run to obtain one million trials?

Context: Models of cognitive processes require very large datasets to be fitted. Sadly, it is difficult for a single laboratory to achieve this alone. I propose that we work collectively and collaboratively in realizing a 1 million RT experiment. This initiative is described on my ResearchGate page, www.researchgate.net/profile/Denis_Cousineau.

Beyond model fitting, such a dataset will be beneficial for other purposes. Examples: Are parameters describing performances normally distributed across subjects? Does training effect --resulting in faster RT-- imply that the parameters are all evolving in one direction? Does Bayesian estimation more apt than Maximum Likelihood at characterizing the parameters? How can we estimate non-decision times? What is the shape of the RT distributions and how training effect alter its shape? In Bayes again, after how many trials does priors have no longer an impact on the estimates? etc.

Task Requirements: The task must be behavioral only (response times and response choices) as it is not possible to move EEG or fMRI material all around the world. The precise task used is not a critical constraint because the current models of cognition (sampling models such as LBA or Diffusion) are very flexible; they aim at predicting performance in a wide variety of situations.

The tasks proposed will be evaluated based on the following criteria criteria:

• small number of response alternatives
• few conditions
• low error rates
• stimuli that can be characterized for modeling purposes
• possibility of extended training on that task.

Possible candidate tasks could include: A redundant target detection task (Miller, 1982) or a same-different task (e.g., Bamber, 1969). These tasks are simple and accuracy is typically high (above 90% correct); they are possibly performed using some form of parallel processing; response times are generally well below 700 ms.

So in summary:

What task and experimental design would be most suitable given the above constraints?

• I like the idea, but asking for "the best" is rather opinion based, which is a reason for closure of the question. Could you explain a little bit more about what you are researching and why you (and others) want such a large amount of data? Don't cognitive models, and thus the data used, depend on the specific research questions? – Robin Kramer Jan 30 '17 at 19:02
• @RobinKramer: The current models of cognitions (sampling models such as LBA or Diffusion) are very flexible; they aim at predicting performance in a wide variety of situations. Hence, the exact task used is not a critical constraint when testing these models. For example, they can be tested on lexical decision tasks, dot motion tasks, numerosity tasks, matching tasks, etc., all tasks that are very different but make similar assumptions about the underlying processes (accumulation of evidence+thresholds). However, what is critical is the sample size; with small $n$, all models offer good fits. – Denis Cousineau Jan 30 '17 at 19:11
• As of the label "best", that would be with regard to those criteria: small number of response alternatives, few conditions, low error rates, stimuli that can be characterized for modeling purposes, possibility of extended training on that task. Hence,with these objective criteria in mind, the question is no longer opinion-based. – Denis Cousineau Jan 30 '17 at 19:13
• See the help-center: [... open-ended questions diminish the usefulness of our site and push other questions off the front page ].Go to the chat room for chatty conversations. Go to the meta site to find out whether a type of question is appropriate. – AliceD Feb 1 '17 at 9:50
• I think this is a very different kind of question. It does go against some of the rules of the site. It could be adapted further to the norms of the site. That said, Denis (as a senior academic), I would like to welcome you to the site. My preference would be to give this question some latitude, given that we are keen to engage more academics and active researchers on the site. Perhaps we could propose and make edits to the question to bring the question more in line with expectations. – Jeromy Anglim Feb 3 '17 at 1:12

I don't like the idea that the task should be low error, errors are needed for modeling choice. I don't like random dot motion because there are big individual difference making it hard to find a common calibration point over people. I prefer instead a numerosity judgement, e.g., are more of less than 50% of a square array of pixels (usually a large number) blue or orange? You can make the array big enough to get fine grained control of difficulty, and randomly choose which pixels to have in each color (or brightness) on each frame if you want to make particular strategies hard to use.

• Thank you. The proposed task has desired properties: fast responses; difficulty level easily adjustable; training effects are likely under extended practice; only two response alternatives. The experiment will begin May 1st, 2017! – Denis Cousineau Apr 7 '17 at 0:22
• @DenisCousineau Your comment sounds like a task has been chosen. What task was chosen? Is it still possible for researchers to contribute data? – matus Apr 14 '17 at 11:18
• @matus: The task chosen is a discrimination task: squares of two possible colors are shown dynamically, and the decision is to decide which of the two colors are the most common. Yes, you can indeed contribute data to this research. I plan on starting in September 2017. I will contact you at that moment. Thanks. – Denis Cousineau Apr 23 '17 at 23:45

I like this idea! I think the task should be one where the data could be used by many analytic tools. Hence, the data would not only be beneficial for one particular question, for example parameter estimations, but also for other tools such as Systems Factorial Technology (see Townsend & Nozawa, 1995), which provides a deeper understanding of the underlying cognitive sub-processes architecture, and other interesting information such as the process' capacity.

I suggest that the task at least respect a double factorial paradigm. This design is very simple: you have four conditions in a 2 x 2 design, where you vary two variables. It could be any variable, the only restriction is that you have to vary them independently one from another. In other words, each variable should reflect a cognitive sub-process. In one condition, both variables are in scenario where the sub-process should work very effectively (or optimally). In an other condition, both variables are in a scenario where the sub-process should work less effectively (or sub-optimally). And in the two remaining conditions, one variable works optimally and the other sub-optimally, and vice-versa.

As for the very details of the task, such as the stimuli that will be used and the specific variables, I do not have preferences. Letters are very manipulable, but they do carry semantic (more or less). Abstract stimuli could work. We could play with color, shapes, complexity, etc. At Psychonomics this year, they were many experiments which have use this design.

Reference for citation: Townsend, J. T. & Nozawa, G. (1995). Spatio-temporal properties of elementation perception an investigation of parallel, serial, and coactive theories. Journal of Mathematical Psychology, 39(4), 321–359. doi: 10.1006/jmps.1995.1033

• Could you provide more detailed citations, preferably full APA. That will make it easier for readers to find the papers. – Robin Kramer Feb 8 '17 at 6:40

Since one of the advantages of a large data set is high statistical power, it might be a good idea to use a task where the key effect has not been found in previous studies. That would produce a guaranteed result: i.e., a high-power test of an important null result.

A pop-out condition in visual search would be one example (i.e., no effect of display size), but this is not ideal because it would limit the modelling end of the project. That is, it wouldn't be too helpful to model multiple conditions with identical performance (or nearly so).

A better example would be an additive-factor experiment (somewhat as suggested by Marc-André Goulet). I would suggest a simple 2x2 design for which previous studies had reported theoretically important factor additivity (i.e., non-significant interaction). A big study would provide a high-power test of the null interaction. And, as long as both main effects were present, there would be four distinct conditions to model. This design is also interesting because a lot is known about conditions that should be satisfied by the RT distributions under certain models (for examples, see the references below).

A final point: After selecting a task, it might be very useful to test out the planned analyses on simulated data, which are considerably easier to get than real data. This would help make the project's precise goals more explicit and also make sure the goals would be achievable using the proposed data set. It might also help to reveal what aspects of the planned task/conditions were necessary for getting informative results (e.g., minimum effect sizes).

References

Cortese, J. M. & Dzhafarov, E. N. Empirical recovery of response time decomposition rules II. Discriminability of serial and parallel architectures. Journal of Mathematical Psychology, 1996 , 40 , 203-218

Dzhafarov, E. N. & Cortese, J. M. Empirical recovery of response time decomposition rules I. Sample-level decomposition tests. Journal of Mathematical Psychology, 1996 , 40 , 185-202

Dzhafarov, E. N. & Schweickert, R. Decompositions of response times: An almost general theory. Journal of Mathematical Psychology, 1995 , 39 , 285-314

Roberts, S. & Sternberg, S. The meaning of additive reaction-time effects: Tests of three alternatives. In Meyer, D. E. & Kornblum, S. (Eds.) Attention and performance XIV. Synergies in experimental psychology, artificial intelligence, and cognitive neuroscience., MIT Press, 1992 , 611-653

I agree with Marc-André, the selected stimuli should have no semantic association (or as little as possible) which is why I would encourage the use of geometric shapes that vary in colour. Additionally, I believe that the task should be as simple as possible in order to encourage the overall cleanliness of the data.

As you suggested, one such task is the redundant attribute target detection task (Miller, 1982). In this task, participants are asked to keep in mind target dimensions (e.g. blue and square) and as soon as one of the dimensions is presented on screen, the participants respond as quickly and as accurately as possible. Results are typically fast, accurate, and slightly skewed – appropriate for this initiative! We could select a few colours and shapes (if we do go forward with this task) as well as what colour harmony should be selected.

Additionally, having colours that vary in “obviousness” can potentially give us a clearer understanding of the parameters within certain models. For example, an obvious blue should have quicker RT than a muted blue and the rate parameter of each individual channel should reflect that (assuming, of course, that the rate parameter is associated with "obviousness" at all). I am particularly interested in this aspect.

While a go/no would be great for this task, I also strongly believe that a 2AFC task would yield much more interesting results and the 1million RT initiative could help us really understand the “race” in “race models”. This could be as simple as modifying the task as one with “target present” or “target absent” decisions. With clearer trends in RT we could hopefully gather information on the independence of dimensions and whether there is crosstalk between detectors.

Additionally, having such data would not only benefit model fitting but also more solid architecture analyses. This would, in turn, help us get a more concrete understanding of coactive architectures (the architecture that Miller, 1982 suggested was at play in this type of task) which haven’t been as rigorously defined like other architectures (as noted by Houpt & Townsend, 2011).

EDIT:

Houpt, J.W., & Townsend, J.T. (2011). An extension of SIC predictions to the Wiener coactive model. Journal of Mathematical Psychology, 55, 267-270

Miller J. (1982). Divided attention: Evidence for coactivation with redundant signals. Cognitive Psychology, 14,247–279

• Could you provide more detailed citations, preferably full APA. That will make it easier for readers to find the papers. – Robin Kramer Feb 8 '17 at 6:39

I myself am interested in obtaining such a dataset on the Same-Different task (Bamber, 1969). The answer is binary (same or different), accuracy is very high and there is a learning effect (short term, at least). Concerning the number of conditions, we could simplify the original task to go from 14 conditions to 8 by either:

• limiting the number of letters to 4 (instead of 1, 2, 3 and 4)
• using 1, 2 or 4 letters stimuli, and having 1 or 2 mismatches for the 2 letters stimuli, and 2 or 4 mismatches for the 4 letters stimuli

The second option is closest to the original. The first, however, would be very informative on the 4 letters condition, as there is little data on them due to the experiment's design.

• Could you provide more detailed citations, preferably full APA. That will make it easier for readers to find the papers. – Robin Kramer Feb 8 '17 at 6:40

Given many individuals (mostly men) have anomalies of colour perception, you might avoid colour; you also want a simple task with 'easy' stimuli, to enable short trial durations. Visual search paradigms are good for this. I echo the call for a project in which the experimental design would address something of particular interest, and ideally that would be difficult to address with a 'standard' design (e.g., with 10,000 data points, rather than 1,000,000). Estimating the distribution of processing times for a process inferred from indirect measures may be an interesting problem domain.

PJ

• Could you please add references to your statements. At CogSci aim to have well researched and well referenced questions and answers. – Robin Kramer Feb 8 '17 at 6:36

I vote for a simple random dot motion task. This has the advantages that: 1. It's easy to program. It can be delivered over the web via javascript, which makes the "many labs" aspect easy. 2. It's very well studied and well understood. 3. It is easy to set 5-ish coherence levels that will sweep out almost the full range of accuracy, from chance level to 100%. This is important for distinguishing between the models.

• Please add some references to your answer, with examples of or the development of this task. CogSci is a scientific stack, not a discussion board. – Robin Kramer Feb 8 '17 at 6:35

May I ask, have you thought about using existing large datasets, such as those provided by Project Implicit? Dataset.

(sadly, it seems Project Implicit has not stored individual data points but rather a computed 'effect' per participant).

• Thanks @andyw for your input. However, we really need raw data. Many models are best fit using likelihood methods. Also, some effects are visible at the extreme tails of the distributions so that summary statistics (mean, standard deviation, even skewness) are not enough information. – Denis Cousineau Feb 13 '17 at 12:39
• Just a followup. It is likely we run a 10 country, 800 participant per country replication of the gender/career IAT run in project implicit. We would save ALL the data. Perhaps 200 trials per participant (1.6 million RT scores). Of any use? uwahomewardbound.wordpress.com/2016/01/20/… – andyw May 11 '17 at 7:52

May I suggest collecting data online?

With the latest browser technologies, RTs are extremely accurate, in the order of 5 microseconds (discussion). Even with slightly older browser tech, collected RTs can be accurate (ref).

There are several sites where people freely volunteer to do research (discussion).

Collecting data over multiple sites is most certainly easier when there is setup time per location.

• This does not answer the OP's question. He is asking for RT tasks/paradigms. Although the info is interesting, it should have been posted as a comment, or in a new question altogether ( e.g. cogsci.stackexchange.com/q/9584/11318 ) – Robin Kramer Feb 13 '17 at 12:39

A million trials isn't really that many. There are lots of tasks where you can collect 1000 trials an hour. With a ten person subject panel where each person works 2 hours a day (10 hours a week) you could collect a million trials in 10 weeks. Assuming you are paying 15 USD an hour you can collect all the data for 15,000 USD which is a small grant. To run a ten person panel you would ideally want 3 testing stations and an RA to run things.

• You do not provide an answer to the question. It is asking for RT tasks/paradigms. – Denis Cousineau Feb 14 '17 at 2:00
• @DenisCousineau my answer is that your premise that collecting a million trials is difficult is wrong. It does not really qualify as a large N study that needs additional scrutiny or to be crowd sourced. – StrongBad Feb 14 '17 at 2:13
• @StrongBad such a remark would probably best fit as a comment on the question. Answers are reserved to actually answering the questions with statements based on scientific literature. – Robin Kramer Feb 14 '17 at 7:36