How valid are reaction times collected from online studies?

Question

Online studies promise the possibility of greatly increased numbers of and variability in populations to study, but there are many potential concerns and need for validation, and diving in head first seems imprudent.

Here, I am interested in the ability to collect response time data online compared to a standard computer setup (e.g., a PsychToolbox or E-Prime based study with keyboard input, not a response box) in an on-site experiment booth. While these studies have their own limitations, such as being insuitable for experiments where very high fidelity RT data is critical to the study, I am merely interested in whether online studies differ significantly from them.

• Are online RTs different from on-site RTs within tasks?
• Is the theoretical floor on online RTs in general different from that of on-site RTs?

Answer 1

Short answer: The data is likely to be noisier, the absolute reaction time can't be trusted, but given enough power (which is easy to obtain on the Internet) relative reaction time differences should be similar to those in the lab. However, web-based reaction time studies might pose other problems, because you have less control over stimulus presentation and about how participants behave.

Long answer: There is some research that has looked at Internet-based collection of reaction time data using different software approaches. The number of papers is small, but they converge in the conclusion that there will be more noise, but that it can be quite useful depending on the specific research question.

The effect of additional noise

Some noise stems from the fact that hardware and software is widely different "in the wild". For example, using a JAVA - applet Eichstaedt (2001) has shown much variation in reaction times depending on different PCs. Some of this variation between computers is based on factors that add some constant to the reaction time on a specific machine. These constants don't matter if you do within-subjects reaction time comparisons as they are common in cognitive paradigms. Other factors will add random noise. For example, some keyboards only transmit responses with some frequency (e.g. every 20 ms.). Thus, the timing resolution will be bound to this limit. Also, other software running in the background may result in random noise. Nevertheless, given enough trials and enough participants this random noise may be a manageable nuisance.

In fact, using simulations, Brand and Bradley (2012) have found that adding a random 10 to 100 ms delay to response times reduced statistical power only by 1-4% across a range of different effect sizes.

Research that has compared response times collected with online and lab-technologies suggests similar conclusions. For example, using the Flash-based ScriptingRT Schubert et al. (2013, Study 1) have shown that

the SDs of [reaction times] stayed below 7 ms in all three browsers. That value is comparable to many regular keyboards and standard reaction time software. In addition, the constant added by measuring in ScriptingRT was about 60 ms. This result suggests that researchers using ScriptingRT should thus focus primarily on differences between RTs and be cautious when interpreting absolute latencies.

From Study 2:

ScriptingRT resulted in both longer response latencies and a larger standard deviation than all other packages except SuperLab and E-Prime in one configuration. Nevertheless, in absolute terms, the SD of 4.21 is comparable to what was standard for keyboards for a long time [16]. It is thus clear that any test with ScriptingRT should be well powered and used to assess primarily paradigms with a large effect size.

Similarly, comparing JavaScript and Flash-based data collection Reimers and Stewart (2014) concluded that, in general,

within-system reliability was very good for both Flash and HTML5—standard deviations in measured response times and stimulus presentation durations were generally less than 10 ms. External validity was less impressive, with overestimations of response times of between 30 and 100 ms, depending on system. The effect of browser was generally small and nonsystematic, although presentation durations with HTML5 and Internet Explorer tended to be longer than in other conditions. Similarly, stimulus duration and actual response time were relatively unimportant—actual response times of 150-, 300-, and 600-ms response times gave similar overestimations.

Replications of cognitive paradigms with online samples

Several papers have used online data-collection to replicate well known effects stemming from lab-based research.

For example, Schubert et al. (2013) replicated the Stroop-Effect with online-vs. lab technology and found that the size of the effect was independent from the Software used. Using JAVA, Keller et al. (2009) replicate a the results of a self-paced reading paradigm from the psycho-linguistic literature. The most comprehensive replication project has been published by Crump et al. (2013) who replicate Stroop, Switching, Flanker, Simon, Posner Cuing, attentional blink, subliminal priming, and category learning tasks on Amazon's Mechanical Turk.

Other challenges and limitations

There are several other challenges and limitations associated with online response time collection

• A different question is the accuracy with which stimuli can presented online. There will be limits to time resolution (see, e.g., Garaizar et al. 2014, Reimers & Stewart, 2014, Schubert et al., 2013) and visual differences (color and resolution) depending on hardware and environmental light
• Often online samples will be more diverse with regards to age and education, some may have difficulties understanding difficult instructions. Also, in an online study it is easier to abandon boring RT-tasks than in the lab (Crump et al., 2013)
• Participants' hardware may be confounded with other variables, thus that there might be confounds in the absolute reaction times because a systematic RT constant may added to certain demographic groups. This is not a problem for reaction time differences within participants. However, correlations of absolute reaction times with personality variables may be spurious (as warned by Reimers and Stewart (2014)

References

Brand and Bradley (2012). Assessing the Effects of Technical Variance on the Statistical Outcomes of Web Experiments Measuring Response Times. Social Science Computer Review, 30, 350–357. doi:10.1177/0894439311415604

Crump, M. J. C., McDonnell, J. V., & Gureckis, T. M. (2013). Evaluating Amazon’s Mechanical Turk as a Tool for Experimental Behavioral Research. PLoS ONE, 8, e57410. doi:10.1371/journal.pone.0057410

Eichstaedt, J. (2001). An inaccurate-timing filter for reaction time measurement by JAVA applets implementing Internet-based experiments. Behavior Research Methods, Instruments, & Computers, 33, 179–186. doi:10.3758/BF03195364

Garaizar, P., Vadillo, M. A., & López-de-Ipiña, D. (2014). Presentation Accuracy of the Web Revisited: Animation Methods in the HTML5 Era. PLoS ONE, 9, e109812. doi:10.1371/journal.pone.0109812

Keller, F., Gunasekharan, S., Mayo, N., & Corley, M. (2009). Timing accuracy of Web experiments: A case study using the WebExp software package. Behavior Research Methods, 41, 1–12. doi:10.3758/BRM.41.1.12

Reimers, S., & Stewart, N. (2014). Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments. Behavior Research Methods, 1–19. doi:10.3758/s13428-014-0471-1

Schubert, T.W., Murteira, C., Collins, E.C., Lopes, D. (2013). ScriptingRT: A software library for collecting response latencies in online studies of cognition. PLoS ONE 8: e67769. doi:10.1371/journal.pone.0067769

Answer 2

There are a few factors that could contribute to differences between online versus in-lab reaction time measurement.

Hardware variation

Participants in an online experiment will use their own computers to complete the task, which will result in lots of variation in hardware. Many studies have looked at how hardware variations affect response time measurement, and generally find that hardware variations can cause differences in the range of 10-100ms for a single response (e.g. Plant & Turner, 2009).

Software variation

Online studies and lab studies tend to be conducted with different software, since most of the standard lab-based software can't be used to make an online experiment. A popular option for online experiments is JavaScript & HTML. Reimers & Stewart (2014) measured the error in JavaScript response time measurements, and generally found it to be around 25ms, with some variation across different hardware and software. de Leeuw & Motz (2015) ran an experiment in which subjects completed a visual search task in the lab using both a JavaScript and MATLAB (Psychophysics Toolbox) version of the experiment, and found that JavaScript measured response times that were about 25ms slower. However, both JavaScript and MATLAB had equivalent variance in the measurements and both software systems were equally sensitive to experimental manipulations of the visual search task at the sample sizes used for the experiment.

Online versus in the lab

Hilbig (in press) randomly assigned participants to complete an experiment in the lab using standard lab software (E-prime), in the lab using a web browser, or online at a location of the participant's choice. They measured response times in a standard lexical decision task, and found that there were no significant differences among the three groups. The effect was on the order of 120-220ms. The effect is relatively large (d' ~ 1.5), but given the current literature there is no reason to doubt that it would.

Does it matter?

The last part of the answer is: does it really matter if response times collected online are noisier than those collected in the lab? It turns out that even for pretty noisy measurements, moderate sample sizes will counteract the additional noise of the measurement. Reimers & Stewart (2014) simulated the sample size necessary to find an effect of 50ms with and without the additional noise in response times caused by using online methods. They found that only about 10% more subjects were needed in their model to have an equivalent likelihood of detecting the effect. Ulrich & Giray (1989) came to a similar conclusion in a different modeling context.

References

• de Leeuw, J. R., & Motz, B. A. (2015). Psychophysics in a Web browser? Comparing response times collected with JavaScript and Psychophysics Toolbox in a visual search task. Behavior Research Methods. doi:10.3758/s13428-015-0567-2
• Hilbig, B. E. (in press). Reaction time effects in lab- versus web-based research: Experimental evidence. Behavior Research Methods. doi:10.3758/s13428-015-0678-9
• Plant, R., & Turner, G. (2009). Millisecond precision psychological research in a world of commodity computers: new hardware, new problems? Behavior Research Methods, 41(3), 598-614.
• Reimers, S., & Stewart, N. (2014). Presentation and response timing accuracy in Adobe Flash and HTML5/JavaScript Web experiments. Behavior Research Methods
• Ulrich, R., & Giray, M. (1989). Time resolution of clocks: Effects on reaction time measurement - Good news for bad clocks. British Journal of Mathematical and Statistical Psychology, 42, 1-12.

Answer 3

We cover a discussion on this in an article we've submitted for peer-review. Here is the preprint.

I will cite this stackExchange question/answers in the manuscript (post peer review now) as there are some lovely discussions going on, and doubtless, more to follow.

Tangentially relevant to this discussion is a simulation we did in the paper exploring how not knowing screen refresh impacts on stimulus timing (consider that if RT starts being recorded from a stimulus presentation, error in stimulus timing mucks around with RT):

We tested this appearance issue in a simulation where we varied the duration of visual stimulus, starting at a random time during the refresh cycle (10,000 virtual presentations per stimulus duration). Figure 5 shows the likelihood of short duration stimuli being shown at all, or being shown for the wrong duration, or starting / stopping at the wrong time (https://github.com/andytwoods/refreshSimulation; available to run / tweak online here http://jsfiddle.net/andytwoods/0f56hmaf/).

Below is the abstract:

This article provides an overview of the literature on the use of internet-based testing to address questions in perception research. Internet-based testing has several advantages over in-lab research, including the ability to reach a relatively broad set of participants and to quickly and inexpensively collect large amounts of empirical data. In many cases, the quality of online data appears to match that collected in laboratory research. Generally speaking, online participants tend to be more representative of the population at large than laboratory based participants. There are, though, some important caveats, when it comes to collecting data online. It is obviously much more difficult to control the exact parameters of stimulus presentation (such as display characteristics) in online research. There are also some thorny ethical considerations that need to be considered by experimenters. Strengths and weaknesses of the online approach, relative to others, are highlighted, and recommendations made for those researchers who might be thinking about conducting their own studies using this increasingly-popular approach to research in the psychological sciences.

Answer 4

Depending on how you collect the data, reaction times collected "online" will likely be different from those collected "on-site". When considering reaction times, it is important to decide if the reaction time is being used as a trigger, as the time to a response, or the difference in the time to response.

Consider an experiment which displays a random series of images for a 1/2 second each and the analysis consists of averaging the images that resulted in key presses. If your online system introduces a 1 s delay, you will not be averaging the images that led to the keypress, but the random image after it.

Consider an experiment which displays a random series of images for a 1/2 second each and every once in a while a target image is displayed and the analysis focuses on average amount of time it takes to react to the target image. In this case your reaction time will be 1 second longer than it should be and your data will be effectively meaningless.

Consider an experiment which displays a random series of images for a 1/2 second each and every once in a while one of two target images is displayed and the analysis focuses on difference in the average amount of time it takes to react to the target image. In this case the reaction time to each target will be 1 second longer than it should be, but the difference in reaction will be accurate. If in this experiment, in addition to the 1 s delay, there is also a variable delay (e.g., a Gaussian jitter with mean 0 and variance 1 s). This jitter will add noise to the data and make seeing small difference difficult. This noise, however, will average out across the trials and participants.

As with most psychological measures, the experimenter can trade fidelity of the measurement, number of measurements on each participant, and number of participants against each other. Online studies give up fidelity, and to an extent number of measurements on each participant, for large numbers of participants.

In a lab with a dedicated button box on fast dedicated hardware response latencies of under 1 ms are possible with tiny jitters. This of course ignores the human subject making the response. Wagenmakers et al (2005) suggest jitter of about 100 ms in the best case. Adding keyboard and network jitter of 100 ms (which would be pretty bad) means the online experiment would need 2x increase in the number of subjects to have the same statistical power; if the variance is twice as big (which happens when you add two independent sources of noise with equal variance), you need an N 2 times bigger to have the same standard error of the mean.