I want to estimate the properties of a response time distribution (mean, variance, tail-length) for individual subjects.
How many samples (trials) do I need to collect to get reliable estimates?
Does it depend on the method of estimation (i.e. maximum likelihood versus hierarchical Bayesian)?
The number of samples that are necessary for a good parameter estimation does indeed depend on the estimation method. I am not aware of a simple rule of thumb to determine an optimal sample size, but there has been a lot of literature on this topic. A paper that might be a good starting point for a literature search is
Van Zandt T. (2000) How to fit a response time distribution. Psychonomic Bulletin & Review 7(3): 424-465. [Link]
Disclaimer: I'm not generally doing experiments where reaction time is the primary DV. But I thought I'd look at this issue and explored RTs from a neuroimaging dataset, and I think the findings are relevant to the question.
I think without further qualification, this question doesn't have an answer. Here I've plotted the estimation of reaction time/RT over the time course of an experiment (248 trials total), based on the RTs collected up to each time point, for a single subject:
For each point, the CI is the CI of the mean for all trials up to the respective trial number; so at x = 10, it's the CI for the mean based on the first 10 trials. Incremental mean is the mean over all trials up to this point; it is not the mean of means, but the mean of RTs. The total mean is the mean over all trials collected in this experiment.
I'm ignoring that the data is probably not normal for now.
As you can see, the width of the confidence interval of the mean asymptotes fairly quickly and doesn't get much better after 100 or so trials. However, you also see that the "incremental mean" (the mean of all trials up to this point) has not yet reached the true ("total") mean yet. This is likely because of a significant practice effect; the longer the experiment goes on, the faster the subject becomes. So while the precision of the estimate does not improve much after the 100th trial, the CI doesn't actually contain the "true" value- because the "true" value is non-stationary!
So maybe 200, 250 trials is not enough!? But I don't think the mean would have become stationary at a certain time point/trial count. Rather, eventually, fatigue would have overtaken the practice effect and RTs would have started rising again. d' would probably worsen, and variance rise, etc.
Metaphorically, we could call this Heisenberg's Uncertainty Principle in experimental psychology, in that we change what we measure by the process of measurement.
Now as you can see, this experiment is fairly special in that RTs are quite high - around 1500 msec. This is because my paradigm is comparatively complicated. Maybe practice would matter less, and the estimates asymptote more quickly, and more stable, in a different paradigm; for example, a very simple low-level signal detection experiment. This is why I'm saying there is no straight-forward answer so far, as the specific behavior of RT depends on the paradigm.
Maybe you could try to account for the time effect - for example, with a regression with time as a predictor and estimation of the subject-specific intercept for RT independent of the time term. But then, that's "subject-specific RT intercept after removal of covariates" or something like that, not "reaction time mean".
