I am a Master's student focusing on psycholinguistics and I am preparing for my thesis. What I will be looking at are register effects and for that I will be conducting 2 online experiments using Ibex (participants will hear context sentences, be presented with a set of visual stimuli and then react to the visual stimuli). As far as I know, online RT studies are not as reliable as those which are conducted in the lab in the sense that they are noisier and less accurate. There is a great thread which discusses this topic How valid are reaction times collected from online studies? and where it is claimed that online RTs studies can be reliable if they are well powered. All in all it makes sense but I am confused as to how many participants are too little and how many are just enough. Your help is very much appreciated!

  • 2
    $\begingroup$ Just to add that another way to increase power is through good exclusion criteria. In an experiment that I was involved in recently, we excluded participants with too-low RTs or Accuracy, and even a subset of about half the original participants still had vastly more power than the entire dataset. $\endgroup$ – Arnon Weinberg Sep 1 '20 at 19:21

There are four variables of interest in a statistical power analysis: power (often chosen as 0.8 or 0.9 by convention, this is the probability of correctly rejecting the null hypothesis if it is false), tolerated false-positive rate (alpha, often chosen as 0.05 by convention), sample size, and the effect size (which in turn depends on the variability of the observations and the difference between groups).

When you do a power analysis, you can choose any one of those variables to calculate provided you have all the others. In your case, you want to know the sample size you need so you need to choose or estimate the other numbers.

If you know what statistical test you are using, there are numerous online calculators to help you choose a sample size. Personally, I like to use the free software G*Power.

Power and alpha are typically chosen by convention, so what you still need to know to determine your necessary sample size is the effect size you are looking for. If you have an experiment that you expect to have a large impact on reaction times, you don't need as many participants. If you are looking for a subtle effect, you need more. You can choose that threshold based on what is reasonably interesting/clinically relevant (this is ideal but sometimes frustratingly difficult to settle on), or you can base it on pilot data (can be dangerous when pilots are based on small sample sizes).

If you don't have any preliminary data, the best way to estimate the variance in observations would be to look at other similar studies collecting online reaction time data.

Overall, if you want to make sure you have a sufficiently powered study, it's good to use conservative estimates. If you guess that your effect is smaller and the variability is larger than what you hope, then you have a better chance of a sufficiently powered study.

  • 1
    $\begingroup$ G*Power is great, but I find it's losing its light-weight character of late. Do you know of alternatives? $\endgroup$ – AliceD Sep 1 '20 at 18:28
  • 1
    $\begingroup$ @AliceD If not G*Power, I use R (either from the native stats library or from popular packages I trust for a particular use-case). In practice I also end up simulating power analyses a lot because there is no better alternative , e.g. for mixed effects models. Not sure if that fits the criterion of "more light-weight than G*Power" :) $\endgroup$ – Bryan Krause Sep 1 '20 at 18:46
  • 2
    $\begingroup$ @AliceD Same way you'd simulate anything else, you just have to apply the random effects to all the relevant observations. The hardest part, like in all power analyses, is choosing reasonable parameters to start with. I find the random effects to be the hardest to assume, and especially when you have correlated random effects. $\endgroup$ – Bryan Krause Sep 1 '20 at 19:00
  • 3
    $\begingroup$ CrossValidated is useful, a quick search gave me what seem like some useful results: stats.stackexchange.com/questions/187981/… stats.stackexchange.com/questions/483509/… $\endgroup$ – Bryan Krause Sep 1 '20 at 19:00
  • 2
    $\begingroup$ @AliceD Ah, this is actually the one I was looking for: stats.stackexchange.com/questions/82615/… Ben Bolker is always a good reference for LME at StackExchange. $\endgroup$ – Bryan Krause Sep 1 '20 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.