3
$\begingroup$

I have accuracy data, where every subject gives a correct, i.e. 1 or wrong, i.e. 0 answer. However every subject performs 190 trials in each session, with 10 sessions in total.

To analyze the data I use R, and group the results by session, i.e. more simply put every subject responds 190 times in each session, however after grouping, it gets only one value for each session (I use the group_by function of package dplyr), based on the average of all its trials in that session (could be 1.00 if all the answers are correct or 0.95, 0.85 etc. if the subject has mistakes).

My reasoning is that in this way the data is no longer categorical, but quantitative, therefore I was hoping to use repeated-measures ANOVA for the anlysis. However the normality tests that I did on the data showed to be signifcant and a graphical representation of the data also shows that is very strongly skewed to the right (wich is completly logical of course, since there are much more correct answers than wrong).

My question is: What can I use to analyze this data and what is used in the scientific community in general in those situations, since it is not an uncommon problem, accuracy being often measured? I checked some articles, but there for accuracy researchers simply report percentage differences, and state that they are (or aren't) significant. I know that I could use Friedman's ANOVA, but it isn't that powerfull so I'm looking for another way. Also researchers often report using ANOVA for accuracy scores, which is leaving me absolutely stunned... Maybe there are some errors in my reasoning, so all comments are welcome!!

Thank you for the answers!!!

Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

ANOVA and linear regression are inappropriate when your outcome is a percentage of correct trials, especially so if numbers are near the boundaries 0% or 100%.

Mixed effects logistic regression may be appropriate.

More detailed answers would require more details about your experiment.

Improve this answer
$\endgroup$
2
$\begingroup$

Based on the percentage of published papers that I read, and those that I review, that use un-transformed accuracy scores, I am going to guess that is what most cogsci people do. When your accuracy is around 0.5, maybe you can get away with this, but at 0.95, I believe it is a total no go. Of the people who transform, I think they typically either convert to d' with a z-transform (inverse of the cumulative distribution of a Gaussian) or use an arcsine-square-root transformation. I am partial to the rationalized arcsine transform proposed by (Studebaker 1985) or even better using a logit mixed model (Jaeger 2008).

The reason that some people think you can get away without transforming when the accuracy is around 0.5 but not around 0.9 is that the transformations (arcsine and logit) are essentially linear near 0.5 but not near 0.95.

Apart from ease of use and historical precedent, there really isn't a good reason not to use a logit mixed model for all statistics involving accuracy data.

Improve this answer
$\endgroup$
14
  • 1
    $\begingroup$ @BryanKrause apart from ease of use and history, I don't think so. That said, using what your reviewers expect can be helpful. I still plot things in RAU during exploratory analyses. $\endgroup$
    – StrongBad
    May 1, 2019 at 17:36
  • 2
    $\begingroup$ @BryanKrause that is how I run my power analysis.... The power analysis said we only needed 12 per group, but the reviewers will scream if we don't have at least 16, so we need funding to run 16. $\endgroup$
    – StrongBad
    May 1, 2019 at 17:54
  • 1
    $\begingroup$ Zscore and ztransform are different. en.m.wikipedia.org/wiki/Fisher_transformation @BLambrev although I'm not sure what exactly StrongBad was referring to. $\endgroup$
    – Bryan Krause ♦
    May 2, 2019 at 13:46
  • 2
    $\begingroup$ @BryanKrause I was using it to mean the inverse of the cumulative distribution of a Gaussian (aka norminv) which is usually denoted as Z(.) en.m.wikipedia.org/wiki/Sensitivity_index I agree the notation was confusing and thought d' would be enough. Will wed it to clarify. $\endgroup$
    – StrongBad
    May 2, 2019 at 13:55
  • 1
    $\begingroup$ Ah that makes more sense in the d' context. I think I was misled by the Fisher transform being similar to the other reshape you suggested. We really need to stop overusing Z everywhere. Z transform has at least one other meaning too en.m.wikipedia.org/wiki/Z-transform $\endgroup$
    – Bryan Krause ♦
    May 2, 2019 at 13:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.