# Combining unbiased hit rates

I am analysing data on an emotion perception task, where participants must decide if a given face is happy, sad, angry or fearful (essentially a forced choice between 4 options).

I am confident that I want to calculate an unbiased hit rate for each emotion (as in Wagner 1993). This takes a confusion matrix (toy example below):

        Happy    Sad    Angry   Fearful
Happy     4       1       0        1
Angry     0       0       6        0
Fearful   2       0       0        4


And the unbiased hit rate is then the correct answers squared, divided by the number of times that answer is used multiplied by the number of trials where that answer was correct. So in the above example:

Hu(Happy) = 4^2 / ((4+0+0+2)*6) = 16/36 = 0.444
Hu(Angry) = 36/48 = 0.75
Hu(Fearful) = 16/36 = 0.444


These are then converted with an arcsine tranformation on the square root for use as the dependent variable in regression.

However, I want to use this data to assess general ability at emotion recognition in faces - what is the correct way to combine the unbiased hit rates? That is, can I generate a single value that accounts for all off these, or should I run four separate analyses and examine the overlap in the results?

Reference

Wagner, H. L. (1993). On measuring performance in category judgment studies of nonverbal behavior. Journal of Nonverbal Behavior, 17(1), 3-28.

The purpose of unbiased hit rate is to avoid invalid conclusions in cases where subjects indiscriminately use one (or only few) response options. To give an example (responses in rows, stimuli in columns):

        Happy    Sad    Angry   Fearful
Happy     1       0       0        1
Angry     0       0       0        0
Fearful   2       0       0        1


Consider sad faces. The uncorrected and biased hit rate for sad faces is $h_b(\mathrm{Sad})=8/8=1$. The unbiased hit rate is $h_b(\mathrm{Sad})=8^2/(8 \cdot24)=1/3$. The latter estimate better corresponds to our intuition. The subject responds almost always $\mathrm{Sad}$ and should be assigned a score slightly better than a random performance ($=1/4$).

You are looking for a total score. By definition a total score will account for this kind of bias as it uses all elements of the confusion matrix. Hence, there is no point in looking for a unbiased total hit rate and there is no point in trying to combine the unbiased hit rates.

Wagner (1993) makes it clear on page 4 that the question regarding a total score is distinct and different from the one he is trying to answer with his unbiased hit rate. In a footnote 1 on pages 27-28 he mentions methods for computing the total score:

There are numerous ways of examining overall performance in judgment studies. In psychophysics, the confusion matrix resulting from the method of absolute identification (as the type of judgment study discussed here is known) has been examined with the tools of information theory (see Baird & Noma, 1978, for a clear exposition of this). [...] Other examples of measures of accuracy for the whole matrix are Cohen's $\kappa$ (Cohen, 1960), and Rosenthal and Rubin's $\pi$ (Rosenthal & Rubin, 1989).

Literature

Baird, J. C., & Noma, E. J. (1978). Fundamentals of scaling and psychophysics. John Wiley & Sons.

Cohen, J. (1960). A coefficient of agreement for nominal scales. Educational and psychological measurement, 20(1), 37-46.

Rosenthal, R., & Rubin, D. B. (1989). Effect size estimation for one-sample multiple-choice-type data: Design, analysis, and meta-analysis. Psychological Bulletin, 106(2), 332.

• Thank you - I am also interested in the individual scores, but this provides an alternative way to examine the overall effect that should be valuable! – Joni Jun 30 '15 at 17:21