# How to confirm that a more frequently labelled object is a better instance of the label

I have some data from a study where humans are shown an object and asked to select all words from a list that apply to the object. What I am trying to do is use the data to confirm that a particular object, $$o_{1}$$, is a better fit of a particular word, $$w$$ , than another object, $$o_{2}$$. I am trying to use hypothesis testing to confirm this, with a null hypothesis that $$o_{2}$$ is an equivalent or better fit than $$o_{1}$$ i.e. the probability of labelling $$o_{2}$$ with $$w$$ is greater than or equal to the probability of labelling $$o_{1}$$ with $$w$$.

There are $$n_{1}$$ annotations for $$o_{1}$$, in which the object is labelled $$x_{1}$$ times with $$w$$. Similarly $$o_{2}$$ is labelled $$x_{2}$$ times in $$n_{2}$$ annotations.

To calculate the p-value I am working out the probability of observing the difference in the proportion of times $$o_{1}$$ and $$o_{2}$$ are labelled with $$w$$, $$P(x_{1}/n_{1}-x_{2}/n_{2}\geq d)$$, where d is the observed difference. This gives quite a long function, which relies on the actual underlying probability, $$p_{1}$$, of labelling $$o_{1}$$ with $$w$$, so now I suppose I have to find $$p_{1}$$ which maximises this function?

I feel uncertain that I'm tackling this in the right way, does this seem correct?

You could possibly use a two-proportion z-test to test against the null hypothesis that $$(x_{1}/n_{1}-x_{2}/n_{2})==0$$, though there may be better methods that consider the overall distribution of words chosen and accounting for differences in raters in terms of how many words they choose (your situation does not fit the assumptions of the test but may be 'close enough' for certain purposes).

• Thanks! This looks along the right lines for what I want but I don't think I can fulfil the sampling assumptions with the data I have, as described here. I think I need a way to calculate the p-value more directly – A. Bollans Apr 26 at 13:34
• Aha I think I need to use Fisher's exact test, in the example replacing 'Men' and 'Women' with $o1$ and $o2$ and 'studying' and 'non-studying' to 'labelled' and 'not labelled'. – A. Bollans Apr 26 at 14:12

Fishers's exact test exists for this purpose which calculates the p-value from a contingency table. In this case the null hypothesis is that $$o_{1}$$ and $$o_{2}$$ are equally likely to be labelled with the word. As noted by @Bryan it may also be possible to use a two-proportion z-test, but this would require a larger dataset.