Quantify degree to which non-diagnostic features bias category-present response

Question

I need a measure of the degree to which each of several features biases participants to respond "yes" in a category present / absent task for each of several categories.

I have stimuli defined along 3 binary feature dimensions, let's call them a, b, and c. Each feature is perfectly diagnostic for 1 of 3 categories, let's call them x, y, and z, and perfectly non-diagnostic for the other 2. In other words, for example, a stimulus belongs to category x if it has feature a and not if it doesn't, and it is equally likely to belong to category y or not, and to category z or not, both when feature a is present and when it is absent. Participants are shown stimuli and asked whether or not they belong to a given category. In a given block of the experiment, they are only asked about 1 of the 3 categories.

So going back to my question, if the feature is the one diagnostic for the category, then correct response would be "yes" 100% of the time the feature is present and "no" 100% of the time the feature is absent. If the feature is one of the two irrelevant ones, then "yes" will be the correct response 50% of the time regardless of feature presence. In fact it turns out that people respond "yes" more than 50% of the time when the irrelevant features are present, and less than 50% of the time when they are absent, indicating that the irrelevant features bias them to respond "yes".

What I need is a way to quantify that effect. Things I've considered. (1) Subtract the positive response rate for feature absent from the positive response rate for feature present. 0% would indicate no bias for irrelevant features. (2) Divide the positive response rate for feature present by the positive response rate for feature absent. 1.0 would indicate no bias for irrelevant features. (3) Combining feature present and absent, divide the positive response rate by the correct response rate (i.e. the response rate of the perfect responder). 1.0 would indicate no bias for irrelevant features.

I am wondering whether there is a good "standard" measure for this kind of situation or, failing that, whether anyone can think of a good reason to prefer any one of the options above, or another one I haven't thought of.

I'm not sure whether or not signal detection theory is relevant here. My sense is that it's not, because I'm not looking for general response bias but rather the bias induced by specific features, and also because I have binary rather than continuous features, but I'm willing to be persuaded otherwise - maybe I just don't understand SDT well enough.

Oh, one other thing in case it matters - the stimuli in question are graphs and tables of data from 2x2 experimental designs, while the features / categories are presence / absence of main effects of each variable and the interaction.

Answer 1

I've been going through your question a couple of times now and I find it quite tricky to get all the details. So please correct me, if I misunderstood you in some point.

Basically you have a test and your subjects have to determine if a stimulus belongs to a certain category or not. Of the three possible features, one is a perfect indicator for a specific category, while the other two are unrelated to it. So far, so good. But I think that there are a couple of problems with the way that you framed the problem.

Correct Response vs. Expected Value

You say that,

if the feature is the one diagnostic for the category, then correct response would be "yes" 100% of the time the feature is present and "no" 100% of the time the feature is absent.

Now, I think that there are three things mixed together in this statement:

• the correct response,
• a certain response rate whith which the correct response might be given and
• the expected value of category membership given the perfect indicator.

These may seem like a subtle differences, but I would argue they are not. While you can determine, what the correct response to a question is, there is not really a correct response rate. When a subject answers a question correctly in 95% of the cases, he gave the wrong answer a couple of times. But the response rate is not right or wrong, it's just what it is. And the response rate does not have to equal the expected value (which is 100% in this case). Jeromy Anglim already commented somehting along these lines as well. This becomes even clearer for the next statement:

If the feature is one of the two irrelevant ones, then "yes" will be the correct response 50% of the time regardless of feature presence.

Again, the expected value of category membership given an unrelated indicator is 50%. And since we know about that, we can formulate the hypothesis, that the subjects will catch on to it. But it's just a hypothesis and it cannot be taken for granted.

Is it really a bias?

Now, administering the test, you have found that

In fact it turns out that people respond "yes" more than 50% of the time when the irrelevant features are present, and less than 50% of the time when they are absent, indicating that the irrelevant features bias them to respond "yes".

I'm not so sure that this really is a bias, at least not one that is caused by the features. As Jeormy Anglim mentioned in the comments, the subjects might come up with all sorts of strategies. A tendency to answer with "yes" might also be some sort of aquiescence bias (Ray, 1983). To be sure, that would be a bias, too, but it would not be caused by the features. But since it is not clear, how the subjects should respond to the test, it does not make sense to speak of a bias.

Quantifying the results

Having said all that, you could still quantify the results. A good way to do that would be to look at the conditional probabilities of a correct response, given a certain feature (which is what you already did). E.g. $$P(Y_x=1|a)$$ would denote the probability of getting a "yes" for an item $Y$ that asks about category $x$ when feature $a$ is present in the stimulus. You could do this for all combinations of categories and features. This would also allow for testing hypothesis about certain response rates like: Is the response rate for a certain combination significantly different from some value $P_0$?

The other suggestions that you make would work, as well. E.g. taking the difference between the conditional probabilities (like in your first suggestion. A result of zero indicates that there is no difference between the proportions.

If you are interested in pairwise comparisons you might take a look at the odds ratio (Agresti, 2007). The odds ratio considering features $a$ and $b$ for a given category would be defined as $$OR = \frac{P(Y=1|a)/(1 - P(Y=1|a))}{P(Y=1|b/(1-P(Y=1|b)}$$ which is the same as $$OR = \frac{odds(a)}{odds(b)}.$$

References:

_{Agresti, A. (2007). An introduction to categorical data analysis (Vol. 423). Wiley-Interscience.
Ray, J. J. (1983). Reviving the problem of acquiescent response bias. The Journal of Social Psychology, 121(1), 81-96.}