4
$\begingroup$

When I present two signals in paired comparison and ask the respondent to select the stronger signal in the pair, I find that it is the signal strength of the second signal that more so determines correct detection (0,1) yet the correlation between probability of correct detection (p) and both signals is significant and negative. Yes, the contrast contributes to the prediction of (p) but I cant figure out why my correlations are negative. It is as if the respondent is anchoring on the first signal and the "smallness" of the second signal determines (p). Does anyone have an explanation for this?

Experiment backgrounds

  • The stimulus is paired comparison words with each having a different factor loading based on it's measurement of agreeableness. So each word taps the construct more or less deeply than the other;
  • The contrast between the words in terms of their factor loading is the signal;
  • A set of 122 such paired comparison tasks was developed and the task before the respondent is to select the more agreeable word.
  • The stronger loading was alternated between being on the first and second word. I did find that the contrast predicted probability of correct detection (r = .20*). I know this is an unusual application of signal detection theory and you might not has seen this before.
Improve this question
$\endgroup$
2
  • $\begingroup$ Interesting problem. I can't answer this easily, the more since I'm only familiar with simple yes/no tasks. However, I'm in the auditory field doing nothing but speech understandings tasks :) So I like this question. $\endgroup$
    – AliceD ♦
    Commented Jun 16, 2017 at 18:42
  • 1
    $\begingroup$ I would advise to perhaps elaborate on key terms like 'factor loading based on it's measurement of agreeableness'. I've added some tags to attract the attention of the other psychophysics folks hanging around here. $\endgroup$
    – AliceD ♦
    Commented Jun 16, 2017 at 18:48

1 Answer 1

Reset to default
2
$\begingroup$

I find that it is the signal strength of the second signal that more so determines correct detection (0,1)

The ideal unbiased observer bases the decision on $X1-X2$ while it sounds like your subjects are basing their decision on $aX1-X2$ where $a < 1$. While it is suboptimal, it is not a big deal. The fact that you know it means either report it or train the subjects out of it.

the correlation between probability of correct detection (p) and both signals is significant and negative.

If the first stimulus is very weak, the difference between it and the second stimulus will on average be larger than if the first stimulus is of moderate strength. One would therefore expect Pc to initially decrease as the strength of the first stimulus decreases. Additionally, if the first stimulus is very strong, the difference between it and the second stimulus will again be larger on average than if the first stimulus is of moderate strength. Therefore, for an ideal observer, the dependency of Pc of the strength of the first signal should be non-monotonic. The same holds for the second stimuli.

We know from the above statement that your subjects are not ideal. If the bias depends on the strength of the first signal (i.e., $a(X2)$ increases with X2) then the non-monotonicity of Pc on the strength of the stimuli could be hidden.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Thank you for elucidating. I am not a signal detection theorist. The respondents are students recruited for the research having varying degrees of sensitivity. Each word in the pair only has a poor estimate (though the best we have a priori) of signal and the contrast should be the salient feature in the task. Would you have any guidance for enhancing subjects' signal sensitivity estimation? Thanks again! $\endgroup$
    – J. Peter Leeds
    Commented Jun 16, 2017 at 22:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.