In a forced-choice task, what proportion of responses is above chance level?

Question

If I give a task in which there are only two response-options, and I want to categorise the participant according to their responses, what is the 'cutoff' or required proportion of one type of response?

E.g.

50-50 would be chance level performance: 50% Type A responses; 50% Type B responses.

100-0 would be a perfect Type A performance: 100% Type A responses.

Presumably a significant number of Type A responses (say, probably around 65%) would qualify as 'Type A' as opposed to 'chance performance'. But how should I decide on what that number is - what is a justifiable criterion and how do I justify using it?

Answer 1

This is a binomial distribution and the answer depends on the total number of Type A and B questions as well as what confidence level you want to use for the cutoff. There is probably a formula for determining what you want, but I don't know what it is. You can compute it yourself using a permutation test.

Generate a null hypothesis distribution of random guesses and then use this to find the cutoff for whatever p-value (probability that randomly guessing will give a score greater than a certain value) you are comfortable with. The easiest way to get that value is to sort all of the null performance scores and get one that is 95% higher than all of the others (say Score #950 if you did 1000 random guesses and sorted the scores).

Here is some Matlab code to estimate this:

n_trials = 50;
p_cutoff = 0.05;
n_permutations = 100000;
% Create a vector of n_trials 'correct' responses and replicate it across n_permutations
expected_values = repmat(randi([0 1],[1 n_trials]),[n_permutations 1]);
% Create an array of n_permutations x n_trials random guesses
random_responses = randi([0 1],[n_permutations n_trials]);
% Compute the percent correct for each permutation
percent_correct = sum(expected_values==random_responses,2)/n_trials;
% Compute and print the cutoff percentage for desired p value
fprintf('\np=%1.3f Cutoff for %d trials: %3.1f%%\n\n', p_cutoff, n_trials, quantile(percent_correct,1-p_cutoff)*100);

Here are some cutoffs for p=0.05 assuming the same number of Type A and Type B questions:

n_trials cutoff
10       80.0%
20       70.0%
30       63.3%
40       62.5%
50       62.0%
75       60.0%
100      58.0%
250      55.2%
500      53.6%

More total questions (trials) and the cutoff gets closer to 50%.

Note this is all assuming that there is no kind of bias, say a participant choosing randomly will be more likely to choose A (like the first answer or something like that).

Here is an online calculator for computing some probabilities with the binomial distribution.

Answer 2

In many cases, we talk about threshold as being the minimum "signal" level needed to "reliably" detect the presence of the signal. This threshold level is well above the signal level for there to be a statistically significant increase in the probability of correct. The idea being that the minimum signal level to be statistically better than chance depends on the number of trials, while the signal level to be better than "threshold" is independent of the number of trials (although the precision, and possibly the accuracy, of the estimated threshold depends on the number of trials).

Historically, threshold was estimated by fitting a psychometric function to measures of the probability of correct (or hit rate). For a 2AFC task, the psychometric function goes from 50 to 100 percent correct and threshold was arbitrarily defined as the signal level needed to get 75 percent correct. This choice was mostly because it made the math easier. An alternative to defining threshold in terms of percent correct is to use d prime. Typically, a d prime of unity is chosen as threshold, which under certain conditions corresponds to 68 percent correct. Adaptive methods like Levitt (1971) track specific points on the psychometric function. For example, a 2-down 1-up procedure tracks 70.7 percent correct and a 3-down 1-up procedure tracks 79.4 percent correct and the QUEST procedure tracks 92 percent correct. All of these values (and others) have been used to define threshold.

While one might think having thresholds defined between 68 and 92 percent correct would cause problems, often the psychometric function is steep enough (the transition from 50%+$\epsilon$ to 100%-$\epsilon$ is rapid) that the differences in the definition of threshold are obscured by the measurement noise.

From a pragmatic approach, chose the definition of threshold that makes the most sense given the measurement procedure you are using.