# Calculate threshold from 'binary' yes/no data

For various reasons, I have a crappy experimental design that I am forced to deal with. I present subjects with 20 yes/no trials, out of which 5 are no-signal catch trials and 15 are signal trials. Each signal presentation has a unique level with unequal spacing (as small as one in arbitrary units and as great as 5). All subjects run the trials in the same order (i.e., the 5th trial is always at signal level, and the 11th and the 6th trial are always catch trials).

To make things more complicated, at the population level, and likely at the individual level, the criterion is not stable such that the probability of saying yes on a no-signal trial depends on the catch trial number (the 1st catch trial has a probability of yes of about 0.05 and the 5th catch trial has a probability of yes of about 0.2). From this mess, I need a single number representing threshold. I don't really care what the threshold corresponds to (e.g., the signal level leading to 50% yes, 50% correct, or a d' of 1 would all be fine). Ideally, the method would provide a confidence interval on the estimate at the individual subject level.

I was thinking of using a Spearman-Karber type approach, but the unequal step sizes seems problematic (along with the variable criteria). I also don't understand the history of Spearman-Karber (i.e., where was it first published and how it relates to Spearman 1908) and kind of hope that maybe in the past 100 years we have improved on things.

• wow. Good luck :-) Are you using an adaptive algorithm? How different are the step sizes? Do you have multiple measures per step? Could you show us a data set?
– AliceD
Aug 14, 2017 at 18:29
• @AliceD everyone runs the trials in the same order (i.e., not adaptive) and there is only a single trial at each signal level. The step sizes vary from 1 to 5. Aug 14, 2017 at 18:45
• Then it is a MOC procedure with binary data (say 0 (no) or 1 (yes)? Can't you fit a Weibul or other psychometric function? Perhaps that is indeed difficult with binary data. Let me know. A picture of a (imaginary) data set would be nice still to visualize your problem
– AliceD
Aug 14, 2017 at 19:49
• @AliceD yes, but with only a single trial per level I am not sure that is the way to go. Additionally, I am not sure how to get a CI from the fitted function with only a single trial per level. Aug 14, 2017 at 19:52
• I have always gathered enough repeats per level to estimate a correct rate. Interesting problem. I will ponder on this. +1'd already. Should be interesting for a lot of people.
– AliceD
Aug 14, 2017 at 19:55

The biggest drawback of your procedure, also point of discussion previously here is the limited number of trials. Apparently, you obtained binary data, namely one answer (yes/no) per trial, basically leaving you with a sparsely sampled, binary data set. Binary data will not allow a curve fitting, which is the conventional way to determine a threshold.

To make matters worse, there is a certain bias in the catch trials. But values anywhere below 25% false positive are, imo fine. So let's forget that issue, as it is likely the least of your problems.

Now, single-stimulus per trial data are referred to as the Method of Limits. Basically what you would normally do to obtain a threshold with MoL is increase the level to see where the subjects starts perceiving the stimulus (YES/NO reversal) and then flip the sequence and see where the person stops answering YES and hits NO. Generally, such a procedure is experimentally used to get a quick handle of a good starting point for the 'real deal' experiments. Or, it is useful in clinical environments. For example, clinical audiometric thresholds are often determined this way - quick and dirty, but accurate by about 5 dB or so.

The response you get in this case is a so-called type 2 response, since you never can tell whether the subject was right or not. You just evaluate when the subject reports to perceive the stimulus. The problem with the method described here is that the S may become habituated (I didn't perceive it now, likely I won't in the next trial) or the S makes the error of expectation (It will probably become apparent now).

• The only way in which I can see to analyze your data is determine the flipping point and take that as a measure. You can then, perhaps take the average between the two intensities where the response flips to get an estimate of the 50% level, the gold standard of the threshold in yes/no tasks.
Likely you will not have an unambiguous flipping point in your data, but I'm sure you can think of something. Perhaps just take the lowest intensity where a flip occurred, or something along those lines.
• Alternatively, you may group your intensities into approximate levels and average out the responses to get values including rates between 0 and 100%. This procedure, however, is likely pretty much doomed to be shot to pieces by any referee or peer in your field, the more since your step sizes are unequal.

Reference
- Kingdom & Prins. Psychophysics. Elsevier 2010

• We work around the habituation by having the levels not be monotonically decreasing such that sometimes a lower level signal is followed by a higher level signal. At the population level, when we sort by trial, we see big jumps in the probability of yes that reflect the changes in the signal level, but when we sort by signal level, we get a nice monotonic psychometric function. Aug 15, 2017 at 13:42
• While threshold in the MoL is often estimated by taking the last level for a yes response, that doesn't make it the best way to do it. At the individual level, while for some subjects there is a single level above which they always responds yes and below which they always respond no, for many subjects that is not the case. Aug 15, 2017 at 13:44
• @StrongBad, I understand that it may not be a sharp tipping point, as included in my answer. My post is simply the best I can come up with after mulling it over. It's certainly not pretty, but neither is your data :)
– AliceD
Aug 15, 2017 at 13:54
• Pooling the data of the population is always an option, but even less pretty imo. It defies the purpose of determining thresholds.
– AliceD
Aug 15, 2017 at 13:56
• We pool the data to show differences between our two groups (control and test). We now want to look at individual subjects to see if the test has diagnostic value. Aug 15, 2017 at 13:59