1
$\begingroup$

I am in a project where we are preparing a visual decision making experiment which requires participants to detect vertical grating in a patch of dynamic noise (the noise pattern is updated on every frame; the experiment design is partly a replication of a published paper). We need to find signal contrast level that would result in 75% accuracy rate, and after multiple attempts with QUEST and the method of constants, finding this level seems impossible — the thresholds estimated by those two methods result in either close to 100% accuracy, or chance performance. The experiment is implemented in PsychoPy.

What could be the reason for these difficulties finding the "right" contrast level for the grating? The image below is an attempt to fit a psychometric function to the experimental data -- the slope is almost vertical. The horizontal orange line is the 0.75 level, the vertical axis shows the proportion of "yes" responses, and the values on the horizontal axis are contrast levels.

Curve fit

$\endgroup$
7
  • $\begingroup$ What is chance accuracy here? $\endgroup$
    – Bryan Krause
    Aug 10 at 21:11
  • $\begingroup$ @BryanKrause 50% $\endgroup$
    – nat
    Aug 10 at 22:40
  • $\begingroup$ Yet on the left of your graph there are several points far below chance. I think something is wrong with your experiment. $\endgroup$
    – Bryan Krause
    Aug 10 at 22:42
  • $\begingroup$ @BryanKrause this plot shows data from a titration procedure where signal is present on every trial, so the vertical axis here shows the proportion of "yes" answers. In the "main" experiment the signal (grating) is always shown with the same level of contrast (it is equal to the threshold estimated for each participant individually during the titration), and signal is present on 50% of all trials, so a participant who always presses the "no" button would achieve 50% accuracy. $\endgroup$
    – nat
    Aug 10 at 22:51
  • 1
    $\begingroup$ Then you shouldn't expect chance to be 50% and shouldn't attempt to fit a psychometric curve that assumes chance is 50%. $\endgroup$
    – Bryan Krause
    Aug 10 at 22:52
1
$\begingroup$

Regardless of what the chance level exactly was, as alluded to in the comments, the scores leap from 0.1 to 0.8. This results in nearly vertical slopes that are prone to convergence errors and hence lead to suboptimal threshold estimates in terms of accuracy. MATLAB always warns me when that happens. I have no experience with PsychoPy unfortunately.

If you wish to achieve better accuracies of your threshold estimates of grating contrast levels between 0.035 and 0.04, you simply need more measurements in that range. Above 0.04 there are about 10 redundant points all basically centered at 100% correct. These measurements should be redistributed to the relevant region around threshold instead.

A convenient way of designing such a threshold experiment is using a dynamic procedure that reduces step size along the way, like a staircase procedure (Levitt, 1971). Staircase procedures may be less suitable if you wish to generate psychometric curves. If this is the case, using a stair case to swiftly determine thresholds and then choosing a number of fixed gratings below and above the approximate threshold to generate a psychometric curve can help a lot.

Reference
- Levitt, JASA (1971); 49 467-77

$\endgroup$
9
  • $\begingroup$ Thank you, this is very helpful. An adaptive staircase (QUEST) was in fact the first thing that we tried in this project. Pilot participants received their threshold estimates from a one up one down staircase, and then during the yes/no task where signal and noise were shown an equal number of times, everyone's accuracy rate was close to 100%. Are there any common mistakes in implementing the staircase procedure that we might be committing? The staircase was implemented as a yes/no task, can it be that a two-interval forced choice is significantly more reliable in this case? $\endgroup$
    – nat
    Aug 24 at 10:01
  • $\begingroup$ @nat Are there any common mistakes in implementing the staircase -- what is the problem exactly? $\endgroup$
    – AliceD
    Aug 24 at 10:36
  • $\begingroup$ Participants achieve near 100% accuracy using the staircase threshold estimate that should result in accuracy rates of 75%, so the threshold estimate from the staircase seems too high $\endgroup$
    – nat
    Aug 24 at 10:40
  • $\begingroup$ @nat on the basis of what do you expect 75%, i.e., what is your golden standard? $\endgroup$
    – AliceD
    Aug 24 at 19:13
  • $\begingroup$ 75% performance would be a participant indicating that there was no grating present on about 25% of all trials where the grating was present, and indicating that they saw a grating on 25% of trials where no grating was shown. $\endgroup$
    – nat
    Aug 24 at 21:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.