2
$\begingroup$

In metacognition literature, why is confidence, regardless of scale (Likert-type or continuous) and definition (e.g. decision confidence as a subjective probability of a decision being correct), mostly measured in binary decision tasks, such as 2AFC?

Is it psychologically valid to obtain confidence in other types of choice tasks, such as multiple-alternatives or ordinal ones, in a similar way?

$\endgroup$
2
  • 1
    $\begingroup$ Confidence gets complicated when you have more than two choices. If you have options ABCD you may be very confident it is either A or B but not actually certain that A is the correct answer. In this case you have a mixture of confidences at different levels of comparison. Two-alternative tasks are easier to analyze for many reasons and often multiple alternative tasks can be distilled to two choice tests anyways. $\endgroup$ – Bryan Krause Mar 20 at 23:44
  • 1
    $\begingroup$ Thank you, Bryan. It’s really an insightful example. But I’m still wondering if people are good at probabilities (Bayesian), an ordinary single-step way for measuring confidence (as probability of choice among ABCD being correct) and a hierarchical way, e.g. first, prob. of AB being correct (e.g. 90% for sure), and then prob of A being correct between A and B (e.g. 50% for sure) make a big difference: i.e. the former may directly yield 45 %, and the latter does 90% x 50% = 45%. It would be appreciated if you or someone else could cite some theories or previous studies if any. Best, $\endgroup$ – Shelly C. Mar 21 at 1:12
0
$\begingroup$

Non-binary choices complicate analysis unnecessarily, as most multiple alternative scenarios can be distilled into multiple binary choices.

Outside of metacognitive literature, I have seen studies that assume the naive Bayesian approach for multi-choice decision confidence, for example, from Satopää et al (2014) in forecasting:

For now, assume that the event can take exactly one of a total of M≥2 different outcomes. Under pure ignorance, the forecaster should assign a probability of 1/M to each outcome. The more ignorant the forecaster is, the more we would expect him to shrink his forecasts towards 1/M.

However, this assumption may not hold in the uncalibrated confidence judgments of metacognitive tasks. There is an interesting paper by Li & Ma (2020):

Experiments on confidence reports have almost exclusively focused on two-alternative decision-making. ... Here, we test ... a three-alternative visual categorization task. We found that confidence reports are best explained by the difference between the posterior probabilities of the best and the next-best options, rather than by the posterior probability of the chosen (best) option alone, or by the overall uncertainty (entropy) of the posterior distribution. Our results upend the leading notion of decision confidence and instead suggest that confidence reflects the observer’s subjective probability that they made the best possible decision.

This would make even 4-choice tasks much more difficult to interpret.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.