In metacognition literature, why is confidence, regardless of scale (Likert-type or continuous) and definition (e.g. decision confidence as a subjective probability of a decision being correct), mostly measured in binary decision tasks, such as 2AFC?
Is it psychologically valid to obtain confidence in other types of choice tasks, such as multiple-alternatives or ordinal ones, in a similar way?
Non-binary choices complicate analysis unnecessarily, as most multiple alternative scenarios can be distilled into multiple binary choices.
Outside of metacognitive literature, I have seen studies that assume the naive Bayesian approach for multi-choice decision confidence, for example, from Satopää et al (2014) in forecasting:
For now, assume that the event can take exactly one of a total of M≥2 different outcomes. Under pure ignorance, the forecaster should assign a probability of 1/M to each outcome. The more ignorant the forecaster is, the more we would expect him to shrink his forecasts towards 1/M.
However, this assumption may not hold in the uncalibrated confidence judgments of metacognitive tasks. There is an interesting paper by Li & Ma (2020):
Experiments on confidence reports have almost exclusively focused on two-alternative decision-making. ... Here, we test ... a three-alternative visual categorization task. We found that confidence reports are best explained by the difference between the posterior probabilities of the best and the next-best options, rather than by the posterior probability of the chosen (best) option alone, or by the overall uncertainty (entropy) of the posterior distribution. Our results upend the leading notion of decision confidence and instead suggest that confidence reflects the observer’s subjective probability that they made the best possible decision.
This would make even 4-choice tasks much more difficult to interpret.
