Confidence rating in cognitive experiments

Question

Does a confidence rating included in a post task measure the likelihood of correctness to the answer or likelihood of occurrence of an event.

To give an example: participants are given a picture of clouds and they are provided with the binary question if they think it might start raining; the answer is thus either a 'yes' or a 'no'. Then they have to give an estimate of how confident they are of their decisions on a rating scale that indicates how confident they were; for example they might say '70%'.

Does the confidence rating reflect their estimate of likelihood of rain, or rather the likelihood that the choice is correct?

Answer 1

If the question is 'Will it start to rain based on this cloud cover?' and the answer is a binary 'yes' or 'no' and the subsequent question is 'How sure are you about that?', then the answer to your question

Does the confidence rating reflect their estimate of likelihood of rain, or rather the likelihood that the choice is correct?

is the latter: it reflects the likelihood that the choice is correct, because when they say 'no it will not rain', with a confidence of 60%, then the chance it will rain is 40% and not 60%. Hence, it reflects the confidence they have in their answer.

Answer 2

The question you ask is at the heart of confidence judgments, i.e. whether they reflect objective truth or an observer belief of the truth. There is currently no answer to your question. All you can do is read the literature and please do experiments to give us an answer to this question. Start with: Fleming & Frith (2014) The Cognitive Neuroscience of Metacognition

Another paper very relevant to your question is: Pouget, A., Drugowitsch, J., & Kepecs, A. (2016). Confidence and certainty: distinct probabilistic quantities for different goals. Nature neuroscience, 19(3), 366.