Confidence rating in cognitive experiments
If the question is 'Will it start to rain based on this cloud cover?' and the answer is a binary 'yes' or 'no' and the subsequent question is 'How sure are you about that?', then the answer to your question


  Does the confidence rating reflect their estimate of likelihood of
  rain, or rather the likelihood that the choice is correct?


is the latter: it reflects the likelihood that the choice is correct, because when they say 'no it will not rain', with a confidence of 60%, then the chance it will rain is 40% and not 60%. Hence, it reflects the confidence they have in their answer.