# How confident is my model? Is there a way to know?

I am interested in finding out how confident my model (say Logistic Regression) is in predicting the label of a new data point. For example, if it is not confident, I better abstain from making a prediction.

Logistic Regression outputs probabilities, which gives you a notion that there is confidence in prediction. But in fact, it is not. $$P(y|x)$$ being 0.3, doesn't tell me that the model is confident in its prediction or not. All, we can say is that it believes 100% that class A is 0.3 and other class is 0.7.

Can we use confidence intervals of LR as some sort of confidence in prediction? E.g., larger the difference between upper and lower bound for a particular data point, less confident it is, and vice-versa?

If not, how can we build confidence in our prediction? Can anybody guide me to some paper or field of study?

• You could use bootstrap samples to attain different values and calculate a CI based on these new outcomes. – peteR Feb 21 at 7:41
• yes you can use confidence interval calculations for logistic regression, and for more complicated models you have to use bootstrap samples. see stats.stackexchange.com/a/354660 – seanv507 Feb 21 at 9:50

## 1 Answer

Your inference about $$y$$ is the Bernoulli distribution $$\mathrm{prob}(y = y^\prime | x = x^\prime, \mathcal{D}, \mathcal{I}) = \begin{cases} 0.3 & y^\prime = 1 \\ 0.7 & y^\prime = 0 \end{cases}$$ The probability $$0.3$$ is itself the measure of your confidence in $$y = 1$$. So you should not confidently predict that $$y = 1$$, nor should you even confidently predict that $$y = 0$$.
{"ft_lang_label":"__label__en","ft_lang_prob":0.9368914,"math_prob":0.9908129,"size":828,"snap":"2019-35-2019-39","text_gpt3_token_len":191,"char_repetition_ratio":0.1553398,"word_repetition_ratio":0.0,"special_character_ratio":0.23067632,"punctuation_ratio":0.1388889,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.99870723,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2019-08-18T21:26:02Z\",\"WARC-Record-ID\":\"<urn:uuid:126fdc4b-5039-4f19-a21f-52dbdf9ebf74>\",\"Content-Length\":\"134396\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:e6e4c0c4-466a-46a3-8cde-f06284cd4ead>\",\"WARC-Concurrent-To\":\"<urn:uuid:0c307f22-87b4-4dab-ac65-66ad484d51e6>\",\"WARC-IP-Address\":\"151.101.65.69\",\"WARC-Target-URI\":\"https://stats.stackexchange.com/questions/393599/how-confident-is-my-model-is-there-a-way-to-know\",\"WARC-Payload-Digest\":\"sha1:AUB3B5QAO2RUEEPBZSSE6LEGTY5MWFWE\",\"WARC-Block-Digest\":\"sha1:ZNOJ4TKOWDYVIHTUYBUYFLUSS4PBW3ML\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2019/CC-MAIN-2019-35/CC-MAIN-2019-35_segments_1566027314130.7_warc_CC-MAIN-20190818205919-20190818231919-00380.warc.gz\"}"}