# Can one leverage the probability difference between the the predicted class vs the original class?

I have P predictors, C classes

After performing training on a training partition I run the test partition through my classifier.

Say for a particular test instance the predicted class is Cpred with probability Ppred while the original label is Corig and the probability of it being that class is computed to be Porig by the trained classifier.

What type of inferences, insights, uses can I draw from (Ppred - Porig)?

Some specific questions?

1. Can I use cases where Ppred - Porig >> mean(Ppred - Porig | Corig, Cpred) to look for mislabelled data?
2. Can I use case where Ppred - Porig >> mean(Ppred - Porig | Corig, Cpred) to further partition the Classes into a new Class Corig,pred -- perhaps the original labels are not expressive enough for the underlying phenomenon?
3. How could one interpret Pi,pred - Pi,orig == Pj,pred - Pj,orig where i and j are two instances in the test set but where Pi,pred >> Pj,pred

Notes:

1. How is Porig calculated? Both Porig and Ppred are produced by the classifier. If there are C classes -- the Classifier will output P1, ... , Pc probabilities, one for each class. The predicted class will correspond to the argmax of P1 ... Pc
2. OK, so how is Porig different from Ppred? Ppred is the max(P1, ... Pc) while Porig is P of the labeled class that we know apriori
3. See R code below for a simple illustration using the R

Example:

set.seed(1)
str(iris)
train<-sample(2,nrow(iris),replace=TRUE)
rf<-randomForest(Species~.,data=iris,subset=train==1)
p.resp<-predict(rf,newdata=iris[train==2,],type="prob")
pred<-predict(rf,newdata=iris[train==2,])
table(iris[train==2,]$Species,pred) virginica_classifiedas_versicolor<-pred=="versicolor" & iris[train==2,]$Species=="virginica"
Porig<-p.resp[virginica_classifiedas_versicolor,"virginica"]
Ppred<-p.resp[virginica_classifiedas_versicolor,"versicolor"]

• "the original label is Corig and the probability of it being that class is computed to be Porig" - How do you compute this probability? – Arun Jose May 4 '17 at 4:45
• @ArunJose -- as computed by the classifier in a multinomial case -- let me update the question – user1172468 May 4 '17 at 5:02
• In that case how is Ppred different from Porig? If both are probabilities from the same classifier? – Arun Jose May 4 '17 at 5:04
• @ArunJose -- I also updated the question with note 2 to make that distinction – user1172468 May 4 '17 at 5:07
• "Porig is P of the labeled class that we know apriori" - This is still confusing. Are you suggesting (sum of class A/ total N classes) = Porig? – Arun Jose May 4 '17 at 5:10
{"ft_lang_label":"__label__en","ft_lang_prob":0.725816,"math_prob":0.7068653,"size":1829,"snap":"2020-45-2020-50","text_gpt3_token_len":485,"char_repetition_ratio":0.1369863,"word_repetition_ratio":0.0625,"special_character_ratio":0.24220885,"punctuation_ratio":0.16239317,"nsfw_num_words":0,"has_unicode_error":false,"math_prob_llama3":0.98541135,"pos_list":[0],"im_url_duplicate_count":[null],"WARC_HEADER":"{\"WARC-Type\":\"response\",\"WARC-Date\":\"2020-11-23T16:40:28Z\",\"WARC-Record-ID\":\"<urn:uuid:a161dcc6-776f-4042-98f1-11ba886c8fbe>\",\"Content-Length\":\"153949\",\"Content-Type\":\"application/http; msgtype=response\",\"WARC-Warcinfo-ID\":\"<urn:uuid:a1cbf961-50c0-4c3e-a205-df561a20aba7>\",\"WARC-Concurrent-To\":\"<urn:uuid:970a5b41-3e17-4e5f-bc67-3c58701734b3>\",\"WARC-IP-Address\":\"151.101.1.69\",\"WARC-Target-URI\":\"https://stats.stackexchange.com/questions/277465/can-one-leverage-the-probability-difference-between-the-the-predicted-class-vs-t\",\"WARC-Payload-Digest\":\"sha1:WQORDERK45UXG7GAYQS5JCXHYLDV4NR6\",\"WARC-Block-Digest\":\"sha1:HSE52DPGWS7F3424IMZ3JSMG5CRMNBBE\",\"WARC-Identified-Payload-Type\":\"text/html\",\"warc_filename\":\"/cc_download/warc_2020/CC-MAIN-2020-50/CC-MAIN-2020-50_segments_1606141163411.0_warc_CC-MAIN-20201123153826-20201123183826-00447.warc.gz\"}"}