On Temperature Scaling and Conformal Prediction of Deep Classifiers
classification;temperature scaling;conformal prediction;conditional coverage;prediction sets
Yes
No ethics review needed.
I have not much to ask or say. The paper was clear!
A typo/English language check would improve it before the next round.
The paper is strong and I recommend acceptance.

I was impressed and surprised by the insights in this paper. Temperature scaling has a huge effect, the experiments bear this out, and the theory provides some explanation as to why, and might be useful to others in the field. (Specifically, the theory about how set size is affected by temperature is fairly general-purpose.)

* The empirical experiments are painstakingly detailed and very scientific.
* The theory is useful and correct.
The paper provides a very detailed theoretical and empirical study of the effect of temperature scaling on prediction set sizes and conditional coverage in conformal prediction. It finds that temperature scaling can (sometimes drastically) increase set sizes for common classification tasks. It explains this phenomenon theoretically and provides guidelines to practitioners about how to set the parameter moving forward.
* The theorems are somewhat weak, and of limited practical value in terms of being applied directly. They seem to be useful mostly for the purpose of post-hoc explanation of why this phenomenon happens.
Yes
No ethics review needed.
1. While Theorem 4.4 provides valuable theoretical insights, it is hard to understand how to use the theoretical results for estimating $T_{critical}$. I would greatly appreciate it if the authors could provide an explicit expression for computing $\hat{T}_{critical}$.
1. The work is well-writing. Basically, the paper is written in a good manner and I believe readers can easily touch the core idea.  

2. This paper provides a theoretical analysis to show how temperature values influence the properties of prediction sets. With the theoretical results, researchers can understand why temperature scaling can affect the conformal prediction.

3. The authors provide empirical validation of their theoretical framework.
In this paper, the authors investigate the interplay between conformal prediction and calibration. Firstly, the paper empirically shows that while temperature scaling improves the class-conditional coverage of adaptive CP methods. Then, the authors establish a mathematical theory that explains this phenomenon. Finally, the paper offers a guideline to effectively combine adaptive CP with calibration.
1. The paper presents an inconsistency in its mathematical derivations. In Eq. (3), the analysis is mainly based on the relationship between $\sum_{i=1}^M\pi_i-\sum_{i=1}^M\pi_{T,i}\quad\text{and}\quad{q}-q_T$ (omit \hat for \pi and q due to the tex support),
which represents the difference of accumulated probability and threshold value before and after applying a temperature. Later, in Eq. (5), the problem above is translated to investigate the relationship between 
$$\sum_{i=1}^M\exp(z_i)-\sum_{i=1}^M\exp(z_i/T)\quad\text{and}\quad\sum_{i=1}^M\exp(z_i^q)-\sum_{i=1}^M\exp(z_i^q/T).$$However, based on the definition on $z^q$, we know that if $M$ is not the true label of $z^q$, then $\sum_{i=1}^M\exp(z_i^q/T)\neq\hat{q}_T$. Therefore, it's unclear how the problem in Eq. (3) can be equivalent to the analysis in Eq. (5). 

2. The assumptions in Theorem 4.4 appear to be unreasonable. Theorem 4.4 in the paper states that if $\Delta z>b(T)$, then rising $z_1$ leads to an increase in the set size. However, the assumption that '$\Delta z$ is preserved as $z_1$ increases' (line 458) is not natural for me because counterexamples exist where $\Delta z$ increases as $z_1$ increases. Furthermore, even if we accept the condition that $\Delta z$ remains constant, the paper's claim that '$z^q$ has a larger dominant entry than typical $z$' lacks proper justification.

3. The theoretical bounds and empirical results show inconsistency. The paper reports an empirical critical temperature of $T^{*}=1.524$. However, this value falls between the theoretical temperature ranges $(0,0.813)$ and $(1.25,4.81)$, suggesting a gap between theory and practice. This inconsistency challenges the paper's claim that 'the bounds in Theorem 4.4 do not require unreasonable values of $\Delta z$ and T' (line 463). Moreover, it indicates that using the median of $\Delta z$ to estimate $T_{critical}$ may not be a reliable approach.
    
4. The proposed approach of using $T_{critical}$ to enhance conditional coverage (as proposed in Section 5) has limitations. As discussed in Weaknesses[3], the theorem may fail to provide accurate estimation of $T_{critical}$. Even though an accurate estimation can be achieved, simply applying $T_{critical}$ falls short of achieving group-wise coverage that Mondrian conformal prediction provides. Furthermore, the paper does not present empirical evidence demonstrating how their proposed guideline enhances conformal prediction performance. Overall, these limitations restrict the practical applicability of the theoretical results.


**Minor Comments:**
1. The paper's analysis is limited to Temperature Scaling, while leaving out other important calibration methods such as histogram binning.
2. The mathematical proofs lack clarity. For example, in proof of Theorem 4.1, the first equation (line 750-755) is stated without proper mathematical justification.
3. typo in Section 5: "guidelinse" should be "guidelines".

I would be open to reconsidering my recommendation if these issues could be addressed in the rebuttal.
Yes
No ethics review needed.
See above.
1. The paper focuses on an interesting question: the influence of temperature scaling on conformal prediction outcomes, concerning group coverage and band length.
2. Empirical evidence is provided to suggest that TS can detrimentally harm the set size of conformal prediction measures such as APS and RAPS, while simultaneously improving group coverage.
3. The authors demonstrate that TS facilitates a trade-off between set size and group coverage.
4. Theoretical insights into the empirical findings are offered, along with practical guidelines for practitioners.
This paper delves into the impact of Temperature Scaling (TS) on conformal prediction methods. 
Traditionally, researchers have applied TS before conducting conformal prediction on the resulting classifiers. 
The paper argues that while it enhances group coverage, it negatively affects set size. 
I find this paper offers valuable experiments and intriguing insights into the interplay between conformal prediction and temperature scaling, and I recommend an acceptance. My hesitation to give a higher score stems from (a) the paper's inherently limited scope to classification tasks, as regression tasks do not typically encounter this issue; (b) it is somehow intuitive that TS might not be the best choice for the set size, which is not particularly surprising.
have few major concerns with this paper. As previously mentioned, my decision not to assign a higher score is based on:
1. The limited scope of the paper to classification tasks, as regression tasks do not typically present this issue.
2. The somewhat expected nature of the finding that TS might impact set size negatively.
We theoretically and empirically analyze the impact of temperature scaling beyond its usual calibration role on key conformal prediction methods.
@inproceedings{
anonymous2024on,
title={On Temperature Scaling and Conformal Prediction of Deep Classifiers},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=MxHgnYbxly},
note={under review}
}
In many classification applications, the prediction of a deep neural network (DNN) based classifier needs to be accompanied by some confidence indication. Two popular approaches for that aim are: 1) *Calibration*: modifies the classifier's softmax values such that the maximal value better estimates the correctness probability; and 2) *Conformal Prediction* (CP): produces a prediction set of candidate labels that contains the true label with a user-specified probability, guaranteeing marginal coverage but not, e.g., per class coverage.  In practice, both types of indications are desirable, yet, so far the interplay between them has not been investigated. 
Focusing on the ubiquitous *Temperature Scaling* (TS) calibration, we start this paper with an extensive empirical study of its effect on prominent CP methods. We show that while TS calibration improves the class-conditional coverage of adaptive CP methods, surprisingly, it negatively affects their prediction set sizes. Motivated by this behavior, we explore the effect of TS on CP *beyond its calibration application* and reveal an intriguing trend under which it allows to trade prediction set size and conditional coverage of adaptive CP methods. Then, we establish a mathematical theory that explains the entire non-monotonic trend.
Finally, based on our experiments and theory, we offer simple guidelines for practitioners to effectively combine adaptive CP with calibration.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
classification temperature scaling conformal prediction conditional coverage prediction sets
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/a6f36bafabba711c93c63d6b929692d848a3a790.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
On Temperature Scaling and Conformal Prediction of Deep Classifiers
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission