Exploring the Link Between Out-of-Distribution Detection and Conformal Prediction with Illustrations of Its Benefits
Out-of-distribution detection;Conformal Prediction;benchmark;nonconformity scores
Yes
No ethics review needed.
1. Is there a scenario where classical metrics like FPR@β or AUROC might fail for OOD evaluation, but your conformal metrics accurately capture the evaluation? How do the experimental results in Section 4 contribute to safer OOD evaluation?

2. When using OOD scores as nonconformity scores in CP, can you still obtain theoretical guarantees for the prediction sets?
* The paper is well-structured, well-written, and easy to read.

* The paper introduces modified metrics for OOD evaluation that are more robust when considered from a hypothesis testing perspective.

* The OOD detection problem is significant for ML research, making this topic relevant to the ICLR community.

* The exploration of the link between OOD and CP is informative.
The paper examines the relationship between conformal prediction (CP) and out-of-distribution (OOD) detection. It approaches the OOD detection problem from a conformal prediction perspective and, inspired by CP, proposes several modified metrics for OOD evaluation, testing them on various OOD benchmarks. Additionally, the paper constructs prediction sets for classification tasks, drawing on techniques from OOD detection.
* The main insight of the paper is that FPR@β can be interpreted as a p-value within a specific statistical hypothesis testing framework. By building on the work of [1], the paper proposes a corrected estimator for this metric to improve robustness. While this approach is interesting, I believe it is insufficient to warrant publication. Firstly, much of the technical foundation relies on [1], and the corrected metric for OOD detection performs similarly to the classical version. Therefore, I do not find the current version sufficiently novel for publication.

* Computing the proposed metric for OOD detection requires access to extra validation sets.

Minor Comments:
Line 22: FPR? (not FRP)

[1]Stephen Bates, Emmanuel Candès, Lihua Lei, Yaniv Romano, and Matteo Sesia. Testing for Outliers
with Conformal p-values, May 2022. URL http://arxiv.org/abs/2104.08279.
Yes
No ethics review needed.
Given the significant overlap with Bates et al. (2022) 's methods, what specific contributions does this paper make beyond adapting CP to OOD tasks?
The paper presents an application by adapting the conformal prediction methods from Bates et al. (2022) to OOD detection.
This paper investigates the relationship between Out-of-Distribution (OOD) detection and Conformal Prediction (CP). They apply conformal prediction methods from prior work by Bates et al. (2022) to the task of Out-of-Distribution (OOD) detection, which is to distinguish in-distribution (ID) data from OOD data. They introduce metrics conformal AUROC and conformal FPR@TPR95 which provide conservative probabilistic guarantees in OOD detection tasks, particularly relevant for safety-critical applications. The authors then empirically validate the performance of these metrics on common OOD benchmarks.
(1) The methodology largely follows the framework established by Bates et al. (2022) in the conformal prediction literature, with limited novel contributions specific to OOD detection. The primary innovation appears to be an application of existing CP methods to OOD tasks rather than a new approach.

(2) While the idea of combining OOD detection with CP is interesting, the approach itself and the resulting metrics are relatively straightforward extensions and may not represent a significant advancement in either OOD or CP methodologies.

minors:
(1) Line 135: In Equation (2), n_val is not defined and weird, should be probability. Similarly, n_val is used as probability in Line 296, Equation (10). Other places, it is used as the number of data points. 
 
(2) A lot of places, for example, section 4.3.1, tau and t are used in mix to refer to the same thing.
Yes
No ethics review needed.
I think that there are two types of uncertainties for a model predicting a new test instance:

1.	Prediction Confidence: Assuming the instance is within the In-distribution, this type of uncertainty relates to the strength of evidence supporting the prediction.

2.	In-Distribution Uncertainty: As discussed in this paper, this refers to uncertainty regarding whether the instance truly belongs in the In-distribution.

I would appreciate further discussion from the authors on the distinctions and connections between these two types of uncertainties. Can they be quantified into a single metric?
•	OOD scores can be unreliable, and employing new statistical developments to improve their reliability is a promising direction. Using the uncertainty estimates from CP appears to be a robust approach to address OOD score unreliability.

•	The paper targets a practical problem, effectively demonstrating key concepts through comprehensive case studies.

•	By linking OOD detection with CP, this work facilitates cross-fertilization that benefits both machine learning and statistics. The authors provide a good introduction to using CP for interpreting and correcting OOD scores.
This paper explores the connection between Out-of-Distribution (OOD) detection scores and the non-conformity scores in Conformal Prediction (CP), showing the potential for cross-fertilization between these methods. Non-conformity scores offer probabilistic interpretation and correction for OOD scores, while OOD scores can enhance the efficiency of prediction sets obtained through CP.
•	Non-conformity scores in CP require an evaluation dataset, and it reliability depends on the choice of this dataset. This introduces a practical challenge in implementing this statistically sound solution.

•	The paper presents limited innovation, as much of the work is based on existing research.
Yes
N/A
No ethics review needed.
See Weaknesses. In addition, Typo at line 69: eas (2023). Shouldn't it be Easa?
The paper makes an extremely interesting parallel between OOD and CP. In addition, I found very interesting the marginal guarantees that the authors were able to find.
The authors ask themselves, and answer positively, to the question whether CP can be beneficial to OOD, and whether the vice versa holds as well. They also validate empirically their findings.
CP is a method for *uncertainty representation*, not uncertainty *quantification*. Indeed, CP *represents* uncertainty via the conformal prediction region. It does not quantify it: there is no real value attached to any kind of predictive uncertainty (e.g. aleatoric or epistemic, AU and EU, respectively). Some claim that the diameter of the conformal prediction region quantifies the uncertainty, but even in that case, it is unable to distinguish between AU and EU. Indeed, the diameter is a positive function of both: it increases as both increase, and hence it cannot be used to distinguish between the two [1]. Please add this clarification in the camera-ready version of the manuscript.

The simplest CP technique is (arguably) transductive CP, not split CP.

Shouldn't $n_\text{val}$ in (2) and in (10) be substituted by $P$? In particular, CP guarantees hold for all exchangeable countably additive probabilities $P$ on the space $\mathcal{Y}$ of outputs.

How does the proposed method relate to the subsequent work by Kaur [2]?

---

[1] https://arxiv.org/abs/2302.09656

[2] https://arxiv.org/abs/2302.11019
We show that Out-of-distribution detection and Conformal Prediction are naturally intertwinded and emphasize some benefits of this link.
@inproceedings{
anonymous2024exploring,
title={Exploring the Link Between Out-of-Distribution Detection and Conformal Prediction with Illustrations of Its Benefits},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=GQhlM0Mavg},
note={under review}
}
Research on Out-Of-Distribution (OOD) detection focuses mainly on building scores that efficiently distinguish OOD data from In Distribution (ID) data. 
    On the other hand, Conformal Prediction (CP) uses non-conformity scores to construct prediction sets with probabilistic coverage guarantees. In other words, the former designs scores, while the latter designs probabilistic guarantees based on scores. Therefore, we claim that these two fields might be naturally intertwined. 
    This work advocates for cross-fertilization between OOD and CP by formalizing their link and emphasizing two benefits of using them jointly.
    First, we show that in standard OOD benchmark settings, evaluation metrics can be overly optimistic due to the test dataset's finite sample size.
    Based on the work of (Bates et al, 2022), we define new *conformal AUROC* and *conformal FRP@TPR$\beta$* metrics, 
    which are corrections that provide probabilistic conservativeness guarantees on the variability of these metrics.
    We show the effect of these corrections on two reference OOD and anomaly detection benchmarks, OpenOOD (Yang et al, 2022) and ADBench (Han et al. 2022). 
    Second, we explore using OOD scores as non-conformity scores and show that they can improve the efficiency of the prediction sets obtained with CP.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Out-of-distribution detection Conformal Prediction benchmark nonconformity scores
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/37499a670ea7e167336ff4bb9978757f7ff8f8db.pdf
alignment, fairness, safety, privacy, and societal considerations
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Exploring the Link Between Out-of-Distribution Detection and Conformal Prediction with Illustrations of Its Benefits
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission