Forte : Finding Outliers with Representation Typicality Estimation
Generative Models;Out-of-Distribution Detection (OOD)
We are truly grateful for your detailed evaluation and positive feedback. We appreciate your recognition of our paper's clarity, implementability, and the comprehensive supporting materials we provided in the appendix. Your suggestion for conference highlighting is very much appreciated.
Note of thanks
Yes
No ethics review needed.
None.  The appendix cleared them up
The paper is very well written, easy to follow, and highly implementable.  An extensive appendix provides supporting information and data.
The authors present a methodology, Forte, which enables the identification of outliers using a rigorous unsupervised algorithm.  The algorithm relies on establishing metrics to identify which samples are in-distribution vs out-of-distribution.  The method is very general, and applicable to many models.  The authors use SVM, KDE, and GMM models with their method, and show that it is capable of distinguishing between real and synthetic data.
None
Yes
No ethics review needed.
1. My main question is how the proposed method really works. If the authors could provide a pseudocode of the algorithm, and how it differs with DoSE, it would be great.
2. See above weaknesses.
3. Overall poor writing and latex formatting, in addition to the issues listed above. E.g., "in Figure 2 FIgure 3 Figure 4", 
3. Minor typos: "near-ood" -> "near-OOD", "Table 1 & 2" -> "Tables 1 & 2".
[The following is based on my guess of the proposed method, which is not well described in paper.]

+ Proposed approach is a simple change over feature-space OOD methods, and appears effective.
+ Experiments seems cover a wide range of scenarios
The paper investigates OOD by modeling distributions in feature space. The paper is poorly written, but my guess is that the proposed method is a variant of DoSE, using models _per-sample metrics_ of the feature space, rather than the feature vectors themselves. Four metrics, including precision and recall, are used. Experiments are performed on real images from distinct classes, synthetic vs. real images, and medical images. Results indicate that the proposed method is effective.
+ The paper is extremely poorly written. I list some major issues here.
    1. None of the math latex in section 3.2 is well formatted. Subscripts and superscripts are wrong.
    2. Variables are used without definition, e.g., $\text{nearest}_k$ in section 3.2. Is it different from the $k$ below?
    3. No description is given on how the four metrics are used. Are they used as the "summary statistics" that the proposed method models?
    4. The method, referred to as "Forte", is never truly defined or mentioned in the method section 3.
    
+ If my understanding of the method is correct, is it simple just taking DoSE and run it on the new set of statistics?
+ Incorrect claims. E.g., GMM is not non-parametric, and I don't think that the four metrics are newly proposed in this paper.
+ Density definition is inconsistent with Fig. 1. As defined, it is just a scaled "recall", which would make it useless to model.
Yes
No ethics review needed.
Could the computational demands of Forte’s ensemble approach limit its applicability in real-time OOD detection scenarios?

Would Forte’s effectiveness in OOD detection promote if one apply multiple models by one self-supervised approach in this framework?
- **Performance**: Forte demonstrates superior OOD detection performance compared to state-of-the-art methods across multiple benchmarks, including synthetic data and medical image datasets, which often present significant OOD detection challenges.
- **Flexibility**: Forte’s unsupervised nature eliminates the need for labeled data or pre-exposure to OOD samples, making it adaptable to various tasks and practical for real-world applications where OOD examples may not be available during training.
- **Comprehensive Evaluation**: Forte is rigorously tested on both synthetic and medical datasets, demonstrating the framework’s versatility and robustness across vastly different domains.
- **Insightful Metrics**: The use of novel per-point summary statistics (e.g., precision, recall, density, and coverage) contributes valuable insight into data distribution, enhancing OOD detection beyond standard density-based methods.
This paper introduces Forte, a novel out-of-distribution (OOD) detection framework that leverages self-supervised learners. Forte enhances detection by combining representation learning methods (e.g., CLIP, ViT-MSN, and DINOv2) with non-parametric density estimators (OCSVM, KDE, GMM) to model the typicality of input samples. The proposed framework emphasizes detecting atypical samples through summary statistics (precision, recall, density, and coverage) to analyze representation distributions. Forte’s performance was evaluated on synthetic datasets generated by Stable Diffusion and various medical image datasets, showcasing its advantages over existing supervised and unsupervised methods.
1. **Paper Structure**: The paper allocates a substantial portion of its Introduction to reviewing existing OOD detection literature and explaining the typicality concept. This approach detracts from an immediate focus on the novel contributions and design of Forte, which may hinder reader engagement and understanding of the primary contributions.
2. **Complexity in Practical Implementation**: The integration of multiple representation learning techniques, combined with non-parametric density estimators, may lead to a higher computational overhead and increased complexity in practical deployment. The paper lacks an illustration figure to clearly explain the proposed framework how to integrate the representations from diverse models.
3. **Insight.**: This work integrates representations from diverse models empirically. It lacks insight into the choice of self-supervised models, such as whether any specific attributes of CLIP, ViT-MSN, or DINOv2 contribute uniquely to Forte’s robustness.
Yes
No ethics review needed.
I miss some comparison with DoSE in some table for completeness.
The paper is clear to understand. The problem is relevant. The method is sound. The experimental framework is correct and complete.
This paper proposes a method for identifying OOD and synthetic data created using generative models. The definition of OOD changes with the advent of foundation models that can generate very plausible data. However, even when this data seems real, there can still be distribution shifts that are difficult to detect. In this paper, the concept of typicality is used to assess OOD. However, computing typicality is challenging. This paper proposes Forte, a method for computing typicality, and they evaluate it and compare it with other methods.
In the related work section I miss some comparison of the current SOTA with Forte.

This method is based on DoSE. I miss a clear statement in the intro about that and the changes over DoSE with an intuition of why. Also, it would be beneficial to clearly state the method to guide the reader on what will come.
@inproceedings{
anonymous2024forte,
title={Forte : Finding Outliers with Representation Typicality Estimation},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=7XNgVPxCiA},
note={under review}
}
Generative models can now produce photorealistic synthetic data which is virtually indistinguishable from the real data used to train it. This is a significant evolution over previous models which could produce reasonable facsimiles of the training data, but ones which could be visually distinguished from the training data by human evaluation. Recent work on OOD detection has raised doubts that generative model likelihoods are optimal OOD detectors due to issues involving likelihood misestimation, entropy in the generative process, and typicality. We speculate that generative OOD detectors also failed because their models focused on the pixels rather than the semantic content of the data, leading to failures in near-OOD cases where the pixels may be similar but the information content is significantly different. We hypothesize that estimating typical sets using self-supervised learners leads to better OOD detectors. We introduce a novel approach that leverages representation learning, and informative summary statistics based on manifold estimation, to address all of the aforementioned issues. Our method outperforms other unsupervised approaches and achieves state-of-the art performance on well-established challenging benchmarks, and new synthetic data detection tasks.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Generative Models Out-of-Distribution Detection (OOD)
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/31eeab036ca84fe71023b6abb7d5b5eac276e9c4.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Forte : Finding Outliers with Representation Typicality Estimation
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission