Zero-shot Outlier Detection via Synthetically Pretrained Transformers: Model Selection Bygone!
zero-shot outlier detection;prior-data fitted networks
Yes
No ethics review needed.
- I wonder if it would be possible to do some sort of finetuning when a training data comes in, so that we don't need to feed in the entire training data into the model?
- The authors of the paper tackles an important yet very challenging problem of unsupervised anomaly detection. 
- The proposed method is technically sound. 
- The paper is well written and easy to follow. 
- The authors of the paper conduct ample experiments to demonstrate the effectiveness of the proposed method.
This paper introduces FoMo-0D, a novel zero-shot approach for outlier detection (OD) that bypasses the traditional model selection process, a major obstacle in unsupervised OD tasks. Model selection, including algorithm and hyperparameter tuning, has long been a challenge due to the unsupervised nature of OD. FoMo-0D leverages Prior-data Fitted Networks (PFNs), a Transformer-based model trained on a large synthetic dataset from a prior distribution, allowing it to make direct outlier predictions on new datasets without further tuning or training. As a foundation model for zero-shot OD on tabular data, FoMo-0D delivers outlier predictions with a single forward pass, eliminating the need for manual algorithm selection or hyperparameter adjustments. The authors of the paper conduct extensive experiments to demonstrate the effectiveness of the proposed method.
- Despite the optimization done to scale the proposed method up, it is still not immediately clear to me how scalable the proposed method is. Does the proposed method still work if the training data contains millions of samples? Moreover, what is the inference latency and memory overhead? Would the proposed method incur very heavy computational burden if I have to handle a large number of samples during inference? Such a scenarios is arguably common in real world anomaly detection settings. 
- How limiting is the use of GMM for synthetic data generation for pre-training. Does such a method scale to more complex datasets?
Yes
No ethics review needed.
See Weaknesses.
1. The paper presentation is clear. The proposed method is well-motivated.
2. The experimental results are highly sufficient and also demonstrate the effectiveness of the proposed foundation model.
This paper present FoMo-0D, the first foundation model for zero-shot OD on tabular datasets. FoMo-0D is pretrained on many synthetically generated datasets drawn from a novel data prior that the author introduce to capture various inlier and outlier distributions. Feeding the FoMo-0D with a new training set and a text point, it perform zero-shot inference on a new dataset via a single forward pass, fully abolishing the need for both model training on a new dataset and hyperparameter selection.
1. While it is noted that "finding a prior that supports a sufficiently large subset of possible [data generating] functions isn’t trivial," the paper demonstrates that the initial attempts were adequate to achieve remarkable performance even with a simple data prior. Could you provide some in-depth insights into this result?

2. The paper conducted experiments on several benchmark datasets for out-of-distribution (OOD) detection, including CIFAR-10, Fashion-MNIST, MNIST-C, MVTec-AD, and SVHN. I am curious about the performance of FoMo-0D on more complex datasets, such as the ImageNet-level benchmarks recently proposed in OpenOOD v1.5 [1].

[1] OpenOOD v1.5: Enhanced Benchmark for Out-of-Distribution Detection, NeurIPS 2023 Workshop on Distribution Shifts
Yes
No ethics review needed.
see above
1. The paper is written well and is easy to understand.

2. The studied problem is very important.

3. The results seem to outperform state-of-the-art.
This paper leverages Prior-data Fitted Networks, which can directly predict the (outlier/inlier) label of any test data at inference time, by merely a single forward pass—making obsolete the need for choosing an algorithm/architecture and tuning its associated hyperparameters, besides requiring no training of model parameters when given a new OD dataset.
1. The baselines compared in this paper are not competitive enough. There are much more advanced OD algorithms that are unsupervised or semi-supervised and do not require retraining on the new test set. Please refer to [1] for more advanced algorithms. There are plenty of strong post-hoc algorithms that can be efficient and effective at the same time. 

2. The inlier piror and outlier synthesis idea is explored in [2]. Could the authors clarify the differences? For me, I did not see any significant technical differences from the prior selection and the synthesis approach, even though there are naunces in model architecture and the focused problem. 

3. Is it possible for the authors to show the actual detection metrics rather than the rank so that it makes the readers easy to see the algorithm performance.



[1] Yang et al., Generalized Out-of-Distribution Detection: A Survey

[2] Du et al., VOS: Learning What You Don't Know by Virtual Outlier Synthesis
Yes
No ethics review needed.
1. As the foundation model already achieves comparative performance compared with baselines, can the performance be continuously improved if we conduct post-training with specific real data, a common practice in NLP or CV?
1. This paper presents a paradigm shift for outlier detection, training a foundation model for various downstream tasks without model selection and hyperparameter tuning.
2. The paper is well-written, which makes it easy to follow.
3. The experiments are sufficient for elucidating the intrinsic mechanisms of the proposed FoMo-0D.
The paper proposes to pretrain a foundation model on synthetic data for outlier detection, which utilizes in-context learning for the downstream detection tasks without hyperparameter tuning or post-training. The authors leverage the "router mechanism" to facilitate architecture scaling up and random linear transformation to speed up the synthesis process.
1. While FoMo-0D doesn't require training on specific real datasets, it has inferior inference time compared to several baselines in Table 3, due to the forward pass of the whole training dataset during inference. If the training dataset scales up, there will be a high computation load during inference, undermining FoMo-0D's efficiency in time-critical inference scenarios.
@inproceedings{
anonymous2024zeroshot,
title={Zero-shot Outlier Detection via Synthetically Pretrained Transformers: Model Selection Bygone!},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=gRXLa6LS3J},
note={under review}
}
Outlier detection (OD) has a vast literature as it finds numerous applications in
environmental monitoring, security, manufacturing, and finance to name a few.
Being an inherently unsupervised task, model selection is a key bottleneck for OD
(both algorithm and hyperparameter selection) without label supervision. There is
a long list of techniques to choose from – both classical algorithms and deep neural
architectures – and while several studies report their hyperparameter sensitivity, the
literature remains quite slim on unsupervised model selection—limiting the effective use of OD in practice. In this paper we present FoMo-0D, for zero/0-shot OD
exploring a transformative new direction that bypasses the hurdle of model selection
altogether (!), thus breaking new ground. The fundamental idea behind FoMo-0D is
the Prior-data Fitted Networks, recently introduced by Müller et al. (2022), which
trains a Transformer model on a large body of synthetically generated data from a
prior data distribution. In essence, FoMo-0D is a pretrained Foundation Model
for zero/0-shot OD on tabular data, which can directly predict the (outlier/inlier)
label of any test data at inference time, by merely a single forward pass—making
obsolete the need for choosing an algorithm/architecture and tuning its associated
hyperparameters, besides requiring no training of model parameters when given a
new OD dataset. Extensive experiments on 57 public benchmark datasets against
26 baseline methods show that FoMo-0D performs statistically no different from the
2nd top baseline, while significantly outperforming the majority of the baselines,
with an average inference time of 7.7 ms per test sample.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
zero-shot outlier detection prior-data fitted networks
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/71da07b5f6d7c26935ecec1bf2eff14a7c2555ca.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Zero-shot Outlier Detection via Synthetically Pretrained Transformers: Model Selection Bygone!
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission