AnoLLM: Large Language Models for Tabular Anomaly Detection
Anomaly detection;tabular data;large language models
Yes
No ethics review needed.
- What is an example of an anomaly detection in tabular data? 
- How does the time comparison look for the baselines? 
- What is "50% of normal samples for training"? It seems that the proposed model is unsupervised and 50% is used to train the unsupervised loss, but what does it mean for other models?
- Have the authors considered dealing numerical columns separately? If LLM falls behind in modeling numerical values, it would be interesting to a simple concatenation of numerical values with the output of 'serialization + LLM' and applying the proposed loss.
- Have the authors considered other encoding methods for categorical variables? (e.g., Target Encoder or the package of Skrub)
- The paper is generally well-written and easy to follow.
- The paper provides theoretical grounds on their model and shows competitive performances compared to other baselines.
- The paper provides some interesting insights on anomaly detection on tabular data with large-language models.
The paper proposes AnoLLM, a large language model (LLM) based framework for unsupervised tabular anomaly detection. It utilizes serialization of tabular data into sentences, and finetunes an LLM with the causal language modeling loss. The model have been applied to numerous datasets and shows competitive performances compared to other baselines.
- An example for anomaly detection (in Figure 1) would be good for reader's understanding of what is an anomaly detection for tabular data. (Possibly an example explicitly included in the dataset used for the experiments)
- From the results, it is difficult to determine which model performs the best without explicit standard deviation (which is provided in the appendix). It would be better to have some simple plots (e.g., critical difference plots) that incorporates some statistical testing to determine which model is performing better.
- While the authors state there is a similar trends for other metrics, it would be good to see the actual results (at least in the appendix), since there is a high imbalance of normal/anomaly.
- The time comparison between the models and the discussion on the trade-off would be very interesting.
- It is confusing to see "50% of normal samples for training" in the evaluation protocol. Clearer description would help better understanding of the experiments.
- One interesting direction for the future works might be to pretrain a model suited for the anomaly detection. Moreover, more elaboration on impacts for the proposed model would be interesting for the conclusion.
Yes
No ethics review needed.
W1, W2, W3
S1. The problem statement is innovative.

S2. The paper is quite well written. It is easy to follow and logical.

S3. The results are good.

S4. Instead of simply using large LLMs, small variants are explored, and it is shown that they are no less better.
The paper proposes an innovative use of LLMs, that of detecting anomalies from tabular data. It is well written, and gives good results.
W1. The effect of number of decimal digits should have been explored in greater detail.

W2. Similarly, the effect of normalization could have been explored in more detail. Although, the effect of raw numbers is seen, how about simply rounding raw number to x number of decimal digits (and not normalizing) to reduce effect of long decimal numbers, and then using them directly?

W3. What is the effect of not permuting the column names, and having a canonical ordering? Are they not supposed to give even better results?

W4. It will be good to highlight some failure cases, both false positives and false negatives, and try and analyze why that happened.
Yes
No ethics review needed.
- Did the other baseline also use the ensemble of permutations at inference time?
- What explains the discrepancy in the results of ICL in this paper vs the original paper, which was also tested on OODS?
- How did you choose hyperparameters for the baselines?
- Given the observed trend that larger pretrained models do not seem to benefit AnoLLM, this raises the question: what would happen if we trained a model from scratch using the AnoLLM framework? This feels like a natural question that should have been explored. Did you try this? This might weaken the understanding of the text feature of the model, but it would be interesting to see the impact it has on numerical values.
- What is the total computational cost of the experiments?
- Was experimenting with contaminated training data (the truly unsupervised setting) considered? I ask this because reproducing the paper is not easy, and this is also an important task for anomaly detection.
- What steps are you taking to ensure reproducibility? Will the code be released?
- *See weaknesses*.
- The paper introduces a novel model type for anomaly detection using a large language model.
- The method provides an effective way to handle text and categorical data as features for anomaly detection, which are typically challenging to manage. To do so, they proposed a method to mitigate length-bias in the LLM’s output probabilities and theoretically validated it.
- The authors demonstrate a preprocessing technique for tabular data, facilitating effective LLM fine-tuning.
- The work is easy to follow and the motivation is clear.
This paper presents a new framework, AnoLLM, for unsupervised anomaly detection by fine-tuning a pretrained large language model (LLM). The authors use a predefined template to serialize, i.e., convert tabular data into text for the LLM, along with preprocessing to mitigate limitations related to the model's autoregressive nature. They employ the negative log-likelihood across different column permutations to compute an anomaly score for each sample in the test set. The method is compared against various classical and deep learning methods on the ODDS datasets and six new datasets featuring mixed types of attributes. Overall, the approach demonstrates strong performance against baselines, particularly for datasets containing text features.
The paper claims to outperform certain deep learning methods; however, in my experience, some of these methods perform similarly or even better than KNN (which is reported to have results comparable to the proposed method). For example, ICL outperforms KNN on the ODDS benchmark (Shenkar and Wolf, 2022), as does DTE (Livernoche et al., 2024), which was cited but not included as a baseline. The use of column permutations in the paper can be seen as a sort of ensemble strategy, a technique known to slightly improve anomaly detection performance. To ensure a fair comparison, the baselines should also be evaluated using these same permutations, as Appendix C suggests that this step may not be critical, or specific, to AnoLLM. In a small test I conducted, implementing this strategy led to performance improvements in other deep learning methods as well. Including F1-score or AUC-PR results as supplemental material would be helpful, as these metrics are more sensitive to class imbalances, which are common in anomaly detection. Scoring metrics can influence the relative ranking of methods on benchmarks. This claim that AnoLLM outperform deep learning methods should be more cautiously framed.

One key limitation mentioned at the end of the paper is the computational expensiveness of the proposed method. 7 A100 GPUs were used for LLM fine-tuning, this makes it difficult for others to access this model or replicate results of the paper. Since no code is provided, it is even more challenging to verify the reported results. Most anomaly detection methods can run on basic GPUs, or only on CPUs, a significant contrast with AnoLLM. A section discussing inference and training times would help clarify this limitation. I consider this to be the paper’s biggest weakness: its most significant limitation is not addressed at all.

In anomaly detection literature, a clear distinction exists between unsupervised and semi-supervised (or uncontaminated unsupervised) anomaly detection. While we can call them unsupervised methods, since they can be applied in both context, it should be noted that the experiments were conducted in a semi-supervised setting. The distinction lies in whether the training set contains anomalies (unsupervised) or not (semi-supervised). Section 2.1 should be revised to clarify this distinction.

**Minor Comments:**

- In the introduction's first line, "specicious" should be corrected to "specious."
- There is a double colon on line 213 ("equation::").
- Figure 2’s title is missing a space between "yellow" and the parentheses.
- Please use conference or journal citations rather than arXiv versions where possible. Below is a list of those I identified:

  - Liron Bergman and Yedid Hoshen. Classification-based anomaly detection for general data. (ICLR 2020)
  - Vadim Borisov, Kathrin Seßler, Tobias Leemann, Martin Pawelczyk, and Gjergji Kasneci. Language models are realistic tabular data generators. (ICLR 2023)
  - Sungwon Han, Jinsung Yoon, Sercan O Arik, and Tomas Pfister. Large language models can automatically engineer features for few-shot tabular learning. (ICML 2024)
  - Edward J Hu, Yelong Shen, Phillip Wallis, Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang, and Weizhu Chen. Lora: Low-rank adaptation of large language models. (ICLR 2022)
  - Nayoung Lee, Kartik Sreenivasan, Jason D Lee, Kangwook Lee, and Dimitris Papailiopoulos. Teaching arithmetic to small transformers. (ICLR 2024)
  - Xuannan Liu, Peipei Li, Huaibo Huang, Zekun Li, Xing Cui, Jiahao Liang, Lixiong Qin, Weihong Deng, and Zhaofeng He. Fakenewsgpt4: Advancing multimodal fake news detection through knowledge-augmented lvlms. (MM2024)
  - Victor Livernoche, Vineet Jain, Yashar Hezaveh, and Siamak Ravanbakhsh. On diffusion modeling for anomaly detection. (ICLR 2024)
  - Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. (ICLR 2019)
  - Tomás Mikolov, Kai Chen, Greg Corrado, Jeffrey Dean. Efficient estimation of word representations in vector space. (ICLR Workshop 2013)
  - Hu Wang, Guansong Pang, Chunhua Shen, and Congbo Ma. Unsupervised representation learning by predicting random distances. (AJCAI'20)
  - Jiahuan Yan, Bo Zheng, Hongxia Xu, Yiheng Zhu, Danny Chen, Jimeng Sun, Jian Wu, and Jintai Chen. Making pre-trained language models great on tabular prediction. (ICLR 2024)
  - Tianping Zhang, Shaowen Wang, Shuicheng Yan, Jian Li, and Qian Liu. Generative table pretraining empowers models for tabular prediction. (EMNLP 2023)
  - Bingzhao Zhu, Xingjian Shi, Nick Erickson, Mu Li, George Karypis, and Mahsa Shoaran. Xtab: Cross-table pretraining for tabular transformers. (ICML 2023)
  - Yaqi Zhu, Shaofeng Cai, Fang Deng, and Junran Wu. Do LLMs understand visual anomalies? uncovering LLM capabilities in zero-shot anomaly detection. (MM2024)
Yes
No ethics review needed.
See weakness.
It’s good to consider different types in tabular data, such as textual, numerical and categorical columns.

Experiment results show good potential for AnoLLM.
This paper proposes AnoLLM, a new framework that uses large language models (LLMs) for unsupervised tabular anomaly detection. AnoLLM assigns anomaly scores based on the negative log likelihood for anomaly detection. The authors claim that AnoLLM detects anomal samples in raw features and can deal with textual and categorical features. Experimental results show that AnoLLM achieves good performance on six benchmark datasets with mixed feature types. AnoLLM is also competitive with KNN.
The method seems too simple and lack novelty. It’s easy and direct to consider to use the negative log likelihood for anomaly detection. Even though authors consider different types of columns in tabular data, the process seems to be the same.

I think it’s inappropriate to claim that you are the first to apply LLMs to tabular anomaly detection. There are many works in this area, such as “Anomaly detection of tabular data using llms”, and other corresponding works such as “LLMClean: Context-Aware Tabular Data Cleaning via LLM-Generated OFDs”, “Enhancing Anomaly Detection in Financial Markets with an LLM-based Multi-Agent Framework”.

The backbone LLM is not so famous such as Llama, Qwen, Mistral, etc. Why authors do not use these LLMs?

I don’t understand what the differences in contribution 3 and 4. It seems they are all talking about the experiment.

In experiments, only one of the datasets here has text columns. More such datasets and the dataset with more attributes (features) should be considered.

More ablation studies should be provided such as the impact of Random column permutations. Case study is also missing, I cannot recognize which samples are anomaly.

By the way, the formatting on line 249 could use some improvement.
We introduce AnoLLM, a novel framework that leverages large language models for unsupervised tabular anomaly detection.
@inproceedings{
anonymous2024anollm,
title={Ano{LLM}: Large Language Models for Tabular Anomaly Detection},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=7VkHffT5X2},
note={under review}
}
We introduce AnoLLM, a novel framework that leverages large language models (LLMs) for unsupervised tabular anomaly detection. By converting tabular data into a standardized text format, we further adapt a pre-trained LLM with this serialized data, and assign anomaly scores based on the negative log likelihood generated by the LLM. Unlike traditional methods that can require extensive feature engineering, and often lose textual information during data processing, AnoLLM preserves data integrity and streamlines the preprocessing required for tabular anomaly detection. This approach can effectively handle mixed-type data, especially those containing textual features. Our empirical results indicate that AnoLLM delivers the best performance on six benchmark datasets with mixed feature types. Additionally, across 30 datasets from the ODDS library, which are predominantly numerical, AnoLLM outperforms all other candidate deep-learning based approaches on average, and equals the performance of the top classical method, k-nearest neighbors. This study marks one of the first successful application of LLMs to tabular anomaly detection.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Anomaly detection tabular data large language models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/cf43263d37281e94ec64be5da4cbbc4381dfc0a8.pdf
other topics in machine learning (i.e., none of the above)
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/86c1ff5fc50c3aef45694a1c7e225f44abaaf9ed.pdf
AnoLLM: Large Language Models for Tabular Anomaly Detection
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission