DEALING WITH OUT OF DISTRIBUTION IN PREDICTION PROBLEM
representation learning;tabular data;out of distribution
```
are not appropriate for this study, as these datasets do not display distribution shifts,
```
We deliberately use a common database. The reason behind this is In the real world, users can only use the databases that are available. For that reason, we also cover detection and steps to be taken for OOD.
```
Many of the main claims in the abstract lack support or are difficult to verify -- for example, "Addressing OOD data requires extensive fine-tuning and experimental trials" and "Deep learning has been sug- gested as a solution and has shown significant improvements".
```
We will add this in our revision. Thank you.

```
 The paper is missing references to many relevant works, including:
```
We will add this in our revision. Thank you.

```
The paper is also missing references to several relevant empirical studies for tabular data, including:
```
Our work focuses on how to detect and predict out-of-distribution. TCL is part of the paper but not all about TCL. Our work uses a regular dataset, predicts OOD and proof, and predicts classification or regression problems on it. Adding this will max out page limitation or deduct technical details of our paper. 

```
The paper frames its use of contrastive learning as a novel contribution, however, there are two problems with this:
...
The paper simply applies an existing method (Contrastive Federated Learning, or CFL) to tabular data. This is not sufficient for acceptance, given the concerns with the empirical evaluations described above.
```
CFL focuses on a Federated learning network, while ours is common tabular data. In CFL, while exhibiting similar name,  they use partial data augmentation as part of federated learning concept. In our CFL we use a complete full raw matrix representation. Our TCL also did not came from base understanding of CFL that mention about data in a silo similar to partial data augmentation in original contratstive learning for image. TCL is part of the paper, the whole paper is A-Z discussing about OOD treatment for general user.
```
The experimental results are not particularly impressive. While TCL shows high efficiency, it only achieved optimal performance on three datasets compared to FT-T and ResNet.
```
We have done follow-up experiments with better results. We will upload it in the next revision. Thank you

```
The paper lacks a comprehensive comparison of methods. It would benefit from including comparisons with GBDT methods (e.g., CatBoost[1], XGBoost[2]), other state-of-the-art deep learning models (e.g., TabR[3], ExcelFormer[4], Trompt[5]), and alternative self-supervised learning algorithms (e.g., Scarf[6], VIME[7]) besides SubTab.
```

We have added GBDT to our experiment. We will release it in our revision We can not include every available method due to our limitation. We will add Scarf in our next revision. We will consider others, but probably for the next paper. thank you


```
More details should be included in the paper, such as the specific structure of TCL, the downstream classifier used, hyperparameters of the comparison methods, and whether experiments were conducted multiple times to present average results.
```

This work focuses on how to detect and predict out-of-distribution. TCL is part of the paper but not all about TCL. Our work uses a regular dataset, predicts OOD and proof, and predicts classification or regression problems on it. This is what real-world users actually seek. In addition, we add TCL as a prediction tool for general users without GPU.

```
The experiment comparing dot product and Euclidean distance indicates that dot product is more efficient on the hardware they used. However, this work does not discuss how using these two distance metrics impacts the performance of TCL and how significant the computational time for distance calculation is within the entire 
```

Our distinctive feature is the use of a full matrix representation in the design of TCL. This approach eliminates the need for data slicing and matrix multiplication. To optimize for CPU compatibility, we refine the loss function by employing only dot matrices and narrow layers. The core concept of contrastive learning involves adding noise and enhancing generalization. By applying this strategy, we can achieve performance comparable to larger models that typically demand more powerful hardware.


```
TCL process.What criteria did the authors use to select different detectors for OOD detection across various datasets?
```
We tried many detection algorithms, but we only used algorithms that can empirically be proven by the next step mentioned in our paper after detection. 

```
Authors may want to assess the performance of TCL using the recently published benchmark, TabRed [1], which is a Benchmark of Tabular Machine Learning in-the-Wild with Real-World Industry-Grade Tabular Datasets.
```
We will definitely consider this in our revision, but for now, we will add Scraf as the method gain more popularity.
```
The paper presentation should be improved. In the abstract, for instance, the issues in existing OOD works do not appear novel. Extensive fine-tuning and experimental trials are not unique to this specific OOD problem but relate to many general issues. The authors need to outline issues more specific to the OOD problem.
```

We will revise this later. Thank you for your input.

```
The review of related works requires significant improvement. There is a substantial body of existing work on OOD in tabular domains; however, the authors only discuss two methods in Sec. 2.1 and one other in L38, with the most recent paper referenced from 2017. The authors should put more effort into the literature review, for example, [1].
```

Our related work is actually more than this; however, we've done a substantial page deduction due to the page limit. Thank you; we will add this to the revision.

```
The proposed TCL employs a common strategy: contrastive learning with two views generated from a sample for self-supervised learning, followed by a head for classification/regression. This framework has been used in various contrastive learning-based tabular models, such as SubTab, which modifies contrastive targets with slicing techniques for tabular structures. The performance is not superior to the selected baselines, which limits both the novelty and effectiveness of the proposed approach.
```

We are using full matrix representation. We have done an upcoming experiment with better results, and we will release it this week. We also include XGB, Catboost and Lightgbm.


```
The selected baselines are based on the work of Gorishniy et al., 2021, which was published in 2021. Since then, many advanced tabular prediction models have emerged and should be included in this paper, such as those summarized in [2]. Note that some of these models serve as tabular foundation models and may not experience the OOD issues described in this paper.
```

We started this project in January, while the survey paper was listed in February. We are not aware of this. Our goal is to develop an algorithm that is light enough to be used by a general user without a GPU but capable of nearing or exceeding the capability of a user with a GPU. The methods mentioned in the survey paper are based on MLP and transformer. While we do not cover the methods mentioned in the survey paper, we include base MLP and Transformer.


```
The authors used CPU(s) to train their model but used an H100 GPU for the baselines, which is confusing. I didn't see any specific design element that would make the proposed method more efficient; on the contrary, matrix augmentation appears to require more computational resources. Although Table 4 presents training durations, it is unclear how this table was created. For example, is the parameter space the same across all models?
```

Our distinctive feature is the use of a full matrix representation in the design of TCL. This approach eliminates the need for data slicing and matrix multiplication. To optimize for CPU compatibility, we refine the loss function by employing only dot matrices and narrow layers. The core concept of contrastive learning involves adding noise and enhancing generalization. By applying this strategy, we can achieve performance comparable to larger models that typically demand more powerful hardware. We will add parameters specification on the next revision.
```
Why is the Contrastive Federated Learning (CFL) reference in an anonymized link? 
```

-> It is due to the anonymized rule, correspondence to one or more author of this paper.


```
Why does the title miss some keywords like Tabular Contrastive Learning? 
```

-> This work focuses on how to detect and predict out-of-distribution. TCL is part of the paper but not all about TCL. Our work uses a regular dataset, predicts OOD and proof, and predicts classification or regression problems on it. This is what actually real-world users seek. 


```
Why is there no experiment for baseline models running on CPU (Apple) to fairly compare with your TCL?
```

-> According to the original paper, baseline models are best performed within GPU. To support that, we follow the original paper's architecture to fit their best setting. We have a follow-up experiment with a CPU with better results. Our point is to show that with only a consumer CPU, we can achieve similar results with advanced models trained with GPU.
Yes
No ethics review needed.
See above.
# Overall assessment

In general, the writing in the paper could be improved considerably. The paper is missing references to many relevant works in the tabular prediction literature, and fails to compare to many relevant baselines (most notably, GBDT methods like XGBoost, LightGBM, and CatBoost). The datasets selected for the empirical comparisons are not appropriate for this study, as these datasets do not display distribution shift, while the authors do not use or acknowledge several existing benchmarks for OOD prediction in tabular data.

# Major comments

* Many of the main claims in the abstract lack support or are difficult to verify -- for example, "Addressing OOD data requires extensive fine-tuning and experimental trials" and "Deep learning has been sug-
gested as a solution and has shown significant improvements".

* The paper is missing references to many relevant works, including:
  - Malinin, Andrey, et al. "Shifts 2.0: Extending the dataset of real distributional shifts." arXiv preprint arXiv:2206.15407 (2022). 
  - Gardner, Josh, Zoran Popovic, and Ludwig Schmidt. "Benchmarking distribution shift in tabular data with tableshift." Advances in Neural Information Processing Systems 36 (2024).
  - Liu, Jiashuo, et al. "On the need for a language describing distribution shifts: Illustrations on tabular datasets." Advances in Neural Information Processing Systems 36 (2024).

  All three of the above papers propose benchmarks for OOD-related tabular tasks. These would be much more appropriate than the 10 toy datasets selected for the empirical studies.

* The paper is also missing references to several relevant empirical studies for tabular data, including: 
  - Gardner, Josh, Zoran Popovic, and Ludwig Schmidt. "Subgroup robustness grows on trees: An empirical baseline investigation." Advances in Neural Information Processing Systems 35 (2022): 9939-9954.
  - Grinsztajn, Léo, Edouard Oyallon, and Gaël Varoquaux. "Why do tree-based models still outperform deep learning on typical tabular data?." Advances in neural information processing systems 35 (2022): 507-520.
  - Kadra, Arlind, et al. "Well-tuned simple nets excel on tabular datasets." Advances in neural information processing systems 34 (2021): 23928-23941.

  In particular, the first two studies above suggest that GBDTs are perhaps the most appropriate baseline for this study, but they are omitted from the study completely.

* The paper frames its use of contrastive learning as a novel contribution, however, there are two problems with this:
  1. Contrastive learning has already been widely used for tabular data; See e.g. (Bahri, D., Jiang, H., Tay, Y., & Metzler, D. (2021). Scarf: Self-supervised contrastive learning using random feature corruption. arXiv preprint arXiv:2106.15147.) and the survey of (Rabbani, Shourav B., Ivan V. Medri, and Manar D. Samad. "Attention versus Contrastive Learning of Tabular Data--A Data-centric Benchmarking." arXiv preprint arXiv:2401.04266 (2024).) In particular I would note that its performance has generally not been competitive with strong baselines and adoption of the method has been limited as a result.
  2. The paper simply applies an existing method (Contrastive Federated Learning, or CFL) to tabular data. This is not sufficient for acceptance, given the concerns with the empirical evaluations described above.

# Minor comments

* More detail should be provided on the datasets in the main text (note that I suggest completely changing the datasets to more appropriate distribution shift datasets). In particular, the authors should explain why these generic tabular datasets (e.g. Adult, California Housing) are appropriate for studying the very specific task of OOD detection. Some descriptive statustucs about the datasets would also be useful.

# Typos etc.

* The paper is quite difficult to read in places due to grammatical issues. I would suggest having a native English speaker proofread the manuscript, or using a grammar checking tool.
* Definition 4: "eucledian"
The paper applies an existing method (Contrastive Federated Learning, or CFL) to tabular data prediction tasks. The authors use 10 toy tabular datasets to evaluate this method and compare with a set of deep learning-based baselines on various classification and regression metrics.
See above.
Yes
No ethics review needed.
- Why is the Contrastive Federated Learning (CFL) reference in an anonymized link?
- Why does the title miss some keywords like Tabular Contrastive Learning?
- Why is there no experiment for baseline models running on CPU (Apple) to fairly compare with your TCL?
+ Multiple Dataset and Models

+ Impressing Experiments and Results

The manuscript tackles an important challenge in machine learning by focusing on out-of-distribution (OOD) data handling for tabular datasets, and introduces Tabular Contrast Learning (TCL) as a novel solution. This is a well-defined problem with significant practical implications, especially for general users with computational limitations. The authors have provided a clear motivation for their work, particularly in making OOD detection and handling accessible without high-end hardware requirements. The proposed TCL method appears promising for addressing efficiency and accuracy in tabular OOD tasks.
This study focuses on detecting and predicting OOD data in tabular datasets. It provides methods for identifying OOD data, guidance on evaluating OOD selections, and introduces Tabular Contrast Learning (TCL), a technique optimized for tabular predictions. 
TCL seems to be more computationally efficient than baseline models, making it suitable for general users with limited computational power. The study also compares TCL with existing approaches, emphasizing both accuracy and efficiency.
However, it would benefit from a more detailed comparison with recent advancements in tabular contrastive learning. Notably, two works could provide relevant baselines and enhance the context for TCL.
- Best of Both Worlds: Multimodal Contrastive Learning With Tabular and Imaging Data, CVPR23
- TabContrast: A Local-Global Level Method for Tabular Contrastive Learning, NIPS23
Yes
No ethics review needed.
1. See weaknesses
2. What criteria did the authors use to select different detectors for OOD detection across various datasets? 
3. Authors may want to assess the performance of TCL using the recently published benchmark, TabRed [1], which is a Benchmark of Tabular Machine Learning in-the-Wild with Real-World Industry-Grade Tabular Datasets.

[1] Ivan Rubachev, Nikolay Kartashev, Yury Gorishniy, Artem Babenko: TabReD: A Benchmark of Tabular Machine Learning in-the-Wild.
1. The research addresses an important issue in machine learning—handling OOD data, which is crucial for ensuring model reliability and robustness.
2. TCL demonstrates a very efficient approach that reduces computational costs, making it viable for deployment in environments with limited resources.
The article presents a method called tabular contrast learning (TCL) aimed at improving model performance on tabular datasets, particularly in the context of Out-of-Distribution (OOD) tabular data. The authors highlight TCL's efficiency in reducing computational costs and its potential for deployment in resource-limited environments. The paper discusses the model's training process and experimental results across 10 datasets.
1. The experimental results are not particularly impressive. While TCL shows high efficiency, it only achieved optimal performance on three datasets compared to FT-T and ResNet.
2. The paper lacks a comprehensive comparison of methods. It would benefit from including comparisons with GBDT methods (e.g., CatBoost[1], XGBoost[2]), other state-of-the-art deep learning models (e.g., TabR[3], ExcelFormer[4], Trompt[5]), and alternative self-supervised learning algorithms (e.g., Scarf[6], VIME[7]) besides SubTab.
3. More details should be included in the paper, such as the specific structure of TCL, the downstream classifier used, hyperparameters of the comparison methods, and whether experiments were conducted multiple times to present average results.
4. The experiment comparing dot product and Euclidean distance indicates that dot product is more efficient on the hardware they used. However, this work does not discuss how using these two distance metrics impacts the performance of TCL and how significant the computational time for distance calculation is within the entire TCL process.

[1] Liudmila Ostroumova Prokhorenkova, Gleb Gusev, Aleksandr Vorobev, Anna Veronika Dorogush, and Andrey Gulin. Catboost: unbiased boosting with categorical features. In NeurIPS

[2] Tianqi Chen and Carlos Guestrin. Xgboost: A scalable tree boosting system. In KDD

[3] Yury Gorishniy, Ivan Rubachev, Nikolay Kartashev, Daniil Shlenskii, Akim Kotelnikov, and Artem Babenko. Tabr: Tabular deep learning meets nearest neighbors in 2023. In ICLR

[4] Jintai Chen, Jiahuan Yan, Danny Ziyi Chen, Jian Wu: ExcelFormer: A Neural Network Surpassing GBDTs on Tabular Data. In KDD

[5] Kuan-Yu Chen, Ping-Han Chiang, Hsin-Rung Chou, Ting-Wei Chen, Tien-Hao Chang: Trompt: Towards a Better Deep Neural Network for Tabular Data. In ICML

[6] Dara Bahri, Heinrich Jiang, Yi Tay, Donald Metzler:Scarf: Self-Supervised Contrastive Learning using Random Feature Corruption. In ICLR

[7] Jinsung Yoon, Yao Zhang, James Jordon, Mihaela van der Schaar: VIME: Extending the Success of Self- and Semi-supervised Learning to Tabular Domain. In NeurIPS
Yes
No ethics review needed.
Please see above comments.
1. The proposed method is easy to understand, with a clear problem statement and definition.
2. Simaltaneously considering the OOD problem with efficiency may be an interesting direction, especially in the realm of large-scale models.
This paper tackles the OOD issue for tabular data by using a framework consisting of contrastive learning. To separate in-domain and out-of-domain samples for experiments, the authors adopt OpenMax and Temperature Scaling with manually assigning the thresholds. The proposed TCL shows higher speed-accuracy trade-off scores, which was only trained on a CPU.
Generally, this paper requires extensive improvements in terms of motivations, related works, proposed method, as well as the experiments. Detailed comments are provided below:

1. The paper presentation should be improved. In the abstract, for instance, the issues in existing OOD works do not appear novel. Extensive fine-tuning and experimental trials are not unique to this specific OOD problem but relate to many general issues. The authors need to outline issues more specific to the OOD problem.
2. The review of related works requires significant improvement. There is a substantial body of existing work on OOD in tabular domains; however, the authors only discuss two methods in Sec. 2.1 and one other in L38, with the most recent paper referenced from 2017. The authors should put more effort into the literature review, for example, [1].
3. The proposed TCL employs a common strategy: contrastive learning with two views generated from a sample for self-supervised learning, followed by a head for classification/regression. This framework has been used in various contrastive learning-based tabular models, such as SubTab, which modifies contrastive targets with slicing techniques for tabular structures. The performance is not superior to the selected baselines, which limits both the novelty and effectiveness of the proposed approach.
4. The selected baselines are based on the work of Gorishniy et al., 2021, which was published in 2021. Since then, many advanced tabular prediction models have emerged and should be included in this paper, such as those summarized in [2]. Note that some of these models serve as tabular foundation models and may not experience the OOD issues described in this paper.
5. The authors used CPU(s) to train their model but used an H100 GPU for the baselines, which is confusing. I didn't see any specific design element that would make the proposed method more efficient; on the contrary, matrix augmentation appears to require more computational resources. Although Table 4 presents training durations, it is unclear how this table was created. For example, is the parameter space the same across all models?

[1] Multi-Class Data Description for Out-of-distribution Detection. KDD 2020.

[2] A Survey on Self-Supervised Learning for Non-Sequential Tabular Data. ACML 2024.
@inproceedings{
anonymous2024dealing,
title={{DEALING} {WITH} {OUT} {OF} {DISTRIBUTION} {IN} {PREDICTION} {PROBLEM}},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=i28ZjVxl81},
note={under review}
}
Open world assumption in model development means that a model may not have enough information to effectively handle data that is completely different or out of distribution (OOD). When a model encounters OOD data, it may suffer a significant decrease in performance. Addressing OOD data requires extensive fine-tuning and experimental trials, which in turn require substantial computational resources. Deep learning has been suggested as a solution and has shown significant improvements, but it often requires high-specification hardware, particularly GPUs, which may not always be readily available to general users. Additionally, there is a lack of clear guidance for common users on how to select and evaluate OOD data.
This study delves into detection, evaluation, and prediction tasks within the context of OOD on tabular datasets. It demonstrates how common users can identify OOD data from real datasets and provides guidance on evaluating the OOD selection through experiments and visualizations. Furthermore, the study introduces tabular contrast learning (TCL), an enhanced technique specifically designed for tabular prediction tasks. TCL is more efficient compared to other baseline models, making it useful for general machine learning user  with computational limitation on dealing with OOD problems. The study also includes a comprehensive comparison with existing approaches, focusing on both accuracy and computational efficiency.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
representation learning tabular data out of distribution
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/93a26a44c87ade7ad886eac8d7e9bc32dad39914.pdf
unsupervised, self-supervised, semi-supervised, and supervised representation learning
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/53c01024cfd24201613e6df9d06a2909b510ac19.zip
DEALING WITH OUT OF DISTRIBUTION IN PREDICTION PROBLEM
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission