LLMs Boost the Performance of Decision Trees on Tabular Data across Sample Sizes
Tabular data;large language models;decision trees;ensembling
Yes
No ethics review needed.
1. What is the performance with large-sized datasets, instead of small and medium sized datasets?
2. line 201 "weather" -> "whether"
3. Missing inference:
- Iida, Hiroshi, Dung Thai, Varun Manjunatha, and Mohit Iyyer. "Tabbie: Pretrained representations of tabular data." arXiv preprint arXiv:2105.02584 (2021).
- Chen, Pei, Soumajyoti Sarkar, Leonard Lausen, Balasubramaniam Srinivasan, Sheng Zha, Ruihong Huang, and George Karypis. "HYTREL: Hypergraph-enhanced tabular data representation learning." Advances in Neural Information Processing Systems 36 (2024).
- **Enhanced Performance:** LLM-Boost shows superior classification performance compared to traditional GBDTs and other ensemble methods.
- **Model Agnostic:** The boosting mechanism can be applied to various high-performing tabular architectures beyond LLMs.
- **Efficiency in Training:** Once LLM outputs are pre-computed, the training cost aligns with that of standard GBDT training.
The paper introduces the LLM-Boost algorithm, which integrates large language models (LLMs) with gradient-boosted decision trees (GBDTs) to enhance classification performance on tabular datasets. The method involves extracting LLM logits for each row of data and using these predictions to augment the GBDT model, allowing it to learn the residuals to the true labels. The authors demonstrate that LLM-Boost outperforms traditional models and other ensemble techniques across various dataset sizes, showcasing its potential in automating predictive modeling pipelines.
- **Dependence on Interpretability:** The method requires interpretable text descriptors as column headers, which may necessitate prompt engineering for some datasets.
- **Pre-computation Costs:** For very large datasets, the initial cost of pre-computing LLM outputs may become significant.
Yes
No ethics review needed.
1. Can close-sourced LLMs such as GPT-family models perform well on such tasks?
2. What is the rationale for choosing Flan-T5 for the major experiments?
1. This paper combines LLM and decision trees for tabular classification tasks. It shows certain novelty in the proposed method.
This paper combines both LLM and decision trees to do the classification task on the tabular data. The major experiment is to explore how LLM, decision tree and the combined method work across different data sizes.
1. The major baseline is quite strange. It would be better to separately show how LLM and decision tree work and treat both as the baselines, rather than mix them into one baseline 'Select'.
2. The experiment is not comprehensive. This paper simply lists the performance of different data sizes. It would be better to show some error analysis and cases to indicate when the proposed method can excel.
3. The selection of the LLM (eg, Flan-T5) is not very convincing.
Yes
No ethics review needed.
1. How does LLM-Boost perform when column headers are semantically ambiguous or incorrect? Would it still provide performance benefits over standalone GBDTs?

2. Why can’t we simply compare prediction accuracies to evaluate performance?

3. Typos: In Section 3.2, “replacing the first tree int he” → “replacing the first tree in the”.
1. **Novelty.** Demonstrating that combining LLMs with GBDTs for tabular prediction tasks in such a simple manner can be effective is a valuable contribution.

2. **Discussion on limitations and future work.** A thorough discussion of limitations and future work is often overlooked, but it is something researchers reading the paper can greatly appreciate.
The paper introduces LLM-Boost, a method that combines large language models (LLMs) and gradient-boosted decision trees (GBDTs) to improve performance on tabular data. The approach leverages LLMs' ability to interpret column headers and GBDTs' efficiency on large datasets. LLMs are used to extract predictions and residuals, which are then refined by GBDT models, leading to improvements, particularly on small to medium-sized datasets.
1. **Computational overhead.** It is unclear how practical it is to use LLM inference for tabular prediction tasks. An alternative use of LLMs in the tabular domain, which arguably is much simpler conceptually, is to use them for automatic feature engineering and training GBDTs on the augmented set of features for prediction [1, 2].

2. **Missing experiments.** Several crucial baselines, such as comparing LLM-Boost against simply increasing the number of trees in the GBDT, are missing. Additionally, it would be useful to evaluate how the method scales to datasets with significantly larger feature sets.

---
[1] Hollmann et al. “Large language models for automated data science: Introducing caafe for context-aware automated feature engineering.” (2023).\
[2] Nam et al. “Optimized Feature Generation for Tabular Data via LLMs with Decision Tree Reasoning.” (2024).
Yes
No ethics review needed.
1. Did the authors try other LLMs, such as Llama 3?

2. Why did the authors use Flan-T5 for the main experiment?

3. Can the proposed LLM-Boost be used with commercial-level LLMs like GPT-4 (which is a black-box API)?

4. Have you tried a dataset with more samples? For example, a dataset with a million samples.
1. Great motivation. I also agree that injecting linguistic context is important to improve tabular predictive models, and to that end, many researchers are trying to incorporate LLM, which is an important research direction in the tabular learning community.

2. Simple method. If I understood the paper clearly, the main idea of LLM-Boost is to change the first step of GBDT determination to LLM's prediction, which seems really simple and intuitive to me.

3. The authors conducted extensive experiments on a variety of datasets.
The authors propose LLM-Boost, a fusion of LLM and decision tree algorithms. The main motivation here is the idea that GBDT is known to be the best performing predictive model in the tabular domain, but it cannot handle linguistic context, so it would be useful to be able to inject linguistic context into GBDT.
1. Visualizations for the main results are too difficult to understand.

2. In some cases, the performance improvement is too small or even no improvement. While the authors claim that it is important to integrate LLM and GBDT as the dataset size increases, results such as Table 3 (Appendix) do not support this. When the training set size is at its maximum, the performance of LLM-Boost is almost identical to XGBoost.
We fuse LLMs and gradient-boosted decision trees for a single learner that performs well on tabular datasets of all sizes.
@inproceedings{
anonymous2024llms,
title={{LLM}s Boost the Performance of Decision Trees on Tabular Data across Sample Sizes},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=gp5tRHkz9B},
note={under review}
}
Large language models (LLMs) perform remarkably well on tabular datasets in zero- and few-shot settings, since they can extract meaning from natural language column headers that describe features and labels. In contrast to LLMs, gradientboosted decision trees (GBDTs) must learn the relationships among columns from scratch, increasing their data requirements. Meanwhile, LLMs are not competitive with GBDTs on medium or large datasets, and their scalability is capped by their limited context lengths. In this paper, we propose LLM-Boost, a simple and lightweight approach for fusing large language models with gradientboosted decision trees, which enables larger datasets to benefit from the natural language capabilities of LLMs than was previously shown. LLM-Boost outperforms both LLMs and GBDTs on a wide range of dataset sizes. We demonstrate state-of-the-art performance against numerous baselines and ensembling approaches, and we also show how to fuse GBDTs with TabPFN, a recent nonLLM model for in-context learning on tabular data. We find that this combination achieves the best performance on large datasets.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Tabular data large language models decision trees ensembling
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/7a737969ed62b0325accee684dc0d274b6be061d.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/cba538f5e706d1a91eb4c531ad834a3888ec63f5.zip
LLMs Boost the Performance of Decision Trees on Tabular Data across Sample Sizes
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission