ProtoLLM: Training and Example-free LLMs for Few-shot Tabular Learning
Tabular data;Few-shot learning;Large language models
Yes
No ethics review needed.
The questions are in the weakness part.
**Strength**

1. The overall performance is strong.

2. The method may offer novelty from one perspective: generating features with LLMs already exists [1], but using them additionally for building prototypes is relatively new.

**Reference**\
[1] Large Language Models Can Automatically Engineer Features for Few-Shot Tabular Learning, ICML 2024
This paper suggests generating individual tabular feature values with LLM (which they denote as Oracle feature generation) per class, then building a prototype, which is used to build a non-parametric classifier. This prototype can be jointly used with few-shot samples by building prototypes with few-shot samples (i.e., averaged feature value of few-shot samples). The author shows that the method is effective for zero to low-shot (e.g., 64-shot) classification for tabular dataset.
**Weakness**

1. The writing lacks clarity and is not mathematically formulated. For example, there are several questions:
- What is the feature value z? This value should be mathematically defined. Currently, it appears to be the feature generated by the LLM. However, it is not clearly defined, making it difficult for readers to understand.
- In Figure 3, 4, can the authors clarify what each color represents?
- Why is the LLM-generated feature referred to as "Oracle"? "Oracle" is a term that requires careful use; I would suggest "LLM-generated feature" instead, as the LLM is not an Oracle.

2. Most of the datasets focus on binary classification, which can be considered a toy example (also, how did the authors compute AUC for the car dataset, which seems to be a multi-class classification task?). The authors should consider including more multi-class classification datasets, such as those in STUNT [1]. Extending this approach to regression tasks is also essential, as it’s a significant application area for tabular learning.

3. The experiment section could be strengthened.
- Variance should be reported for few-shot (or low-shot) learning experiments, as stability is also a crucial aspect of the work.
- Reporting full-shot accuracy and discussing limitations in this setup would add valuable insights.

4. There is a concern regarding dataset contamination and whether the method will work on truly novel datasets. While it’s acknowledged that the authors have considered two datasets new to the LLM, tabular datasets often contain novel features (e.g., from fields like factory settings or biology). It is unclear if this method would generalize well to such data.

5. There are existing works focused on feature generation that can be combined with other effective methods (e.g., XGBoost) to enhance performance further [2,3,4,5]. Typically, in this context, few-shot learning is not the primary focus, as it is somewhat constrained or limited in application.

6. Lack of comparison with LLM used Tabular learning frameworks [6,7,8].

**Summary**\
The paper has certain strengths; however, currently, the weaknesses are more prominent. In this context, I respectfully request the authors to address these issues during the rebuttal.

**Reference**\
[1] STUNT: Few-shot Tabular Learning with Self-generated Tasks from Unlabeled Tables, ICLR 2023\
[2] Large Language Models for Automated Data Science: Introducing CAAFE for Context-Aware Automated Feature Engineering, NeurIPS 2023\
[3] Generalized and Heuristic-Free Feature Construction for Improved Accuracy, SIAM Data Mining 2010\
[4] Learning a Data-Driven Policy Network for Pre-Training Automated Feature Engineering, ICLR 2023\
[5] OpenFE: Automated Feature Generation with Expert-level Performance, ICML 2023\
[6] LIFT: Language-Interfaced Fine-Tuning for Non-Language Machine Learning Tasks, NeurIPS 2022\
[7] Tabular Transfer Learning via Prompting LLMs, COLM 2024\
[8] Language models are weak learners, NeurIPS 2023
Yes
No ethics review needed.
* How does the method deal with continuous features? Is it similar to FeatLLM?
* How many tokens are consumed for an average iteration of ProtoLLM?
* How is the distance function used in ProtoLLM? What is meant by 'logistics' in this sentence: "All we need to calculate the logistics is the Euclidean distance between the input samples and obtained prototype"?
* The authors provide some intriguing indications that LLMs can solve certain tabular classification problems which are well specified using natural language zero-shot, simply by predicting likely feature values for each class.
* The authors include some helpful ablation studies and appendix material, including a complete results table.
* The choice of few-shot baseline methods is comprehensive and strong.
Conventional machine learning models for tabular data rely on labeled samples, and performance tends to degrade when few samples are available. It has recently been argued that LLMs may learn useful information for certain tabular classification tasks during pretraining. This may make them superior classifiers in the extreme few-shot setting (which, for tabular data, could roughly be classified as fewer than 100 labeled datapoints available for training). TabLLM and FeatLLM constitute two important precursor works in this line of research.

The authors of ProtoLLM remove the few-shot examples from the prompt template and prompt LLMs to generate oracle feature values according to the LLM's inherent knowledge about the given tasks. They query LLMs feature by feature, rather than generating all features of a tabular sample simultaneously. Instead of generating a single value for each feature, they query LLMs K = 10 times for each feature across all datasets. Results are provided for zero-shot and few-shot.

The authors claim their method leads to robust and superior performance on a range of datasets.
MAJOR

* This paper argues for the scientific relevance of a method whose main results are demonstrated only on ChatGPT, a closed-source LLM. Closed models make bad baselines, are not reproducible, and bias future research [https://hackingsemantics.xyz/2023/closed-baselines/]. To make the claim that ProtoLLM is significant, it is necessary to show that it is doing something qualitatively different which improves results while controlling for model size, the type and amount of data, and important hyperparameters. Since we don’t have any of this information for GPT, it is not possible to claim ProtoLLM improves on prior work; apparent improvements could simply reflect a quirk of the chosen GPT checkpoint, for example. At a bare minimum, the authors should show results on a small open-source LLM such as T0 or LLAMA-3-8B in addition to GPT.
* The choice of datasets is small and non-standard; using a large established benchmark in the literature would lend more credibility to the findings [https://arxiv.org/abs/2305.02997]. Data contamination is a major concern here, as most of the datasets included are old. On the two datasets where data contamination is less likely (NHANES and Cultivars), a simple logistic regression baseline with just 64 data points beats ProtoLLM.
* Some datasets in Fig. 5 have missing baselines.
* Something is wrong with the formatting of this paper; line numbers don't align with the lines.
* There are few zero-shot baselines provided; DSPy would have been a natural choice [https://github.com/stanfordnlp/dspy]. The one zero-shot baselines that is provided (TabLLM GPT-3.5) beats ProtoLLM on 4 of 9 datasets; given that the authors argue their contributions are most significant in the zero-shot regime, this is troubling.
* TabLLM GPT-3.5 results should be made available for few shot as well as zero-shot for a fair comparison between methods.

MINOR

* Fig. 5 is hard to read; it should be larger.
* Please provide links to the datasets used in the appendix; there are many versions of these datasets in common use.
* Many typos (lines 283, 300, Table 1, Sec. 4.1 header, line 337, etc).
Yes
No ethics review needed.
## Typos:
Authors might consider going over their manuscript to correct typos which occur quite frequently. In particular, citations should be placed after a word and separated with by a whitespace (e.g. line 346 "*STUNT(Nam et al., 2023),*" -> "*STUNT (Nam et al., 2023),*").


## Questions
- **(i)** The $\tau$ parameter in equation $3$ appears critical. **How is $\tau$ selected ?**
- **(ii) Have the authors investigated the impact of the number of paramaters of the base LLM on the overall performance?** While one would (of course) expect the performance to drop as the the number of parameters of the LLM decreases, it might be interesting to analyse the relationship between performance and number of parameters. In particular, if the performance drop is small between large models and smaller models (e.g. 13B parameter models), this might prove to be a significant argument in favor of ProtoLLM.
- **(iii)** As mentionned as a weaknesses of the proposed method, generating the "*golden prototypes*" requires several iterations that may cause a significant "*training*" time. The authors should consider comparing the overall time required to train the competing models vs the time required to construct the golden prototypes used in inference.
- **(iv)** In figure 5 as the reported results are averaged for different seeds, I would be curious to see the figures including the standard deviations/confidence intervals to assess statistical significance of performance gains. **Have the authors performed any statistcal tests (e.g. Wilcoxon signed-rank test) to compare performance with the other best performing alternative?**
- **(v)** As mentionned as a weakness of the proposed approach, ProtoLLM cannot directly be applied for regression. **Have the authors considered modifications/adaptation to the proposed setup to possibly perform regression tasks?** If so, could they share their opinion on the faisability?
- **(vi)** In [1], the authors analyze the impact of spurious correlation on the performance of their proposed method in comparison to existing methods. While ProtoLLM works for zero-shot classification, **have the authors analyzed the effect of including those noisy samples when generating the golden prototypes in the few-shot setting?**
- **(vii)** I might be interesting to analyze how ProtoLLM performs on datasets that have a high number of features, e.g. Arrhythmia (https://archive.ics.uci.edu/dataset/5/arrhythmia) or other tabular datasets the authors might think of.
- **(viii)** For the few-shot settings, to assess the added value of the proposed method, one may consider the following setting: For each class, construct the *prototype* using **only** the samples and excluding the prototype generated by the LLM. That way one could assess the explicit added value of the prototype in the few-shot setting as done for the zero-shot setting. 

Given the listed strength and weaknesses, as well as questions and interrogations listed above, we lean towards reject. However, we are open to discussion and would happily increase our score if the authors address our concerns and interrogations. 

[1] Sungwon Han, Jinsung Yoon, Sercan O Arik, and Tomas Pfister Large language models can automatically engineer features for few-shot tabular learning. In Forty-first International Conference on Machine Learning, 2024.
- The proposed method does not require any training and inference is perfomed only based on a distance measure between the sample of interest and the generated prototypes.
- The proposed approach is relevant for both zero-shot and few-shot learning set-ups.
- Proposed approach has (to our knowledge) significant novelty.
- The experiments are extensive and fair, and show that ProtoLLM offers strong performance on the tested benchmark. 
- Ablation study is rather exhaustive and investigates most of the relevant factors.
- Work is reproductible.
This paper presents ProtoLLM a novel approach to leverage LLM for supervised classification on tabular data. More precisely, the presented method focuses on few-shot and zero-shot learning for tabular data.

The authors observe that that few-shot learning of LLMs for supervised tasks on tabular data can perform poorly if the few examples fed to the model are not properly selected. The authors thus propose to generate a prototype solely based on the existing knowledge of the LLM through careful prompting, and proceed to use this prototype in a training free maner to compute a prediction for a sample of interest. Their approach can also be extended to few-shot learning by aggregating existing examples with the generated prototype (e.g. averaging the feature values). 

The authors test their approach on a tabular data benchmark composed of 10 datasets following previous work, and compare to existing methods. They show that their approach performs strongly based on the AUROC metric and also perform several ablations studies to further demonstrate the relevance of their proposed method.
- The paper is sometimes hard to follow and wordy. The overall style could be improved, and part of the paper contains either typos or odd turns of phrase (e.g. line 356:"*To validate the seasoning ability of our ProtoLLM on tabular data*", line 300: "*where $\tau$ is a temperature hype-parameter*", line 381: "*We attribute this superior to our example-free prototype generation*".).
- While the proposed approach is novel and interesting, the **contribution is relatively thin** for a conference like ICLR.
- The proposed approach requires prompting an LLM $d \times K \times C$ times, where $d$ is the number of features, $K$ the number of iteration considered to construct the prototype, and $C$ the number of classes. This might restrains this type of approach to datasets with reasonable number of features or classes. Moreover, depending on the set-up, this might be very costly for tabular data classification when considering hardware requirements to run LLMs.
- the proposed approach seems restrained to applications on which LLMs may have relevant knowledge, hence not too specialized or novel. While this out of the scope of this work, it might be interesting to investigate whether humans may generate prototypes as relevant as the LLMs on some datasets.
- The proposed approach is restrained to classification and cannot be directly adapted for regression tasks.
We use LLMs to generate oracle features for tabular data without relying on examples, thereby boosting few-shot performance through a training-free approach.
@inproceedings{
anonymous2024protollm,
title={Proto{LLM}: Training and Example-free {LLM}s for Few-shot Tabular Learning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=kymuzakf7V},
note={under review}
}
Recent breakthroughs in large language models (LLMs) have opened the door to in-depth investigation of their potential in tabular data modeling. However, the paradigm for effectively utilizing advanced LLMs in few-shot and even unseen scenarios remains to be explored. We observed an unusual phenomenon: directly using LLMs for data augmentation or rule generation by feeding a few examples significantly degrades the reasoning ability in tabular data understanding. We identified two main obstacles behind this issue: overfitting to the examples and knowledge disruption. Specifically, the provided examples may introduce noisy patterns that interfere with the model's prior knowledge, leading to unexpected and less reliable results. To this end, we propose an example-free framework to leverage the inherent knowledge of LLMs. Our key idea is to prompt the LLM for oracle feature generation based solely on task and feature description. Without such example pollution, each output feature is treated as a standard guideline, and they together act as a prototype for each class. To transfer the LLM's knowledge to a given task, we further design an efficient fusion strategy to integrate the prototype with example features, showing impressive generalizability in the few-shot setting. Importantly, our pipeline requires no learnable variables, resulting in a desired training-free property. Extensive comparisons and ablations on multiple tabular datasets demonstrate the improvements of our simple framework.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
Tabular data Few-shot learning Large language models
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/90758bd0fd5ba2947ec3aae112192f25453dcda3.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/dba42b30ed689620b9da1bc3a9ec56e560c5d1ed.zip
ProtoLLM: Training and Example-free LLMs for Few-shot Tabular Learning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission