Large Language Models Engineer Too Many Simple Features for Tabular Data
LLMs;feature engineering;bias;tabular data;automated data science
Yes
No ethics review needed.
I’m a little confused about the experimental setup regarding operations. If I understand correctly, the authors are comparing the distribution of operators generated by the LLM and by OpenFE. If OpenFE is considered ground truth, why not compare directly to OpenFE final generated feature set? For example, rather than just counting the number of times we see the operation “GroupByThanRank”, why not look at the original features that were input into this operation?
It is good to see more examples of evaluation of downstream LLM tasks "in the wild".

I appreciate that the authors were rigorous in removing datasets that were thought to be memorized or in the training data of the LLM, even though they did not have access to the training data itself.
The authors investigate how well LLMs can engineer features for tabular datasets. Specifically, they look at the frequencies of operators, and find that there is bias toward simpler features rather than more interesting or useful ones. They also evaluate on the downstream accuracy of the models trained with and without the engineered features.
To me, this doesn’t seem like an influential enough contribution. Not only is it tackling a very narrow problem, but it is also only evaluating a specific method for addressing that problem. While there is some prior work around using LLMs for feature engineering, I’m not convinced that this work’s method for feature engineering is necessarily representative of all the methods for using LLMs for this task. 

Specifically, the authors only use one prompting strategy, on a snapshot of models at the current time that this paper is being written. A few examples of people using LLMs for feature engineering are cited (Hatch, 2024; Türkmen, 2024), but it is unclear what methods these citations used– is the author’s method the same, or inspired by them? Should data scientists conclude from this paper that they should never use LLMs for feature engineering, even if they use a different method? Overall, I think this is an interesting use case to evaluate, but the work is not broad enough to be included in ICLR.

Nits:
Typo: “which is send to the LLM” → sent
Yes
No ethics review needed.
1. What is the statistical significance of the results shown in Table 3?
2. Why aren't larger models in GPT and Gemini family not explored?
The authors experimentally show the limitations of LLMs for feature engineering. The experimental setting is convincing.
The paper explores the featuring engineering powers of LLM with OpenFE as the baseline. The authors perform experiments on 27 datasets and 4 LLMs. The primary findings are the following.

1. LLM perform worse than the baseline.
2. Proprietary "Large" models perform worse than small "open" models.
1. The conclusions of the paper are along expected lines and are not surprising. A more notable contribution would be to address the limitations.
2. The statistical significance of the results is not provided.
3. The term "bias" is too strong for the problem explored. The authors can use the word "limitation".
Yes
No ethics review needed.
1. Why would I use an LLM for feature engineering anyway, if there are existing SOTA automated systems that already do it and perform much better? 
2. If your answer to #1 is that I probably wouldn't, then the main question about publishing this paper would be - why would I read a paper about the biases such a solution might have? There could be several answers (e.g., to inspire a similar analysis of LLM solutions for other problems) but they need to be clear within the paper.  

Small issues:
1. Please explain that the improvement in "Predictive Performance Improvement" is improvement compared to a system without FE earlier in the document, e.g. before table 3.
2. While the random experiment is fun and adds to the paper, I don't think it is at all accurate to say that it tests "whether our prompt template influenced our results" - seeing as the prompt template itself did not change in this experiment, only the names of the features. I don't think it shows anything about the prompting strategy - but rather that the nature of the bias depends on the feature naming. 
3. Caught typo: The API usage costed -> cost
The paper is based on solid experimental work, testing using several LLMs and across many datasets, testing for memorization issues separately to check for bias explicitly. 
The paper is an interesting and easy to follow read. Problematic properties of LLM solution paths for different problems are always appreciated, as we develop more and more systems that significantly rely on this tool, we must strive to understand the biases this seemingly easy fix-all solution of asking an LLM brings into our work and the times it might fail completely. It is also interesting that the large models have failed worse at adding features that helped with the downstream system's results compared to the smaller models, which did help a little.
The paper tests LLM bias in the task of feature engineering for training a downstream learning system. The paper uses several LLM settings across 27 datasets, to demonstrate that LLMs do indeed have bias for this task, which indeed seems to lead to poor performance compared to an existing SOTA automated feature engineering solution. Some further discussion and experimentation into the properties of the bias shows that the LLM seems to prefer simpler features when the features have meaningful names, but also doesn't sample uniformly when the features have nondescript random names.
The main issue with this paper is that it is rather unclear why the usage of LLMs for this task was explored at all. It seems that when feature engineering is done by an LLM, the downstream system's performance is worse than existing SOTA systems -  and sometimes even worse than doing any feature engineering at all. Frankly, it's also not a task that I would intuitively expect LLMs to be good at, as general knowledge, common sense and language knowledge is probably not what humans would use for feature engineering, but rather math/engineering skills and perhaps expert domain knowledge or lived experience - all usually less strong qualities of LLMs. The paper does not call this issue out or justify it. Usually, checking for biases of a solution might have one of two purposes: 1. call out poor performance that happens in a way that isn't expected or measured in other ways, so for example, if the system had seemingly good downstream performance, checking for biases or other issues might help guard us from using a problematic solution that looks good in our metrics. 2. try to improve the performance of the biased system by somehow mitigating the bias. It seems that option 1 in this case is unnecessary, since the LLMs have worse performance, and no actual attempt is made towards the #2 target.
Yes
No ethics review needed.
1, Could you clarify the source of the original features—were they extracted or provided with the datasets?
2, Have you considered experimenting with prompts that encourage the use of complex features, perhaps by emphasizing intricate relationships between original features?
3, What methods were used to validate the effectiveness, fairness, and implications of the operator frequency metric?
4, How did you account for the stochastic nature of LLM responses, where identical prompts might yield different operators and features?
5, Would it also be informative to evaluate model performance using only the generated features, excluding original features? Maybe you can try this.
6, Have you conducted feature-level analysis of the constructed features? Specifically:
Classification Performance Level: Identifying dominant features in both LLM and OpenFE-generated sets
Feature Level: Analyzing the characteristics of successful versus unsuccessful generated features
Combining classification-level, feature-level, and operator-level analyses to strengthen conclusions about LLMs' feature engineering capabilities.
7, A potential typo in Hypothesis 1: “HYPOTHESIS 1: FEATURE ENGINEERING WITH LARGE LANGUAGE MODELS IS BIASED TOWARD SIMPLE OPERATES.” The last word should be “OPERATORS”?
The paper presents a novel investigation into LLMs' feature engineering capabilities. The authors introduce an innovative evaluation metric—operator frequency distribution—which effectively quantifies the patterns in operator selection during feature construction. This metric provides valuable insights into how feature engineering tools, particularly LLMs, exhibit preferences for certain operators under different task contexts and prompt conditions. Furthermore, the study's comprehensive evaluation across 27 tabular datasets, with careful consideration for LLM memorization effects, demonstrates robust experimental design and systematic methodology.
This paper investigates Large Language Models' (LLMs) capabilities in feature engineering for tabular data. The study examines four LLMs across 27 tabular classification datasets that were specifically selected to minimize potential memorization effects. For each dataset, the LLMs were tasked with recursively generating 20 new features, using prompts that contained task context, descriptions of original features, and available operators. The study benchmarks these results against OpenFE, an open-source feature engineering tool, using identical operators and original features. To evaluate the effectiveness of the engineered features, a LightGBM model was trained and tested on datasets combining original and constructed features, with classification accuracy as the performance metric. The results demonstrate that OpenFE produces more consistently beneficial feature sets for classification tasks. Through analyzing operator frequency in feature construction across both LLMs and OpenFE, the authors conclude that LLMs exhibit a bias toward simpler operators when no explicit operator preferences are specified in the prompts.
The paper's analysis lacks sufficient depth in several crucial areas. While the proposed operator frequency metric is interesting, it requires further validation in terms of:

Effectiveness: There is no analysis comparing the variability and information content of features generated by simple versus complex operators.
Fairness: The operator-level analysis overlooks that identical operators applied to different features can yield vastly different outcomes, making tool comparisons based solely on operator frequency potentially misleading.
Implications: The study lacks experimental evidence linking complex operator usage to improved classification performance.

The paper's conclusion about LLMs' preference for basic operators requires additional validation. The authors did not explore prompting strategies to encourage complex operator usage, nor did they analyze the specific features and operators suggested by LLMs.
The narrative structure could be improved. For instance, the abstract's discussion of LLM bias in text generation appears tangential to the core focus on feature engineering. Similarly, the section on 'Other Applications of Large Language Models for Tabular Data' would be better integrated into the literature review rather than appearing as a standalone paragraph.
We analyze the bias of LLMs when used for feature engineerng and found that LLMs are biased toward creating simple features.
@inproceedings{
anonymous2024large,
title={Large Language Models Engineer Too Many Simple Features for Tabular Data},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=1JhSJIYX3p},
note={under review}
}
Tabular machine learning problems often require time-consuming and labor-intensive feature engineering.
Recent efforts have focused on using large language models (LLMs) to capitalize on their potential domain knowledge. 
At the same time, researchers have observed ethically concerning negative biases in other LLM-related use cases, such as text generation. These developments motivated us to investigate whether LLMs exhibit a bias that negatively impacts the performance of feature engineering. While not ethically concerning, such a bias could hinder practitioners from fully utilizing LLMs for automated data science. 
Therefore, we propose a method to detect potential biases by detecting anomalies in the frequency of operators (e.g., adding two features) suggested by LLMs when engineering new features. Our experiments evaluate the bias of four LLMs, two big frontier and two small open-source models, across 27 tabular datasets. Our results indicate that LLMs are biased toward simple operators, such as addition, and can fail to utilize more complex operators, such as grouping followed by aggregations. Furthermore, the bias can negatively impact the predictive performance when using LLM-generated features. Our results call for mitigating bias when using LLMs for feature engineering.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
LLMs feature engineering bias tabular data automated data science
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/dc5d16a2eaf96859f4e64e5c9642091171026ab5.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/fec95a0ca55f91e171acb49ba5c79cdb8aee79ff.zip
Large Language Models Engineer Too Many Simple Features for Tabular Data
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission