Rethinking Table Instruction Tuning
table instruction tuning;table understanding
Yes
No ethics review needed.
Whether the findings still hold while applying lora or qlora?
- Comprehensive Analysis: The paper provides a thorough analysis of existing table LLMs, highlighting the importance of hyperparameter tuning, which is often neglected in other studies.

- Empirical Findings: The findings are helpful to subsequential research in this field of applying LLMs into tabular tasks. The effects of learning rate is close to my previous experimence in hyper-parameter tuning while further training LLMs over tables.

- Based on the finding, the proposed TAMA demonstrates an impressive improvement in performance.
This paper presents a thorough evaluation of existing instruction-tuned LLM for table-related tasks, focusing on the often-overlooked aspects of hyper-parameter choices and their impact on model performance. The authors identify the clear declines in the general knowledge obtained by the base model. Through systematic analysis, this work demonstrates that proper learning rates and fewer training instances can enhance table understanding without the sacrifice of LLM's exiting capabilities. Based on these findings, the proposed TAMA can be trained with fewer examples and achieves impressive performance on several representative tabular tasks.
These experiments were only based on Llama models. Further examination is needed for other LLMs, e.g., Qwen-* series, Phi-* series, etc. Whether the findings still hold requires further analysis.
Yes
No ethics review needed.
I really appreciate the motivation and ideas presented in this work, and addressing the questions raised in the weaknesses section would support a higher rating. I am curious whether the conclusion that a small number of samples can achieve good results is based on fine-tuning for specific fields. Because the SFT in these subdivided tableqa fields seems to be the same as what we did in the BERT era, so large-scale pre-training + small-scale fine-tuning can achieve good results. I am curious whether this conclusion still holds true when large models need general capabilities such as reasoning, long text answers, and RAG?

I know it's not easy to answer the question. It would be appreciated if some insights can be provided.
1. Identification of Over-Specialization Issues in Table LLMs: This work reveals a critical trend in current table LLMs, where excessive fine-tuning for table-specific tasks often compromises the models' generalization capabilities. By shedding light on the potential trade-offs, the study encourages the community to reconsider whether the performance gains claimed in state-of-the-art table LLMs justify the loss of general capabilities. This insight points to a promising direction for future research on balancing task-specific and general abilities in LLMs.

2. Extensive Experiments: The study conducts an extensive range of experiments across multiple tasks, including FeTaQA, HiTab, TabFact, and others, providing a broad evaluation of hyperparameter impacts on table-specific and general capabilities. This depth of experimentation allows for a more complete view of the challenges in table LLM tuning and establishes a robust foundation for future research in this area.

3. Identification of Effective Hyperparameter Settings: Through systematic exploration of hyperparameters—such as learning rate, training sample size, and number of epochs—the authors identify an optimal configuration for the LLaMA 3.1-8B model that supports strong table-specific performance while preserving general capabilities. By using significantly fewer training samples, they demonstrate an efficient tuning approach that achieves competitive results, establishing a valuable hyperparameter baseline for future table LLM research.
1. This paper presents a comprehensive study revealing that current open-source table LLMs tend to be overly specialized for table-specific tasks, leading to a decrease in out-of-domain table understanding, instruction-following accuracy, and the general capabilities of the base model. These models, such as TableLLM and TableBenchLLM, typically employ high learning rates (e.g., 2e-5), extended training epochs (e.g. 6 epochs), and massive datasets (often exceeding 2 million instances) that lack diversity. This approach results in significant performance drops—up to 20% on general benchmarks like AI2ARC—compared to their base models.

2. The authors argue that careful hyperparameter tuning is necessary to balance table understanding with general capabilities. Through detailed experiments on tasks like FeTaQA, HiTab, and TabFact, they demonstrate how hyperparameters such as learning rate, epochs, and training sample size affect table LLM performance. As a result, they find that a learning rate of 1.0e-6 and two training epochs optimize the LLaMA 3.1-8B model for both table and general task performance.

3. Based on these findings, the authors sample 200 data instances each from 12 diverse table tasks, including WikiSQL, TATQA, and FEVEROUS, totaling only 2,600 samples—just 0.1% of the data typically used in other table LLMs. This data reduction, achieved by randomly sampling representative examples from each task, allows them to train TAMA, a new table LLM. TAMA matches or surpasses GPT-3.5 and, in some cases, GPT-4 on standard benchmarks, achieving 77.4% on FEVEROUS (3% higher than TableLLM) and 66.9% on MMLU, close to the base model’s score. TAMA’s efficient tuning strategy preserves broad capabilities, offering a resource-efficient yet powerful approach to table instruction tuning.
1. Limited Reproducibility Due to Closed Code and Weights: While the study’s findings are promising, the lack of open-source training and evaluation scripts, data, and TAMA’s model parameters during the review period restricts reproducibility. Although the study’s conclusions align with similar issues I observed in table LLMs like TableLLaMA and TableLLM, having access to TAMA’s scripts and parameters would allow a comprehensive verification of the training process and performance claims of TAMA, underscoring the importance of direct open-sourcing resources for rigorous validation.

2. Title-Content Discrepancy on Table Data Modality: The title, “Rethinking Table Instruction Tuning,” suggests a focus on unique aspects of table data, yet the study does not examine if the observed trade-offs are specific to table modality or generalizable. Discussion on the influence of table-specific features, such as heterogeneity, permutation invariance, or numeric density, could better align the study’s scope with the title.

3. Limited Novelty and Incomplete Analysis Relative to Zhou et al. (2024): While this work applies Zhou et al. (2024) [1]’s approach of using a small set of tuning data to enhance table LLM capabilities, as reflected in lines 246-249, it offers limited additional insights. Although the application in the table domain adds value, the study could have further clarified whether observed improvements are due to pre-trained knowledge or specific learning during fine-tuning. 
Additionally, it does not explore whether models with stronger general capabilities require less tuning data to maximize performance in table tasks, which could provide valuable insights for future table LLM research. Finally, examining differences in pre-training corpora, such as LLaMA 1 vs. LLaMA 2 vs. LLaMA 3.1, could clarify whether observed gains stem from pre-trained abilities or are specific to instruction tuning. More experiments, such as comparing results across diverse pre-trained models, would enhance originality and provide practical insights.

4. Lack of Validation Across Multiple Base Models and Hyperparameter Guidance: While the hyperparameter settings for LLaMA 3.1-8B provide a useful baseline, extending these findings to other mainstream base models is essential to confirm generalizability. Applying the study’s tuning approach across diverse models, such as Mistral [2] and Qwen [3], and establishing a set of reliable hyperparameter baselines across mainstream LLMs—much like the impact achieved by the work [4]—would add significant, long-term value. Such efforts would enhance the study’s applicability and offer future table LLM research a practical framework for effective tuning.

[1] Zhou, Chunting, et al. "Lima: Less is more for alignment." Advances in Neural Information Processing Systems 36 (2024).

[2] https://huggingface.co/mistralai

[3] https://huggingface.co/Qwen

[4] Gorishniy, Yury, et al. "Revisiting deep learning models for tabular data." Advances in Neural Information Processing Systems 34 (2021): 18932-18943.
Yes
No ethics review needed.
N/A
+ The paper illuminates the limitations/weaknesses of existing table LLMs.
+ TAMA is a good contribution to the open-source LLM community, which has a good performance. People can do follow-up work on TAMA to study LLM's specific capabilities.
The paper first evaluates existing table LLMs and finds that they have poor OOD table understanding capability, poor instruction-following capability, and comprised general capabilities. The paper then studies the influence of hyperparameters on training table LLMs. Finally, the paper trains TAMA from LLaMA-3.1-8B-Instruct with great table understanding capability while preserving the general capabilities.
While TAMA is a good contribution to the community, I am personally not sure about the contribution of this paper itself. The takeaway for training TAMA seems to be just a good hyperparameter choice + multi-task training. The latter is a standard way and the former does not provide too much insight - I am not saying hyperparameter tuning/search is meaningless, and my point is that the current Section 3 is more like just an experimental report, where people (at least I) might want to see some fresh insights (e.g., training dynamics, capability analysis, etc). If it is just that the previous table LLM training + careful hyperparameter selection = TAMA, I would doubt the contributions of the paper.

I am open to different opinions. If the authors can make a rebuttal that I feel is convincing, I would be happy to increase my score.
We reveal the limitations of the current table instruction tuning, explore the technical details, and fine-tune our own model based on our findings.
@inproceedings{
anonymous2024rethinking,
title={Rethinking Table Instruction Tuning},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=GLmqHCwbOJ},
note={under review}
}
Recent advances in table understanding have focused on instruction-tuning large language models (LLMs) for table-related tasks. 
However, existing research has overlooked the impact of hyperparameter choices and lacks a comprehensive evaluation of the out-of-domain table understanding ability and the general capabilities of these table LLMs. 
In this paper, we evaluate these abilities in existing table LLMs, and reveal significant declines in both out-of-domain table understanding and general capabilities compared to their base models. 
Through systematic analysis, we show that hyperparameters, such as learning rate, can significantly influence both table-specific and general capabilities.
Contrary to the existing table instruction-tuning works, we demonstrate that smaller learning rates and fewer training instances can enhance table understanding while preserving general capabilities.
Based on our findings, we introduce **TAMA**, a **TA**ble LLM instruction-tuned from LLa**MA** 3.1 8B Instruct, which achieves performance on par with, or surpassing GPT-3.5 and GPT-4 on table tasks, while maintaining strong out-of-domain generalization and general capabilities. 
Our findings highlight the potential for reduced data annotation costs and more efficient model development through careful hyperparameter selection.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
table instruction tuning table understanding
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/93de25ff4821e6293b603353622d1c281ea3576f.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Rethinking Table Instruction Tuning
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission