Triples as the Key: Structuring Makes Decomposition and Verification Easier in LLM-based TableQA
TableQA;Triples
Yes
No ethics review needed.
1. Given that this is an additional step along the pipeline of TableQA steps, what does this step do to the average response time and number of tokens used? 
2. Does this method of generating triples work for mathematical questions such as average age/salary etc? Does this mean that the subquestions would require triples for every entry in the table?
1. Incorporating triples in multiple stages of TableQA is a novel idea and as shown by the evidence in the paper, helps improve the overall QA performance. 
2. The authors demonstrate the effectiveness of TIDE by performing ablation study which is critical in identifying the impact of this method.
3. This method works alongside DP and adds an additional step of generating triples, so this keeps all the benefits of DP as well.
The authors use Triples in different phases of TableQA. The generation of triples during Question decomposition phase helps in extracting key subquestions and conversion of answers during answer verification phase helps remove LLM bias in verification and enables better answer verification.
1. Inorder to demonstrate the impact of TIDE, it could have been useful to see TIDE with other standard LLMs as well. The ablation study shows the impact of TIDE but is this method tailored to work with GPT 3.5 in any specific way?
Yes
No ethics review needed.
The appendix lacks cases, including successful cases compared to other methods, error cases analysis, etc. I think showing some cases can better explain the working mechanism of the entire pipeline.
1. The motivation of the work is clear. It leverages triple extraction as a tool to enhance understanding and validation of TableQA questions.
  
2. The experiment is sufficient, which compares TIDE with many TableQA baselines. Further study can reveal the properties of the proposed method.
This paper proposed a decomposition and verification strategy based on triple to address TableQA tasks. Specifically, it decomposes and transforms questions using triples to capture more key information, while utilizes explicit structure for better verification. By combining LLM-based reasoning and symbolic reasoning based on structured representations, TIDE improves the performance in the TableQA task
1. The contribution is limited; it only proposes a framework based on LLMs with triple extraction. It would be better if it could be demonstrated on some open-source models like Llama3.
  
2. The conclusion drawn from the experimental results that TIDE improves the effectiveness of TableQA problems is questionable. The results of DP and Agent baseline for both datasets are not SOTA. So, can it be considered that the gain in this result comes from additional reasoning integration of DP and Agen rather than the introduction of triplet?
Yes
No ethics review needed.
This goes back to the clarity of the verification stage.The example authors provided seems to be a ‘COUNT’ type questions. How does the system handle other types of numeric aggregation? For questions that are not numeric in nature (e.g. compose-style, multi-hop questions), how does the final stage work after verification? Does this approach only work on aggregation type questions? What algorithms are you using to unify the triples for each subquestion?

My initial recommendation is to reject this paper. First, there are some clarity issues so it's difficult for me to determine to what extent the contributions made by this paper are significant. Second, there are some missing analyses and experiments that, if presented, could bring more value to the community.
Incorporating triples as intermediate steps is an interesting approach, and it intuitively makes sense as a natural intermediate representation to tackle the verification of subquestions. The authors effectively combine existing ideas from different problems and apply it in the settings of table QA, showing SOTA results on challenging table qa datasets.
The paper proposed an inference time approach, TIDE, to address table question answering. Their method builds on a Decomposition-Reasoning-Verification approach for Table QA using LLM, and they proposed to use semantic triples as intermediate representations instead of natural text. First, during question decomposition, they extract triples from the question then generate subquestions based on those triples. Then, they apply CoT (for DP mode)/generating python code (for agent mode) to answer each subquestions. Finally, in the answer verification stage, they compare the triples from stage 1 with triples from stage 2 to filter correctly answered triples. 

Their contribution is using triples as intermediate representations in order to facilitates decomposition and verification, and showed that it leads to improvements over SOTA.
My main concern is clarity and contribution.
For clarity
1.  Details of how each module is implemented is confusing. The formulation really only has one equation (eq 1), and it is about the Table QA problem. Formal explanation (instead of examples only) for each submodule would help improve clarity. Furthermore, I think the prompt for extracting triples are not provided?

2. It’s unclear to me how the final answer is synthesized after the verification stage. Again, providing formulation for this stage would be more helpful. 

3. In Table 3 of your ablation studies, you have TIDE vs TIDE-decomposition vs TIDE-verification. It is confusing to me how the ablated versions are implemented. Again, I think having formulation would help improve clarity.

In terms of contribution, since this is combining different existing ideas from other tasks (see below for examples), I would appreciate more in-depth analysis of the proposed method.

1.  Would other language models be equally effective with generating triples? When does this assumption breakdown? How does model size affect the result and error propagation through each stage ... etc. Have you tested this with open source large/small sized language models or other information extraction approaches for triple extraction? Is it possible to substitute some LLM API calls with other non-LLM methods without hindering performance?

2. The strong performance could potentially come from data contamination. How do you attribute the success of TIDE vs GPT having seen those tables in pretraining? 

3. In verification stage, you mentioned the LLM is prompted to regenerate if verification failed. I would appreciate more analysis on this part such as showing how often verification fails or how many times max you limit the LLM to regenerate.


PS by combining existing ideas I mean
1. Question decomposition is not new. See [1] [2] for example
2. using LLM for information extraction is not new. See [3] for example

I think there are some merits in combining these insights and seeing that triples could apply to this problem, so I think more analysis could help improve this paper.

[1] Sewon Min, Victor Zhong, Luke Zettlemoyer, and Hannaneh Hajishirzi. Multi-hop Reading Comprehension through Question Decomposition and Rescoring. ACL 2019.

[2] Xiaoyu Yang and Xiaodan Zhu.  Exploring Decomposition for Table-based Fact Verification. EMNLP Findings 2021

[3] Zhiyuan Fan and Shizhu He. 2023. Efficient Data Learning for Open Information Extraction with Pre-trained Language Models.  EMNLP Findings 2023
We propose Triples-Inspired Decomposition and Verification strtegy to improve LLM in TableQA.
@inproceedings{
anonymous2024triples,
title={Triples as the Key: Structuring Makes Decomposition and Verification Easier in {LLM}-based Table{QA}},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=UwcZEoNP19},
note={under review}
}
As the mainstream approach, LLMs have been widely applied and researched in TableQA tasks. Currently, the core of LLM-based TableQA methods typically include three phases: question decomposition, sub-question TableQA reasoning, and answer verification. However, several challenges remain in this process: i) Sub-questions generated by these methods often exhibit significant gaps with the original question due to critical information overlooked during the LLM's direct decomposition; ii) Verification of answers is typically challenging because LLMs tend to generate optimal responses during self-correct. To address these challenges, we propose a Triple-Inspired Decomposition and vErification (TIDE) strategy, which leverages the structural properties of triples to assist in decomposition and verification in TableQA. The inherent structure of triples (head entity, relation, tail entity) requires the LLM to extract as many entities and relations from the question as possible. Unlike direct decomposition methods that may overlook key information, our transformed sub-questions using triples encompass more critical details. Additionally, this explicit structure facilitates verification. By comparing the triples derived from the answers with those from the question decomposition, we can achieve easier and more straightforward validation than when relying on the LLM's self-correct tendencies. By employing triples alongside established LLM modes, Direct Prompting and Agent modes, TIDE achieves state-of-the-art performance across multiple TableQA datasets, demonstrating the effectiveness of our method.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
TableQA Triples
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/677c6beafdaa2a517320149d5723452fdd6b59cd.pdf
generative models
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
/attachment/e6bc13d1819d8a1e5101eda4c0a97ff874c40a9e.zip
Triples as the Key: Structuring Makes Decomposition and Verification Easier in LLM-based TableQA
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission