A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding
LLM;DocAI;Visually Rich Document Understanding;KIE
Yes
No ethics review needed.
(a) Are the reported numbers for LayoutLLM_CoT in Table 3 inaccurate? The current numbers seem to correspond to Vicuna 1.5 7B. Would it be more appropriate to report the results based on Llama2-7B for a fairer comparison?

(b) Why does LayTextLLM show significantly inferior performance on the ChartQA dataset, even compared to OCR-free methods?
- LayTextLLM demonstrate that mapping the coordinates to a single bbox embedding can achieve delightful performance on document understanding while mitigating long sequence issues.
- Table 1 and Table show that LayTextLLM demonstrate performance improvement over several baseline methods on DocVQA, InfoVQA, and KIE tasks.
This paper presents LayTexLLM, which interleaves layout and text in an LLM for document understanding. There are two key ideas:
- Spatial layout projector (SLP), which project the coordinate of each bbox (_i.e._, [x_min, y_min, x_max, y_max]) derived by an off-the-shelf OCR module into a $d$-dimensional embedding, so that the layout information can be input to the LLM and learned together with other text tokens via next token prediction (layout-aware).
- Shuffled-OCR supervised fine-tuning (SSFT), which randomly shuffles the order of OCR-derived text in a certain proportion of examples to mitigate the bias that OCR engines typically output text bboxes in a way of top-to-bottom and left-to-right. 
Experiments on DocVQA, InfoVQA, and KIE tasks show that LayTextLLM outperforms or achieves comparable performance to baselines, while experiencing a significant performance drop on the ChartQA dataset.
1. The performance of the model heavily relies on the quality of the OCR engine, which poses a risk of significant performance drop, especially for handwritten documents or low-resource languages where OCR performance can be subpar. Moreover, incorporating an OCR module can negatively impact inference latency, making real-time or high-throughput applications challenging.

2. As indicated in Table 3, LayTextLLM_zero and even LayTextLLM_all demonstrate inferior performance on Document-Oriented VQA compared to LayoutLLM (Luo et al., 2024). Additionally, Table 8 reveals that LayTextLLM shows a substantial drop in performance on the ChartQA dataset. Specifically, OCR-free methods generally achieve scores between 50-59.9, whereas the best performance achieved by LayTextLLM_all is only 35.4.

3. The ablation study requires a clearer description of the variants involved. The first line in Table 4 lacks clarity—does it refer to a setting that uses naive coordinate-based pretraining (e.g., normalized [x1, y1, x2, y2]) with the proposed SSFT strategy? If not, it would be beneficial to include such a variant to demonstrate the effectiveness of the Spatial Layout Projector.
Yes
No ethics review needed.
See weakness.

Moreover, I'd like to reuqest for a fair comparison to baseline. Most vision LLMs listed in the paper are not the SOTA. For example, InternVL is not a very good model for OCR, while internvl2 is super strong. Moreover, Cambrian-1 is also a better baseline.
- The empriical performance shows a imporlvement on downstream document understanding tasks
The paper use one token (generated by a small NNs called SLP) to replace the four coordinate representation of location for normal document understanding tasks. With such compression, the model can accept more tokens and longer context than a vanilla llm. The method show better results than some document understanding MLLMs.
Overall, the paper propose an emprical trick to compress four localization tokens into one token, however, the contribution is thin. 

- The one-token representation demonstrates effective text localization. To strengthen the paper's contribution, it would be beneficial if this capability could be generalized beyond document understanding to other spatial understanding tasks to suit in ICLR. Otherwise, the paper is more suitable to conference focusing on doucment understanding -- e.g. ICDAR.

-  The rationale for retaining explicit coordinates is to predict bounding boxes for unlabelled texts, which also a important sub-task of document undrstanding. The application of the Spatial Layout Projector (SLP) on these coordinates seems to restrict the model’s extensibility. I would suggest the author also explore the perfomance of training an inverse SLP to convert the single-token representation back to coordinates.
Yes
No ethics review needed.
refer to weakness
1. The paper proposes the adoption of a unified embedding method in LLM for the first time, interleaving the spatial layout with the text data.
2. By reducing the sequence length and making full use of the autoregressive characteristics, the performance of document understanding is significantly improved.
3. Comprehensive experimental results and ablation studies are provided, demonstrating the effectiveness of the proposed method.
Traditional methods for document understanding usually rely on the text extracted by OCR and spatial layout information, but they have limitations when dealing with long text sequences. This paper proposes a method named LayTextLLM, which reduces the sequence length problem by mapping each bounding box to a single embedding and interleaving it with the text, while taking advantage of the autoregressive characteristics of LLM. Two training tasks are proposed: layout-aware next token prediction and supervised fine-tuning of shuffled OCR, to enhance the generality of the model. In Key Information Extraction (KIE) and Visual Question Answering (VQA) tasks, LayTextLLM significantly outperforms the existing state-of-the-art methods. In the zero-shot scenario, the performance of the KIE task is improved by 27.0% and that of the VQA task is improved by 19.8%.
Compared with the existing VLM papers, the paradigm is actually quite similar. Although it is not highly innovative, it offers a new perspective for handling OCR document understanding problems in this field. Another point is that perhaps the number of baselines for comparison seems a bit small. I wonder if there are more and newer works related to Text + Coordinates for comparison.
Additionally, there is a slight error in the bolded text in Table 1.
Yes
na
No ethics review needed.
1.The different training tasks such as Layout-aware Next Token Prediction mentioned，how many tokens of training data were used respectively?
2.Why does the chosen lora structure work as expected? Do other loras have the same effect, and how are ablation trials conducted?
3.Is there any analysis examining the impact on the llama models' ability to understand language after incorporating spatial information?
1.This work is the first work to employ a unified embedding approach that interleaves spatial layouts directly with textual data within a LLM.
2.They propose two tailored training tasks: (1) Layout-aware Next Token Prediction  and (2) Shuffled-OCR Supervised Fine-tuning task to better elicit the model generalizability in downstream tasks.
3.Experimental results sounds good.
This paper introduce Interleaving Layout and Text in a Large Language Model (LayTextLLM) for document understanding, which projects each bounding box to a single embedding and interleaves it with text, efficiently avoiding long sequence issues while leveraging autoregressive traits of LLMs. They also propose a tailored pre-training task—Layout-aware Next Token Prediction—a completely self-supervised task that enhances the alignment between layout and textual . LayTextLLM significantly improves performance on downstream document-related VQA and KIE tasks.
1.The methods adopt a P-LoRA structure, which is introduced by existing work InternLM-XCompoer2, limiting the novelty of this work‘s.
2.The analysis of the impact of SLP training on the effectiveness of the model remains incomplete.
@inproceedings{
anonymous2024a,
title={A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=lFijzkTUNB},
note={under review}
}
Recently, many studies have demonstrated that exclusively incorporating OCR-derived text and spatial layouts with large language models (LLMs) can be highly effective for document understanding tasks. However, existing methods that integrate spatial layouts with text have limitations, such as producing overly long text sequences or failing to fully leverage the autoregressive traits of LLMs. In this work, we introduce Interleaving Layout and Text in a Large Language Model (LayTextLLM)} for document understanding. In particular, LayTextLLM projects each bounding box to a single embedding and interleaves it with text, efficiently avoiding long sequence issues while leveraging autoregressive traits of LLMs. LayTextLLM not only streamlines the interaction of layout and textual data but also shows enhanced performance in Key Information Extraction (KIE) and Visual Question Answering (VQA). Comprehensive benchmark evaluations reveal significant improvements, with a 27.2% increase on KIE tasks and 12.0% on VQA tasks compared to previous state-of-the-art document understanding MLLMs, as well as a 15.1% improvement over other SOTA OCR-based LLMs on KIE tasks.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
LLM DocAI Visually Rich Document Understanding KIE
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/75231c9126bd49c07d83b9326b00e2214ff1a6f9.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
A Bounding Box is Worth One Token: Interleaving Layout and Text in a Large Language Model for Document Understanding
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission