Decoding Reading Goals from Eye Movements
application of language models to cognitive science and psycholinguistic;eye movements in reading;reading goals;cognitive state decoding;multimodal models (eye movements and text)
Yes
No ethics review needed.
- Why didn't you report AUROC?
The paper was a pleasure to read. It introduces all relevant background regarding eye movements and reading, has a clear outline and follows a nice story.

Related work is exhaustive and paints a good picture of both machine learning models used in any kind of reading setting as well as eye-tracking-while-reading in general. They then introduce several machine learning models which consume either eye movements, the text that was read during the recording, or both.
In their paper, the authors explore the option of predicting the goal of a person while reading a text. They distuingish between two different natural reading options. The first setting is a ordinary, general reading for pleasure, context, or prose. The second setting is a sharp turn, where the reader has a specfic goal in mind, answering in question, mimicing application scenarios for every person in daily life.

These are the two labels, for several binary classification models that try to distuingish, given the eye movements, and optionally the text, the goal of the reader. They also introduce a new way of interpreting the model's results using linear mixed models.
The only true weakness is that the authors do not introduce a new model which exploits the eye movement while reading setting the authors investigate. 

Standard error not report, additionally, no statistical significance tests were done between trainable models, e.g. best model vs rest. The critical span (8./9. in the linear mixed effect model) for interpretability only works for already known texts. For binary classification AUROC would also be a great metric to report. 

Unfortunately, the anonymous repository is not accessible, as well as the data.
Yes
No ethics review needed.
Could the authors clarify how reading goal classification would perform in real-world applications where eye-tracking calibration is variable, such as web-based eye tracking?
Have the authors considered testing whether other task-specific goals (e.g., skimming) could also be accurately predicted using these models?
Originality: The study introduces a novel problem - decoding reading goals from eye movements - that has not been widely explored. This new application area could encourage further research in cognitive science and multimodal data analysis.
Quality: The use of diverse models and a comprehensive evaluation protocol ensures the quality and reliability of the findings. The error analysis is particularly valuable, as it provides insights into the factors that influence classification success, such as the presence of critical spans and reading time.
Clarity: The paper explains the decoding task well and provides a logical flow of ideas. The inclusion of various model types and a mixed-effects model analysis demonstrates a comprehensive and thought-out approach.
Significance: This study lays a foundation for understanding goal-oriented reading behaviors. While applications are currently exploratory, the findings could aid cognitive science researchers and developers of educational tools who aim to personalize learning experiences based on user behavior.
The paper explores a novel task of predicting reading goals - specifically distinguishing between information-seeking and ordinary reading - based on eye movement data. Utilizing an extensive dataset and advanced machine learning models, the authors demonstrate that eye movement patterns contain valuable signals for goal decoding. The study systematically evaluates several state-of-the-art models, including both eye-movement-only and multimodal (eye movements + text) approaches, and introduces an ensemble model that further improves prediction accuracy. The paper provides an in-depth error analysis, identifying key challenges in goal decoding and highlighting textual and participant-specific factors affecting classification difficulty.
Complexity in Model Descriptions: Some model descriptions lack clarity, especially in multimodal integration approaches. Providing visual diagrams or more intuitive breakdowns could make these sections more understandable.
Limited Scope of Goal Types: The study only explores two reading goals (information seeking and ordinary reading). Extending this research to other goals (e.g., skimming, proofreading) could make the findings more broadly applicable.
Yes
No ethics review needed.
see above
Originality. The paper proposes an interesting research question, i.e. if one can distinguish between two reading tasks based on eye movements (+ text). This can be seen as an extension of previous work, in particular [1]. The authors suggest that their paper has broader scope and ecological validity than [1]. While this represents an important and valid extension, it is not particularly stark in originality. The methodologies are all adapted from previous works, as well as the data. While taking a different perspective, the conceptual framework is almost identical to previous work [1,2].

Quality. The methods used are appropriate and the analysis performed is appropriate, well presented, and interpreted. The authors released their code, which together with the implementation details provided in the paper should make the results reproducible.

Clarity. The paper is well written and exhaustively reports implementation details. To further improve clarity, I would add a more clear explanation of the difference between the proposed task (reading goal classification) and reading comprehension. 

Significance. The paper represents an interesting contribution to the field of psycholinguistics, exploring a new task. The analysis in Section 6 highlights important features to consider when studying eye movements on text. 

[1] Hollenstein, Nora, et al. "ZuCo 2.0: A Dataset of Physiological Recordings During Natural Reading and Annotation." Proceedings of the Twelfth Language Resources and Evaluation Conference. 2020.
[2] Shubi, Omer, et al. "Fine-Grained Prediction of Reading Comprehension from Eye Movements." arXiv preprint arXiv:2410.04484 (2024).
This paper adopts an existing dataset and existing methods to test reading goal prediction, which consists in predicting if a human is performing ordinary reading or information seeking (binary classification task). In addition to testing existing methods, the authors introduce a model ensemble which shows improved performance, highlighting that different models capture different information. Finally, the authors study how different trial features affect the model's prediction. Reading time before/after critical span, critical span length and paragraph position seem to be the most crucial features.
- As discussed above, the paper is an interesting extension of previous work but does not particularly shine in terms of originality.

- The authors do not discuss limitations of current methods or dataset. The authors briefly mention the need for different tasks and datasets in Section 8, but I believe a more structured and systematic discussion of limitations is necessary. For example:
  - The paragraphs considered in the study are all short. Do the findings generalise to longer text? 
  - Being taken from newspaper articles, all the paragraphs are expository - they provide information. Do the authors expect to find similar results for, e.g., narrative texts written to entertain? 
  - There are differences between different reading comprehension tasks [1]. For example, students that read a text when doing an exam have likely a different behaviour than students that read just to understand the text. Are these differences relevant to this paper?  

- The scientific implications of the authors’ findings can be better discussed. Based on their findings, how can we improve current methods? How can we fully exploit the potential of eye movements for such a task? 

- While the authors state that reading is an ubiquitous and essential skill, the paper lacks an explanation of why reading goal prediction is an important task to study and which are the applications or areas that will benefit the most from that. 

- Overall, this paper presents interesting research in psycholinguistics. However, I believe its contributions are not strong and original enough to be published at ICLR. In addition, the topic would probably fit better a psycholinguistics or computational linguistics venue.

[1] Mar, Raymond A., et al. "Memory and comprehension of narrative versus expository texts: A meta-analysis." Psychonomic Bulletin & Review 28 (2021): 732-749.
Yes
No ethics review needed.
no questions
The article is very well written and easy to follow.
The experiments are extensive, sound, and deliver some meaningful insights.
The dataset can be a valuable resource for the community.
The authors investigate automatic classification of reading behaviour into information seeking and "ordinary reading". They recorded a new eye tracking dataset for this task and evaluated several relevant approaches.
The novelty is limited: the author conduct a number of evaluations of existing methods an a new dataset.
The novelty of the task is also very marginal and incorrectly portrayed as the authors miss highly relevant previous work. Xiuge et al. (2023) distinguished deep from skim reading using eye tracking, which is very similar to what the authors claim as novelty.

Further relevant previous work that should be discussed is by Kunze et al. (2013) who distinguished different document types.
It would also be meaningful to compare the reading goal recognition task to similar tasks that were investigated with eye tracking data. One example is informational versus navigational intent classification (Sharma et al., 2023).

In my opinion, if the authors re-formulate their contribution, taking all the relevant related work into account, the paper could still be interesting. I would not see it at ICLR due to the limited contribution w.r.t. to deep learning, but rather at a more HCI-focused venue such as ETRA, ICMI, IUI,...

References:
-----------

Chen, Xiuge, et al. "Characteristics of Deep and Skim Reading on Smartphones vs. Desktop: A Comparative Study." Proceedings of the 2023 CHI Conference on Human Factors in Computing Systems. 2023.

Kunze, Kai, et al. "I know what you are reading: recognition of document types using mobile eye tracking." Proceedings of the 2013 international symposium on wearable computers. 2013.

Sharma, Mansi, et al. "Implicit Search Intent Recognition using EEG and Eye Tracking: Novel Dataset and Cross-User Prediction." Proceedings of the 25th International Conference on Multimodal Interaction. 2023.
Decoding Reading Goals from Eye Movements.
@inproceedings{
anonymous2024decoding,
title={Decoding Reading Goals from Eye Movements},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=ZjuEPZJsa3},
note={under review}
}
Readers can have different goals with respect to the text they are reading. Can these goals be decoded from the pattern of their eye movements over the text? In this work, we examine for the first time whether it is possible to decode two types of reading goals that are common in daily life: information seeking and ordinary reading. Using large scale eye-tracking data, we apply to this task a wide range of state-of-the-art models for eye movements and text that cover different architectural and data representation strategies, and further introduce a new model ensemble. We systematically evaluate these models at three levels of generalization: new textual item, new participant, and the combination of both. We find that eye movements contain highly valuable signals for this task. We further perform an error analysis which builds on prior empirical findings on differences between ordinary reading and information seeking and leverages rich textual annotations. This analysis reveals key properties of textual items and participant eye movements that contribute to the difficulty of the task.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
application of language models to cognitive science and psycholinguistic eye movements in reading reading goals cognitive state decoding multimodal models (eye movements and text)
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/f3ca51801945928098e3a31db2aa746670aae0ee.pdf
applications to neuroscience & cognitive science
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Decoding Reading Goals from Eye Movements
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission