Inductive Linguistic Reasoning with Large Language Models
language models;linguistic reasoning;prompting;analogical reasoning;linguistics puzzles
Yes
No ethics review needed.
1. The paper briefly mentions that frontier models like GPT-4o and Llama-3.1-405B-Instruct often successfully identify language families. How accurately do LLMs identify language families, and how often do they correctly solve queries when the language family identification is accurate?

2. The results show that the mixture setting—where analogical exemplars are generated by one model and applied by another—outperforms the self-generation setting, but the paper does not delve deeply into why this occurs.
**Originality**

The paper introduces an innovative approach to evaluating linguistic reasoning in LLMs through analogical prompting. It applies this method to extremely low-resource languages and further evaluates generating exemplars through a different LLM, increasing overall performance.

**Quality**

The paper presents experimentation across multiple models and prompting strategies.

**Clarity**

The paper is well-structured, with clear explanations of each experimental setup, metric, and finding.

**Significance**

The paper highlights advancing the understanding of LLMs' reasoning capabilities across diverse languages. The focus on low-resource languages underscores the broader implications of this work for multilingual AI and low-resource language preservation.
The paper explores LLMs' linguistic reasoning using linguistic puzzles on extremely low-resource languages. Its key contribution is a two-stage analogical prompting method, where the model first generates examples from related languages and then applies these to deduce grammar rules in a target language.
1. Section 4 mentions that each response was manually evaluated to provide exact match scores, but this evaluation process lacks details. Specifically, there’s no mention of how many responses were reviewed, how many LLMs were involved, the number of evaluators, or their inter-annotator agreement. Without this, it’s challenging to assess the reliability of the manual evaluation.

2. Section 5.2 mentions other linguistic reasoning datasets, yet these were not utilized in the experiments. Incorporating additional benchmarks would provide more reliable and generalizable results.
Yes
No ethics review needed.
Please describe in more detail the test linguistic problems in this study.

What are rationales in the particular case of linguistic problems?
The 2 stage analogical prompt is interesting and suggests that perhaps models might leverage information about related but more represented languages to solve the given linguistic problems in the test set.  There is also an interesting difference between larger models like Llama 405B or GPT4o and smaller models; the analogical exemplars work for the larger models but not the smaller ones, pointing to an ability of the larger models to adapt the analogical examples to the given linguistic problem that the smaller models lack.
This paper investigates the possibility of using few shot learning so that LLMs can generalize their knowledge to new and highly under resourced languages at inference time.  The authors introduce a new method of prompting called 2 stage analogical prompting, according to which for a given language problem P for a novel or very under resourced language L they first get a model to infer what family of languages L belongs to, then they get the model to select languages in L's family and to produce language problems similar to the given one P.  These results are then fed into either the same or a different model to then solve the original language problem P.  The authors show that their 2 stage analogical prompting delivers superior results to other  state of the art prompting (CoT) and methods without CoT.
The paper's main weakness is the disconnect between the empirical investigations, which seem sound enough, and the desired conclusion that is given here: "In summary, our results suggest that the ability of the model to deduce from inductively learned rules is the key performance driver."  In other parts of the paper the rules referred to here would seem to be grammar rules.  There is little in the paper to suggest in the results that any grammar rules have been really learned or what the form of the grammar rules might be.  For example the rules could involve simple agreement or complex long distance effects governing ellipsis, gapping, or some other complex grammatical phenomenon.   At least this reviewer would like to see a much more detailed study in which (i) the grammar rules at issue are clearly stated, (ii) we have results for patterns that are governed by the rules (iii) we have results for constructed examples that violate those rules.  I would expect that for examples that violate the rules the models would either fail to produce an output or flag it in some way, if they had learned the grammatical rules.  The paper provides no such data, and so we can't really conclude anything about the mechanism that the models used to infer correct solutions to the language problems posed.

Another problem with this paper is the reference to linguistic problems that aren't really very well described.  One can gather that at least some of the problems are translation problems.  But are they all translation problems?  If so, how on an olympiad test would a participant be able to get a good translation for a completely unknown language without any clues?   The whole experimental basis of the paper is kind of murky and needs to be cleaned up for those readers who are unfamiliar with the linguistic olympiads.  

The strengths of the paper could be improved by looking into more detail as to what 2 step analogical reasoning is doing.
I might of missed it but it seems that the paper itself doesn't contain a discussion of what happens when the language family is omitted but the examples are provided.  It would have been nice to have a more detailed stucy of the analogical reasoning itself.
Yes
No ethics review needed.
1. In Table 1, zero-shot scores are near zero across all models, which is unexpected, given that BLEU metrics are relatively lenient. Any insights into why this might be the case?

2. Line 311-313: "Our findings suggest that when equipped with the right tools (analogical demonstrations) from effective multilingual reasoners, strong deducers can thrive.". However, in Table 2, using Aya-23-35B as the generator yields better results than Llama-405B (which performed better in prior evaluations) when GPT-4o is the deducer. Does this imply that Aya excels at language identification rather than machine translation?
1. There are limited works on solving Linguistics Olympiad problems.  This paper's methodology is valuable as a benchmark for future studies.
2. The study presents comprehensive experiments across various models and prompting techniques, with a clear presentation of results.
The paper explores analogical prompting to solve modeLing (Chi et al., 2024), a dataset containing International Linguistics Olympiad-style problems. Through experiments with proprietary and open-source models using different prompting strategies, the authors demonstrate that few-shot chain-of-thought prompting with explanatory rationales yields optimal performance. They further suggest including analogical exemplars ( language family information obtained through LLM prompting) in prompts can enhance model performance.
1. The paper's contribution is primarily empirical, with limited conceptual innovation. The approach of using analogical prompting to boost performance is not very inspiring, as it mainly involves augmenting prompts with self-generated information [1].

2. The authors tested their method only on machine translation tasks, overlooking other question formats in IOL, such as multiple-choice and cloze questions. A more suitable benchmark than modeLing would be [2] or [3].

3. It is widely known that closely related languages help with cross-lingual transfer [4] [5]. This paper, however, does not seem to provide any novel insights in this area.


References:

[1] Sun, Z., Wang, X., Tay, Y., Yang, Y., & Zhou, D. (2022). Recitation-Augmented Language Models. ICLR 2023.
 
[2] Sánchez, E., Alastruey, B., Ropers, C., Stenetorp, P., Artetxe, M., & Costa-jussà, M. R. (2024). Linguini: A benchmark for language-agnostic linguistic reasoning. arXiv preprint arXiv:2409.12126.

[3] Bean, A. M., Hellsten, S., Mayne, H., Magomere, J., Chi, E. A., Chi, R., ... & Kirk, H. R. (2024). LINGOLY: A Benchmark of Olympiad-Level Linguistic Reasoning Puzzles in Low-Resource and Extinct Languages. arXiv preprint arXiv:2406.06196.

[4] Dan Malkin, Tomasz Limisiewicz, and Gabriel Stanovsky. 2022. A Balanced Data Approach for Evaluating Cross-Lingual Transfer: Mapping the Linguistic Blood Bank. NAACL 2022.

[5] Vésteinn Snæbjarnarson, Annika Simonsen, Goran Glavaš, and Ivan Vulić. 2023. Transfer to a Low-Resource Language via Close Relatives: The Case Study on Faroese.NoDaLiDa 2023.
Yes
No ethics review needed.
How did you conduct the evaluation?

Do you plan to release the code publicly?
The idea is interesting because using the analogical reasoning approach proposed in ‘Large language models as analogical reasoners’ on multilingual tasks is a methodology that has been little explored and apparently shows promise.


The authors propose a good experimental setting and a comprehensive discussion, however with difficulty one understands some passages
The paper explores the capabilities of LLM in performing linguistic reasoning on low-resource languages through language puzzles. This study uses the ‘analogical prompting’ approach, which enhances the reasoning capabilities of these models by using analogy-generated examples to improve performance in translation tasks, particularly in low-resource languages.

The idea is very interesting, and this is the first contribution that transfers the idea beyond English. However, there are some really serious points that emerge (detailed below). This does not put the paper in a good light and it strongly needs revision.
Among the paper's weaknesses are:

- The image Figure 1 is very confusing and is really difficult to read as it is of very poor quality.

- Many steps should be carefully explained e.g. the heart section (section 2 introduces the method that should emulate multilingual analogical reasoning). No examples are given in this section and the problem is not formalised, confusing the reader.

- The experiments, although many, are poorly introduced and the thread is not understood.
We study LLM performance on linguistics puzzles with analogical reasoning, generating auxiliary exemplars from the same language family for translation tasks.
@inproceedings{
anonymous2024inductive,
title={Inductive Linguistic Reasoning with Large Language Models},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=8XQ1hLbwmU},
note={under review}
}
Evaluating large language models (LLMs) on their linguistic reasoning capabilities is an important task to understand the gaps in their skills that may surface during large-scale adoption. In this work, we investigate the abilities of such models to perform abstract multilingual reasoning through the lens of linguistic puzzles on extremely low-resource languages. As these translation tasks involve inductive and deductive reasoning from reference instances, we examine whether diverse auxiliary demonstrations can be automatically induced from seed exemplars, through analogical prompting. We employ a two-stage procedure, first generating analogical exemplars with a language model, and then applying them in-context along with provided target language exemplars. We explore various combinations of language models as analogical generators and reasoning agents, testing different model sizes and specialized multilingual LLMs. Our results on the modeLing dataset show that analogical prompting is effective in eliciting models' knowledge of language grammar similarities, boosting the performance of GPT-4o by as much as 8.1\% and Llama-3.1-405B by 5.9\% over chain-of-thought approaches. These gains are realized with self-generated analogical demonstrations as well as those generated by weaker multilingual models. We also report several findings about interesting phenomena which drive linguistic reasoning performance, suggesting that such puzzles are a valuable benchmark for new reasoning methods.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
language models linguistic reasoning prompting analogical reasoning linguistics puzzles
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/34dd2d3d0fd2f40a12a3f3c5766e9d1496eab981.pdf
applications to computer vision, audio, language, and other modalities
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Inductive Linguistic Reasoning with Large Language Models
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission