Hierarchical Autoregressive Transformers for Tokenizer-Free Language Modelling
transformer;autoregressive;generative;language modelling;tokenizer-free;byte-level;hierarchical
Yes
No ethics review needed.
- I'm curious how the model works when doing inference in the same language but when the text is from a different distrubtion (out-of-domain).
- A method of tokenizer-free langauge modelling that is more robust to input perturbation and input domain shift.
The paper proposed a technique for tokenizer-free language modeling. They design hierarchical processing to aggregate character-level embeddings into word-level embeddings. They achieved similar performance as models with word-level tokenizers while being more robust to input pertubation. They improved the continued pretraining speed when switching domain from English to German while maintaining the performance.
- The proposed method only works for character-based languages.
Yes
No ethics review needed.
Have the authors considered that people switch registers depending on pragmatics, and that a AAVE speaker when interacting via text with an LLM might naturally switch to something closer to Standard English by default? 

Also some of the datasets used do not neatly fit into the type of problems that would require a lot of variation in vernacular eg. maths problems would be quite similar no matter what vernacular used and using a very different query for a maths problem in AAVE might be artificial and forced. The examples in Figure 1 do seem to be forced (especially the first three) and I think an AAVE speaker would feel like they are forcing a more significant difference than would be natural. I realise you have a naturalness check in the annotation pipeline, but this could just be that people thought it would plausibly be AAVE not that it *should* be AAVE - if annotators are given instruction to rewrite in AAVE they might do so even it is not very authentic.  Some discussion of AAVE being different under different communication scenarios and tasks would be welcomed.  

It was not clear if the answers were also rewritten in AAVE or just the queries as the description says it is just the queries, but the answers in AAVE would also be useful, measuring generation capability as well as understanding and reasoning.
Nice dataset for measuring LLMs robustness to dialects
Study showing brittleness of models to AAVE
This paper explores how robust language models are to language variation and dialects. They create a test set with 1.2k query pairs in Standard English and African American Vernacular English (AAVE). They show that a variety of standard large language models perform worse on AAVE than on standard English, and on misspelled English. Prompting the LLM to rephrase quieries in Standard English improves performance but does not close the gap in performance. This is a nice contribution to the field and the dataset could be of general interest.
Lack of discussion about the appropriateness of AAVE in different communications scenarios/tasks
I would have expected some more experiments investigating different robustness techniques for LLMs to mitigate the problem with dialects
Yes
No ethics review needed.
Please see the weaknesses.
1. Compared with subword tokenization, word-level tokenization reduces the lengths of sequences processed by the backbone LLM and allows a larger number of parameters under the same computation budget.
2. Experiments show the proposed method is more robust to input perturbation and cross-lingual continued pretraining.
The authors propose a method to avoid large lookup table in word-level tokenization by utilizing a small character-level lookup mechanism. They employ a light weight character level encoder to convert character sequences into word embeddings, which are then processed by a word-level backbone model and decoded back into characters. Compared with subword tokenization, word-level tokenization with character-level lookup reduces the lengths of sequences processed by the backbone LLM and allows a larger number of parameters under the same computation budget.
1. **I disagree with the claim that this method is tokenization-free.** Actually, it does employ a word-level tokenization approach. It heavily relies on a space-based tokenization mechanism. The backbone autoregressive model operates on word-level embeddings.
2. **The primary contribution of this work seems to be more like "how to avoid a large lookup table in word-level tokenization." The proposed method addresses this issue by utilizing a small character-level lookup table.** It aggregates the embeddings of individual characters within a word during encoding and employs an autoregressive model to decode each character from a word-level embedding during inference.
3. Clearly, this method cannot be applied to languages without using an alphabet system (e.g., Chinese, Japanese) or without a space tokenization mechanism (e.g., Arabic, Thai). Therefore, the adaptation experiments in Section 4.6 is not very convincing, which chooses languages from the same language family.
4. The contribution appears incremental compared to Megabyte [1]. **While Megabyte employs a fixed patch size, this work utilizes a predefined tokenization mechanism to segment each patch. However, this predefined tokenization approach reduces the flexibility of the method, preventing it from being considered tokenization-free.**

[1]  MEGABYTE: predicting million-byte sequences with multiscale transformers.
@inproceedings{
anonymous2024hierarchical,
title={Hierarchical Autoregressive Transformers for Tokenizer-Free Language Modelling},
author={Anonymous},
booktitle={Submitted to The Thirteenth International Conference on Learning Representations},
year={2024},
url={https://openreview.net/forum?id=tU074jg2vS},
note={under review}
}
Tokenization is a fundamental step in natural language processing, breaking text into units that computational models can process. While learned subword tokenizers have become the de-facto standard, they present challenges such as large vocabularies, limited adaptability to new domains or languages, and sensitivity to spelling errors and variations. To overcome these limitations, we propose a hierarchical architecture for autoregressive language modelling that combines character-level and word-level processing. Our approach employs a lightweight character-level encoder to convert character sequences into word embeddings, which are then processed by a word-level backbone model and decoded back into characters via a compact character-level decoder. This method retains the sequence compression benefits of word-level tokenization without relying on a rigid, predefined vocabulary. We demonstrate, at scales up to 7 billion parameters, that hierarchical transformers match the downstream task performance of subword-tokenizer-based models while exhibiting significantly greater robustness to input perturbations. Additionally, during continued pretraining on an out-of-domain language, our model trains almost twice as fast, achieves superior performance on the target language, and retains more of its previously learned knowledge. Hierarchical transformers pave the way for NLP systems that are more robust, flexible, and generalizable across languages and domains.
I certify that there is no URL (e.g., github page) that could be used to find authors’ identity.
I acknowledge that I and all co-authors of this work have read and commit to adhering to the ICLR Code of Ethics.
transformer autoregressive generative language modelling tokenizer-free byte-level hierarchical
I certify that there is no acknowledgement section in this submission for double blind review.
/pdf/04263a8324d7ff5d2f1543f36700eb7e48d0cd7e.pdf
foundation or frontier models, including LLMs
I understand the reciprocal reviewing requirement as described on https://iclr.cc/Conferences/2025/CallForPapers. If none of the authors are registered as a reviewer, it may result in a desk rejection at the discretion of the program chairs. To request an exception, please complete this form at https://forms.gle/Huojr6VjkFxiQsUp6.
I certify that this submission complies with the submission instructions as described on https://iclr.cc/Conferences/2025/AuthorGuide.
Hierarchical Autoregressive Transformers for Tokenizer-Free Language Modelling
ICLR 2025 Conference Submission
ICLR.cc/2025/Conference/Submission