Tokens do not typically represent words, but instead often represent subword tokens. For 
example the word “incompetence” may be broken into three tokens: “in-,” “competent,” and “-
ence.” This approach to tokenization enables representation of grammar and word variations, 
effectively allowing a high degree of language generalizability.  7
In recent years, LLM research has successfully been able to scale up models by pre-training 
on a large number of tokens. In turn, this has allowed a higher degree of language 
generalizability in the resulting model. For example, OpenAI’s ChatGPT trained on hundreds 
of billions of tokens, allowing it to model language in a very general way. The resulting 
models an then be ﬁne-tuned for speci ﬁc tasks using training data representing a particular 
corpus, such as software code.    8
 McKinsey provides an overview of the different types of tokens that may be used by AI models. 7
McKinsey. “What Is Tokenization? | McKinsey.” Mckinsey.com, 2023, www.mckinsey.com/featured-
insights/mckinsey-explainers/what-is-tokenization.
 There are certain technical challenges in using books in AI training as well, given the nature of the 8
format. First, one must address whether a book is already in digital form. For the vast majority of books, 
that is not the case. One ﬁrst needs to digitize the book, and convert it to a digital text ﬁle using optical 
character recognition (OCR), or use a born-digital version (although we return to speciﬁc limitations on 
that approach below). Second, once a book is in digital text form, it must be converted into a text format 
that is suitable for AI training. Text conversion tools transfer the content of books into complete text 
ﬁles, which is akin to the type of conversion that must be done between a Microsoft Word or Adobe PDF 
ﬁle format and a simple .txt format. This conversion is generally not adequate for the purpose of AI 
training; researchers have found that post-processing is required to ensure these text ﬁles are properly 
formatted for the purposes of tokenization. For example, when building the dataset known as The Pile, 
researchers had to modify an existing epub-to-text converter tool to ensure that document structure 
across chapters was preserved to match the table of contents, that tables of data were correctly 
rendered, to convert numbered lists from digitally legible lists of “1\.” to “1.”, and to replace unicode 
punctuation with ascii punctuation. See Discussion in 4.3.2 in Bandy, Jack, and Nicholas Vincent. 
Addressing “Documentation Debt” in Machine Learning Research: A Retrospective Datasheet for 
BookCorpus. 2021, https://arxiv.org/pdf/2105.05241.pdf. and C.16 of The Pile documentation in Gao, 
Leo, et al. The Pile: An 800GB Dataset of Diverse Text for Language Modeling, https://arxiv.org/pdf/
2101.00027.pdf.
Towards a Books Data Commons for AI Training 4