2. Basics of AI Training and Technical Challenges 
of Including Books  
It’s critical to understand that LLMs are not trained on text “as is” – meaning that the model 
is not digesting the text in a way humans would, front to back. The text does not represent a 
copy of the original text in its original form. Instead, the text is processed in smaller chunks 
of text, which are then shuﬄed and “tokenized,” as we explain further below.  
One way to conceptualize the chunking, shu ﬄing and tokenizing process is to imagine a 900 
page book, which has 400,000 words. To feed into an AI model, the book will ﬁrst be cut into 
manageable chunks of text that represent up to several thousand tokens; such a process 
might result in around 50 “chunks” of text. Each of those chunks will contain long sections of 
narrative content; however, the chunks themselves will then be randomized, and fed into the 
AI model out of sequence from each other; the ﬁrst chunk may be text from Chapters 9 and 
10, while the initial text in Chapter 1 may be in the 30th chunk. Within these chunks, the text 
itself will be understood by the model as tokens.  
In the example below, 252 characters of human-readable text are shown in tokenized form as 
57 distinct tokens, the relationships between which then form the basis of building an AI 
model. The illustration shows a block of human-readable text as it would be tokenized for AI 
training; different colors are used in this visualization merely to differentiate one token from 
another within the string of text. As the visualization makes clear, not all of the tokens 
directly correspond to a single word; tokens merely represent characters that often appear 
together in the training data.    6
 
 OpenAI’s Tokenizer tool at https://platform.openai.com/tokenizer explains how ChatGPT uses tokens 6
and provides a tool to visualize examples. As noted on their site, the tokenization process is different for 
every model, this is merely an illustrative example. The visual below represents an example of how 
OpenAI’s ChatGPT creates tokens from English text. 
Towards a Books Data Commons for AI Training 3