text: ***2. Basics of AI Training and Technical Challenges***
  ***of Including Books***
  
  It’s critical to understand that LLMs are not trained on text “as is” – meaning that the model 
  is not digesting the text in a way humans would, front to back. The text does not represent a 
  copy of the original text in its original form. Instead, the text is processed in smaller chunks 
  of text, which are then shuﬄed and “tokenized,” as we explain further below. 
  
  One way to conceptualize the chunking, shuﬄing and tokenizing process is to imagine a 900 
  page book, which has 400,000 words. To feed into an AI model, the book will ﬁrst be cut into 
  manageable chunks of text that represent up to several thousand tokens; such a process 
  might result in around 50 “chunks” of text. Each of those chunks will contain long sections of 
  narrative content; however, the chunks themselves will then be randomized, and fed into the 
  AI model out of sequence from each other; the ﬁrst chunk may be text from Chapters 9 and 
  10, while the initial text in Chapter 1 may be in the 30th chunk. Within these chunks, the text 
  itself will be understood by the model as tokens. 