.md
.pdf
Data Augmented Generation
 Contents 
Overview
Related Literature
Fetching
Text Splitting
Relevant Documents
Augmenting
Use Cases
Data Augmented Generation#
Overview#
Language models are trained on large amounts of unstructured data, which makes them fantastic at general purpose text generation. However, there are many instances where you may want the language model to generate text based not on generic data but rather on specific data. Some common examples of this include:
Summarization of a specific piece of text (a website, a private document, etc.)
Question answering over a specific piece of text (a website, a private document, etc.)
Question answering over multiple pieces of text (multiple websites, multiple private documents, etc.)
Using the results of some external call to an API (results from a SQL query, etc.)
All of these examples are instances when you do not want the LLM to generate text based solely on the data it was trained over, but rather you want it to incorporate other external data in some way. At a high level, this process can be broken down into two steps:
Fetching: Fetching the relevant data to include.
Augmenting: Passing the data in as context to the LLM.
This guide is intended to provide an overview of how to do this. This includes an overview of the literature, as well as common tools, abstractions and chains for doing this.
Related Literature#
There are a lot of related papers in this area. Most of them are focused on end-to-end methods that optimize the fetching of the relevant data as well as passing it in as context. These are a few of the papers that are particularly relevant:
RAG: Retrieval Augmented Generation.