# embedding_metadata

key: _node_content
string_value:
{"id_": "2d7bc133-52b4-4d7f-a871-ef925e7f3bbb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Clustering.ipynb", "title": "imports", "tokens": 1743, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "89b0fbc6-8468-4f59-8fc0-46c772ded6cc", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Clustering.ipynb", "title": "imports", "tokens": 1743, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "33b195d3788ce86941015464446578b737d50abff856f45799d369fc2fbb2fc5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "069ed1dd-6412-4fd1-a2c4-d8cb3becbade", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Clustering.ipynb", "title": "imports", "tokens": 1743, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f8637740e73377aed40ad4601c217b1023e980e1d9ec51191991e7db884995d7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4586, "end_char_idx": 7023, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
of co
    ----------------------------------------------------------------------------------------------------
    Cluster 1 Theme: Pet food reviews
    2, Messy and apparently undelicious:   My cat is not a huge fan. Sure, she'll lap up the gravy, but leaves th
    4, The cats like it:   My 7 cats like this food but it is a little yucky for the human. Piece
    5, cant get enough of it!!!:   Our lil shih tzu puppy cannot get enough of it. Everytime she sees the
    1, Food Caused Illness:   I switched my cats over from the Blue Buffalo Wildnerness Food to this
    5, My furbabies LOVE these!:   Shake the container and they come running. Even my boy cat, who isn't 
    ----------------------------------------------------------------------------------------------------
    Cluster 2 Theme: All the reviews are about different types of coffee.
    5, Fog Chaser Coffee:   This coffee has a full body and a rich taste. The price is far below t
    5, Excellent taste:   This is to me a great coffee, once you try it you will enjoy it, this 
    4, Good, but not Wolfgang Puck good:   Honestly, I have to admit that I expected a little better. That's not 
    5, Just My Kind of Coffee:   Coffee Masters Hazelnut coffee used to be carried in a local coffee/pa
    5, Rodeo Drive is Crazy Good Coffee!:   Rodeo Drive is my absolute favorite and I'm ready to order more!  That
    ----------------------------------------------------------------------------------------------------
    Cluster 3 Theme: The theme of these customer reviews is food and drink products.
    5, Wonderful alternative to soda pop:   This is a wonderful alternative to soda pop.  It's carbonated for thos
    5, So convenient, for so little!:   I needed two vanilla beans for the Love Goddess cake that my husbands 
    2, bot very cheesy:   Got this about a month ago.first of all it smells horrible...it tastes
    5, Delicious!:   I am not a huge beer lover.  I do enjoy an occasional Blue Moon (all o
    3, Just ok:   I bought this brand because it was all they had at Ranch 99 near us. I
    ----------------------------------------------------------------------------------------------------


It's important to note that clusters will not necessarily match what you intend to use them for. A larger amount of clusters will focus on more specific patterns, whereas a small number of clusters will usually focus on largest discrepencies in the data.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/Entity_extraction_for_long_documents.ipynb


key: _node_content
string_value:
{"id_": "9f38adbb-a52f-49d8-8bb4-7d807efacb81", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Entity_extraction_for_long_documents.ipynb", "title": "Long Document Content Extraction", "tokens": 2562, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f9bae17-92ff-4a94-b0ff-c442e0292a3e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Entity_extraction_for_long_documents.ipynb", "title": "Long Document Content Extraction", "tokens": 2562, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "381db3223923ff44d2e90b9f6c7028e0d381b0d3c769c3586bf8b4a299d35de4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "436d98dc-334a-4fff-b571-515c3da9551d", "node_type": "1", "metadata": {}, "hash": "6fb7099abf2140b0f2f18b1a886560b7f2662bdb69fa2106cbd4ed9affe4f491", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3115, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Long Document Content Extraction

GPT-3 can help us extract key figures, dates or other bits of important content from documents that are too big to fit into the context window. One approach for solving this is to chunk the document up and process each chunk separately, before combining into one list of answers. 

In this notebook we'll run through this approach:
- Load in a long PDF and pull the text out
- Create a prompt to be used to extract key bits of information
- Chunk up our document and process each chunk to pull any answers out
- Combine them at the end
- This simple approach will then be extended to three more difficult questions

## Approach

- **Setup**: Take a PDF, a Formula 1 Financial Regulation document on Power Units, and extract the text from it for entity extraction. We'll use this to try to extract answers that are buried in the content.
- **Simple Entity Extraction**: Extract key bits of information from chunks of a document by:
    - Creating a template prompt with our questions and an example of the format it expects
    - Create a function to take a chunk of text as input, combine with the prompt and get a response
    - Run a script to chunk the text, extract answers and output them for parsing
- **Complex Entity Extraction**: Ask some more difficult questions which require tougher reasoning to work out

## Setup


```python
!pip install textract
!pip install tiktoken
```


```python
import textract
import os
import openai
import tiktoken

client = openai.OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))

# Extract the raw text from each PDF using textract
text = textract.process('data/fia_f1_power_unit_financial_regulations_issue_1_-_2022-08-16.pdf', method='pdfminer').decode('utf-8')
clean_text = text.replace("  ", " ").replace("\n", "; ").replace(';',' ')
```

## Simple Entity Extraction


```python
# Example prompt - 
document = '<document>'
template_prompt=f'''Extract key pieces of information from this regulation document.
If a particular piece of information is not present, output \"Not specified\".
When you extract a key piece of information, include the closest page number.
Use the following format:\n0. Who is the author\n1. What is the amount of the "Power Unit Cost Cap" in USD, GBP and EUR\n2. What is the value of External Manufacturing Costs in USD\n3. What is the Capital Expenditure Limit in USD\n\nDocument: \"\"\"<document>\"\"\"\n\n0. Who is the author: Tom Anderson (Page 1)\n1.'''
print(template_prompt)
```

    Extract key pieces of information from this regulation document.
    If a particular piece of information is not present, output "Not specified".
    When you extract a key piece of information, include the closest page number.
    Use the following format:
    0. Who is the author
    1. What is the amount of the "Power Unit Cost Cap" in USD, GBP and EUR
    2. What is the value of External Manufacturing Costs in USD
    3. What is the Capital Expenditure Limit in USD
    
    Document: """<document>"""
    
    0. Who is the author: Tom Anderson (Page 1)
    1.


key: _node_content
string_value:
{"id_": "436d98dc-334a-4fff-b571-515c3da9551d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Entity_extraction_for_long_documents.ipynb", "title": "Long Document Content Extraction", "tokens": 2562, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f9bae17-92ff-4a94-b0ff-c442e0292a3e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Entity_extraction_for_long_documents.ipynb", "title": "Long Document Content Extraction", "tokens": 2562, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "381db3223923ff44d2e90b9f6c7028e0d381b0d3c769c3586bf8b4a299d35de4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9f38adbb-a52f-49d8-8bb4-7d807efacb81", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Entity_extraction_for_long_documents.ipynb", "title": "Long Document Content Extraction", "tokens": 2562, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e4425026bb0ed2e30d8f132097de6f9c07907c4ea6e616fe3bd6b75121b1082f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5138e72e-ddc1-4e9c-9edd-53a5ea2a3bcd", "node_type": "1", "metadata": {}, "hash": "151324bbc2bdf37a45e302a88ddc1a8962a1565d3cf0652c856c2558bfa14f16", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1898, "end_char_idx": 4891, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Example prompt - 
document = '<document>'
template_prompt=f'''Extract key pieces of information from this regulation document.
If a particular piece of information is not present, output \"Not specified\".
When you extract a key piece of information, include the closest page number.
Use the following format:\n0. Who is the author\n1. What is the amount of the "Power Unit Cost Cap" in USD, GBP and EUR\n2. What is the value of External Manufacturing Costs in USD\n3. What is the Capital Expenditure Limit in USD\n\nDocument: \"\"\"<document>\"\"\"\n\n0. Who is the author: Tom Anderson (Page 1)\n1.'''
print(template_prompt)
```

    Extract key pieces of information from this regulation document.
    If a particular piece of information is not present, output "Not specified".
    When you extract a key piece of information, include the closest page number.
    Use the following format:
    0. Who is the author
    1. What is the amount of the "Power Unit Cost Cap" in USD, GBP and EUR
    2. What is the value of External Manufacturing Costs in USD
    3. What is the Capital Expenditure Limit in USD
    
    Document: """<document>"""
    
    0. Who is the author: Tom Anderson (Page 1)
    1.



```python
# Split a text into smaller chunks of size n, preferably ending at the end of a sentence
def create_chunks(text, n, tokenizer):
    tokens = tokenizer.encode(text)
    """Yield successive n-sized chunks from text."""
    i = 0
    while i < len(tokens):
        # Find the nearest end of sentence within a range of 0.5 * n and 1.5 * n tokens
        j = min(i + int(1.5 * n), len(tokens))
        while j > i + int(0.5 * n):
            # Decode the tokens and check for full stop or newline
            chunk = tokenizer.decode(tokens[i:j])
            if chunk.endswith(".") or chunk.endswith("\n"):
                break
            j -= 1
        # If no end of sentence found, use n tokens as the chunk size
        if j == i + int(0.5 * n):
            j = min(i + n, len(tokens))
        yield tokens[i:j]
        i = j

def extract_chunk(document,template_prompt):
    prompt = template_prompt.replace('<document>',document)

    messages = [
            {"role": "system", "content": "You help extract information from documents."},
            {"role": "user", "content": prompt}
            ]

    response = client.chat.completions.create(
            model='gpt-4', 
            messages=messages,
            temperature=0,
            max_tokens=1500,
            top_p=1,
            frequency_penalty=0,
            presence_penalty=0
        )
    return "1." + response.choices[0].message.content
```


```python
# Initialise tokenizer
tokenizer = tiktoken.get_encoding("cl100k_base")

results = []
    
chunks = create_chunks(clean_text,1000,tokenizer)
text_chunks = [tokenizer.decode(chunk) for chunk in chunks]

for chunk in text_chunks:
    results.append(extract_chunk(chunk,template_prompt))
    #print(chunk)
    print(results[-1])

```


key: _node_content
string_value:
{"id_": "5138e72e-ddc1-4e9c-9edd-53a5ea2a3bcd", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Entity_extraction_for_long_documents.ipynb", "title": "Long Document Content Extraction", "tokens": 2562, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f9bae17-92ff-4a94-b0ff-c442e0292a3e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Entity_extraction_for_long_documents.ipynb", "title": "Long Document Content Extraction", "tokens": 2562, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "381db3223923ff44d2e90b9f6c7028e0d381b0d3c769c3586bf8b4a299d35de4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "436d98dc-334a-4fff-b571-515c3da9551d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Entity_extraction_for_long_documents.ipynb", "title": "Long Document Content Extraction", "tokens": 2562, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e8bb8575446e109ea97a6a7795ee97de4e1af66c86e8aa7e637438d6b3a30165", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b7bf15a3-7b40-422c-a123-d61fd8e0ab24", "node_type": "1", "metadata": {}, "hash": "1bfd9e3d24ef35de5fc0d193e2b9b9fae4757dc6571aaf858f457069ea1c473f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4551, "end_char_idx": 7063, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Initialise tokenizer
tokenizer = tiktoken.get_encoding("cl100k_base")

results = []
    
chunks = create_chunks(clean_text,1000,tokenizer)
text_chunks = [tokenizer.decode(chunk) for chunk in chunks]

for chunk in text_chunks:
    results.append(extract_chunk(chunk,template_prompt))
    #print(chunk)
    print(results[-1])

```


```python
groups = [r.split('\n') for r in results]

# zip the groups together
zipped = list(zip(*groups))
zipped = [x for y in zipped for x in y if "Not specified" not in x and "__" not in x]
zipped
```




    ['1. What is the amount of the "Power Unit Cost Cap" in USD, GBP and EUR: USD 95,000,000 (Page 2); GBP 76,459,000 (Page 2); EUR 90,210,000 (Page 2)',
     '2. What is the value of External Manufacturing Costs in USD: US Dollars 20,000,000 in respect of each of the Full Year Reporting Periods ending on 31 December 2023, 31 December 2024 and 31 December 2025, adjusted for Indexation (Page 10)',
     '3. What is the Capital Expenditure Limit in USD: US Dollars 30,000,000 (Page 32)']



## Complex Entity Extraction


```python
# Example prompt - 
template_prompt=f'''Extract key pieces of information from this regulation document.
If a particular piece of information is not present, output \"Not specified\".
When you extract a key piece of information, include the closest page number.
Use the following format:\n0. Who is the author\n1. How is a Minor Overspend Breach calculated\n2. How is a Major Overspend Breach calculated\n3. Which years do these financial regulations apply to\n\nDocument: \"\"\"<document>\"\"\"\n\n0. Who is the author: Tom Anderson (Page 1)\n1.'''
print(template_prompt)
```

    Extract key pieces of information from this regulation document.
    If a particular piece of information is not present, output "Not specified".
    When you extract a key piece of information, include the closest page number.
    Use the following format:
    0. Who is the author
    1. How is a Minor Overspend Breach calculated
    2. How is a Major Overspend Breach calculated
    3. Which years do these financial regulations apply to
    
    Document: """<document>"""
    
    0. Who is the author: Tom Anderson (Page 1)
    1.



```python
results = []

for chunk in text_chunks:
    results.append(extract_chunk(chunk,template_prompt))
    
groups = [r.split('\n') for r in results]

# zip the groups together
zipped = list(zip(*groups))
zipped = [x for y in zipped for x in y if "Not specified" not in x and "__" not in x]
zipped
```


key: _node_content
string_value:
{"id_": "b7bf15a3-7b40-422c-a123-d61fd8e0ab24", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Entity_extraction_for_long_documents.ipynb", "title": "Long Document Content Extraction", "tokens": 2562, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f9bae17-92ff-4a94-b0ff-c442e0292a3e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Entity_extraction_for_long_documents.ipynb", "title": "Long Document Content Extraction", "tokens": 2562, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "381db3223923ff44d2e90b9f6c7028e0d381b0d3c769c3586bf8b4a299d35de4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5138e72e-ddc1-4e9c-9edd-53a5ea2a3bcd", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Entity_extraction_for_long_documents.ipynb", "title": "Long Document Content Extraction", "tokens": 2562, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d9c21d1109291eb68983e7c206e4383e6901381fde00051d736d7c2e91b10c0a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "df40e0f1-e075-4c2d-8010-cda51c406821", "node_type": "1", "metadata": {}, "hash": "82da62b1640d4e5ac2daabf3916d5d5843d910ccdc3b814c2614bf2117cdd480", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5594, "end_char_idx": 9651, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Complex Entity Extraction


```python
# Example prompt - 
template_prompt=f'''Extract key pieces of information from this regulation document.
If a particular piece of information is not present, output \"Not specified\".
When you extract a key piece of information, include the closest page number.
Use the following format:\n0. Who is the author\n1. How is a Minor Overspend Breach calculated\n2. How is a Major Overspend Breach calculated\n3. Which years do these financial regulations apply to\n\nDocument: \"\"\"<document>\"\"\"\n\n0. Who is the author: Tom Anderson (Page 1)\n1.'''
print(template_prompt)
```

    Extract key pieces of information from this regulation document.
    If a particular piece of information is not present, output "Not specified".
    When you extract a key piece of information, include the closest page number.
    Use the following format:
    0. Who is the author
    1. How is a Minor Overspend Breach calculated
    2. How is a Major Overspend Breach calculated
    3. Which years do these financial regulations apply to
    
    Document: """<document>"""
    
    0. Who is the author: Tom Anderson (Page 1)
    1.



```python
results = []

for chunk in text_chunks:
    results.append(extract_chunk(chunk,template_prompt))
    
groups = [r.split('\n') for r in results]

# zip the groups together
zipped = list(zip(*groups))
zipped = [x for y in zipped for x in y if "Not specified" not in x and "__" not in x]
zipped
```




    ['1. How is a Minor Overspend Breach calculated: A Minor Overspend Breach arises when a Power Unit Manufacturer submits its Full Year Reporting Documentation and Relevant Costs reported therein exceed the Power Unit Cost Cap by less than 5% (Page 24)',
     '2. How is a Major Overspend Breach calculated: A Material Overspend Breach arises when a Power Unit Manufacturer submits its Full Year Reporting Documentation and Relevant Costs reported therein exceed the Power Unit Cost Cap by 5% or more (Page 25)',
     '3. Which years do these financial regulations apply to: 2026 onwards (Page 1)',
     '3. Which years do these financial regulations apply to: 2023, 2024, 2025, 2026 and subsequent Full Year Reporting Periods (Page 2)',
     '3. Which years do these financial regulations apply to: 2022-2025 (Page 6)',
     '3. Which years do these financial regulations apply to: 2023, 2024, 2025, 2026 and subsequent Full Year Reporting Periods (Page 10)',
     '3. Which years do these financial regulations apply to: 2022 (Page 14)',
     '3. Which years do these financial regulations apply to: 2022 (Page 16)',
     '3. Which years do these financial regulations apply to: 2022 (Page 19)',
     '3. Which years do these financial regulations apply to: 2022 (Page 21)',
     '3. Which years do these financial regulations apply to: 2026 onwards (Page 26)',
     '3. Which years do these financial regulations apply to: 2026 (Page 2)',
     '3. Which years do these financial regulations apply to: 2022 (Page 30)',
     '3. Which years do these financial regulations apply to: 2022 (Page 32)',
     '3. Which years do these financial regulations apply to: 2023, 2024 and 2025 (Page 1)',
     '3. Which years do these financial regulations apply to: 2022 (Page 37)',
     '3. Which years do these financial regulations apply to: 2026 onwards (Page 40)',
     '3. Which years do these financial regulations apply to: 2022 (Page 1)',
     '3. Which years do these financial regulations apply to: 2026 to 2030 seasons (Page 46)',
     '3. Which years do these financial regulations apply to: 2022 (Page 47)',
     '3. Which years do these financial regulations apply to: 2022 (Page 1)',
     '3. Which years do these financial regulations apply to: 2022 (Page 1)',
     '3. Which years do these financial regulations apply to: 2022 (Page 56)',
     '3. Which years do these financial regulations apply to: 2022 (Page 1)',
     '3. Which years do these financial regulations apply to: 2022 (Page 16)',
     '3. Which years do these financial regulations apply to: 2022 (Page 16)']


key: _node_content
string_value:
{"id_": "df40e0f1-e075-4c2d-8010-cda51c406821", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Entity_extraction_for_long_documents.ipynb", "title": "Long Document Content Extraction", "tokens": 2562, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f9bae17-92ff-4a94-b0ff-c442e0292a3e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Entity_extraction_for_long_documents.ipynb", "title": "Long Document Content Extraction", "tokens": 2562, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "381db3223923ff44d2e90b9f6c7028e0d381b0d3c769c3586bf8b4a299d35de4", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b7bf15a3-7b40-422c-a123-d61fd8e0ab24", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Entity_extraction_for_long_documents.ipynb", "title": "Long Document Content Extraction", "tokens": 2562, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "2840c5109d1270ea5e841856a87f5bee08464b6ccb5b158cea4ce31bdd145a63", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9655, "end_char_idx": 10502, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Consolidation

We've been able to extract the first two answers safely, while the third was confounded by the date that appeared on every page, though the correct answer is in there as well.

To tune this further you can consider experimenting with:
- A more descriptive or specific prompt
- If you have sufficient training data, fine-tuning a model to find a set of outputs very well
- The way you chunk your data - we have gone for 1000 tokens with no overlap, but more intelligent chunking that breaks info into sections, cuts by tokens or similar may get better results

However, with minimal tuning we have now answered 6 questions of varying difficulty using the contents of a long document, and have a reusable approach that we can apply to any long document requiring entity extraction. Look forward to seeing what you can do with this!


key: _node_content
string_value:
{"id_": "6aa9db15-4523-4937-aefe-02693a331f4f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b3208930-1208-4c3e-897b-d2d9a6d9751d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c5aafccd5656c280e661d3ef3fcbfcc93b5dbf7c14ea277183d9a8f594db66c9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "321f0f05-b099-4408-ab06-b59fce59b998", "node_type": "1", "metadata": {}, "hash": "21b4b95f8086b8bc0a8d00f03fc67fb41afaabe28e7b17ee6009a282eb917812", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3808, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative


A lot of enterprise data is unstructured and locked up in difficult-to-use formats, e.g. PDFs, PPT, PNG, that are not optimized for use with LLMs or databases. As a result this type of data tends to be underutilized for analysis and product development, despite it being so valuable. The traditional way of extracting information from unstructured or non-ideal formats has been to use OCR, but OCR struggles with complex layouts and can have limited multilingual support. Moreover, manually applying transforms to data can be cumbersome and timeconsuming. 

The multi-modal capabilities of GPT-4o enable new ways to extract and transform data because of GPT-4o's ability to adapt to different types of documents and to use reasoning for interpreting the content of documents. Here are some reasons why you would choose GPT-4o for your extraction and transformation workflows over traditional methods. 


| **Extraction**                                               | **Transformation**                                              |
|---------------------------------------------------------------|------------------------------------------------------------------|
| **Adaptable**: Handles complex document layouts better, reducing errors | **Schema Adaptability**: Easily transforms data to fit specific schemas for database ingestion |
| **Multilingual Support**: Seamlessly processes documents in multiple languages | **Dynamic Data Mapping**: Adapts to different data structures and formats, providing flexible transformation rules |
| **Contextual Understanding**: Extracts meaningful relationships and context, not just text | **Enhanced Insight Generation**: Applies reasoning to create more insightful transformations, enriching the dataset with derived metrics, metadata and relationships |
| **Multimodality**: Processes various document elements, including images and tables |  |


This cookbook has three parts:
1. How to extract data from multilingual PDFs 
2. How to transform data according to a schema for loading into a database
3. How to load transformed data into a database for downstream analysis

We're going to mimic a simple ELT workflow where data is first extracted from PDFs into JSON using GPT-4o, stored in an unstructured format somewhere like a data lake, transformed to fit a schema using GPT-4o, and then finally ingested into a relational database for querying. It's worth noting that you can do all of this with the BatchAPI if you're interested in lowering the cost of this workflow. 

![](../images/elt_workflow.png)

The data we'll be using is a set of publicly available 2019 hotel invoices from Germany available on [Jens Walter's GitHub](https://github.com/JensWalter/my-receipts/tree/master/2019/de/hotel), (thank you Jens!). Though hotel invoices generally contain similar information (reservation details, charges, taxes etc.), you'll notice that the invoices present itemized information in different ways and are multilingual containing both German and English. Fortunately GPT-4o can adapt to a variety of different document styles without us having to specify formats and it can seamlessly handle a variety of languages, even in the same document. 
Here is what one of the invoices looks like: 

![](../images/sample_hotel_invoice.png)

## Part 1: Extracting data from PDFs using GPT-4o's vision capabilities
GPT-4o doesn't natively handle PDFs so before we extract any data we'll first need to convert each page into an image and then encode the images as base64. 


```python
from openai import OpenAI
import fitz  # PyMuPDF
import io
import os
from PIL import Image
import base64
import json

api_key = os.getenv("OPENAI_API_KEY")
client = OpenAI(api_key=api_key)


key: _node_content
string_value:
{"id_": "321f0f05-b099-4408-ab06-b59fce59b998", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b3208930-1208-4c3e-897b-d2d9a6d9751d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c5aafccd5656c280e661d3ef3fcbfcc93b5dbf7c14ea277183d9a8f594db66c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6aa9db15-4523-4937-aefe-02693a331f4f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "6e88793457b1108dbfa97efb38b788420f9c473279d82fbde8783c12a7fcfbde", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6197227d-ecca-4f5b-bb67-3b4cf4e93df4", "node_type": "1", "metadata": {}, "hash": "edb704ccd4f00fa63d1f595c7a017798a77aa32674ec2b8dfd5c210ea2df79af", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3610, "end_char_idx": 4957, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from openai import OpenAI
import fitz  # PyMuPDF
import io
import os
from PIL import Image
import base64
import json

api_key = os.getenv("OPENAI_API_KEY")
client = OpenAI(api_key=api_key)


@staticmethod
def encode_image(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode("utf-8")


def pdf_to_base64_images(pdf_path):
    #Handles PDFs with multiple pages
    pdf_document = fitz.open(pdf_path)
    base64_images = []
    temp_image_paths = []

    total_pages = len(pdf_document)

    for page_num in range(total_pages):
        page = pdf_document.load_page(page_num)
        pix = page.get_pixmap()
        img = Image.open(io.BytesIO(pix.tobytes()))
        temp_image_path = f"temp_page_{page_num}.png"
        img.save(temp_image_path, format="PNG")
        temp_image_paths.append(temp_image_path)
        base64_image = encode_image(temp_image_path)
        base64_images.append(base64_image)

    for temp_image_path in temp_image_paths:
        os.remove(temp_image_path)

    return base64_images
```

We can then pass each base64 encoded image in a GPT-4o LLM call, specifying a high level of detail and JSON as the response format. We're not concerned about enforcing a schema at this step, we just want all of the data to be extracted regardless of type.


key: _node_content
string_value:
{"id_": "6197227d-ecca-4f5b-bb67-3b4cf4e93df4", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b3208930-1208-4c3e-897b-d2d9a6d9751d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c5aafccd5656c280e661d3ef3fcbfcc93b5dbf7c14ea277183d9a8f594db66c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "321f0f05-b099-4408-ab06-b59fce59b998", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1d43fc296867b7c2124834bab0923037a776d418510fb69ae64309ba683b7f89", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "71cb5c8e-7579-4b1c-a4d5-8b5db94de899", "node_type": "1", "metadata": {}, "hash": "cdcc1014fa7593ba314c07f6ca9e0cc8c9278cef62446c9d15fd71b2a4e14931", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3610, "end_char_idx": 7089, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from openai import OpenAI
import fitz  # PyMuPDF
import io
import os
from PIL import Image
import base64
import json

api_key = os.getenv("OPENAI_API_KEY")
client = OpenAI(api_key=api_key)


@staticmethod
def encode_image(image_path):
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode("utf-8")


def pdf_to_base64_images(pdf_path):
    #Handles PDFs with multiple pages
    pdf_document = fitz.open(pdf_path)
    base64_images = []
    temp_image_paths = []

    total_pages = len(pdf_document)

    for page_num in range(total_pages):
        page = pdf_document.load_page(page_num)
        pix = page.get_pixmap()
        img = Image.open(io.BytesIO(pix.tobytes()))
        temp_image_path = f"temp_page_{page_num}.png"
        img.save(temp_image_path, format="PNG")
        temp_image_paths.append(temp_image_path)
        base64_image = encode_image(temp_image_path)
        base64_images.append(base64_image)

    for temp_image_path in temp_image_paths:
        os.remove(temp_image_path)

    return base64_images
```

We can then pass each base64 encoded image in a GPT-4o LLM call, specifying a high level of detail and JSON as the response format. We're not concerned about enforcing a schema at this step, we just want all of the data to be extracted regardless of type.


```python
def extract_invoice_data(base64_image):
    system_prompt = f"""
    You are an OCR-like data extraction tool that extracts hotel invoice data from PDFs.
   
    1. Please extract the data in this hotel invoice, grouping data according to theme/sub groups, and then output into JSON.

    2. Please keep the keys and values of the JSON in the original language. 

    3. The type of data you might encounter in the invoice includes but is not limited to: hotel information, guest information, invoice information,
    room charges, taxes, and total charges etc. 

    4. If the page contains no charge data, please output an empty JSON object and don't make up any data.

    5. If there are blank data fields in the invoice, please include them as "null" values in the JSON object.
    
    6. If there are tables in the invoice, capture all of the rows and columns in the JSON object. 
    Even if a column is blank, include it as a key in the JSON object with a null value.
    
    7. If a row is blank denote missing fields with "null" values. 
    
    8. Don't interpolate or make up data.

    9. Please maintain the table structure of the charges, i.e. capture all of the rows and columns in the JSON object.

    """
    
    response = client.chat.completions.create(
        model="gpt-4o",
        response_format={ "type": "json_object" },
        messages=[
            {
                "role": "system",
                "content": system_prompt
            },
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": "extract the data in this hotel invoice and output into JSON "},
                    {"type": "image_url", "image_url": {"url": f"data:image/png;base64,{base64_image}", "detail": "high"}}
                ]
            }
        ],
        temperature=0.0,
    )
    return response.choices[0].message.content

```

Because invoice data can span multiple pages in a PDF, we're going to produce JSON objects for each page in the invoice and then append them together. The final invoice extraction will be a single JSON file.


key: _node_content
string_value:
{"id_": "71cb5c8e-7579-4b1c-a4d5-8b5db94de899", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b3208930-1208-4c3e-897b-d2d9a6d9751d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c5aafccd5656c280e661d3ef3fcbfcc93b5dbf7c14ea277183d9a8f594db66c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6197227d-ecca-4f5b-bb67-3b4cf4e93df4", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b8d5ffde8077abb76f53980531a48dc513c9dbe84dab347767388c3d7137e000", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "eedff53d-a28a-47c8-be61-4e014f7474d7", "node_type": "1", "metadata": {}, "hash": "fd83903713e7f52cbbf8b53db6f352762af8bbb7002e5f8c92aa997269f680cd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7092, "end_char_idx": 10103, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def extract_from_multiple_pages(base64_images, original_filename, output_directory):
    entire_invoice = []

    for base64_image in base64_images:
        invoice_json = extract_invoice_data(base64_image)
        invoice_data = json.loads(invoice_json)
        entire_invoice.append(invoice_data)

    # Ensure the output directory exists
    os.makedirs(output_directory, exist_ok=True)

    # Construct the output file path
    output_filename = os.path.join(output_directory, original_filename.replace('.pdf', '_extracted.json'))
    
    # Save the entire_invoice list as a JSON file
    with open(output_filename, 'w', encoding='utf-8') as f:
        json.dump(entire_invoice, f, ensure_ascii=False, indent=4)
    return output_filename


def main_extract(read_path, write_path):
    for filename in os.listdir(read_path):
        file_path = os.path.join(read_path, filename)
        if os.path.isfile(file_path):
            base64_images = pdf_to_base64_images(file_path)
            extract_from_multiple_pages(base64_images, filename, write_path)


read_path= "./data/hotel_invoices/receipts_2019_de_hotel"
write_path= "./data/hotel_invoices/extracted_invoice_json"

main_extract(read_path, write_path)

```

Each invoice JSON will have different keys depending on what data the original invoice contained, so at this point you can store the unschematized JSON files in a data lake that can handle unstructured data. For simplicity though, we're going to store the files in a folder. Here is what one of the extracted JSON files looks like, you'll notice that even though we didn't specify a schema, GPT-4o was able to understand German and group similar information together. Moreover, if there was a blank field in the invoice GPT-4o transcribed that as "null". 


```python
[
    {
        "Hotel Information": {
            "Name": "Hamburg City (Zentrum)",
            "Address": "Willy-Brandt-Straße 21, 20457 Hamburg, Deutschland",
            "Phone": "+49 (0) 40 3039 379 0"
        },
        "Guest Information": {
            "Name": "APIMEISTER CONSULTING GmbH",
            "Guest": "Herr Jens Walter",
            "Address": "Friedrichstr. 123, 10117 Berlin"
        },
        "Invoice Information": {
            "Rechnungsnummer": "GABC19014325",
            "Rechnungsdatum": "23.09.19",
            "Referenznummer": "GABC015452127",
            "Buchungsnummer": "GABR15867",
            "Ankunft": "23.09.19",
            "Abreise": "27.09.19",
            "Nächte": 4,
            "Zimmer": 626,
            "Kundereferenz": 2
        },
        "Charges": [
            {
                "Datum": "23.09.19",
                "Uhrzeit": "16:36",
                "Beschreibung": "Übernachtung",
                "MwSt.%": 7.0,
                "Betrag": 77.0,
                "Zahlung": null
            },
            {
                "Datum": "24.09.19",
                "Uhrzeit": null,
                "Beschreibung": "Übernachtung",
                "MwSt.%": 7.0,


key: _node_content
string_value:
{"id_": "eedff53d-a28a-47c8-be61-4e014f7474d7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b3208930-1208-4c3e-897b-d2d9a6d9751d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c5aafccd5656c280e661d3ef3fcbfcc93b5dbf7c14ea277183d9a8f594db66c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "71cb5c8e-7579-4b1c-a4d5-8b5db94de899", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f4a611220a5e7d5fdc7d5f9f57c46bd3f01f40d9b74459e8c34017e7b65a9e1d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2e2f78fb-5246-41b7-b3d4-ac79de86dab8", "node_type": "1", "metadata": {}, "hash": "0203665dd1f8aef346d506b29a65d2ddc8b9ba9f29bb15061d29b1d21ee4df96", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8881, "end_char_idx": 11401, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
[
    {
        "Hotel Information": {
            "Name": "Hamburg City (Zentrum)",
            "Address": "Willy-Brandt-Straße 21, 20457 Hamburg, Deutschland",
            "Phone": "+49 (0) 40 3039 379 0"
        },
        "Guest Information": {
            "Name": "APIMEISTER CONSULTING GmbH",
            "Guest": "Herr Jens Walter",
            "Address": "Friedrichstr. 123, 10117 Berlin"
        },
        "Invoice Information": {
            "Rechnungsnummer": "GABC19014325",
            "Rechnungsdatum": "23.09.19",
            "Referenznummer": "GABC015452127",
            "Buchungsnummer": "GABR15867",
            "Ankunft": "23.09.19",
            "Abreise": "27.09.19",
            "Nächte": 4,
            "Zimmer": 626,
            "Kundereferenz": 2
        },
        "Charges": [
            {
                "Datum": "23.09.19",
                "Uhrzeit": "16:36",
                "Beschreibung": "Übernachtung",
                "MwSt.%": 7.0,
                "Betrag": 77.0,
                "Zahlung": null
            },
            {
                "Datum": "24.09.19",
                "Uhrzeit": null,
                "Beschreibung": "Übernachtung",
                "MwSt.%": 7.0,
                "Betrag": 135.0,
                "Zahlung": null
            },
            {
                "Datum": "25.09.19",
                "Uhrzeit": null,
                "Beschreibung": "Übernachtung",
                "MwSt.%": 7.0,
                "Betrag": 82.0,
                "Zahlung": null
            },
            {
                "Datum": "26.09.19",
                "Uhrzeit": null,
                "Beschreibung": "Übernachtung",
                "MwSt.%": 7.0,
                "Betrag": 217.0,
                "Zahlung": null
            },
            {
                "Datum": "24.09.19",
                "Uhrzeit": "9:50",
                "Beschreibung": "Premier Inn Frühstücksbuffet",
                "MwSt.%": 19.0,
                "Betrag": 9.9,
                "Zahlung": null
            },
            {
                "Datum": "25.09.19",
                "Uhrzeit": "9:50",
                "Beschreibung": "Premier Inn Frühstücksbuffet",
                "MwSt.%": 19.0,
                "Betrag": 9.9,
                "Zahlung": null
            },
            {
                "Datum": "26.09.19",
                "Uhrzeit": "9:50",
                "Beschreibung": "Premier Inn Frühstücksbuffet",
                "MwSt.%": 19.0,
                "Betrag": 9.9,


key: _node_content
string_value:
{"id_": "2e2f78fb-5246-41b7-b3d4-ac79de86dab8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b3208930-1208-4c3e-897b-d2d9a6d9751d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c5aafccd5656c280e661d3ef3fcbfcc93b5dbf7c14ea277183d9a8f594db66c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "eedff53d-a28a-47c8-be61-4e014f7474d7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ba58a757e26943eac125cd331825fde886c7415897bb35a7430c94138767b9a7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a5dc91b1-00bb-4bcb-92a9-3e7107475ce6", "node_type": "1", "metadata": {}, "hash": "5716c6457082c3f49eee969fd6abe79868d768629b07904ab59ef0b29ca5bd8f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10101, "end_char_idx": 12728, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0,
                "Betrag": 135.0,
                "Zahlung": null
            },
            {
                "Datum": "25.09.19",
                "Uhrzeit": null,
                "Beschreibung": "Übernachtung",
                "MwSt.%": 7.0,
                "Betrag": 82.0,
                "Zahlung": null
            },
            {
                "Datum": "26.09.19",
                "Uhrzeit": null,
                "Beschreibung": "Übernachtung",
                "MwSt.%": 7.0,
                "Betrag": 217.0,
                "Zahlung": null
            },
            {
                "Datum": "24.09.19",
                "Uhrzeit": "9:50",
                "Beschreibung": "Premier Inn Frühstücksbuffet",
                "MwSt.%": 19.0,
                "Betrag": 9.9,
                "Zahlung": null
            },
            {
                "Datum": "25.09.19",
                "Uhrzeit": "9:50",
                "Beschreibung": "Premier Inn Frühstücksbuffet",
                "MwSt.%": 19.0,
                "Betrag": 9.9,
                "Zahlung": null
            },
            {
                "Datum": "26.09.19",
                "Uhrzeit": "9:50",
                "Beschreibung": "Premier Inn Frühstücksbuffet",
                "MwSt.%": 19.0,
                "Betrag": 9.9,
                "Zahlung": null
            },
            {
                "Datum": "27.09.19",
                "Uhrzeit": "9:50",
                "Beschreibung": "Premier Inn Frühstücksbuffet",
                "MwSt.%": 19.0,
                "Betrag": 9.9,
                "Zahlung": null
            }
        ],
        "Payment Information": {
            "Zahlung": "550,60",
            "Gesamt (Rechnungsbetrag)": "550,60",
            "Offener Betrag": "0,00",
            "Bezahlart": "Mastercard-Kreditkarte"
        },
        "Tax Information": {
            "MwSt.%": [
                {
                    "Rate": 19.0,
                    "Netto": 33.28,
                    "MwSt.": 6.32,
                    "Brutto": 39.6
                },
                {
                    "Rate": 7.0,
                    "Netto": 477.57,
                    "MwSt.": 33.43,
                    "Brutto": 511.0
                }
            ]
        }
    }
]
```

## Part 2: Transforming data according to a schema 

You've extracted data from PDFs and have likely loaded the unstructured extractions as JSON objects in a data lake. The next step in our ELT workflow is to use GPT-4o to transform the extractions according to our desired schema. This will enable us to ingest any resulting tables into a database.


key: _node_content
string_value:
{"id_": "a5dc91b1-00bb-4bcb-92a9-3e7107475ce6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b3208930-1208-4c3e-897b-d2d9a6d9751d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c5aafccd5656c280e661d3ef3fcbfcc93b5dbf7c14ea277183d9a8f594db66c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2e2f78fb-5246-41b7-b3d4-ac79de86dab8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ec2567bfe00d21f6efaf478810abf320e6ea29bccca8917b7152a9ab317a269f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "db9739ce-7174-4286-8a50-e3c215d53027", "node_type": "1", "metadata": {}, "hash": "a973d98aa1e3718e6f3fd2f385a287b1324024271a0ec60c31338ef2caf2c3ae", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11355, "end_char_idx": 13088, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"MwSt.%": 19.0,
                "Betrag": 9.9,
                "Zahlung": null
            },
            {
                "Datum": "27.09.19",
                "Uhrzeit": "9:50",
                "Beschreibung": "Premier Inn Frühstücksbuffet",
                "MwSt.%": 19.0,
                "Betrag": 9.9,
                "Zahlung": null
            }
        ],
        "Payment Information": {
            "Zahlung": "550,60",
            "Gesamt (Rechnungsbetrag)": "550,60",
            "Offener Betrag": "0,00",
            "Bezahlart": "Mastercard-Kreditkarte"
        },
        "Tax Information": {
            "MwSt.%": [
                {
                    "Rate": 19.0,
                    "Netto": 33.28,
                    "MwSt.": 6.32,
                    "Brutto": 39.6
                },
                {
                    "Rate": 7.0,
                    "Netto": 477.57,
                    "MwSt.": 33.43,
                    "Brutto": 511.0
                }
            ]
        }
    }
]
```

## Part 2: Transforming data according to a schema 

You've extracted data from PDFs and have likely loaded the unstructured extractions as JSON objects in a data lake. The next step in our ELT workflow is to use GPT-4o to transform the extractions according to our desired schema. This will enable us to ingest any resulting tables into a database. We've decided upon the following schema that broadly covers most of the information we would have seen across the different invoices. This schema will be used to process each raw JSON extraction into our desired schematized JSON and can specify particular formats such as "date": "YYYY-MM-DD". We're also going to translate the data into English at this step.


key: _node_content
string_value:
{"id_": "db9739ce-7174-4286-8a50-e3c215d53027", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b3208930-1208-4c3e-897b-d2d9a6d9751d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c5aafccd5656c280e661d3ef3fcbfcc93b5dbf7c14ea277183d9a8f594db66c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a5dc91b1-00bb-4bcb-92a9-3e7107475ce6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b3bf9dc8185b731b88e6659f57d620228910601d5b1beebdf4b1d189fee4ffa3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b9d43aa3-8c09-410b-b312-69d1ba8a5a8d", "node_type": "1", "metadata": {}, "hash": "41eb3815c43db3942efe7545245f747249cf906321e11dd9c568688d222e875b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11551, "end_char_idx": 14751, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"Beschreibung": "Premier Inn Frühstücksbuffet",
                "MwSt.%": 19.0,
                "Betrag": 9.9,
                "Zahlung": null
            }
        ],
        "Payment Information": {
            "Zahlung": "550,60",
            "Gesamt (Rechnungsbetrag)": "550,60",
            "Offener Betrag": "0,00",
            "Bezahlart": "Mastercard-Kreditkarte"
        },
        "Tax Information": {
            "MwSt.%": [
                {
                    "Rate": 19.0,
                    "Netto": 33.28,
                    "MwSt.": 6.32,
                    "Brutto": 39.6
                },
                {
                    "Rate": 7.0,
                    "Netto": 477.57,
                    "MwSt.": 33.43,
                    "Brutto": 511.0
                }
            ]
        }
    }
]
```

## Part 2: Transforming data according to a schema 

You've extracted data from PDFs and have likely loaded the unstructured extractions as JSON objects in a data lake. The next step in our ELT workflow is to use GPT-4o to transform the extractions according to our desired schema. This will enable us to ingest any resulting tables into a database. We've decided upon the following schema that broadly covers most of the information we would have seen across the different invoices. This schema will be used to process each raw JSON extraction into our desired schematized JSON and can specify particular formats such as "date": "YYYY-MM-DD". We're also going to translate the data into English at this step. 



```python
[
    {
        "hotel_information": {
            "name": "string",
            "address": {
                "street": "string",
                "city": "string",
                "country": "string",
                "postal_code": "string"
            },
            "contact": {
                "phone": "string",
                "fax": "string",
                "email": "string",
                "website": "string"
            }
        },
        "guest_information": {
            "company": "string",
            "address": "string",
            "guest_name": "string"
        },
        "invoice_information": {
            "invoice_number": "string",
            "reservation_number": "string",
            "date": "YYYY-MM-DD",  
            "room_number": "string",
            "check_in_date": "YYYY-MM-DD",  
            "check_out_date": "YYYY-MM-DD"  
        },
        "charges": [
            {
                "date": "YYYY-MM-DD", 
                "description": "string",
                "charge": "number",
                "credit": "number"
            }
        ],
        "totals_summary": {
            "currency": "string",
            "total_net": "number",
            "total_tax": "number",
            "total_gross": "number",
            "total_charge": "number",
            "total_credit": "number",
            "balance_due": "number"
        },
        "taxes": [
            {
                "tax_type": "string",
                "tax_rate": "string",
                "net_amount": "number",
                "tax_amount": "number",
                "gross_amount": "number"
            }
        ]
    }
]

```


key: _node_content
string_value:
{"id_": "b9d43aa3-8c09-410b-b312-69d1ba8a5a8d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b3208930-1208-4c3e-897b-d2d9a6d9751d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c5aafccd5656c280e661d3ef3fcbfcc93b5dbf7c14ea277183d9a8f594db66c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "db9739ce-7174-4286-8a50-e3c215d53027", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7f19e8b4a78ddca1fd7fbaa3fe7cba959ff20f582466df8ebd2c050dcded12b3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "28df222e-23e6-4c58-b7be-ed756822404b", "node_type": "1", "metadata": {}, "hash": "9eb1e0a24f3d52e555cc8d653af7d597d48c078ad28ce290f5d89f6b9ed717e5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12863, "end_char_idx": 16009, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This schema will be used to process each raw JSON extraction into our desired schematized JSON and can specify particular formats such as "date": "YYYY-MM-DD". We're also going to translate the data into English at this step. 



```python
[
    {
        "hotel_information": {
            "name": "string",
            "address": {
                "street": "string",
                "city": "string",
                "country": "string",
                "postal_code": "string"
            },
            "contact": {
                "phone": "string",
                "fax": "string",
                "email": "string",
                "website": "string"
            }
        },
        "guest_information": {
            "company": "string",
            "address": "string",
            "guest_name": "string"
        },
        "invoice_information": {
            "invoice_number": "string",
            "reservation_number": "string",
            "date": "YYYY-MM-DD",  
            "room_number": "string",
            "check_in_date": "YYYY-MM-DD",  
            "check_out_date": "YYYY-MM-DD"  
        },
        "charges": [
            {
                "date": "YYYY-MM-DD", 
                "description": "string",
                "charge": "number",
                "credit": "number"
            }
        ],
        "totals_summary": {
            "currency": "string",
            "total_net": "number",
            "total_tax": "number",
            "total_gross": "number",
            "total_charge": "number",
            "total_credit": "number",
            "balance_due": "number"
        },
        "taxes": [
            {
                "tax_type": "string",
                "tax_rate": "string",
                "net_amount": "number",
                "tax_amount": "number",
                "gross_amount": "number"
            }
        ]
    }
]

```


```python
def transform_invoice_data(json_raw, json_schema):
    system_prompt = f"""
    You are a data transformation tool that takes in JSON data and a reference JSON schema, and outputs JSON data according to the schema.
    Not all of the data in the input JSON will fit the schema, so you may need to omit some data or add null values to the output JSON.
    Translate all data into English if not already in English.
    Ensure values are formatted as specified in the schema (e.g. dates as YYYY-MM-DD).
    Here is the schema:
    {json_schema}

    """
    
    response = client.chat.completions.create(
        model="gpt-4o",
        response_format={ "type": "json_object" },
        messages=[
            {
                "role": "system",
                "content": system_prompt
            },
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": f"Transform the following raw JSON data according to the provided schema. Ensure all data is in English and formatted as specified by values in the schema. Here is the raw JSON: {json_raw}"}
                ]
            }
        ],
        temperature=0.0,
    )
    return json.loads(response.choices[0].message.content)


key: _node_content
string_value:
{"id_": "28df222e-23e6-4c58-b7be-ed756822404b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b3208930-1208-4c3e-897b-d2d9a6d9751d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c5aafccd5656c280e661d3ef3fcbfcc93b5dbf7c14ea277183d9a8f594db66c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b9d43aa3-8c09-410b-b312-69d1ba8a5a8d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e30fcbd0d02641d03e5cecdb0a5bae76aa1a0958f6d229bf479433e77af76440", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aa8e372a-cf1f-45e6-8991-c4608423670d", "node_type": "1", "metadata": {}, "hash": "f996a492764e68041203a34087e4cce37dd28dddfcc3296ea1a8a325c75c647a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14754, "end_char_idx": 17787, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def transform_invoice_data(json_raw, json_schema):
    system_prompt = f"""
    You are a data transformation tool that takes in JSON data and a reference JSON schema, and outputs JSON data according to the schema.
    Not all of the data in the input JSON will fit the schema, so you may need to omit some data or add null values to the output JSON.
    Translate all data into English if not already in English.
    Ensure values are formatted as specified in the schema (e.g. dates as YYYY-MM-DD).
    Here is the schema:
    {json_schema}

    """
    
    response = client.chat.completions.create(
        model="gpt-4o",
        response_format={ "type": "json_object" },
        messages=[
            {
                "role": "system",
                "content": system_prompt
            },
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": f"Transform the following raw JSON data according to the provided schema. Ensure all data is in English and formatted as specified by values in the schema. Here is the raw JSON: {json_raw}"}
                ]
            }
        ],
        temperature=0.0,
    )
    return json.loads(response.choices[0].message.content)



def main_transform(extracted_invoice_json_path, json_schema_path, save_path):
    # Load the JSON schema
    with open(json_schema_path, 'r', encoding='utf-8') as f:
        json_schema = json.load(f)

    # Ensure the save directory exists
    os.makedirs(save_path, exist_ok=True)

    # Process each JSON file in the extracted invoices directory
    for filename in os.listdir(extracted_invoice_json_path):
        if filename.endswith(".json"):
            file_path = os.path.join(extracted_invoice_json_path, filename)

            # Load the extracted JSON
            with open(file_path, 'r', encoding='utf-8') as f:
                json_raw = json.load(f)

            # Transform the JSON data
            transformed_json = transform_invoice_data(json_raw, json_schema)

            # Save the transformed JSON to the save directory
            transformed_filename = f"transformed_{filename}"
            transformed_file_path = os.path.join(save_path, transformed_filename)
            with open(transformed_file_path, 'w', encoding='utf-8') as f:
                json.dump(transformed_json, f, ensure_ascii=False, indent=2)

   
    extracted_invoice_json_path = "./data/hotel_invoices/extracted_invoice_json"
    json_schema_path = "./data/hotel_invoices/invoice_schema.json"
    save_path = "./data/hotel_invoices/transformed_invoice_json"

    main_transform(extracted_invoice_json_path, json_schema_path, save_path)
```

## Part 3: Loading transformed data into a database 

Now that we've schematized all of our data, we can segment it into tables for ingesting into a relational database. In particular, we're going to create four tables: Hotels, Invoices, Charges and Taxes. All of the invoices pertained to one guest, so we won't create a guest table.


key: _node_content
string_value:
{"id_": "aa8e372a-cf1f-45e6-8991-c4608423670d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b3208930-1208-4c3e-897b-d2d9a6d9751d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c5aafccd5656c280e661d3ef3fcbfcc93b5dbf7c14ea277183d9a8f594db66c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "28df222e-23e6-4c58-b7be-ed756822404b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "eebbb3079ae0f0ea4931d303c8b85141a44c57f311c6b346bcae1fcbcb930131", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0894407e-994a-4592-bce6-b932c6fe7df7", "node_type": "1", "metadata": {}, "hash": "46c7fcf323fc762bc6bf73ede1fabf6d65400ced1ebcca66e2d4029d3895960a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16013, "end_char_idx": 20164, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
def main_transform(extracted_invoice_json_path, json_schema_path, save_path):
    # Load the JSON schema
    with open(json_schema_path, 'r', encoding='utf-8') as f:
        json_schema = json.load(f)

    # Ensure the save directory exists
    os.makedirs(save_path, exist_ok=True)

    # Process each JSON file in the extracted invoices directory
    for filename in os.listdir(extracted_invoice_json_path):
        if filename.endswith(".json"):
            file_path = os.path.join(extracted_invoice_json_path, filename)

            # Load the extracted JSON
            with open(file_path, 'r', encoding='utf-8') as f:
                json_raw = json.load(f)

            # Transform the JSON data
            transformed_json = transform_invoice_data(json_raw, json_schema)

            # Save the transformed JSON to the save directory
            transformed_filename = f"transformed_{filename}"
            transformed_file_path = os.path.join(save_path, transformed_filename)
            with open(transformed_file_path, 'w', encoding='utf-8') as f:
                json.dump(transformed_json, f, ensure_ascii=False, indent=2)

   
    extracted_invoice_json_path = "./data/hotel_invoices/extracted_invoice_json"
    json_schema_path = "./data/hotel_invoices/invoice_schema.json"
    save_path = "./data/hotel_invoices/transformed_invoice_json"

    main_transform(extracted_invoice_json_path, json_schema_path, save_path)
```

## Part 3: Loading transformed data into a database 

Now that we've schematized all of our data, we can segment it into tables for ingesting into a relational database. In particular, we're going to create four tables: Hotels, Invoices, Charges and Taxes. All of the invoices pertained to one guest, so we won't create a guest table. 


```python
import os
import json
import sqlite3

def ingest_transformed_jsons(json_folder_path, db_path):
    conn = sqlite3.connect(db_path)
    cursor = conn.cursor()

    # Create necessary tables
    cursor.execute('''
    CREATE TABLE IF NOT EXISTS Hotels (
        hotel_id INTEGER PRIMARY KEY AUTOINCREMENT,
        name TEXT,
        street TEXT,
        city TEXT,
        country TEXT,
        postal_code TEXT,
        phone TEXT,
        fax TEXT,
        email TEXT,
        website TEXT
    )
    ''')

    cursor.execute('''
    CREATE TABLE IF NOT EXISTS Invoices (
        invoice_id INTEGER PRIMARY KEY AUTOINCREMENT,
        hotel_id INTEGER,
        invoice_number TEXT,
        reservation_number TEXT,
        date TEXT,
        room_number TEXT,
        check_in_date TEXT,
        check_out_date TEXT,
        currency TEXT,
        total_net REAL,
        total_tax REAL,
        total_gross REAL,
        total_charge REAL,
        total_credit REAL,
        balance_due REAL,
        guest_company TEXT,
        guest_address TEXT,
        guest_name TEXT,
        FOREIGN KEY(hotel_id) REFERENCES Hotels(hotel_id)
    )
    ''')

    cursor.execute('''
    CREATE TABLE IF NOT EXISTS Charges (
        charge_id INTEGER PRIMARY KEY AUTOINCREMENT,
        invoice_id INTEGER,
        date TEXT,
        description TEXT,
        charge REAL,
        credit REAL,
        FOREIGN KEY(invoice_id) REFERENCES Invoices(invoice_id)
    )
    ''')

    cursor.execute('''
    CREATE TABLE IF NOT EXISTS Taxes (
        tax_id INTEGER PRIMARY KEY AUTOINCREMENT,
        invoice_id INTEGER,
        tax_type TEXT,
        tax_rate TEXT,
        net_amount REAL,
        tax_amount REAL,
        gross_amount REAL,
        FOREIGN KEY(invoice_id) REFERENCES Invoices(invoice_id)
    )
    ''')

    # Loop over all JSON files in the specified folder
    for filename in os.listdir(json_folder_path):
        if filename.endswith(".json"):
            file_path = os.path.join(json_folder_path, filename)

            # Load the JSON data
            with open(file_path, 'r', encoding='utf-8') as f:
                data = json.load(f)

            # Insert Hotel Information
            cursor.execute('''
            INSERT INTO Hotels (name, street, city, country, postal_code, phone, fax, email, website) 
            VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?)


key: _node_content
string_value:
{"id_": "0894407e-994a-4592-bce6-b932c6fe7df7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b3208930-1208-4c3e-897b-d2d9a6d9751d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c5aafccd5656c280e661d3ef3fcbfcc93b5dbf7c14ea277183d9a8f594db66c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aa8e372a-cf1f-45e6-8991-c4608423670d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f7037b64d1790500a8d6713ad58ca596d5cc436bd11b2ac4ff173d98ae47bc45", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "11cef8ec-2ee4-4f6d-9e4a-7dfaac13210d", "node_type": "1", "metadata": {}, "hash": "aa996aac1153766ead665b28cf095b8f0d5297ffb55a8e6b498a3e04e88715c1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20177, "end_char_idx": 23714, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
''', (
                data["hotel_information"]["name"],
                data["hotel_information"]["address"]["street"],
                data["hotel_information"]["address"]["city"],
                data["hotel_information"]["address"]["country"],
                data["hotel_information"]["address"]["postal_code"],
                data["hotel_information"]["contact"]["phone"],
                data["hotel_information"]["contact"]["fax"],
                data["hotel_information"]["contact"]["email"],
                data["hotel_information"]["contact"]["website"]
            ))
            hotel_id = cursor.lastrowid

            # Insert Invoice Information
            cursor.execute('''
            INSERT INTO Invoices (hotel_id, invoice_number, reservation_number, date, room_number, check_in_date, check_out_date, currency, total_net, total_tax, total_gross, total_charge, total_credit, balance_due, guest_company, guest_address, guest_name)
            VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)
            ''', (
                hotel_id,
                data["invoice_information"]["invoice_number"],
                data["invoice_information"]["reservation_number"],
                data["invoice_information"]["date"],
                data["invoice_information"]["room_number"],
                data["invoice_information"]["check_in_date"],
                data["invoice_information"]["check_out_date"],
                data["totals_summary"]["currency"],
                data["totals_summary"]["total_net"],
                data["totals_summary"]["total_tax"],
                data["totals_summary"]["total_gross"],
                data["totals_summary"]["total_charge"],
                data["totals_summary"]["total_credit"],
                data["totals_summary"]["balance_due"],
                data["guest_information"]["company"],
                data["guest_information"]["address"],
                data["guest_information"]["guest_name"]
            ))
            invoice_id = cursor.lastrowid

            # Insert Charges
            for charge in data["charges"]:
                cursor.execute('''
                INSERT INTO Charges (invoice_id, date, description, charge, credit) 
                VALUES (?, ?, ?, ?, ?)
                ''', (
                    invoice_id,
                    charge["date"],
                    charge["description"],
                    charge["charge"],
                    charge["credit"]
                ))

            # Insert Taxes
            for tax in data["taxes"]:
                cursor.execute('''
                INSERT INTO Taxes (invoice_id, tax_type, tax_rate, net_amount, tax_amount, gross_amount) 
                VALUES (?, ?, ?, ?, ?, ?)
                ''', (
                    invoice_id,
                    tax["tax_type"],
                    tax["tax_rate"],
                    tax["net_amount"],
                    tax["tax_amount"],
                    tax["gross_amount"]
                ))

    conn.commit()
    conn.close()


```

Now let's check that we've correctly ingested the data by running a sample SQL query to determine the most expensive hotel stay and the same of the hotel!  
You can even automate the generation of SQL queries at this step by using function calling, check out our [cookbook on function calling with model generated arguments](https://cookbook.openai.com/examples/how_to_call_functions_with_chat_models#how-to-call-functions-with-model-generated-arguments) to learn how to do that.


key: _node_content
string_value:
{"id_": "11cef8ec-2ee4-4f6d-9e4a-7dfaac13210d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b3208930-1208-4c3e-897b-d2d9a6d9751d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c5aafccd5656c280e661d3ef3fcbfcc93b5dbf7c14ea277183d9a8f594db66c9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0894407e-994a-4592-bce6-b932c6fe7df7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Data_extraction_transformation.ipynb", "title": "Data Extraction and Transformation in ELT Workflows using GPT-4o as an OCR Alternative", "tokens": 5588, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b1aeb5cf768435f14a7c4bcc423742864da53c08513e8135043d17c39b503a06", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 22465, "end_char_idx": 25559, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
''', (
                    invoice_id,
                    charge["date"],
                    charge["description"],
                    charge["charge"],
                    charge["credit"]
                ))

            # Insert Taxes
            for tax in data["taxes"]:
                cursor.execute('''
                INSERT INTO Taxes (invoice_id, tax_type, tax_rate, net_amount, tax_amount, gross_amount) 
                VALUES (?, ?, ?, ?, ?, ?)
                ''', (
                    invoice_id,
                    tax["tax_type"],
                    tax["tax_rate"],
                    tax["net_amount"],
                    tax["tax_amount"],
                    tax["gross_amount"]
                ))

    conn.commit()
    conn.close()


```

Now let's check that we've correctly ingested the data by running a sample SQL query to determine the most expensive hotel stay and the same of the hotel!  
You can even automate the generation of SQL queries at this step by using function calling, check out our [cookbook on function calling with model generated arguments](https://cookbook.openai.com/examples/how_to_call_functions_with_chat_models#how-to-call-functions-with-model-generated-arguments) to learn how to do that. 


```python

def execute_query(db_path, query, params=()):
    """
    Execute a SQL query and return the results.

    Parameters:
    db_path (str): Path to the SQLite database file.
    query (str): SQL query to be executed.
    params (tuple): Parameters to be passed to the query (default is an empty tuple).

    Returns:
    list: List of rows returned by the query.
    """
    try:
        # Connect to the SQLite database
        conn = sqlite3.connect(db_path)
        cursor = conn.cursor()

        # Execute the query with parameters
        cursor.execute(query, params)
        results = cursor.fetchall()

        # Commit if it's an INSERT/UPDATE/DELETE query
        if query.strip().upper().startswith(('INSERT', 'UPDATE', 'DELETE')):
            conn.commit()

        return results
    except sqlite3.Error as e:
        print(f"An error occurred: {e}")
        return []
    finally:
        # Close the connection
        if conn:
            conn.close()


# Example usage
transformed_invoices_path = "./data/hotel_invoices/transformed_invoice_json"
db_path = "./data/hotel_invoices/hotel_DB.db"
ingest_transformed_jsons(transformed_invoices_path, db_path)

query = '''
    SELECT 
        h.name AS hotel_name,
        i.total_gross AS max_spent
    FROM 
        Invoices i
    JOIN 
        Hotels h ON i.hotel_id = h.hotel_id
    ORDER BY 
        i.total_gross DESC
    LIMIT 1;
    '''

results = execute_query(db_path, query)
for row in results:
    print(row)
```

    ('Citadines Michel Hamburg', 903.63)


To recap in this cookbook we showed you how to use GPT-4o for extracting and transforming data that would otherwise be inaccessible for data analysis. If you don't need these workflows to happen in real-time, you can take advantage of OpenAI's BatchAPI to run jobs asynchronously at a much lower cost!


key: _node_content
string_value:
{"id_": "f40f496f-4d26-4b2f-8850-e42d3b697304", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_correct_misspelling.ipynb", "title": "Addressing transcription misspellings: prompt vs post-processing", "tokens": 3290, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "aae74d07-7672-4eb7-a7b3-ab3900f787b3", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_correct_misspelling.ipynb", "title": "Addressing transcription misspellings: prompt vs post-processing", "tokens": 3290, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "39c2d8f1b8978601ba8107c272786d8fcace4edc2e74c2d8e8fef500e3320128", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2858e862-67eb-4e04-b2ce-098fbd85b845", "node_type": "1", "metadata": {}, "hash": "7b518ba9e713ebc66e02c467319789a95edfc35d2c051671e94c5413a284cb7d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3242, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Addressing transcription misspellings: prompt vs post-processing

We are addressing the problem of enhancing the precision of transcriptions, particularly when it comes to company names and product references. Our solution involves a dual strategy that utilizes both the Whisper prompt parameter and GPT-4's post-processing capabilities. 

Two approaches to correct inaccuracies are:

- We input a list of correct spellings directly into Whisper's prompt parameter to guide the initial transcription.

- We utilized GPT-4 to fix misspellings post transcription, again using the same list of correct spellings in the prompt.

These strategies aimed at ensuring precise transcription of unfamilar proper nouns.

## Setup

To get started, let's:

- Import the OpenAI Python library (if you don't have it, you'll need to install it with ```pip install openai```)
- Download the audio file example


```python
# imports
from openai import OpenAI  # for making OpenAI API calls
import urllib  # for downloading example audio files
import os  # for accessing environment variables

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))
```


```python
# set download paths
ZyntriQix_remote_filepath = "https://cdn.openai.com/API/examples/data/ZyntriQix.wav"


# set local save locations
ZyntriQix_filepath = "data/ZyntriQix.wav"

# download example audio files and save locally
urllib.request.urlretrieve(ZyntriQix_remote_filepath, ZyntriQix_filepath)

```




    ('data/ZyntriQix.wav', <http.client.HTTPMessage at 0x10559a910>)



## Setting our baseline with a fictitious audio recording

Our reference point is a monologue, which was generated by ChatGPT from prompts given by the author. The author then voiced this content. So, the author both guided the ChatGPT's output with prompts and brought it to life by speaking it.

Our fictitious company, ZyntriQix, offers a range of tech products. These include Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, OrbitalLink Seven, and DigiFractal Matrix. We also spearhead several initiatives such as PULSE, RAPT, B.R.I.C.K., Q.U.A.R.T.Z., and F.L.I.N.T.


```python
# define a wrapper function for seeing how prompts affect transcriptions
def transcribe(prompt: str, audio_filepath) -> str:
    """Given a prompt, transcribe the audio file."""
    transcript = client.audio.transcriptions.create(
        file=open(audio_filepath, "rb"),
        model="whisper-1",
        prompt=prompt,
    )
    return transcript.text

```


```python
# baseline transcription with no prompt
transcribe(prompt="", audio_filepath=ZyntriQix_filepath)
```




    "Have you heard of ZentricX? This tech giant boasts products like Digi-Q+, Synapse 5, VortiCore V8, Echo Nix Array, and not to forget the latest Orbital Link 7 and Digifractal Matrix. Their innovation arsenal also includes the Pulse framework, Wrapped system, they've developed a brick infrastructure court system, and launched the Flint initiative, all highlighting their commitment to relentless innovation. ZentricX, in just 30 years, has soared from a startup to a tech titan, serving us tech marvels alongside a stimulating linguistic challenge. Quite an adventure, wouldn't you agree?"


key: _node_content
string_value:
{"id_": "2858e862-67eb-4e04-b2ce-098fbd85b845", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_correct_misspelling.ipynb", "title": "Addressing transcription misspellings: prompt vs post-processing", "tokens": 3290, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "aae74d07-7672-4eb7-a7b3-ab3900f787b3", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_correct_misspelling.ipynb", "title": "Addressing transcription misspellings: prompt vs post-processing", "tokens": 3290, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "39c2d8f1b8978601ba8107c272786d8fcace4edc2e74c2d8e8fef500e3320128", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f40f496f-4d26-4b2f-8850-e42d3b697304", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_correct_misspelling.ipynb", "title": "Addressing transcription misspellings: prompt vs post-processing", "tokens": 3290, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c82ac258985a205e1af48d8530015cc25ed4ec3c9a7c3eefe25926a99f171d77", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c3eecdda-d294-46b5-88d2-104e4ca303b8", "node_type": "1", "metadata": {}, "hash": "fd1aa4e1e4d91e7b080e337c003778275707283af1308e288d0bfba1c798ae51", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1579, "end_char_idx": 4437, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Setting our baseline with a fictitious audio recording

Our reference point is a monologue, which was generated by ChatGPT from prompts given by the author. The author then voiced this content. So, the author both guided the ChatGPT's output with prompts and brought it to life by speaking it.

Our fictitious company, ZyntriQix, offers a range of tech products. These include Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, OrbitalLink Seven, and DigiFractal Matrix. We also spearhead several initiatives such as PULSE, RAPT, B.R.I.C.K., Q.U.A.R.T.Z., and F.L.I.N.T.


```python
# define a wrapper function for seeing how prompts affect transcriptions
def transcribe(prompt: str, audio_filepath) -> str:
    """Given a prompt, transcribe the audio file."""
    transcript = client.audio.transcriptions.create(
        file=open(audio_filepath, "rb"),
        model="whisper-1",
        prompt=prompt,
    )
    return transcript.text

```


```python
# baseline transcription with no prompt
transcribe(prompt="", audio_filepath=ZyntriQix_filepath)
```




    "Have you heard of ZentricX? This tech giant boasts products like Digi-Q+, Synapse 5, VortiCore V8, Echo Nix Array, and not to forget the latest Orbital Link 7 and Digifractal Matrix. Their innovation arsenal also includes the Pulse framework, Wrapped system, they've developed a brick infrastructure court system, and launched the Flint initiative, all highlighting their commitment to relentless innovation. ZentricX, in just 30 years, has soared from a startup to a tech titan, serving us tech marvels alongside a stimulating linguistic challenge. Quite an adventure, wouldn't you agree?"



Whisper transcribed our company name, product names, and miscapitalized our acronyms incorrectly. Let's pass the correct names as a list in the prompt. 


```python
# add the correct spelling names to the prompt
transcribe(
    prompt="ZyntriQix, Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, OrbitalLink Seven, DigiFractal Matrix, PULSE, RAPT, B.R.I.C.K., Q.U.A.R.T.Z., F.L.I.N.T.",
    audio_filepath=ZyntriQix_filepath,
)

```




    "Have you heard of ZyntriQix? This tech giant boasts products like Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, and not to forget the latest OrbitalLink Seven and DigiFractal Matrix. Their innovation arsenal also includes the PULSE framework, RAPT system. They've developed a B.R.I.C.K. infrastructure, Q.U.A.R.T. system, and launched the F.L.I.N.T. initiative, all highlighting their commitment to relentless innovation. ZyntriQix in just 30 years has soared from a startup to a tech titan, serving us tech marvels alongside a stimulating linguistic challenge. Quite an adventure, wouldn't you agree?"



When passing the list of product names, some of the product names are transcribed correctly while others are still misspelled.


key: _node_content
string_value:
{"id_": "c3eecdda-d294-46b5-88d2-104e4ca303b8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_correct_misspelling.ipynb", "title": "Addressing transcription misspellings: prompt vs post-processing", "tokens": 3290, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "aae74d07-7672-4eb7-a7b3-ab3900f787b3", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_correct_misspelling.ipynb", "title": "Addressing transcription misspellings: prompt vs post-processing", "tokens": 3290, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "39c2d8f1b8978601ba8107c272786d8fcace4edc2e74c2d8e8fef500e3320128", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2858e862-67eb-4e04-b2ce-098fbd85b845", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_correct_misspelling.ipynb", "title": "Addressing transcription misspellings: prompt vs post-processing", "tokens": 3290, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "120eb3beefd1a332e5b68ab37aa743165ec35d9d832587a1e4ab4bb7e7f63bd4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "54d8facf-c37f-41a1-8f49-39914a816da7", "node_type": "1", "metadata": {}, "hash": "d9d0e44e73f00959b21b3088dfdbf91c5264dd0bd68ed2cf6c3966c6a65d04f4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3246, "end_char_idx": 5845, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Whisper transcribed our company name, product names, and miscapitalized our acronyms incorrectly. Let's pass the correct names as a list in the prompt. 


```python
# add the correct spelling names to the prompt
transcribe(
    prompt="ZyntriQix, Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, OrbitalLink Seven, DigiFractal Matrix, PULSE, RAPT, B.R.I.C.K., Q.U.A.R.T.Z., F.L.I.N.T.",
    audio_filepath=ZyntriQix_filepath,
)

```




    "Have you heard of ZyntriQix? This tech giant boasts products like Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, and not to forget the latest OrbitalLink Seven and DigiFractal Matrix. Their innovation arsenal also includes the PULSE framework, RAPT system. They've developed a B.R.I.C.K. infrastructure, Q.U.A.R.T. system, and launched the F.L.I.N.T. initiative, all highlighting their commitment to relentless innovation. ZyntriQix in just 30 years has soared from a startup to a tech titan, serving us tech marvels alongside a stimulating linguistic challenge. Quite an adventure, wouldn't you agree?"



When passing the list of product names, some of the product names are transcribed correctly while others are still misspelled. 


```python
# add a full product list to the prompt
transcribe(
    prompt="ZyntriQix, Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, OrbitalLink Seven, DigiFractal Matrix, PULSE, RAPT, AstroPixel Array, QuantumFlare Five, CyberPulse Six, VortexDrive Matrix, PhotonLink Ten, TriCircuit Array, PentaSync Seven, UltraWave Eight, QuantumVertex Nine, HyperHelix X, DigiSpiral Z, PentaQuark Eleven, TetraCube Twelve, GigaPhase Thirteen, EchoNeuron Fourteen, FusionPulse V15, MetaQuark Sixteen, InfiniCircuit Seventeen, TeraPulse Eighteen, ExoMatrix Nineteen, OrbiSync Twenty, QuantumHelix TwentyOne, NanoPhase TwentyTwo, TeraFractal TwentyThree, PentaHelix TwentyFour, ExoCircuit TwentyFive, HyperQuark TwentySix, B.R.I.C.K., Q.U.A.R.T.Z., F.L.I.N.T.",
    audio_filepath=ZyntriQix_filepath,
)

```




    "Have you heard of ZentricX? This tech giant boasts products like DigiCube Plus, Synapse 5, VortiCore V8, EchoNix Array, and not to forget the latest Orbital Link 7 and Digifractal Matrix. Their innovation arsenal also includes the PULSE framework, RAPT system. They've developed a brick infrastructure court system and launched the F.L.I.N.T. initiative, all highlighting their commitment to relentless innovation. ZentricX in just 30 years has soared from a startup to a tech titan, serving us tech marvels alongside a stimulating linguistic challenge. Quite an adventure, wouldn't you agree?"


key: _node_content
string_value:
{"id_": "54d8facf-c37f-41a1-8f49-39914a816da7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_correct_misspelling.ipynb", "title": "Addressing transcription misspellings: prompt vs post-processing", "tokens": 3290, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "aae74d07-7672-4eb7-a7b3-ab3900f787b3", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_correct_misspelling.ipynb", "title": "Addressing transcription misspellings: prompt vs post-processing", "tokens": 3290, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "39c2d8f1b8978601ba8107c272786d8fcace4edc2e74c2d8e8fef500e3320128", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c3eecdda-d294-46b5-88d2-104e4ca303b8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_correct_misspelling.ipynb", "title": "Addressing transcription misspellings: prompt vs post-processing", "tokens": 3290, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0d604539fcecff4a4682430a4c4c45f690d9414319ffd51782dab42fac52fbd4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "12f53668-dd28-4729-8c11-e1f0ff23ba1c", "node_type": "1", "metadata": {}, "hash": "34e3709e95c510a3fbac983bed67593d5d1728b131ac4787947e0b123661bed9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4441, "end_char_idx": 7027, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# add a full product list to the prompt
transcribe(
    prompt="ZyntriQix, Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, OrbitalLink Seven, DigiFractal Matrix, PULSE, RAPT, AstroPixel Array, QuantumFlare Five, CyberPulse Six, VortexDrive Matrix, PhotonLink Ten, TriCircuit Array, PentaSync Seven, UltraWave Eight, QuantumVertex Nine, HyperHelix X, DigiSpiral Z, PentaQuark Eleven, TetraCube Twelve, GigaPhase Thirteen, EchoNeuron Fourteen, FusionPulse V15, MetaQuark Sixteen, InfiniCircuit Seventeen, TeraPulse Eighteen, ExoMatrix Nineteen, OrbiSync Twenty, QuantumHelix TwentyOne, NanoPhase TwentyTwo, TeraFractal TwentyThree, PentaHelix TwentyFour, ExoCircuit TwentyFive, HyperQuark TwentySix, B.R.I.C.K., Q.U.A.R.T.Z., F.L.I.N.T.",
    audio_filepath=ZyntriQix_filepath,
)

```




    "Have you heard of ZentricX? This tech giant boasts products like DigiCube Plus, Synapse 5, VortiCore V8, EchoNix Array, and not to forget the latest Orbital Link 7 and Digifractal Matrix. Their innovation arsenal also includes the PULSE framework, RAPT system. They've developed a brick infrastructure court system and launched the F.L.I.N.T. initiative, all highlighting their commitment to relentless innovation. ZentricX in just 30 years has soared from a startup to a tech titan, serving us tech marvels alongside a stimulating linguistic challenge. Quite an adventure, wouldn't you agree?"



## You can use GPT-4 to fix spelling mistakes

Leveraging GPT-4 proves especially useful when the speech content is unknown beforehand and we have a list of product names readily available.

The post-processing technique using GPT-4 is notably more scalable than depending solely on Whisper's prompt parameter, which has a token limit of 244. GPT-4 allows us to process larger lists of correct spellings, making it a more robust method for handling extensive product lists.

However, this post-processing technique isn't without limitations. It's constrained by the context window of the chosen model, which may pose challenges when dealing with vast numbers of unique terms. For instance, companies with thousands of SKUs may find that the context window of GPT-4 is insufficient to handle their requirements, and they might need to explore alternative solutions.

Interestingly, the GPT-4 post-processing technique seems more reliable than using Whisper alone. This method, which leverages a product list, enhances the reliability of our results. However, this increased reliability comes at a price, as using this approach can increase costs and can result in higher latency.


key: _node_content
string_value:
{"id_": "12f53668-dd28-4729-8c11-e1f0ff23ba1c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_correct_misspelling.ipynb", "title": "Addressing transcription misspellings: prompt vs post-processing", "tokens": 3290, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "aae74d07-7672-4eb7-a7b3-ab3900f787b3", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_correct_misspelling.ipynb", "title": "Addressing transcription misspellings: prompt vs post-processing", "tokens": 3290, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "39c2d8f1b8978601ba8107c272786d8fcace4edc2e74c2d8e8fef500e3320128", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "54d8facf-c37f-41a1-8f49-39914a816da7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_correct_misspelling.ipynb", "title": "Addressing transcription misspellings: prompt vs post-processing", "tokens": 3290, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3c4cc78804260ad43e3a3419ab78868f66578662a62093d5143f6c83462fc708", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5e6fae27-501d-4f01-8007-49756ff0fbaf", "node_type": "1", "metadata": {}, "hash": "08631eea4fe162b308a406487cd4892769003c55a54090867b11928d676b21b7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5250, "end_char_idx": 7992, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"Have you heard of ZentricX? This tech giant boasts products like DigiCube Plus, Synapse 5, VortiCore V8, EchoNix Array, and not to forget the latest Orbital Link 7 and Digifractal Matrix. Their innovation arsenal also includes the PULSE framework, RAPT system. They've developed a brick infrastructure court system and launched the F.L.I.N.T. initiative, all highlighting their commitment to relentless innovation. ZentricX in just 30 years has soared from a startup to a tech titan, serving us tech marvels alongside a stimulating linguistic challenge. Quite an adventure, wouldn't you agree?"



## You can use GPT-4 to fix spelling mistakes

Leveraging GPT-4 proves especially useful when the speech content is unknown beforehand and we have a list of product names readily available.

The post-processing technique using GPT-4 is notably more scalable than depending solely on Whisper's prompt parameter, which has a token limit of 244. GPT-4 allows us to process larger lists of correct spellings, making it a more robust method for handling extensive product lists.

However, this post-processing technique isn't without limitations. It's constrained by the context window of the chosen model, which may pose challenges when dealing with vast numbers of unique terms. For instance, companies with thousands of SKUs may find that the context window of GPT-4 is insufficient to handle their requirements, and they might need to explore alternative solutions.

Interestingly, the GPT-4 post-processing technique seems more reliable than using Whisper alone. This method, which leverages a product list, enhances the reliability of our results. However, this increased reliability comes at a price, as using this approach can increase costs and can result in higher latency.


```python
# define a wrapper function for seeing how prompts affect transcriptions
def transcribe_with_spellcheck(system_message, audio_filepath):
    completion = client.chat.completions.create(
        model="gpt-4",
        temperature=0,
        messages=[
            {"role": "system", "content": system_message},
            {
                "role": "user",
                "content": transcribe(prompt="", audio_filepath=audio_filepath),
            },
        ],
    )
    return completion.choices[0].message.content

```

Now, let's input the original product list into GPT-4 and evaluate its performance. By doing so, we aim to assess the AI model's ability to correctly spell the proprietary product names, even with no prior knowledge of the exact terms to appear in the transcription. In our experiment, GPT-4 was successful in correctly spelling our product names, confirming its potential as a reliable tool for ensuring transcription accuracy.


key: _node_content
string_value:
{"id_": "5e6fae27-501d-4f01-8007-49756ff0fbaf", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_correct_misspelling.ipynb", "title": "Addressing transcription misspellings: prompt vs post-processing", "tokens": 3290, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "aae74d07-7672-4eb7-a7b3-ab3900f787b3", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_correct_misspelling.ipynb", "title": "Addressing transcription misspellings: prompt vs post-processing", "tokens": 3290, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "39c2d8f1b8978601ba8107c272786d8fcace4edc2e74c2d8e8fef500e3320128", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "12f53668-dd28-4729-8c11-e1f0ff23ba1c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_correct_misspelling.ipynb", "title": "Addressing transcription misspellings: prompt vs post-processing", "tokens": 3290, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "21dee670c7a32672241e95d158bfdf1370b36c9158cde574de22b46ea8a8bab3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5ec2bde4-3989-4440-a4ce-6f7e834eae11", "node_type": "1", "metadata": {}, "hash": "e37ba35fed2a810b7c0bfda41a09100fc726f6062b6574ec1f734284e3a3dc02", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7030, "end_char_idx": 9500, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# define a wrapper function for seeing how prompts affect transcriptions
def transcribe_with_spellcheck(system_message, audio_filepath):
    completion = client.chat.completions.create(
        model="gpt-4",
        temperature=0,
        messages=[
            {"role": "system", "content": system_message},
            {
                "role": "user",
                "content": transcribe(prompt="", audio_filepath=audio_filepath),
            },
        ],
    )
    return completion.choices[0].message.content

```

Now, let's input the original product list into GPT-4 and evaluate its performance. By doing so, we aim to assess the AI model's ability to correctly spell the proprietary product names, even with no prior knowledge of the exact terms to appear in the transcription. In our experiment, GPT-4 was successful in correctly spelling our product names, confirming its potential as a reliable tool for ensuring transcription accuracy.


```python
system_prompt = "You are a helpful assistant for the company ZyntriQix. Your task is to correct any spelling discrepancies in the transcribed text. Make sure that the names of the following products are spelled correctly: ZyntriQix, Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, OrbitalLink Seven, DigiFractal Matrix, PULSE, RAPT, B.R.I.C.K., Q.U.A.R.T.Z., F.L.I.N.T."
new_text = transcribe_with_spellcheck(system_prompt, audio_filepath=ZyntriQix_filepath)
print(new_text)

```

    Have you heard of ZyntriQix? This tech giant boasts products like Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, and not to forget the latest OrbitalLink Seven and DigiFractal Matrix. Their innovation arsenal also includes the PULSE framework, RAPT system, they've developed a B.R.I.C.K. infrastructure court system, and launched the F.L.I.N.T. initiative, all highlighting their commitment to relentless innovation. ZyntriQix, in just 30 years, has soared from a startup to a tech titan, serving us tech marvels alongside a stimulating linguistic challenge. Quite an adventure, wouldn't you agree?


In this case, we supplied a comprehensive product list that included all the previously used spellings, along with additional new names. This scenario simulates a real-life situation where we have a substantial SKU list and uncertain about the exact terms to appear in the transcription. Feeding this extensive list of product names into the system resulted in a correctly transcribed output.


key: _node_content
string_value:
{"id_": "5ec2bde4-3989-4440-a4ce-6f7e834eae11", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_correct_misspelling.ipynb", "title": "Addressing transcription misspellings: prompt vs post-processing", "tokens": 3290, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "aae74d07-7672-4eb7-a7b3-ab3900f787b3", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_correct_misspelling.ipynb", "title": "Addressing transcription misspellings: prompt vs post-processing", "tokens": 3290, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "39c2d8f1b8978601ba8107c272786d8fcace4edc2e74c2d8e8fef500e3320128", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5e6fae27-501d-4f01-8007-49756ff0fbaf", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_correct_misspelling.ipynb", "title": "Addressing transcription misspellings: prompt vs post-processing", "tokens": 3290, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f51832ab7e5e77be96791ddd6af50989883525df05d289cd6a19b792b5ec37db", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "52e72084-e565-4d03-b0fa-387272348ae7", "node_type": "1", "metadata": {}, "hash": "3fbc8c2519418cd8c7731db08c34443dd02a39e7d926e674b5ff3d47bfebccaa", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7995, "end_char_idx": 11339, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
system_prompt = "You are a helpful assistant for the company ZyntriQix. Your task is to correct any spelling discrepancies in the transcribed text. Make sure that the names of the following products are spelled correctly: ZyntriQix, Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, OrbitalLink Seven, DigiFractal Matrix, PULSE, RAPT, B.R.I.C.K., Q.U.A.R.T.Z., F.L.I.N.T."
new_text = transcribe_with_spellcheck(system_prompt, audio_filepath=ZyntriQix_filepath)
print(new_text)

```

    Have you heard of ZyntriQix? This tech giant boasts products like Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, and not to forget the latest OrbitalLink Seven and DigiFractal Matrix. Their innovation arsenal also includes the PULSE framework, RAPT system, they've developed a B.R.I.C.K. infrastructure court system, and launched the F.L.I.N.T. initiative, all highlighting their commitment to relentless innovation. ZyntriQix, in just 30 years, has soared from a startup to a tech titan, serving us tech marvels alongside a stimulating linguistic challenge. Quite an adventure, wouldn't you agree?


In this case, we supplied a comprehensive product list that included all the previously used spellings, along with additional new names. This scenario simulates a real-life situation where we have a substantial SKU list and uncertain about the exact terms to appear in the transcription. Feeding this extensive list of product names into the system resulted in a correctly transcribed output.


```python
system_prompt = "You are a helpful assistant for the company ZyntriQix. Your task is to correct any spelling discrepancies in the transcribed text. Make sure that the names of the following products are spelled correctly: ZyntriQix, Digique Plus, CynapseFive, VortiQore V8, EchoNix Array,  OrbitalLink Seven, DigiFractal Matrix, PULSE, RAPT, AstroPixel Array, QuantumFlare Five, CyberPulse Six, VortexDrive Matrix, PhotonLink Ten, TriCircuit Array, PentaSync Seven, UltraWave Eight, QuantumVertex Nine, HyperHelix X, DigiSpiral Z, PentaQuark Eleven, TetraCube Twelve, GigaPhase Thirteen, EchoNeuron Fourteen, FusionPulse V15, MetaQuark Sixteen, InfiniCircuit Seventeen, TeraPulse Eighteen, ExoMatrix Nineteen, OrbiSync Twenty, QuantumHelix TwentyOne, NanoPhase TwentyTwo, TeraFractal TwentyThree, PentaHelix TwentyFour, ExoCircuit TwentyFive, HyperQuark TwentySix, GigaLink TwentySeven, FusionMatrix TwentyEight, InfiniFractal TwentyNine, MetaSync Thirty, B.R.I.C.K., Q.U.A.R.T.Z., F.L.I.N.T. Only add necessary punctuation such as periods, commas, and capitalization, and use only the context provided."
new_text = transcribe_with_spellcheck(system_prompt, audio_filepath=ZyntriQix_filepath)
print(new_text)

```

    Have you heard of ZyntriQix? This tech giant boasts products like Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, and not to forget the latest OrbitalLink Seven and DigiFractal Matrix. Their innovation arsenal also includes the PULSE framework, RAPT system, they've developed a B.R.I.C.K. infrastructure court system, and launched the F.L.I.N.T. initiative, all highlighting their commitment to relentless innovation. ZyntriQix, in just 30 years, has soared from a startup to a tech titan, serving us tech marvels alongside a stimulating linguistic challenge. Quite an adventure, wouldn't you agree?


key: _node_content
string_value:
{"id_": "52e72084-e565-4d03-b0fa-387272348ae7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_correct_misspelling.ipynb", "title": "Addressing transcription misspellings: prompt vs post-processing", "tokens": 3290, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "aae74d07-7672-4eb7-a7b3-ab3900f787b3", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_correct_misspelling.ipynb", "title": "Addressing transcription misspellings: prompt vs post-processing", "tokens": 3290, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "39c2d8f1b8978601ba8107c272786d8fcace4edc2e74c2d8e8fef500e3320128", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5ec2bde4-3989-4440-a4ce-6f7e834eae11", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Whisper_correct_misspelling.ipynb", "title": "Addressing transcription misspellings: prompt vs post-processing", "tokens": 3290, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "163524f91e9753f42b8f52ccbc5af73aa663b419d3d1dba8823afe22a5e08870", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11342, "end_char_idx": 13546, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We are employing GPT-4 as a spell checker, using the same list of correct spellings that was previously used in the prompt.


```python
system_prompt = "You are a helpful assistant for the company ZyntriQix. Your first task is to list the words that are not spelled correctly according to the list provided to you and to tell me the number of misspelled words. Your next task is to insert those correct words in place of the misspelled ones. List: ZyntriQix, Digique Plus, CynapseFive, VortiQore V8, EchoNix Array,  OrbitalLink Seven, DigiFractal Matrix, PULSE, RAPT, AstroPixel Array, QuantumFlare Five, CyberPulse Six, VortexDrive Matrix, PhotonLink Ten, TriCircuit Array, PentaSync Seven, UltraWave Eight, QuantumVertex Nine, HyperHelix X, DigiSpiral Z, PentaQuark Eleven, TetraCube Twelve, GigaPhase Thirteen, EchoNeuron Fourteen, FusionPulse V15, MetaQuark Sixteen, InfiniCircuit Seventeen, TeraPulse Eighteen, ExoMatrix Nineteen, OrbiSync Twenty, QuantumHelix TwentyOne, NanoPhase TwentyTwo, TeraFractal TwentyThree, PentaHelix TwentyFour, ExoCircuit TwentyFive, HyperQuark TwentySix, GigaLink TwentySeven, FusionMatrix TwentyEight, InfiniFractal TwentyNine, MetaSync Thirty, B.R.I.C.K., Q.U.A.R.T.Z., F.L.I.N.T."
new_text = transcribe_with_spellcheck(system_prompt, audio_filepath=ZyntriQix_filepath)
print(new_text)

```

    The misspelled words are: ZentricX, Digi-Q+, Synapse 5, VortiCore V8, Echo Nix Array, Orbital Link 7, Digifractal Matrix, Pulse, Wrapped, brick, Flint, and 30. The total number of misspelled words is 12.
    
    The corrected paragraph is:
    
    Have you heard of ZyntriQix? This tech giant boasts products like Digique Plus, CynapseFive, VortiQore V8, EchoNix Array, and not to forget the latest OrbitalLink Seven and DigiFractal Matrix. Their innovation arsenal also includes the PULSE framework, RAPT system, they've developed a B.R.I.C.K. infrastructure court system, and launched the F.L.I.N.T. initiative, all highlighting their commitment to relentless innovation. ZyntriQix, in just MetaSync Thirty years, has soared from a startup to a tech titan, serving us tech marvels alongside a stimulating linguistic challenge. Quite an adventure, wouldn't you agree?


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb


key: _node_content
string_value:
{"id_": "4448a0e0-0064-4694-aa19-685711d16e03", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7975c8a7-a812-476d-8b57-ec5955526040", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "75578ea97075a1210a2c40a6776cf0b9153a7aeb2fb6dd52d5f0f84c303bd94b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6abf644d-e803-4bc9-a4c7-068dc8da10ab", "node_type": "1", "metadata": {}, "hash": "8613fc587dc5f0687342e251142ddb1a0570a2d2168a234a4d50623cff80ac4d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 646, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to count tokens with tiktoken

[`tiktoken`](https://github.com/openai/tiktoken/blob/main/README.md) is a fast open-source tokenizer by OpenAI.

Given a text string (e.g., `"tiktoken is great!"`) and an encoding (e.g., `"cl100k_base"`), a tokenizer can split the text string into a list of tokens (e.g., `["t", "ik", "token", " is", " great", "!"]`).

Splitting text strings into tokens is useful because GPT models see text in the form of tokens. Knowing how many tokens are in a text string can tell you (a) whether the string is too long for a text model to process and (b) how much an OpenAI API call costs (as usage is priced by token).


key: _node_content
string_value:
{"id_": "6abf644d-e803-4bc9-a4c7-068dc8da10ab", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7975c8a7-a812-476d-8b57-ec5955526040", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "75578ea97075a1210a2c40a6776cf0b9153a7aeb2fb6dd52d5f0f84c303bd94b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4448a0e0-0064-4694-aa19-685711d16e03", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d21c256f0142935079ecb204caa0fe0182e8872428f564aa7a1e96e52953d79a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e71ff8b0-df40-452a-a575-1de982615cd3", "node_type": "1", "metadata": {}, "hash": "10214300116a4c8c575510b9d1bd2b46991a22d87826684dd10576fb6696694e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2854, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to count tokens with tiktoken

[`tiktoken`](https://github.com/openai/tiktoken/blob/main/README.md) is a fast open-source tokenizer by OpenAI.

Given a text string (e.g., `"tiktoken is great!"`) and an encoding (e.g., `"cl100k_base"`), a tokenizer can split the text string into a list of tokens (e.g., `["t", "ik", "token", " is", " great", "!"]`).

Splitting text strings into tokens is useful because GPT models see text in the form of tokens. Knowing how many tokens are in a text string can tell you (a) whether the string is too long for a text model to process and (b) how much an OpenAI API call costs (as usage is priced by token).


## Encodings

Encodings specify how text is converted into tokens. Different models use different encodings.

`tiktoken` supports three encodings used by OpenAI models:

| Encoding name           | OpenAI models                                       |
|-------------------------|-----------------------------------------------------|
| `cl100k_base`           | `gpt-4`, `gpt-3.5-turbo`, `text-embedding-ada-002`, `text-embedding-3-small`, `text-embedding-3-large`  |
| `p50k_base`             | Codex models, `text-davinci-002`, `text-davinci-003`|
| `r50k_base` (or `gpt2`) | GPT-3 models like `davinci`                         |

You can retrieve the encoding for a model using `tiktoken.encoding_for_model()` as follows:
```python
encoding = tiktoken.encoding_for_model('gpt-3.5-turbo')
```

Note that `p50k_base` overlaps substantially with `r50k_base`, and for non-code applications, they will usually give the same tokens.

## Tokenizer libraries by language

For `cl100k_base` and `p50k_base` encodings:
- Python: [tiktoken](https://github.com/openai/tiktoken/blob/main/README.md)
- .NET / C#: [SharpToken](https://github.com/dmitry-brazhenko/SharpToken), [TiktokenSharp](https://github.com/aiqinxuancai/TiktokenSharp)
- Java: [jtokkit](https://github.com/knuddelsgmbh/jtokkit)
- Golang: [tiktoken-go](https://github.com/pkoukk/tiktoken-go)
- Rust: [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs)

For `r50k_base` (`gpt2`) encodings, tokenizers are available in many languages.
- Python: [tiktoken](https://github.com/openai/tiktoken/blob/main/README.md) (or alternatively [GPT2TokenizerFast](https://huggingface.co/docs/transformers/model_doc/gpt2#transformers.GPT2TokenizerFast))
- JavaScript: [gpt-3-encoder](https://www.npmjs.com/package/gpt-3-encoder)
- .NET / C#: [GPT Tokenizer](https://github.com/dluc/openai-tools)
- Java: [gpt2-tokenizer-java](https://github.com/hyunwoongko/gpt2-tokenizer-java)
- PHP: [GPT-3-Encoder-PHP](https://github.com/CodeRevolutionPlugins/GPT-3-Encoder-PHP)
- Golang: [tiktoken-go](https://github.com/pkoukk/tiktoken-go)
- Rust: [tiktoken-rs](https://github.com/zurawiki/tiktoken-rs)

(OpenAI makes no endorsements or guarantees of third-party libraries.)


key: _node_content
string_value:
{"id_": "e71ff8b0-df40-452a-a575-1de982615cd3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7975c8a7-a812-476d-8b57-ec5955526040", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "75578ea97075a1210a2c40a6776cf0b9153a7aeb2fb6dd52d5f0f84c303bd94b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6abf644d-e803-4bc9-a4c7-068dc8da10ab", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e14ca7e9409b843d24ecb8140be860f9a04babf89455793fcae1eb4ebfe89a77", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e396d4f6-66b2-4d94-a5e4-b0bc872760a9", "node_type": "1", "metadata": {}, "hash": "5f20f0adf8ce5bceb5ac15a2519bd4e9c85075280992b6b721c8234f6bbc2a9d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2857, "end_char_idx": 5611, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## How strings are typically tokenized

In English, tokens commonly range in length from one character to one word (e.g., `"t"` or `" great"`), though in some languages tokens can be shorter than one character or longer than one word. Spaces are usually grouped with the starts of words (e.g., `" is"` instead of `"is "` or `" "`+`"is"`). You can quickly check how a string is tokenized at the [OpenAI Tokenizer](https://beta.openai.com/tokenizer), or the third-party [Tiktokenizer](https://tiktokenizer.vercel.app/) webapp.

## 0. Install `tiktoken`

If needed, install `tiktoken` with `pip`:


```python
%pip install --upgrade tiktoken
%pip install --upgrade openai
```

## 1. Import `tiktoken`


```python
import tiktoken
```

## 2. Load an encoding

Use `tiktoken.get_encoding()` to load an encoding by name.

The first time this runs, it will require an internet connection to download. Later runs won't need an internet connection.


```python
encoding = tiktoken.get_encoding("cl100k_base")

```

Use `tiktoken.encoding_for_model()` to automatically load the correct encoding for a given model name.


```python
encoding = tiktoken.encoding_for_model("gpt-3.5-turbo")
```

## 3. Turn text into tokens with `encoding.encode()`



The `.encode()` method converts a text string into a list of token integers.


```python
encoding.encode("tiktoken is great!")

```




    [83, 1609, 5963, 374, 2294, 0]



Count tokens by counting the length of the list returned by `.encode()`.


```python
def num_tokens_from_string(string: str, encoding_name: str) -> int:
    """Returns the number of tokens in a text string."""
    encoding = tiktoken.get_encoding(encoding_name)
    num_tokens = len(encoding.encode(string))
    return num_tokens

```


```python
num_tokens_from_string("tiktoken is great!", "cl100k_base")

```




    6



## 4. Turn tokens into text with `encoding.decode()`

`.decode()` converts a list of token integers to a string.


```python
encoding.decode([83, 1609, 5963, 374, 2294, 0])

```




    'tiktoken is great!'



Warning: although `.decode()` can be applied to single tokens, beware that it can be lossy for tokens that aren't on utf-8 boundaries.

For single tokens, `.decode_single_token_bytes()` safely converts a single integer token to the bytes it represents.


```python
[encoding.decode_single_token_bytes(token) for token in [83, 1609, 5963, 374, 2294, 0]]

```




    [b't', b'ik', b'token', b' is', b' great', b'!']



(The `b` in front of the strings indicates that the strings are byte strings.)

## 5. Comparing encodings

Different encodings vary in how they split words, group spaces, and handle non-English characters. Using the methods above, we can compare different encodings on a few example strings.


key: _node_content
string_value:
{"id_": "e396d4f6-66b2-4d94-a5e4-b0bc872760a9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7975c8a7-a812-476d-8b57-ec5955526040", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "75578ea97075a1210a2c40a6776cf0b9153a7aeb2fb6dd52d5f0f84c303bd94b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e71ff8b0-df40-452a-a575-1de982615cd3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "83d0d3ab376671102c64307526a4a1c9732831c178981ec59269372844d5b798", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3c36ceb9-588f-49fd-8371-5d115edaa79f", "node_type": "1", "metadata": {}, "hash": "a90d6f1720873c9224df6a0a2322a0f531f2c1ac4743b5f2246e57fac9b830ad", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4093, "end_char_idx": 6398, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The `.encode()` method converts a text string into a list of token integers.


```python
encoding.encode("tiktoken is great!")

```




    [83, 1609, 5963, 374, 2294, 0]



Count tokens by counting the length of the list returned by `.encode()`.


```python
def num_tokens_from_string(string: str, encoding_name: str) -> int:
    """Returns the number of tokens in a text string."""
    encoding = tiktoken.get_encoding(encoding_name)
    num_tokens = len(encoding.encode(string))
    return num_tokens

```


```python
num_tokens_from_string("tiktoken is great!", "cl100k_base")

```




    6



## 4. Turn tokens into text with `encoding.decode()`

`.decode()` converts a list of token integers to a string.


```python
encoding.decode([83, 1609, 5963, 374, 2294, 0])

```




    'tiktoken is great!'



Warning: although `.decode()` can be applied to single tokens, beware that it can be lossy for tokens that aren't on utf-8 boundaries.

For single tokens, `.decode_single_token_bytes()` safely converts a single integer token to the bytes it represents.


```python
[encoding.decode_single_token_bytes(token) for token in [83, 1609, 5963, 374, 2294, 0]]

```




    [b't', b'ik', b'token', b' is', b' great', b'!']



(The `b` in front of the strings indicates that the strings are byte strings.)

## 5. Comparing encodings

Different encodings vary in how they split words, group spaces, and handle non-English characters. Using the methods above, we can compare different encodings on a few example strings.


```python
def compare_encodings(example_string: str) -> None:
    """Prints a comparison of three string encodings."""
    # print the example string
    print(f'\nExample string: "{example_string}"')
    # for each encoding, print the # of tokens, the token integers, and the token bytes
    for encoding_name in ["r50k_base", "p50k_base", "cl100k_base"]:
        encoding = tiktoken.get_encoding(encoding_name)
        token_integers = encoding.encode(example_string)
        num_tokens = len(token_integers)
        token_bytes = [encoding.decode_single_token_bytes(token) for token in token_integers]
        print()
        print(f"{encoding_name}: {num_tokens} tokens")
        print(f"token integers: {token_integers}")
        print(f"token bytes: {token_bytes}")
        
```


key: _node_content
string_value:
{"id_": "3c36ceb9-588f-49fd-8371-5d115edaa79f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7975c8a7-a812-476d-8b57-ec5955526040", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "75578ea97075a1210a2c40a6776cf0b9153a7aeb2fb6dd52d5f0f84c303bd94b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e396d4f6-66b2-4d94-a5e4-b0bc872760a9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3c928bba23d1fcc40943b534432d3c96a95f29d25514e572de8415f0d15118c8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a4e7d9c7-1444-4c92-bd8a-96b69fccba5a", "node_type": "1", "metadata": {}, "hash": "1039ed747ba0ca3a4d91e1254dbe0f924726afa76a334329a3b983e79a8c10d2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4807, "end_char_idx": 6990, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
encoding.decode([83, 1609, 5963, 374, 2294, 0])

```




    'tiktoken is great!'



Warning: although `.decode()` can be applied to single tokens, beware that it can be lossy for tokens that aren't on utf-8 boundaries.

For single tokens, `.decode_single_token_bytes()` safely converts a single integer token to the bytes it represents.


```python
[encoding.decode_single_token_bytes(token) for token in [83, 1609, 5963, 374, 2294, 0]]

```




    [b't', b'ik', b'token', b' is', b' great', b'!']



(The `b` in front of the strings indicates that the strings are byte strings.)

## 5. Comparing encodings

Different encodings vary in how they split words, group spaces, and handle non-English characters. Using the methods above, we can compare different encodings on a few example strings.


```python
def compare_encodings(example_string: str) -> None:
    """Prints a comparison of three string encodings."""
    # print the example string
    print(f'\nExample string: "{example_string}"')
    # for each encoding, print the # of tokens, the token integers, and the token bytes
    for encoding_name in ["r50k_base", "p50k_base", "cl100k_base"]:
        encoding = tiktoken.get_encoding(encoding_name)
        token_integers = encoding.encode(example_string)
        num_tokens = len(token_integers)
        token_bytes = [encoding.decode_single_token_bytes(token) for token in token_integers]
        print()
        print(f"{encoding_name}: {num_tokens} tokens")
        print(f"token integers: {token_integers}")
        print(f"token bytes: {token_bytes}")
        
```


```python
compare_encodings("antidisestablishmentarianism")

```

    
    Example string: "antidisestablishmentarianism"
    
    r50k_base: 5 tokens
    token integers: [415, 29207, 44390, 3699, 1042]
    token bytes: [b'ant', b'idis', b'establishment', b'arian', b'ism']
    
    p50k_base: 5 tokens
    token integers: [415, 29207, 44390, 3699, 1042]
    token bytes: [b'ant', b'idis', b'establishment', b'arian', b'ism']
    
    cl100k_base: 6 tokens
    token integers: [519, 85342, 34500, 479, 8997, 2191]
    token bytes: [b'ant', b'idis', b'establish', b'ment', b'arian', b'ism']


key: _node_content
string_value:
{"id_": "a4e7d9c7-1444-4c92-bd8a-96b69fccba5a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7975c8a7-a812-476d-8b57-ec5955526040", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "75578ea97075a1210a2c40a6776cf0b9153a7aeb2fb6dd52d5f0f84c303bd94b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3c36ceb9-588f-49fd-8371-5d115edaa79f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c1669c9b11ae9f19e393246178c7e2ae0f6d8c2b6daa3493324d93bd4a49b667", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "670420e5-8e8d-4feb-a498-240d887a74ca", "node_type": "1", "metadata": {}, "hash": "fcdb6a13cd4eaaab477aacdaf26f46c51eb9596640f2e6e919a676a3abf1fa52", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5614, "end_char_idx": 7477, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def compare_encodings(example_string: str) -> None:
    """Prints a comparison of three string encodings."""
    # print the example string
    print(f'\nExample string: "{example_string}"')
    # for each encoding, print the # of tokens, the token integers, and the token bytes
    for encoding_name in ["r50k_base", "p50k_base", "cl100k_base"]:
        encoding = tiktoken.get_encoding(encoding_name)
        token_integers = encoding.encode(example_string)
        num_tokens = len(token_integers)
        token_bytes = [encoding.decode_single_token_bytes(token) for token in token_integers]
        print()
        print(f"{encoding_name}: {num_tokens} tokens")
        print(f"token integers: {token_integers}")
        print(f"token bytes: {token_bytes}")
        
```


```python
compare_encodings("antidisestablishmentarianism")

```

    
    Example string: "antidisestablishmentarianism"
    
    r50k_base: 5 tokens
    token integers: [415, 29207, 44390, 3699, 1042]
    token bytes: [b'ant', b'idis', b'establishment', b'arian', b'ism']
    
    p50k_base: 5 tokens
    token integers: [415, 29207, 44390, 3699, 1042]
    token bytes: [b'ant', b'idis', b'establishment', b'arian', b'ism']
    
    cl100k_base: 6 tokens
    token integers: [519, 85342, 34500, 479, 8997, 2191]
    token bytes: [b'ant', b'idis', b'establish', b'ment', b'arian', b'ism']



```python
compare_encodings("2 + 2 = 4")

```

    
    Example string: "2 + 2 = 4"
    
    r50k_base: 5 tokens
    token integers: [17, 1343, 362, 796, 604]
    token bytes: [b'2', b' +', b' 2', b' =', b' 4']
    
    p50k_base: 5 tokens
    token integers: [17, 1343, 362, 796, 604]
    token bytes: [b'2', b' +', b' 2', b' =', b' 4']
    
    cl100k_base: 7 tokens
    token integers: [17, 489, 220, 17, 284, 220, 19]
    token bytes: [b'2', b' +', b' ', b'2', b' =', b' ', b'4']


key: _node_content
string_value:
{"id_": "670420e5-8e8d-4feb-a498-240d887a74ca", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7975c8a7-a812-476d-8b57-ec5955526040", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "75578ea97075a1210a2c40a6776cf0b9153a7aeb2fb6dd52d5f0f84c303bd94b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a4e7d9c7-1444-4c92-bd8a-96b69fccba5a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a9023ee099a1f5924470f3e973643aae7c362640a60961cdaee43476721d1e80", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7d36950b-3740-41e2-8dbe-9ac584771765", "node_type": "1", "metadata": {}, "hash": "3d309d032a9d1d00db9e8c43a44bfff0e5da75aac5aa1f59a8afc31d3bd971fe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6994, "end_char_idx": 8509, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
compare_encodings("2 + 2 = 4")

```

    
    Example string: "2 + 2 = 4"
    
    r50k_base: 5 tokens
    token integers: [17, 1343, 362, 796, 604]
    token bytes: [b'2', b' +', b' 2', b' =', b' 4']
    
    p50k_base: 5 tokens
    token integers: [17, 1343, 362, 796, 604]
    token bytes: [b'2', b' +', b' 2', b' =', b' 4']
    
    cl100k_base: 7 tokens
    token integers: [17, 489, 220, 17, 284, 220, 19]
    token bytes: [b'2', b' +', b' ', b'2', b' =', b' ', b'4']



```python
compare_encodings("お誕生日おめでとう")

```

    
    Example string: "お誕生日おめでとう"
    
    r50k_base: 14 tokens
    token integers: [2515, 232, 45739, 243, 37955, 33768, 98, 2515, 232, 1792, 223, 30640, 30201, 29557]
    token bytes: [b'\xe3\x81', b'\x8a', b'\xe8\xaa', b'\x95', b'\xe7\x94\x9f', b'\xe6\x97', b'\xa5', b'\xe3\x81', b'\x8a', b'\xe3\x82', b'\x81', b'\xe3\x81\xa7', b'\xe3\x81\xa8', b'\xe3\x81\x86']
    
    p50k_base: 14 tokens
    token integers: [2515, 232, 45739, 243, 37955, 33768, 98, 2515, 232, 1792, 223, 30640, 30201, 29557]
    token bytes: [b'\xe3\x81', b'\x8a', b'\xe8\xaa', b'\x95', b'\xe7\x94\x9f', b'\xe6\x97', b'\xa5', b'\xe3\x81', b'\x8a', b'\xe3\x82', b'\x81', b'\xe3\x81\xa7', b'\xe3\x81\xa8', b'\xe3\x81\x86']
    
    cl100k_base: 9 tokens
    token integers: [33334, 45918, 243, 21990, 9080, 33334, 62004, 16556, 78699]
    token bytes: [b'\xe3\x81\x8a', b'\xe8\xaa', b'\x95', b'\xe7\x94\x9f', b'\xe6\x97\xa5', b'\xe3\x81\x8a', b'\xe3\x82\x81', b'\xe3\x81\xa7', b'\xe3\x81\xa8\xe3\x81\x86']


key: _node_content
string_value:
{"id_": "7d36950b-3740-41e2-8dbe-9ac584771765", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7975c8a7-a812-476d-8b57-ec5955526040", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "75578ea97075a1210a2c40a6776cf0b9153a7aeb2fb6dd52d5f0f84c303bd94b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "670420e5-8e8d-4feb-a498-240d887a74ca", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "9254417df6db12cea38894e11aefd727dbe57ea073d75ff5c3d6c10aed996732", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7e631eff-5953-44b8-ab78-c2125a747bee", "node_type": "1", "metadata": {}, "hash": "c28e869a313ffc9a25786287c99c77ba796ee77d7bca037aa452a37ea352531d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8512, "end_char_idx": 10942, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## 6. Counting tokens for chat completions API calls

ChatGPT models like `gpt-3.5-turbo` and `gpt-4` use tokens in the same way as older completions models, but because of their message-based formatting, it's more difficult to count how many tokens will be used by a conversation.

Below is an example function for counting tokens for messages passed to `gpt-3.5-turbo` or `gpt-4`.

Note that the exact way that tokens are counted from messages may change from model to model. Consider the counts from the function below an estimate, not a timeless guarantee.

In particular, requests that use the optional functions input will consume extra tokens on top of the estimates calculated below.


```python
def num_tokens_from_messages(messages, model="gpt-3.5-turbo-0613"):
    """Return the number of tokens used by a list of messages."""
    try:
        encoding = tiktoken.encoding_for_model(model)
    except KeyError:
        print("Warning: model not found. Using cl100k_base encoding.")
        encoding = tiktoken.get_encoding("cl100k_base")
    if model in {
        "gpt-3.5-turbo-0613",
        "gpt-3.5-turbo-16k-0613",
        "gpt-4-0314",
        "gpt-4-32k-0314",
        "gpt-4-0613",
        "gpt-4-32k-0613",
        }:
        tokens_per_message = 3
        tokens_per_name = 1
    elif model == "gpt-3.5-turbo-0301":
        tokens_per_message = 4  # every message follows <|start|>{role/name}\n{content}<|end|>\n
        tokens_per_name = -1  # if there's a name, the role is omitted
    elif "gpt-3.5-turbo" in model:
        print("Warning: gpt-3.5-turbo may update over time. Returning num tokens assuming gpt-3.5-turbo-0613.")
        return num_tokens_from_messages(messages, model="gpt-3.5-turbo-0613")
    elif "gpt-4" in model:
        print("Warning: gpt-4 may update over time. Returning num tokens assuming gpt-4-0613.")
        return num_tokens_from_messages(messages, model="gpt-4-0613")
    else:
        raise NotImplementedError(
            f"""num_tokens_from_messages() is not implemented for model {model}."""
        )
    num_tokens = 0
    for message in messages:
        num_tokens += tokens_per_message
        for key, value in message.items():
            num_tokens += len(encoding.encode(value))
            if key == "name":
                num_tokens += tokens_per_name
    num_tokens += 3  # every reply is primed with <|start|>assistant<|message|>
    return num_tokens

```


key: _node_content
string_value:
{"id_": "7e631eff-5953-44b8-ab78-c2125a747bee", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7975c8a7-a812-476d-8b57-ec5955526040", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "75578ea97075a1210a2c40a6776cf0b9153a7aeb2fb6dd52d5f0f84c303bd94b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7d36950b-3740-41e2-8dbe-9ac584771765", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_count_tokens_with_tiktoken.ipynb", "title": "How to count tokens with tiktoken", "tokens": 4040, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "91ee38ccde15f2fa87758bf7a9897dca83a6be988aaf6bbc36a3c0b23373ed6a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10945, "end_char_idx": 13748, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# let's verify the function above matches the OpenAI API response

from openai import OpenAI
import os

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))

example_messages = [
    {
        "role": "system",
        "content": "You are a helpful, pattern-following assistant that translates corporate jargon into plain English.",
    },
    {
        "role": "system",
        "name": "example_user",
        "content": "New synergies will help drive top-line growth.",
    },
    {
        "role": "system",
        "name": "example_assistant",
        "content": "Things working well together will increase revenue.",
    },
    {
        "role": "system",
        "name": "example_user",
        "content": "Let's circle back when we have more bandwidth to touch base on opportunities for increased leverage.",
    },
    {
        "role": "system",
        "name": "example_assistant",
        "content": "Let's talk later when we're less busy about how to do better.",
    },
    {
        "role": "user",
        "content": "This late pivot means we don't have time to boil the ocean for the client deliverable.",
    },
]

for model in [
    "gpt-3.5-turbo-0301",
    "gpt-3.5-turbo-0613",
    "gpt-3.5-turbo",
    "gpt-4-0314",
    "gpt-4-0613",
    "gpt-4",
    ]:
    print(model)
    # example token count from the function defined above
    print(f"{num_tokens_from_messages(example_messages, model)} prompt tokens counted by num_tokens_from_messages().")
    # example token count from the OpenAI API
    response = client.chat.completions.create(model=model,
    messages=example_messages,
    temperature=0,
    max_tokens=1)
    print(f'{response.usage.prompt_tokens} prompt tokens counted by the OpenAI API.')
    print()

```

    gpt-3.5-turbo-0301
    127 prompt tokens counted by num_tokens_from_messages().
    127 prompt tokens counted by the OpenAI API.
    
    gpt-3.5-turbo-0613
    129 prompt tokens counted by num_tokens_from_messages().
    129 prompt tokens counted by the OpenAI API.
    
    gpt-3.5-turbo
    Warning: gpt-3.5-turbo may update over time. Returning num tokens assuming gpt-3.5-turbo-0613.
    129 prompt tokens counted by num_tokens_from_messages().
    129 prompt tokens counted by the OpenAI API.
    
    gpt-4-0314
    129 prompt tokens counted by num_tokens_from_messages().
    129 prompt tokens counted by the OpenAI API.
    
    gpt-4-0613
    129 prompt tokens counted by num_tokens_from_messages().
    129 prompt tokens counted by the OpenAI API.
    
    gpt-4
    Warning: gpt-4 may update over time. Returning num tokens assuming gpt-4-0613.
    129 prompt tokens counted by num_tokens_from_messages().
    129 prompt tokens counted by the OpenAI API.
    



```python

```


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb


key: _node_content
string_value:
{"id_": "155c7bbe-4604-452a-8f64-fe96718a3ff6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e401e314-adda-45d3-8bdb-63677aa5f69d", "node_type": "1", "metadata": {}, "hash": "768ce9614a37a0961decf109e2f9951a976a1b692d4b3178fedb2ff0c78da317", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3070, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to build a tool-using agent with LangChain

This notebook takes you through how to use LangChain to augment an OpenAI model with access to external tools. In particular, you'll be able to create LLM agents that use custom tools to answer user queries.


## What is Langchain?
[LangChain](https://python.langchain.com/en/latest/index.html) is a framework for developing applications powered by language models. Their framework enables you to build layered LLM-powered applications that are context-aware and able to interact dynamically with their environment as agents, leading to simplified code for you and a more dynamic user experience for your customers.

## Why do LLMs need to use Tools?
One of the most common challenges with LLMs is overcoming the lack of recency and specificity in their training data - answers can be out of date, and they are prone to hallucinations given the huge variety in their knowledge base. Tools are a great method of allowing an LLM to answer within a controlled context that draws on your existing knowledge bases and internal APIs - instead of trying to prompt engineer the LLM all the way to your intended answer, you allow it access to tools that it calls on dynamically for info, parses, and serves to customer. 

Providing LLMs access to tools can enable them to answer questions with context directly from search engines, APIs or your own databases. Instead of answering directly, an LLM with access to tools can perform intermediate steps to gather relevant information. Tools can also be used in combination. [For example](https://python.langchain.com/en/latest/modules/agents/agents/examples/mrkl_chat.html), a language model can be made to use a search tool to lookup quantitative information and a calculator to execute calculations.

## Notebook Sections

- **Setup:** Import packages and connect to a Pinecone vector database.
- **LLM Agent:** Build an agent that leverages a modified version of the [ReAct](https://react-lm.github.io/) framework to do chain-of-thought reasoning.
- **LLM Agent with History:** Provide the LLM with access to previous steps in the conversation.
- **Knowledge Base:** Create a knowledge base of "Stuff You Should Know" podcast episodes, to be accessed through a tool.
- **LLM Agent with Tools:** Extend the agent with access to multiple tools and test that it uses them to answer questions.


```python
%load_ext autoreload
%autoreload 2
```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload


# Setup

Import libraries and set up a connection to a [Pinecone](https://www.pinecone.io) vector database.

You can substitute Pinecone for any other vectorstore or database - there are a [selection](https://python.langchain.com/en/latest/modules/indexes/vectorstores.html) that are supported by Langchain natively, while other connectors will need to be developed yourself.


```python
!pip install openai
!pip install pinecone-client
!pip install pandas
!pip install typing
!pip install tqdm
!pip install langchain
!pip install wget
```


key: _node_content
string_value:
{"id_": "e401e314-adda-45d3-8bdb-63677aa5f69d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "155c7bbe-4604-452a-8f64-fe96718a3ff6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "1f433d1a2417f70ff7e35944d4073bb747e2c6237e18e0354c34d9525a188be8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8532508b-1775-4c10-879a-cf52a4799325", "node_type": "1", "metadata": {}, "hash": "516dc9106fd66ade7ab5d0e495a38626a671e05ac5a9e8398db05b02a19ac6b3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2383, "end_char_idx": 4468, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
%load_ext autoreload
%autoreload 2
```

    The autoreload extension is already loaded. To reload it, use:
      %reload_ext autoreload


# Setup

Import libraries and set up a connection to a [Pinecone](https://www.pinecone.io) vector database.

You can substitute Pinecone for any other vectorstore or database - there are a [selection](https://python.langchain.com/en/latest/modules/indexes/vectorstores.html) that are supported by Langchain natively, while other connectors will need to be developed yourself.


```python
!pip install openai
!pip install pinecone-client
!pip install pandas
!pip install typing
!pip install tqdm
!pip install langchain
!pip install wget
```


```python
import datetime
import json
import openai
import os
import pandas as pd
import pinecone
import re
from tqdm.auto import tqdm
from typing import List, Union
import zipfile

# Langchain imports
from langchain.agents import Tool, AgentExecutor, LLMSingleActionAgent, AgentOutputParser
from langchain.prompts import BaseChatPromptTemplate, ChatPromptTemplate
from langchain import SerpAPIWrapper, LLMChain
from langchain.schema import AgentAction, AgentFinish, HumanMessage, SystemMessage
# LLM wrapper
from langchain.chat_models import ChatOpenAI
from langchain import OpenAI
# Conversational memory
from langchain.memory import ConversationBufferWindowMemory
# Embeddings and vectorstore
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import Pinecone

# Vectorstore Index
index_name = 'podcasts'
```

For acquiring an API key to connect with Pinecone, you can set up a [free account](https://app.pinecone.io/) and store it in the `api_key` variable below or in your environment variables under `PINECONE_API_KEY`


```python
api_key = os.getenv("PINECONE_API_KEY") or "PINECONE_API_KEY"

# find environment next to your API key in the Pinecone console
env = os.getenv("PINECONE_ENVIRONMENT") or "PINECONE_ENVIRONMENT"

pinecone.init(api_key=api_key, environment=env)
pinecone.whoami()
```


```python
pinecone.list_indexes()
```




    ['podcasts']


key: _node_content
string_value:
{"id_": "8532508b-1775-4c10-879a-cf52a4799325", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e401e314-adda-45d3-8bdb-63677aa5f69d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c6e0f40c3571b202b6679870739189583ad0baf9cc2a079065e5ae32fd0c2ff9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "570d61ff-b15d-48d4-963c-af923782747c", "node_type": "1", "metadata": {}, "hash": "2e6a215fee870c25a15ef52856536ccc0ae11a09e95c6914720444f40132400d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2909, "end_char_idx": 6927, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
!pip install openai
!pip install pinecone-client
!pip install pandas
!pip install typing
!pip install tqdm
!pip install langchain
!pip install wget
```


```python
import datetime
import json
import openai
import os
import pandas as pd
import pinecone
import re
from tqdm.auto import tqdm
from typing import List, Union
import zipfile

# Langchain imports
from langchain.agents import Tool, AgentExecutor, LLMSingleActionAgent, AgentOutputParser
from langchain.prompts import BaseChatPromptTemplate, ChatPromptTemplate
from langchain import SerpAPIWrapper, LLMChain
from langchain.schema import AgentAction, AgentFinish, HumanMessage, SystemMessage
# LLM wrapper
from langchain.chat_models import ChatOpenAI
from langchain import OpenAI
# Conversational memory
from langchain.memory import ConversationBufferWindowMemory
# Embeddings and vectorstore
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import Pinecone

# Vectorstore Index
index_name = 'podcasts'
```

For acquiring an API key to connect with Pinecone, you can set up a [free account](https://app.pinecone.io/) and store it in the `api_key` variable below or in your environment variables under `PINECONE_API_KEY`


```python
api_key = os.getenv("PINECONE_API_KEY") or "PINECONE_API_KEY"

# find environment next to your API key in the Pinecone console
env = os.getenv("PINECONE_ENVIRONMENT") or "PINECONE_ENVIRONMENT"

pinecone.init(api_key=api_key, environment=env)
pinecone.whoami()
```


```python
pinecone.list_indexes()
```




    ['podcasts']



Run this code block if you want to clear the index, or if the index doesn't exist yet

```
# Check whether the index with the same name already exists - if so, delete it
if index_name in pinecone.list_indexes():
    pinecone.delete_index(index_name)
    
# Creates new index
pinecone.create_index(name=index_name, dimension=1536)
index = pinecone.Index(index_name=index_name)

# Confirm our index was created
pinecone.list_indexes()
```

## LLM Agent

An [LLM agent](https://python.langchain.com/docs/modules/agents/) in Langchain has many configurable components, which are detailed in the Langchain documentation.

We'll employ a few of the core concepts to make an agent that talks in the way we want, can use tools to answer questions, and uses the appropriate language model to power the conversation.
- **Prompt Template:** The input template to control the LLM's behaviour and how it accepts inputs and produces outputs - this is the brain that drives your application ([docs](https://python.langchain.com/en/latest/modules/prompts/prompt_templates.html)).
- **Output Parser:** A method of parsing the output from the prompt. If the LLM produces output using certain headers, you can enable complex interactions where variables are generated by the LLM in their response and passed into the next step of the chain ([docs](https://python.langchain.com/en/latest/modules/prompts/output_parsers.html)).
- **LLM Chain:** A Chain brings together a prompt template with an LLM that will execute it - in this case we'll be using ```gpt-3.5-turbo``` but this framework can be used with OpenAI completions models, or other LLMs entirely ([docs](https://python.langchain.com/en/latest/modules/chains.html)).
- **Tool:** An external service that the LLM can use to retrieve information or execute commands should the user require it ([docs](https://python.langchain.com/en/latest/modules/agents/tools.html)).
- **Agent:** The glue that brings all of this together, an agent can call multiple LLM Chains, each with their own tools. Agents can be extended with your own logic to allow retries, error handling and any other methods you choose to add reliability to your application ([docs](https://python.langchain.com/en/latest/modules/agents.html)).

**NB:** Before using this cookbook with the Search tool you'll need to sign up on https://serpapi.com/ and generate an API key. Once you have it, store it in an environment variable named ```SERPAPI_API_KEY```


key: _node_content
string_value:
{"id_": "570d61ff-b15d-48d4-963c-af923782747c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8532508b-1775-4c10-879a-cf52a4799325", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "69d2dc254a35579e1796ad65754008992082faeb6e4c54d457deaa424355b431", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "74116c2f-6c86-4660-ba03-dac4bc72c31b", "node_type": "1", "metadata": {}, "hash": "a5902baf3d62b4762b17a3c3128b311b534d4d05d86c950bdc695e8338fa4a90", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6930, "end_char_idx": 9600, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Initiate a Search tool - note you'll need to have set SERPAPI_API_KEY as an environment variable as per the above instructions
search = SerpAPIWrapper()

# Define a list of tools
tools = [
    Tool(
        name = "Search",
        func=search.run,
        description="useful for when you need to answer questions about current events"
    )
]
```


```python
# Set up the prompt with input variables for tools, user input and a scratchpad for the model to record its workings
template = """Answer the following questions as best you can, but speaking as a pirate might speak. You have access to the following tools:

{tools}

Use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question

Begin! Remember to speak as a pirate when giving your final answer. Use lots of "Arg"s

Question: {input}
{agent_scratchpad}"""
```


```python
# Set up a prompt template
class CustomPromptTemplate(BaseChatPromptTemplate):
    # The template to use
    template: str
    # The list of tools available
    tools: List[Tool]
    
    def format_messages(self, **kwargs) -> str:
        # Get the intermediate steps (AgentAction, Observation tuples)
        
        # Format them in a particular way
        intermediate_steps = kwargs.pop("intermediate_steps")
        thoughts = ""
        for action, observation in intermediate_steps:
            thoughts += action.log
            thoughts += f"\nObservation: {observation}\nThought: "
            
        # Set the agent_scratchpad variable to that value
        kwargs["agent_scratchpad"] = thoughts
        
        # Create a tools variable from the list of tools provided
        kwargs["tools"] = "\n".join([f"{tool.name}: {tool.description}" for tool in self.tools])
        
        # Create a list of tool names for the tools provided
        kwargs["tool_names"] = ", ".join([tool.name for tool in self.tools])
        formatted = self.template.format(**kwargs)
        return [HumanMessage(content=formatted)]
    
prompt = CustomPromptTemplate(
    template=template,
    tools=tools,
    # This omits the `agent_scratchpad`, `tools`, and `tool_names` variables because those are generated dynamically
    # This includes the `intermediate_steps` variable because that is needed
    input_variables=["input", "intermediate_steps"]
)
```


key: _node_content
string_value:
{"id_": "74116c2f-6c86-4660-ba03-dac4bc72c31b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "570d61ff-b15d-48d4-963c-af923782747c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4ff239492596d2ec5aa6d12c2e23483c2b123cbd1653502de2670561c2d9f3c2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5319a987-40da-4ad6-b371-915fd1336700", "node_type": "1", "metadata": {}, "hash": "9f19d647a7e2edfb89b7fc39aa6f7719c85fbeccf00f5b83101b78b1c8b70a2b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8132, "end_char_idx": 10904, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Set up a prompt template
class CustomPromptTemplate(BaseChatPromptTemplate):
    # The template to use
    template: str
    # The list of tools available
    tools: List[Tool]
    
    def format_messages(self, **kwargs) -> str:
        # Get the intermediate steps (AgentAction, Observation tuples)
        
        # Format them in a particular way
        intermediate_steps = kwargs.pop("intermediate_steps")
        thoughts = ""
        for action, observation in intermediate_steps:
            thoughts += action.log
            thoughts += f"\nObservation: {observation}\nThought: "
            
        # Set the agent_scratchpad variable to that value
        kwargs["agent_scratchpad"] = thoughts
        
        # Create a tools variable from the list of tools provided
        kwargs["tools"] = "\n".join([f"{tool.name}: {tool.description}" for tool in self.tools])
        
        # Create a list of tool names for the tools provided
        kwargs["tool_names"] = ", ".join([tool.name for tool in self.tools])
        formatted = self.template.format(**kwargs)
        return [HumanMessage(content=formatted)]
    
prompt = CustomPromptTemplate(
    template=template,
    tools=tools,
    # This omits the `agent_scratchpad`, `tools`, and `tool_names` variables because those are generated dynamically
    # This includes the `intermediate_steps` variable because that is needed
    input_variables=["input", "intermediate_steps"]
)
```


```python
class CustomOutputParser(AgentOutputParser):
    
    def parse(self, llm_output: str) -> Union[AgentAction, AgentFinish]:
        
        # Check if agent should finish
        if "Final Answer:" in llm_output:
            return AgentFinish(
                # Return values is generally always a dictionary with a single `output` key
                # It is not recommended to try anything else at the moment :)
                return_values={"output": llm_output.split("Final Answer:")[-1].strip()},
                log=llm_output,
            )
        
        # Parse out the action and action input
        regex = r"Action: (.*?)[\n]*Action Input:[\s]*(.*)"
        match = re.search(regex, llm_output, re.DOTALL)
        
        # If it can't parse the output it raises an error
        # You can add your own logic here to handle errors in a different way i.e. pass to a human, give a canned response
        if not match:
            raise ValueError(f"Could not parse LLM output: `{llm_output}`")
        action = match.group(1).strip()
        action_input = match.group(2)
        
        # Return the action and action input
        return AgentAction(tool=action, tool_input=action_input.strip(" ").strip('"'), log=llm_output)
    
output_parser = CustomOutputParser()
```


key: _node_content
string_value:
{"id_": "5319a987-40da-4ad6-b371-915fd1336700", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "74116c2f-6c86-4660-ba03-dac4bc72c31b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "b790999ad5e81a3ed4676b0b0cd58359e11d3b65ab99a4ddac74646b59d13fbd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bbeec2a8-7c74-44b6-b042-61ad438a1fa8", "node_type": "1", "metadata": {}, "hash": "9e544c0adb82452014db2ebad37b3332a6cc0b5d566103bd9ce65f378ec20021", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9603, "end_char_idx": 11750, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
class CustomOutputParser(AgentOutputParser):
    
    def parse(self, llm_output: str) -> Union[AgentAction, AgentFinish]:
        
        # Check if agent should finish
        if "Final Answer:" in llm_output:
            return AgentFinish(
                # Return values is generally always a dictionary with a single `output` key
                # It is not recommended to try anything else at the moment :)
                return_values={"output": llm_output.split("Final Answer:")[-1].strip()},
                log=llm_output,
            )
        
        # Parse out the action and action input
        regex = r"Action: (.*?)[\n]*Action Input:[\s]*(.*)"
        match = re.search(regex, llm_output, re.DOTALL)
        
        # If it can't parse the output it raises an error
        # You can add your own logic here to handle errors in a different way i.e. pass to a human, give a canned response
        if not match:
            raise ValueError(f"Could not parse LLM output: `{llm_output}`")
        action = match.group(1).strip()
        action_input = match.group(2)
        
        # Return the action and action input
        return AgentAction(tool=action, tool_input=action_input.strip(" ").strip('"'), log=llm_output)
    
output_parser = CustomOutputParser()
```


```python
# Initiate our LLM - default is 'gpt-3.5-turbo'
llm = ChatOpenAI(temperature=0)

# LLM chain consisting of the LLM and a prompt
llm_chain = LLMChain(llm=llm, prompt=prompt)

# Using tools, the LLM chain and output_parser to make an agent
tool_names = [tool.name for tool in tools]

agent = LLMSingleActionAgent(
    llm_chain=llm_chain, 
    output_parser=output_parser,
    # We use "Observation" as our stop sequence so it will stop when it receives Tool output
    # If you change your prompt template you'll need to adjust this as well
    stop=["\nObservation:"], 
    allowed_tools=tool_names
)
```


```python
# Initiate the agent that will respond to our queries
# Set verbose=True to share the CoT reasoning the LLM goes through
agent_executor = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True)
```


key: _node_content
string_value:
{"id_": "bbeec2a8-7c74-44b6-b042-61ad438a1fa8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5319a987-40da-4ad6-b371-915fd1336700", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4e764be97f53cc4d0d95320617c4f4e543badd232fbe700f15642ccaf555bf90", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4bb22ea0-59f3-4c70-ba43-902c85710fc5", "node_type": "1", "metadata": {}, "hash": "05ed5f0bdcf08d03187e4a332d9936a064a28b5773dfb464bd2d261aa18d5536", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10907, "end_char_idx": 12702, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Initiate our LLM - default is 'gpt-3.5-turbo'
llm = ChatOpenAI(temperature=0)

# LLM chain consisting of the LLM and a prompt
llm_chain = LLMChain(llm=llm, prompt=prompt)

# Using tools, the LLM chain and output_parser to make an agent
tool_names = [tool.name for tool in tools]

agent = LLMSingleActionAgent(
    llm_chain=llm_chain, 
    output_parser=output_parser,
    # We use "Observation" as our stop sequence so it will stop when it receives Tool output
    # If you change your prompt template you'll need to adjust this as well
    stop=["\nObservation:"], 
    allowed_tools=tool_names
)
```


```python
# Initiate the agent that will respond to our queries
# Set verbose=True to share the CoT reasoning the LLM goes through
agent_executor = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True)
```


```python
agent_executor.run("How many people live in canada as of 2023?")
```

    
    
    [1m> Entering new AgentExecutor chain...[0m
    [32;1m[1;3mThought: Hmm, I be not sure of the answer to that one. Let me think.
    Action: Search
    Action Input: "Canada population 2023"[0m
    
    Observation:[36;1m[1;3m39,566,248[0m[32;1m[1;3mAhoy, that be a lot of people! But I need to make sure this be true.
    Action: Search
    Action Input: "Canada population 2023 official source"[0m
    
    Observation:[36;1m[1;3mThe current population of Canada is 38,664,637 as of Wednesday, April 19, 2023, based on Worldometer elaboration of the latest United Nations data.[0m[32;1m[1;3mArrr, that be the official number! I be confident in me answer now.
    Final Answer: The population of Canada as of 2023 is 38,664,637. Arg![0m
    
    [1m> Finished chain.[0m





    'The population of Canada as of 2023 is 38,664,637. Arg!'


key: _node_content
string_value:
{"id_": "4bb22ea0-59f3-4c70-ba43-902c85710fc5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bbeec2a8-7c74-44b6-b042-61ad438a1fa8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "94645a092007a62f379d8cc18e1ccfaeb1f56512318ca1fb482a3b21ba814a13", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bfc41c2b-179e-473a-a974-c45a1ad69f6f", "node_type": "1", "metadata": {}, "hash": "a24bf5a116023d0d5c7608de4af83b53a2b1d177b1c00c93ae0b53fb8813233b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11524, "end_char_idx": 14159, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Initiate the agent that will respond to our queries
# Set verbose=True to share the CoT reasoning the LLM goes through
agent_executor = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True)
```


```python
agent_executor.run("How many people live in canada as of 2023?")
```

    
    
    [1m> Entering new AgentExecutor chain...[0m
    [32;1m[1;3mThought: Hmm, I be not sure of the answer to that one. Let me think.
    Action: Search
    Action Input: "Canada population 2023"[0m
    
    Observation:[36;1m[1;3m39,566,248[0m[32;1m[1;3mAhoy, that be a lot of people! But I need to make sure this be true.
    Action: Search
    Action Input: "Canada population 2023 official source"[0m
    
    Observation:[36;1m[1;3mThe current population of Canada is 38,664,637 as of Wednesday, April 19, 2023, based on Worldometer elaboration of the latest United Nations data.[0m[32;1m[1;3mArrr, that be the official number! I be confident in me answer now.
    Final Answer: The population of Canada as of 2023 is 38,664,637. Arg![0m
    
    [1m> Finished chain.[0m





    'The population of Canada as of 2023 is 38,664,637. Arg!'




```python
agent_executor.run("How many in 2022?")
```

    
    
    [1m> Entering new AgentExecutor chain...[0m
    [32;1m[1;3mThought: Hmm, I'm not sure what this question is asking about. I better use the search tool.
    Action: Search
    Action Input: "2022 events"[0m
    
    Observation:[36;1m[1;3m8. Humanitarian Crises Deepen · 7. Latin America Moves Left. · 6. Iranians Protest. · 5. COVID Eases. · 4. Inflation Returns. · 3. Climate Change ...[0m[32;1m[1;3mAhoy, it looks like this be a question about what be happenin' in 2022. Let me search again.
    Action: Search
    Action Input: "2022 calendar"[0m
    
    Observation:[36;1m[1;3mUnited States 2022 – Calendar with American holidays. Yearly calendar showing months for the year 2022. Calendars – online and print friendly – for any year ...[0m[32;1m[1;3mShiver me timbers, it looks like this be a question about the year 2022. Let me search one more time.
    Action: Search
    Action Input: "What be happenin' in 2022?"[0m
    
    Observation:[36;1m[1;3m8. Humanitarian Crises Deepen · 7. Latin America Moves Left. · 6. Iranians Protest. · 5. COVID Eases. · 4. Inflation Returns. · 3. Climate Change ...[0m[32;1m[1;3mAvast ye, it looks like the same results be comin' up. I reckon there be no clear answer to this question.
    Final Answer: Arg, I be sorry matey, but I can't give ye a clear answer to that question.[0m
    
    [1m> Finished chain.[0m


key: _node_content
string_value:
{"id_": "bfc41c2b-179e-473a-a974-c45a1ad69f6f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4bb22ea0-59f3-4c70-ba43-902c85710fc5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "db04eaa4b74ebb1dc0da09694f9897273fc605e875175b9cd7d1c2f7bd086713", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "97d231e5-68c8-4aee-9f1c-bd603ed395d5", "node_type": "1", "metadata": {}, "hash": "4011a19a0fd058d63775b68d24f42afb2de7e75092b56cb66bf54f0ab3bf4453", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14169, "end_char_idx": 16441, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"Arg, I be sorry matey, but I can't give ye a clear answer to that question."



## LLM Agent with History

Extend the LLM Agent with the ability to retain a [memory](https://python.langchain.com/en/latest/modules/agents/agents/custom_llm_agent.html#adding-memory) and use it as context as it continues the conversation.

We use a simple ```ConversationBufferWindowMemory``` for this example that keeps a rolling window of the last two conversation turns. LangChain has other [memory options](https://python.langchain.com/en/latest/modules/memory.html), with different tradeoffs suitable for different use cases.


```python
# Set up a prompt template which can interpolate the history
template_with_history = """You are SearchGPT, a professional search engine who provides informative answers to users. Answer the following questions as best you can. You have access to the following tools:

{tools}

Use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question

Begin! Remember to give detailed, informative answers

Previous conversation history:
{history}

New question: {input}
{agent_scratchpad}"""
```


```python
prompt_with_history = CustomPromptTemplate(
    template=template_with_history,
    tools=tools,
    # The history template includes "history" as an input variable so we can interpolate it into the prompt
    input_variables=["input", "intermediate_steps", "history"]
)

llm_chain = LLMChain(llm=llm, prompt=prompt_with_history)
tool_names = [tool.name for tool in tools]
agent = LLMSingleActionAgent(
    llm_chain=llm_chain, 
    output_parser=output_parser,
    stop=["\nObservation:"], 
    allowed_tools=tool_names
)
```


```python
# Initiate the memory with k=2 to keep the last two turns
# Provide the memory to the agent
memory = ConversationBufferWindowMemory(k=2)
agent_executor = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True, memory=memory)
```


key: _node_content
string_value:
{"id_": "97d231e5-68c8-4aee-9f1c-bd603ed395d5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bfc41c2b-179e-473a-a974-c45a1ad69f6f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "1bfcb0beaadb45d96da191962c3c12d0c11c349e0ab69f11facb4c79fb2f1767", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "52ab4778-06a7-41e0-897b-a38f60db992b", "node_type": "1", "metadata": {}, "hash": "d72580d1daba0e695a9dd8feecd2aa7c9c79e736f65b003ca673015c7992b219", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14784, "end_char_idx": 17516, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Set up a prompt template which can interpolate the history
template_with_history = """You are SearchGPT, a professional search engine who provides informative answers to users. Answer the following questions as best you can. You have access to the following tools:

{tools}

Use the following format:

Question: the input question you must answer
Thought: you should always think about what to do
Action: the action to take, should be one of [{tool_names}]
Action Input: the input to the action
Observation: the result of the action
... (this Thought/Action/Action Input/Observation can repeat N times)
Thought: I now know the final answer
Final Answer: the final answer to the original input question

Begin! Remember to give detailed, informative answers

Previous conversation history:
{history}

New question: {input}
{agent_scratchpad}"""
```


```python
prompt_with_history = CustomPromptTemplate(
    template=template_with_history,
    tools=tools,
    # The history template includes "history" as an input variable so we can interpolate it into the prompt
    input_variables=["input", "intermediate_steps", "history"]
)

llm_chain = LLMChain(llm=llm, prompt=prompt_with_history)
tool_names = [tool.name for tool in tools]
agent = LLMSingleActionAgent(
    llm_chain=llm_chain, 
    output_parser=output_parser,
    stop=["\nObservation:"], 
    allowed_tools=tool_names
)
```


```python
# Initiate the memory with k=2 to keep the last two turns
# Provide the memory to the agent
memory = ConversationBufferWindowMemory(k=2)
agent_executor = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True, memory=memory)
```


```python
agent_executor.run("How many people live in canada as of 2023?")
```

    
    
    [1m> Entering new AgentExecutor chain...[0m
    [32;1m[1;3mThought: I need to find the most recent population data for Canada.
    Action: Search
    Action Input: "Canada population 2023"[0m
    
    Observation:[36;1m[1;3m39,566,248[0m[32;1m[1;3mThis data seems reliable, but I should double-check the source.
    Action: Search
    Action Input: "Source of Canada population 2023"[0m
    
    Observation:[36;1m[1;3mThe current population of Canada is 38,664,637 as of Wednesday, April 19, 2023, based on Worldometer elaboration of the latest United Nations data. Canada 2020 population is estimated at 37,742,154 people at mid year according to UN data. Canada population is equivalent to 0.48% of the total world population.[0m[32;1m[1;3mI now know the final answer
    Final Answer: As of April 19, 2023, the population of Canada is 38,664,637.[0m
    
    [1m> Finished chain.[0m





    'As of April 19, 2023, the population of Canada is 38,664,637.'


key: _node_content
string_value:
{"id_": "52ab4778-06a7-41e0-897b-a38f60db992b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "97d231e5-68c8-4aee-9f1c-bd603ed395d5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "59ccb3997ac9896ba9fffee0e46dea53a187d868eb4f9394f19db17f1ef1caef", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1daae759-9412-4d69-b24d-bf763a8ca506", "node_type": "1", "metadata": {}, "hash": "5cd4d08ff5a58f634f6211014b1a1ef9867431cd29e898afa6a9d9e20442f767", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16184, "end_char_idx": 18538, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Initiate the memory with k=2 to keep the last two turns
# Provide the memory to the agent
memory = ConversationBufferWindowMemory(k=2)
agent_executor = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True, memory=memory)
```


```python
agent_executor.run("How many people live in canada as of 2023?")
```

    
    
    [1m> Entering new AgentExecutor chain...[0m
    [32;1m[1;3mThought: I need to find the most recent population data for Canada.
    Action: Search
    Action Input: "Canada population 2023"[0m
    
    Observation:[36;1m[1;3m39,566,248[0m[32;1m[1;3mThis data seems reliable, but I should double-check the source.
    Action: Search
    Action Input: "Source of Canada population 2023"[0m
    
    Observation:[36;1m[1;3mThe current population of Canada is 38,664,637 as of Wednesday, April 19, 2023, based on Worldometer elaboration of the latest United Nations data. Canada 2020 population is estimated at 37,742,154 people at mid year according to UN data. Canada population is equivalent to 0.48% of the total world population.[0m[32;1m[1;3mI now know the final answer
    Final Answer: As of April 19, 2023, the population of Canada is 38,664,637.[0m
    
    [1m> Finished chain.[0m





    'As of April 19, 2023, the population of Canada is 38,664,637.'




```python
agent_executor.run("how about in mexico?")
```

    
    
    [1m> Entering new AgentExecutor chain...[0m
    [32;1m[1;3mThought: I need to search for the current population of Mexico.
    Action: Search
    Action Input: "current population of Mexico"[0m
    
    Observation:[36;1m[1;3mMexico, officially the United Mexican States, is a country in the southern portion of North America. It is bordered to the north by the United States; to the south and west by the Pacific Ocean; to the southeast by Guatemala, Belize, and the Caribbean Sea; and to the east by the Gulf of Mexico.[0m[32;1m[1;3mThat's not the answer to the question, I need to refine my search.
    Action: Search
    Action Input: "population of Mexico 2023"[0m
    
    Observation:[36;1m[1;3m132,709,512[0m[32;1m[1;3mI now know the final answer.
    Final Answer: As of 2023, the population of Mexico is 132,709,512.[0m
    
    [1m> Finished chain.[0m





    'As of 2023, the population of Mexico is 132,709,512.'


key: _node_content
string_value:
{"id_": "1daae759-9412-4d69-b24d-bf763a8ca506", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "52ab4778-06a7-41e0-897b-a38f60db992b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "5d7d4576874d35abbb16473270b0be9b8c4594f7851f5357fd688372d7d09ccc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b31613e3-17df-462b-b104-d66b769651d8", "node_type": "1", "metadata": {}, "hash": "b82962f082c93df6e40a88b28beb18b90be450e954f8f0ff53780ce2f3b0c8c6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17453, "end_char_idx": 20222, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
'As of April 19, 2023, the population of Canada is 38,664,637.'




```python
agent_executor.run("how about in mexico?")
```

    
    
    [1m> Entering new AgentExecutor chain...[0m
    [32;1m[1;3mThought: I need to search for the current population of Mexico.
    Action: Search
    Action Input: "current population of Mexico"[0m
    
    Observation:[36;1m[1;3mMexico, officially the United Mexican States, is a country in the southern portion of North America. It is bordered to the north by the United States; to the south and west by the Pacific Ocean; to the southeast by Guatemala, Belize, and the Caribbean Sea; and to the east by the Gulf of Mexico.[0m[32;1m[1;3mThat's not the answer to the question, I need to refine my search.
    Action: Search
    Action Input: "population of Mexico 2023"[0m
    
    Observation:[36;1m[1;3m132,709,512[0m[32;1m[1;3mI now know the final answer.
    Final Answer: As of 2023, the population of Mexico is 132,709,512.[0m
    
    [1m> Finished chain.[0m





    'As of 2023, the population of Mexico is 132,709,512.'



## Knowledge base

Create a custom vectorstore for the Agent to use as a tool to answer questions with. We'll store the results in [Pinecone](https://docs.pinecone.io/docs/quickstart), which is supported by LangChain ([Docs](https://python.langchain.com/en/latest/modules/indexes/vectorstores/examples/pinecone.html), [API reference](https://python.langchain.com/en/latest/reference/modules/vectorstore.html)). For help getting started with Pinecone or other vector databases, we have a [cookbook](https://github.com/openai/openai-cookbook/blob/colin/examples/vector_databases/Using_vector_databases_for_embeddings_search.ipynb) to help you get started.

You can check the LangChain documentation to see what other [vectorstores](https://python.langchain.com/en/latest/modules/indexes/vectorstores.html) and [databases](https://python.langchain.com/en/latest/modules/chains/examples/sqlite.html) are available.

For this example we'll use the transcripts of the Stuff You Should Know podcast, which was provided thanks to OSF DOI [10.17605/OSF.IO/VM9NT](https://doi.org/10.17605/OSF.IO/VM9NT)


```python
import wget

# Here is a URL to a zip archive containing the transcribed podcasts
# Note that this data has already been split into chunks and embeddings from OpenAI's `text-embedding-3-small` embedding model are included
content_url = 'https://cdn.openai.com/API/examples/data/sysk_podcast_transcripts_embedded.json.zip'

# Download the file (it is ~541 MB so this will take some time)
wget.download(content_url)
```

    100% [......................................................................] 571275039 / 571275039




    'sysk_podcast_transcripts_embedded.json.zip'


key: _node_content
string_value:
{"id_": "b31613e3-17df-462b-b104-d66b769651d8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1daae759-9412-4d69-b24d-bf763a8ca506", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "83b9ffa64a06d84e361033060c1617889008ac076b9251bec6ddc23427bc22c0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dc0be983-9942-4dd4-8b8b-ef0c4a3d5b67", "node_type": "1", "metadata": {}, "hash": "174ae9ba68021ffe4352f49a6370a345e279f5faa5bce4e261b02ae474f9ce54", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Knowledge base

Create a custom vectorstore for the Agent to use as a tool to answer questions with. We'll store the results in [Pinecone](https://docs.pinecone.io/docs/quickstart), which is supported by LangChain ([Docs](https://python.langchain.com/en/latest/modules/indexes/vectorstores/examples/pinecone.html), [API reference](https://python.langchain.com/en/latest/reference/modules/vectorstore.html)). For help getting started with Pinecone or other vector databases, we have a [cookbook](https://github.com/openai/openai-cookbook/blob/colin/examples/vector_databases/Using_vector_databases_for_embeddings_search.ipynb) to help you get started.

You can check the LangChain documentation to see what other [vectorstores](https://python.langchain.com/en/latest/modules/indexes/vectorstores.html) and [databases](https://python.langchain.com/en/latest/modules/chains/examples/sqlite.html) are available.

For this example we'll use the transcripts of the Stuff You Should Know podcast, which was provided thanks to OSF DOI [10.17605/OSF.IO/VM9NT](https://doi.org/10.17605/OSF.IO/VM9NT)


```python
import wget

# Here is a URL to a zip archive containing the transcribed podcasts
# Note that this data has already been split into chunks and embeddings from OpenAI's `text-embedding-3-small` embedding model are included
content_url = 'https://cdn.openai.com/API/examples/data/sysk_podcast_transcripts_embedded.json.zip'

# Download the file (it is ~541 MB so this will take some time)
wget.download(content_url)
```

    100% [......................................................................] 571275039 / 571275039




    'sysk_podcast_transcripts_embedded.json.zip'




```python
# Load podcasts
with zipfile.ZipFile("sysk_podcast_transcripts_embedded.json.zip","r") as zip_ref:
    zip_ref.extractall("./data")
f = open('./data/sysk_podcast_transcripts_embedded.json')
processed_podcasts = json.load(f)
```


```python
# Have a look at the contents
pd.DataFrame(processed_podcasts).head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>filename</th>
      <th>title</th>
      <th>url</th>
      <th>text_chunk</th>
      <th>embedding</th>
      <th>cleaned_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>\n\nSYSK Selects How Crime Scene Cleanup Works</td>
      <td>https://chtbl.com/track/5899E/podtrac.com/pts/.</td>
      <td>Title: sysk_with_transcripts_SYSK Selects How .</td>
      <td>[0.021279960870742798,


key: _node_content
string_value:
{"id_": "dc0be983-9942-4dd4-8b8b-ef0c4a3d5b67", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b31613e3-17df-462b-b104-d66b769651d8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "24cd79ba97cdc5d4fe7d9456886ac72c6d01ee2f22bd45c499bf6aea4d28a233", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cf9cc66b-fb6a-4663-911b-f7800740c75e", "node_type": "1", "metadata": {}, "hash": "f23efa80eb90156ee4238529f9d7f59e891ecc21466e8d08ec56e7cb00d57d4c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
'sysk_podcast_transcripts_embedded.json.zip'




```python
# Load podcasts
with zipfile.ZipFile("sysk_podcast_transcripts_embedded.json.zip","r") as zip_ref:
    zip_ref.extractall("./data")
f = open('./data/sysk_podcast_transcripts_embedded.json')
processed_podcasts = json.load(f)
```


```python
# Have a look at the contents
pd.DataFrame(processed_podcasts).head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>filename</th>
      <th>title</th>
      <th>url</th>
      <th>text_chunk</th>
      <th>embedding</th>
      <th>cleaned_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>\n\nSYSK Selects How Crime Scene Cleanup Works</td>
      <td>https://chtbl.com/track/5899E/podtrac.com/pts/.</td>
      <td>Title: sysk_with_transcripts_SYSK Selects How .</td>
      <td>[0.021279960870742798, -0.005817972123622894, .</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
    </tr>
    <tr>
      <th>1</th>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>\n\nSYSK Selects How Crime Scene Cleanup Works</td>
      <td>https://chtbl.com/track/5899E/podtrac.com/pts/.</td>
      <td>Title: sysk_with_transcripts_SYSK Selects How .</td>
      <td>[0.013859338127076626, 0.00857278611510992, 0.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
    </tr>
    <tr>
      <th>2</th>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>\n\nSYSK Selects How Crime Scene Cleanup Works</td>
      <td>https://chtbl.com/track/5899E/podtrac.com/pts/.</td>
      <td>Title: sysk_with_transcripts_SYSK Selects How .</td>
      <td>[0.015242221765220165, 0.016030369326472282, 0.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.


key: _node_content
string_value:
{"id_": "cf9cc66b-fb6a-4663-911b-f7800740c75e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dc0be983-9942-4dd4-8b8b-ef0c4a3d5b67", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "e7ee0a511fe744537f31528feec32b9ea7f86696756c3b9096d55370649f4013", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "96f04acf-77e2-494a-bfc3-5fd4f811192c", "node_type": "1", "metadata": {}, "hash": "7ac1ce2afddc1e51a1ef78273be050aa7b1f4c8dcd6253f71cfae8d5981988ff", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</td>
      <td>[0.021279960870742798, -0.005817972123622894, .</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
    </tr>
    <tr>
      <th>1</th>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>\n\nSYSK Selects How Crime Scene Cleanup Works</td>
      <td>https://chtbl.com/track/5899E/podtrac.com/pts/.</td>
      <td>Title: sysk_with_transcripts_SYSK Selects How .</td>
      <td>[0.013859338127076626, 0.00857278611510992, 0.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
    </tr>
    <tr>
      <th>2</th>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>\n\nSYSK Selects How Crime Scene Cleanup Works</td>
      <td>https://chtbl.com/track/5899E/podtrac.com/pts/.</td>
      <td>Title: sysk_with_transcripts_SYSK Selects How .</td>
      <td>[0.015242221765220165, 0.016030369326472282, 0.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
    </tr>
    <tr>
      <th>3</th>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>\n\nSYSK Selects How Crime Scene Cleanup Works</td>
      <td>https://chtbl.com/track/5899E/podtrac.com/pts/.</td>
      <td>Title: sysk_with_transcripts_SYSK Selects How .</td>
      <td>[0.004371842369437218, -0.003036574460566044, .</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
    </tr>
    <tr>
      <th>4</th>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>\n\nSYSK Selects How Crime Scene Cleanup Works</td>
      <td>https://chtbl.com/track/5899E/podtrac.com/pts/.</td>
      <td>Title: sysk_with_transcripts_SYSK Selects How .</td>
      <td>[0.017309172078967094, 0.015154214575886726, 0.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
    </tr>
  </tbody>
</table>
</div>


key: _node_content
string_value:
{"id_": "96f04acf-77e2-494a-bfc3-5fd4f811192c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cf9cc66b-fb6a-4663-911b-f7800740c75e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c26059937c7036c1eb048049a482c41b86eb79990b99724aa14c33dd6d120ca6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1a63deb7-c619-48bf-b3d7-d097924ddff0", "node_type": "1", "metadata": {}, "hash": "f3be76813358c0853ae470b01edca39e153751821521806878ac62fadea80158", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
015242221765220165, 0.016030369326472282, 0.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
    </tr>
    <tr>
      <th>3</th>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>\n\nSYSK Selects How Crime Scene Cleanup Works</td>
      <td>https://chtbl.com/track/5899E/podtrac.com/pts/.</td>
      <td>Title: sysk_with_transcripts_SYSK Selects How .</td>
      <td>[0.004371842369437218, -0.003036574460566044, .</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
    </tr>
    <tr>
      <th>4</th>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>\n\nSYSK Selects How Crime Scene Cleanup Works</td>
      <td>https://chtbl.com/track/5899E/podtrac.com/pts/.</td>
      <td>Title: sysk_with_transcripts_SYSK Selects How .</td>
      <td>[0.017309172078967094, 0.015154214575886726, 0.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
    </tr>
  </tbody>
</table>
</div>




```python
# Add the text embeddings to Pinecone

batch_size = 100  # how many embeddings we create and insert at once

for i in tqdm(range(0, len(processed_podcasts), batch_size)):
    # find end of batch
    i_end = min(len(processed_podcasts), i+batch_size)
    meta_batch = processed_podcasts[i:i_end]
    # get ids
    ids_batch = [x['cleaned_id'] for x in meta_batch]
    # get texts to encode
    texts = [x['text_chunk'] for x in meta_batch]
    # add embeddings
    embeds = [x['embedding'] for x in meta_batch]
    # cleanup metadata
    meta_batch = [{
        'filename': x['filename'],
        'title': x['title'],
        'text_chunk': x['text_chunk'],
        'url': x['url']
    } for x in meta_batch]
    to_upsert = list(zip(ids_batch, embeds, meta_batch))
    # upsert to Pinecone
    index.upsert(vectors=to_upsert)
```


```python
# Configuring the embeddings to be used by our retriever to be OpenAI Embeddings, matching our embedded corpus
embeddings = OpenAIEmbeddings()


# Loads a docsearch object from an existing Pinecone index so we can retrieve from it
docsearch = Pinecone.from_existing_index(index_name,embeddings,text_key='text_chunk')
```


```python
retriever = docsearch.as_retriever()
```


```python
query_docs = retriever.get_relevant_documents("can you live without a bank account")
```


key: _node_content
string_value:
{"id_": "1a63deb7-c619-48bf-b3d7-d097924ddff0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "96f04acf-77e2-494a-bfc3-5fd4f811192c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "6ca5e8412817a99ae212c5fb8011707d4ff950865786c8db10dcfa88d287c4c5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "58aa0957-c0c8-4cf8-a817-12f23a88295a", "node_type": "1", "metadata": {}, "hash": "5cbcd43d7fcbee90ae1b82bcfc7638505b02b6f0838d45c0e4d9780e50592c8c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0.015154214575886726, 0.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
    </tr>
  </tbody>
</table>
</div>




```python
# Add the text embeddings to Pinecone

batch_size = 100  # how many embeddings we create and insert at once

for i in tqdm(range(0, len(processed_podcasts), batch_size)):
    # find end of batch
    i_end = min(len(processed_podcasts), i+batch_size)
    meta_batch = processed_podcasts[i:i_end]
    # get ids
    ids_batch = [x['cleaned_id'] for x in meta_batch]
    # get texts to encode
    texts = [x['text_chunk'] for x in meta_batch]
    # add embeddings
    embeds = [x['embedding'] for x in meta_batch]
    # cleanup metadata
    meta_batch = [{
        'filename': x['filename'],
        'title': x['title'],
        'text_chunk': x['text_chunk'],
        'url': x['url']
    } for x in meta_batch]
    to_upsert = list(zip(ids_batch, embeds, meta_batch))
    # upsert to Pinecone
    index.upsert(vectors=to_upsert)
```


```python
# Configuring the embeddings to be used by our retriever to be OpenAI Embeddings, matching our embedded corpus
embeddings = OpenAIEmbeddings()


# Loads a docsearch object from an existing Pinecone index so we can retrieve from it
docsearch = Pinecone.from_existing_index(index_name,embeddings,text_key='text_chunk')
```


```python
retriever = docsearch.as_retriever()
```


```python
query_docs = retriever.get_relevant_documents("can you live without a bank account")
```


```python
# Print out the title and content for the most relevant retrieved documents
print("\n".join(['Title: ' + x.metadata['title'].strip() + '\n\n' + x.page_content + '\n\n' for x in query_docs]))
```

    Title: sysk: Can You Live Without a Bank Account?
    
    Title: sysk_with_transcripts_Can you live without a bank account.json;  And if you had a life, you didn't necessarily rectify your bank checkbook every day. Oh, wait, what is balancing a checkbook mean? Seriously? Yeah. Thank God for my wife. So another reason you might avoid a bank is philosophically. There may be a longstanding distrust of banks in your family that you don't want to put your money in, or you may just want to be like, you know what? I don't want to take part in this modern society. I want to kind of drop out a bit. And a really good first move is to shut your bank account down. That's a big statement. Oh, yeah, it is. But a lot of people that are underbanked and don't have accounts aren't there on purpose. It's not some philosophical statement. A lot of times it's simply because they are poor and they don't have a lot of alternatives. Yeah. And the other thing about not having a bank account, not only do you not have a bank account, you also are, like, basically just avoiding banks altogether. There's plenty of other things that banks offer, like loans and mortgage, lollipops, stuff like that. Yeah. Maybe some free nasty coffee.


key: _node_content
string_value:
{"id_": "58aa0957-c0c8-4cf8-a817-12f23a88295a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1a63deb7-c619-48bf-b3d7-d097924ddff0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "837aa519a6b1c8f1bbb227fd695c04764d1bfd1a5de3fa039e6a16160063d124", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f722ce67-b52d-4376-9f48-6189f22abac1", "node_type": "1", "metadata": {}, "hash": "9f51523ea4d94182ff7118ebe48746e0ecc1de80e1facdef07389e336cbc613b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24777, "end_char_idx": 27953, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
query_docs = retriever.get_relevant_documents("can you live without a bank account")
```


```python
# Print out the title and content for the most relevant retrieved documents
print("\n".join(['Title: ' + x.metadata['title'].strip() + '\n\n' + x.page_content + '\n\n' for x in query_docs]))
```

    Title: sysk: Can You Live Without a Bank Account?
    
    Title: sysk_with_transcripts_Can you live without a bank account.json;  And if you had a life, you didn't necessarily rectify your bank checkbook every day. Oh, wait, what is balancing a checkbook mean? Seriously? Yeah. Thank God for my wife. So another reason you might avoid a bank is philosophically. There may be a longstanding distrust of banks in your family that you don't want to put your money in, or you may just want to be like, you know what? I don't want to take part in this modern society. I want to kind of drop out a bit. And a really good first move is to shut your bank account down. That's a big statement. Oh, yeah, it is. But a lot of people that are underbanked and don't have accounts aren't there on purpose. It's not some philosophical statement. A lot of times it's simply because they are poor and they don't have a lot of alternatives. Yeah. And the other thing about not having a bank account, not only do you not have a bank account, you also are, like, basically just avoiding banks altogether. There's plenty of other things that banks offer, like loans and mortgage, lollipops, stuff like that. Yeah. Maybe some free nasty coffee. So when you don't have a banking account, that's like the most basic unit of the banking world. Right. If you don't have that, you obviously aren't going to be exposed to all these other things that can help. Things like build your credit history through like a revolving loan or a mortgage or a car loan or something like that that you can build up your credit for and ultimately save money. So when you don't have a bank account, for whatever reason, you are effectively out of the banking system. The problem is you can live parallel to the banking system outside of it, but it can be really dangerous, especially if you're just dealing with cash, because that cash has to stay somewhere, whether it's on you or in your mattress or in a coffee can in your backyard. You're exposed for having that readily available to anybody who finds it or comes into your house with a gun to get it. Yes.
    
    
    Title: sysk: Can You Live Without a Bank Account?
    
    Title: sysk_with_transcripts_Can you live without a bank account.json;  And it doesn't have to be an everyday thing. You can host when you want. Like, let's say you're taking a week's vacation. Why not host your home? Because that money could go toward paying for your current vacation or towards your retirement fund or even towards your kids college fund. Yeah. For anything. And listen, if you're worried about your stuff, don't be. Air cover for hosts. Let hosts welcome guests into their home without having to worry. You get $1 million in damage protection anytime you're hosting. Plus pet damage protection and income loss protection, too. And are you ready for this?


key: _node_content
string_value:
{"id_": "f722ce67-b52d-4376-9f48-6189f22abac1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "58aa0957-c0c8-4cf8-a817-12f23a88295a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "d68aa45d23c582c203c50c25c01a88b44ce25691db442efd93b7b1913716a7bd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bc795230-f2c0-462b-90b2-e196ca612884", "node_type": "1", "metadata": {}, "hash": "650d8eaa0dcfee76980ffb53cdb3430b5c6db4a249d25d7695f4fcd2c19e1998", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 26276, "end_char_idx": 29595, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Yeah. Maybe some free nasty coffee. So when you don't have a banking account, that's like the most basic unit of the banking world. Right. If you don't have that, you obviously aren't going to be exposed to all these other things that can help. Things like build your credit history through like a revolving loan or a mortgage or a car loan or something like that that you can build up your credit for and ultimately save money. So when you don't have a bank account, for whatever reason, you are effectively out of the banking system. The problem is you can live parallel to the banking system outside of it, but it can be really dangerous, especially if you're just dealing with cash, because that cash has to stay somewhere, whether it's on you or in your mattress or in a coffee can in your backyard. You're exposed for having that readily available to anybody who finds it or comes into your house with a gun to get it. Yes.
    
    
    Title: sysk: Can You Live Without a Bank Account?
    
    Title: sysk_with_transcripts_Can you live without a bank account.json;  And it doesn't have to be an everyday thing. You can host when you want. Like, let's say you're taking a week's vacation. Why not host your home? Because that money could go toward paying for your current vacation or towards your retirement fund or even towards your kids college fund. Yeah. For anything. And listen, if you're worried about your stuff, don't be. Air cover for hosts. Let hosts welcome guests into their home without having to worry. You get $1 million in damage protection anytime you're hosting. Plus pet damage protection and income loss protection, too. And are you ready for this? Air cover for host is completely free every time you host on airbnb. Free with a capital F, with air cover for Host. It makes hosting a no brainer, and the benefits really start adding up. So learn more and host with peace of mind at Airbnb comaircoverforhosts. Capital One offers commercial solutions you can bank on. Now more than ever, your business faces specific challenges and unique opportunities. That's why Capital One offers a comprehensive suite of financial services custom tailored to your short and long term goals, backed by the expertise, strategy and resources of a top ten commercial bank, a dedicated team works with you to support your success and help you achieve your goals. Explore the possibilities at CapitalOne. comCOMMERCIAL all right, so if you live in modern society today, it is pretty tough to get by without a bank. Most cases these days you have well, I don't know about most cases, but in many cases you have automatic deposits of your work checks. Sure. A lot of people pay their bills wirelessly, online, directly from their bank. You might have a student loan, you might have a car loan, you might have your house mortgage, you might pay your credit card bills. All this stuff is running through a bank, most likely. And you would think it's probably impossible to not have a bank account these days. And I would say pretty much all Americans have them. Not true. Well, pretty much all Americans do. Like 93% do. Yeah, but that's not all. No, it's true.
    
    
    Title: sysk: Can You Live Without a Bank Account?
    
    Title: sysk_with_transcripts_Can you live without a bank account.json;  Yeah.


key: _node_content
string_value:
{"id_": "bc795230-f2c0-462b-90b2-e196ca612884", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f722ce67-b52d-4376-9f48-6189f22abac1", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4a010337d318c0320820282b3d7e975189573e9e3a51f8acd2097090a06c44fa", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1ea6abc5-5124-4ac5-851d-2f5c99ae3d22", "node_type": "1", "metadata": {}, "hash": "8f8519773341e2a6e67457d31b059ed24b095ae5cd64625a4cefefccec2286a5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27866, "end_char_idx": 31049, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Plus pet damage protection and income loss protection, too. And are you ready for this? Air cover for host is completely free every time you host on airbnb. Free with a capital F, with air cover for Host. It makes hosting a no brainer, and the benefits really start adding up. So learn more and host with peace of mind at Airbnb comaircoverforhosts. Capital One offers commercial solutions you can bank on. Now more than ever, your business faces specific challenges and unique opportunities. That's why Capital One offers a comprehensive suite of financial services custom tailored to your short and long term goals, backed by the expertise, strategy and resources of a top ten commercial bank, a dedicated team works with you to support your success and help you achieve your goals. Explore the possibilities at CapitalOne. comCOMMERCIAL all right, so if you live in modern society today, it is pretty tough to get by without a bank. Most cases these days you have well, I don't know about most cases, but in many cases you have automatic deposits of your work checks. Sure. A lot of people pay their bills wirelessly, online, directly from their bank. You might have a student loan, you might have a car loan, you might have your house mortgage, you might pay your credit card bills. All this stuff is running through a bank, most likely. And you would think it's probably impossible to not have a bank account these days. And I would say pretty much all Americans have them. Not true. Well, pretty much all Americans do. Like 93% do. Yeah, but that's not all. No, it's true.
    
    
    Title: sysk: Can You Live Without a Bank Account?
    
    Title: sysk_with_transcripts_Can you live without a bank account.json;  Yeah. 7% of Americans do not have bank accounts. About 9 million people last year in 2015 did not have bank accounts. 9 million people is a lot of people. No, it really is. And apparently that's household sorry, not people. Yeah, right. You're that is a big distinction, too. And the FDIC said, man, that's the lowest since we've been tracking this by far. And someone said, well, how long have you been tracking this? They said, well, the last six years. Really? Yeah, which I'm like. Really? That's when they started tracking it, but apparently so 2009. So if you want another number, the 9 million American households don't have bank accounts at all, then there are 25 million households in addition to that. So that makes almost like 34 million households, which that's a substantial number at this point. Sure. The 25 million are what's called underbanked, meaning they may have a bank account, but they don't use the bank account. Yeah. They don't use it because they are probably afraid of overdraft fees. Or they have maybe a bank account that got grandfathered in so that they don't have to pay minimum amount fees. And who knows? There's all sorts of reasons for people to not use a bank account that they have, but probably cheap among them is overdressed, which you'll talk more about. Yeah. And the majority of these underbank people in the United States are poor, usually. A lot of times they're minorities, a lot of times they're less educated.


key: _node_content
string_value:
{"id_": "1ea6abc5-5124-4ac5-851d-2f5c99ae3d22", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bc795230-f2c0-462b-90b2-e196ca612884", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "09e4ec8af13d5079b6fcf0a16d23216453c1df2dbcf46659b2815c1171bbaabb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2524b0ce-edff-4e39-a7a3-1b189f669bf7", "node_type": "1", "metadata": {}, "hash": "601fd64233a60c86e14375014cf0910febbbc3f0a0096658e509aa8eae6a93ef", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29459, "end_char_idx": 32432, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Title: sysk: Can You Live Without a Bank Account?
    
    Title: sysk_with_transcripts_Can you live without a bank account.json;  Yeah. 7% of Americans do not have bank accounts. About 9 million people last year in 2015 did not have bank accounts. 9 million people is a lot of people. No, it really is. And apparently that's household sorry, not people. Yeah, right. You're that is a big distinction, too. And the FDIC said, man, that's the lowest since we've been tracking this by far. And someone said, well, how long have you been tracking this? They said, well, the last six years. Really? Yeah, which I'm like. Really? That's when they started tracking it, but apparently so 2009. So if you want another number, the 9 million American households don't have bank accounts at all, then there are 25 million households in addition to that. So that makes almost like 34 million households, which that's a substantial number at this point. Sure. The 25 million are what's called underbanked, meaning they may have a bank account, but they don't use the bank account. Yeah. They don't use it because they are probably afraid of overdraft fees. Or they have maybe a bank account that got grandfathered in so that they don't have to pay minimum amount fees. And who knows? There's all sorts of reasons for people to not use a bank account that they have, but probably cheap among them is overdressed, which you'll talk more about. Yeah. And the majority of these underbank people in the United States are poor, usually. A lot of times they're minorities, a lot of times they're less educated. And these communities, there's a few reasons why they may not want to use a bank one. Maybe they don't trust banks. And if you look in the history of the United States or certainly even we're just talking about the Wells Fargo scandal, when you see stuff like that on the news, it should be upsetting to everyone. But obviously if you're poor and you don't have a lot of money, that may scare you into not wanting to use a bank at all. Right? Yeah.
    
    
    Title: sysk: Can You Live Without a Bank Account?
    
    Title: sysk_with_transcripts_Can you live without a bank account.json;  Maybe at the time, I might be making it up. I seem to remember them saying that, and I was like, I don't want that. Just let the check bounce and I'll take it up with them. Yes. The way it was marketed, though, was like, hey, we value you. We want to make sure that you can pay all your bills. So if something happens and you're overdrafted we'll cover it. We're just going to charge you a fee. And it sounds good, but again, when you go from high to low and all of a sudden your overdraft fees go from one to four or five or however many, that's a huge problem. Well, and the people that are overdrafting and the people that are at least able to afford those fees. Exactly. So it's a disproportionate burden on the poor, which makes it, as a scam, one of the more evil scams around. Yes.


key: _node_content
string_value:
{"id_": "2524b0ce-edff-4e39-a7a3-1b189f669bf7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1ea6abc5-5124-4ac5-851d-2f5c99ae3d22", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c2c35f6da0b7649f623e9ae0cd01c549a6e071da0e5b9b2e5c44d354dc174800", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f3a80d8f-e1fb-40c5-9771-dc8a69c2e696", "node_type": "1", "metadata": {}, "hash": "4faa7e8b1aadbf25a859371a3cf53430cd8fd72f31b41cb676e9c34b1b5d3694", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30888, "end_char_idx": 33868, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Yeah. And the majority of these underbank people in the United States are poor, usually. A lot of times they're minorities, a lot of times they're less educated. And these communities, there's a few reasons why they may not want to use a bank one. Maybe they don't trust banks. And if you look in the history of the United States or certainly even we're just talking about the Wells Fargo scandal, when you see stuff like that on the news, it should be upsetting to everyone. But obviously if you're poor and you don't have a lot of money, that may scare you into not wanting to use a bank at all. Right? Yeah.
    
    
    Title: sysk: Can You Live Without a Bank Account?
    
    Title: sysk_with_transcripts_Can you live without a bank account.json;  Maybe at the time, I might be making it up. I seem to remember them saying that, and I was like, I don't want that. Just let the check bounce and I'll take it up with them. Yes. The way it was marketed, though, was like, hey, we value you. We want to make sure that you can pay all your bills. So if something happens and you're overdrafted we'll cover it. We're just going to charge you a fee. And it sounds good, but again, when you go from high to low and all of a sudden your overdraft fees go from one to four or five or however many, that's a huge problem. Well, and the people that are overdrafting and the people that are at least able to afford those fees. Exactly. So it's a disproportionate burden on the poor, which makes it, as a scam, one of the more evil scams around. Yes. It's just wrong, then the idea that if you open an account, you should not opt in for overdraft protection. And it's easy to say when you're talking about checks for, like you're writing a check for a Mountain Dew and some cheetos. Yeah, who cares if you're short for that? You can go without that. But when you're talking about your rent check or like an actual grocery bill or something like that, it sucks that you can't get that stuff. But it's better to have to put a couple of things back than to pay $35 for one $2 item that you went over by, right? Yeah, that's a good point. And this was in my case, too. This is also back in the day when you I mean, a lot of times it was a mystery how much you had in your account. Right. Like, you couldn't just get on your phone before you write the check and be like, oh, well, no, I don't have enough money to cover this. Yeah, because even if you balanced your checkbook, sometimes you forgot to carry the one, it wasn't always 100% accurate.
    
    


## LLM Agent with Tools

Extend our list of tools by creating a [RetrievalQA](https://python.langchain.com/en/latest/modules/chains/index_examples/vector_db_qa.html) chain leveraging our Pinecone knowledge base.


```python
from langchain.chains import RetrievalQA

retrieval_llm = OpenAI(temperature=0)

podcast_retriever = RetrievalQA.from_chain_type(llm=retrieval_llm, chain_type="stuff", retriever=docsearch.as_retriever())
```


key: _node_content
string_value:
{"id_": "f3a80d8f-e1fb-40c5-9771-dc8a69c2e696", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2524b0ce-edff-4e39-a7a3-1b189f669bf7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4eeb28eb70d181b202d91975f673672a6662f3e521665b442dd2a45497143997", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6107c28d-3773-4eb6-9b87-d5e7df4a6a1a", "node_type": "1", "metadata": {}, "hash": "945a04e7dca5025c082282e3c1d34917bbb7e30946a48c1dd9af344a38d41ea8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 32319, "end_char_idx": 35351, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
So it's a disproportionate burden on the poor, which makes it, as a scam, one of the more evil scams around. Yes. It's just wrong, then the idea that if you open an account, you should not opt in for overdraft protection. And it's easy to say when you're talking about checks for, like you're writing a check for a Mountain Dew and some cheetos. Yeah, who cares if you're short for that? You can go without that. But when you're talking about your rent check or like an actual grocery bill or something like that, it sucks that you can't get that stuff. But it's better to have to put a couple of things back than to pay $35 for one $2 item that you went over by, right? Yeah, that's a good point. And this was in my case, too. This is also back in the day when you I mean, a lot of times it was a mystery how much you had in your account. Right. Like, you couldn't just get on your phone before you write the check and be like, oh, well, no, I don't have enough money to cover this. Yeah, because even if you balanced your checkbook, sometimes you forgot to carry the one, it wasn't always 100% accurate.
    
    


## LLM Agent with Tools

Extend our list of tools by creating a [RetrievalQA](https://python.langchain.com/en/latest/modules/chains/index_examples/vector_db_qa.html) chain leveraging our Pinecone knowledge base.


```python
from langchain.chains import RetrievalQA

retrieval_llm = OpenAI(temperature=0)

podcast_retriever = RetrievalQA.from_chain_type(llm=retrieval_llm, chain_type="stuff", retriever=docsearch.as_retriever())
```


```python
expanded_tools = [
    Tool(
        name = "Search",
        func=search.run,
        description="useful for when you need to answer questions about current events"
    ),
    Tool(
        name = 'Knowledge Base',
        func=podcast_retriever.run,
        description="Useful for general questions about how to do things and for details on interesting topics. Input should be a fully formed question."
    )
]
```


```python
# Re-initialize the agent with our new list of tools
prompt_with_history = CustomPromptTemplate(
    template=template_with_history,
    tools=expanded_tools,
    input_variables=["input", "intermediate_steps", "history"]
)
llm_chain = LLMChain(llm=llm, prompt=prompt_with_history)
multi_tool_names = [tool.name for tool in expanded_tools]
multi_tool_agent = LLMSingleActionAgent(
    llm_chain=llm_chain, 
    output_parser=output_parser,
    stop=["\nObservation:"], 
    allowed_tools=multi_tool_names
)
```


```python
multi_tool_memory = ConversationBufferWindowMemory(k=2)
multi_tool_executor = AgentExecutor.from_agent_and_tools(agent=multi_tool_agent, tools=expanded_tools, verbose=True, memory=multi_tool_memory)
```


```python
multi_tool_executor.run("Hi, I'd like to know how you can live without a bank account")
```

    
    
    [1m> Entering new AgentExecutor chain...[0m
    [32;1m[1;3mThought: This is an interesting question. I'm not sure if I have the answer in my knowledge base, so I might need to search for it.


key: _node_content
string_value:
{"id_": "6107c28d-3773-4eb6-9b87-d5e7df4a6a1a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f3a80d8f-e1fb-40c5-9771-dc8a69c2e696", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f6b3188838f0b3a1789e2db275fe43e481e4ee58c4f64c7f469bc01d03f53be4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1c705ab5-586a-4e93-894a-08d88621a223", "node_type": "1", "metadata": {}, "hash": "e5b3c21ec308a8bc1da90066fe09cb8e517991588f5bce019648479121fc6e30", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 33871, "end_char_idx": 37148, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
expanded_tools = [
    Tool(
        name = "Search",
        func=search.run,
        description="useful for when you need to answer questions about current events"
    ),
    Tool(
        name = 'Knowledge Base',
        func=podcast_retriever.run,
        description="Useful for general questions about how to do things and for details on interesting topics. Input should be a fully formed question."
    )
]
```


```python
# Re-initialize the agent with our new list of tools
prompt_with_history = CustomPromptTemplate(
    template=template_with_history,
    tools=expanded_tools,
    input_variables=["input", "intermediate_steps", "history"]
)
llm_chain = LLMChain(llm=llm, prompt=prompt_with_history)
multi_tool_names = [tool.name for tool in expanded_tools]
multi_tool_agent = LLMSingleActionAgent(
    llm_chain=llm_chain, 
    output_parser=output_parser,
    stop=["\nObservation:"], 
    allowed_tools=multi_tool_names
)
```


```python
multi_tool_memory = ConversationBufferWindowMemory(k=2)
multi_tool_executor = AgentExecutor.from_agent_and_tools(agent=multi_tool_agent, tools=expanded_tools, verbose=True, memory=multi_tool_memory)
```


```python
multi_tool_executor.run("Hi, I'd like to know how you can live without a bank account")
```

    
    
    [1m> Entering new AgentExecutor chain...[0m
    [32;1m[1;3mThought: This is an interesting question. I'm not sure if I have the answer in my knowledge base, so I might need to search for it.
    Action: Search
    Action Input: "How to live without a bank account"[0m
    
    Observation:[36;1m[1;3mUnderbanked households have a checking or savings account but also use alternative financial services such as money orders, check cashing, international remittances, payday loans, refund anticipation loans, rent-to-own services, pawnshop loans, or auto title loans, according to the FDIC.[0m[32;1m[1;3mIt seems like there are alternative financial services available for those who don't have a bank account. I should look into this further to provide a more comprehensive answer.
    Action: Search
    Action Input: "Alternative financial services for those without a bank account"[0m
    
    Observation:[36;1m[1;3mInstead, people who are unbanked use alternative financial services—payday loans, money orders, check cashing services, pawnshop loans, and the like—to meet their banking needs. These financial services fill an important role for unbanked individuals, but they can also cause further financial hardship.[0m[32;1m[1;3mIt's important to note that while there are alternatives to having a bank account, relying solely on these services can lead to financial difficulties. I should provide some information on the potential drawbacks of not having a bank account.
    Action: Knowledge Base
    Action Input: "What are the drawbacks of not having a bank account?"[0m
    
    Observation:[33;1m[1;3m Not having a bank account can be dangerous, as the cash has to be stored somewhere and can be exposed to anyone who finds it or comes into the house with a gun to get it. Additionally, not having a bank account means not being exposed to other things that can help, such as building credit history through loans or mortgages, which can ultimately save money.


key: _node_content
string_value:
{"id_": "1c705ab5-586a-4e93-894a-08d88621a223", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6107c28d-3773-4eb6-9b87-d5e7df4a6a1a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "03f90a464312e28a8b4ffd2e8ffce976a3c1a333a4a9cd77eca45193480212fb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "eefeccd3-4aee-4798-84b8-8474d04a1dd3", "node_type": "1", "metadata": {}, "hash": "df88a50aa3030a45df8c7672208a2e90a851eddb94658dc3f7cd005304f8e864", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 35875, "end_char_idx": 39132, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
I should look into this further to provide a more comprehensive answer.
    Action: Search
    Action Input: "Alternative financial services for those without a bank account"[0m
    
    Observation:[36;1m[1;3mInstead, people who are unbanked use alternative financial services—payday loans, money orders, check cashing services, pawnshop loans, and the like—to meet their banking needs. These financial services fill an important role for unbanked individuals, but they can also cause further financial hardship.[0m[32;1m[1;3mIt's important to note that while there are alternatives to having a bank account, relying solely on these services can lead to financial difficulties. I should provide some information on the potential drawbacks of not having a bank account.
    Action: Knowledge Base
    Action Input: "What are the drawbacks of not having a bank account?"[0m
    
    Observation:[33;1m[1;3m Not having a bank account can be dangerous, as the cash has to be stored somewhere and can be exposed to anyone who finds it or comes into the house with a gun to get it. Additionally, not having a bank account means not being exposed to other things that can help, such as building credit history through loans or mortgages, which can ultimately save money. Finally, not having a bank account can be a disproportionate burden on the poor, as overdraft fees can be expensive.[0m[32;1m[1;3mIt's important to provide some resources for those who may be interested in learning more about alternative financial services or how to open a bank account. 
    Action: Knowledge Base
    Action Input: "Resources for alternative financial services or opening a bank account"[0m
    
    Observation:[33;1m[1;3m There are a few resources available for alternative financial services or opening a bank account. Prepaid credit cards are becoming more popular and can be found at convenience stores. Capital One offers commercial solutions and a comprehensive suite of financial services tailored to short and long term goals. Airbnb also offers Air Cover for Hosts, which provides $1 million in damage protection, pet damage protection, and income loss protection.[0m[32;1m[1;3mIt's important to note that while prepaid credit cards and alternative financial services can be helpful, they may not offer the same level of protection and benefits as a traditional bank account. It's also important to do research and compare options before making a decision. 
    Final Answer: While it is possible to live without a bank account by using alternative financial services, it may come with potential drawbacks and limitations. It's important to do research and compare options before making a decision, and there are resources available for those who may be interested in opening a bank account or exploring alternative financial services.[0m
    
    [1m> Finished chain.[0m





    "While it is possible to live without a bank account by using alternative financial services, it may come with potential drawbacks and limitations. It's important to do research and compare options before making a decision, and there are resources available for those who may be interested in opening a bank account or exploring alternative financial services."


key: _node_content
string_value:
{"id_": "eefeccd3-4aee-4798-84b8-8474d04a1dd3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1c705ab5-586a-4e93-894a-08d88621a223", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "cda52334f67825acf10eb4e0fdf530b35d7fe7a4fd6a2a195eec133c0419fbb1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "062e85c4-1529-465c-83f4-59a68c6e249b", "node_type": "1", "metadata": {}, "hash": "b00e1f8a7f410aec0e0be227d1315a814181f8f825f0103aa0a1889aced27f09", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 37149, "end_char_idx": 40870, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Finally, not having a bank account can be a disproportionate burden on the poor, as overdraft fees can be expensive.[0m[32;1m[1;3mIt's important to provide some resources for those who may be interested in learning more about alternative financial services or how to open a bank account. 
    Action: Knowledge Base
    Action Input: "Resources for alternative financial services or opening a bank account"[0m
    
    Observation:[33;1m[1;3m There are a few resources available for alternative financial services or opening a bank account. Prepaid credit cards are becoming more popular and can be found at convenience stores. Capital One offers commercial solutions and a comprehensive suite of financial services tailored to short and long term goals. Airbnb also offers Air Cover for Hosts, which provides $1 million in damage protection, pet damage protection, and income loss protection.[0m[32;1m[1;3mIt's important to note that while prepaid credit cards and alternative financial services can be helpful, they may not offer the same level of protection and benefits as a traditional bank account. It's also important to do research and compare options before making a decision. 
    Final Answer: While it is possible to live without a bank account by using alternative financial services, it may come with potential drawbacks and limitations. It's important to do research and compare options before making a decision, and there are resources available for those who may be interested in opening a bank account or exploring alternative financial services.[0m
    
    [1m> Finished chain.[0m





    "While it is possible to live without a bank account by using alternative financial services, it may come with potential drawbacks and limitations. It's important to do research and compare options before making a decision, and there are resources available for those who may be interested in opening a bank account or exploring alternative financial services."




```python
multi_tool_executor.run('Can you tell me some interesting facts about whether zoos are good or bad for animals')
```

    
    
    [1m> Entering new AgentExecutor chain...[0m
    [32;1m[1;3mThought: This is a complex topic that requires a balanced perspective
    Action: Knowledge Base
    Action Input: "What are the arguments for and against zoos?"[0m
    
    Observation:[33;1m[1;3m The arguments for zoos include that they have gotten a lot better in the last 30-40 years, they participate in research and conservation projects, and they can help save species from extinction. The arguments against zoos include that they are still businesses, they can be counterproductive in terms of educating the public, and they can have a negative impact on the life span of animals in captivity.[0m[32;1m[1;3mIt's important to consider both sides of the argument before coming to a conclusion
    Action: Search
    Action Input: "What are some examples of successful zoo conservation projects?"[0m
    
    Observation:[36;1m[1;3mThere are dedicated species survival programs which have helped species come out from the brink of extinction, good examples of that being the black-footed ferrets, the red wolves, the Przewalski's wild horse, and the California condors.[0m[32;1m[1;3mWhile there are valid arguments on both sides, it seems that zoos can have a positive impact on conservation efforts for endangered species.
    Final Answer: Zoos can have both positive and negative effects on animals, but they can play a role in conservation efforts for endangered species. It's important to consider both sides of the argument and do research before forming an opinion.[0m
    
    [1m> Finished chain.[0m


key: _node_content
string_value:
{"id_": "062e85c4-1529-465c-83f4-59a68c6e249b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a3c7eeaa-5a63-48db-a7d2-497b55dd4a0e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "525158bc025ac66a8380eef18361f30e4fc247905b6ccdcd90aede2dd61cda8d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "eefeccd3-4aee-4798-84b8-8474d04a1dd3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "81c34a5dfdf1061c93400119e42a74b685ee39c86410dd29ffab3eab76d29713", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 40880, "end_char_idx": 41470, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"Zoos can have both positive and negative effects on animals, but they can play a role in conservation efforts for endangered species. It's important to consider both sides of the argument and do research before forming an opinion."



You now have a template to deploy conversational agents with tools. If you want to extend this with a Custom Agent to add your own retry behaviour or treatment of input/output variables, then follow [this article](https://python.langchain.com/en/latest/modules/agents/agents/custom_agent.html).

We look forward to seeing what you build!


```python

```


key: _node_content
string_value:
{"id_": "78a6cd18-b603-46b6-8f75-0f42f4b746b2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb", "title": "Embedding texts that are longer than the model's maximum context length", "tokens": 1474, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "19c5aa20-02f8-484e-bac3-e55a706996cb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb", "title": "Embedding texts that are longer than the model's maximum context length", "tokens": 1474, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "99882b65b5b09072c5526edea134b62b7a8b70cafeb44f4e50de45f4f5d6112b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4ce9e70c-0c9d-44e6-a662-cd7a4e4c8091", "node_type": "1", "metadata": {}, "hash": "090c8a15bb6567b0e145953d5f6c86a3e6efae9385a51a1cdcf228d2482edacd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3065, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Embedding texts that are longer than the model's maximum context length

OpenAI's embedding models cannot embed text that exceeds a maximum length. The maximum length varies by model, and is measured by _tokens_, not string length. If you are unfamiliar with tokenization, check out [How to count tokens with tiktoken](How_to_count_tokens_with_tiktoken.ipynb).

This notebook shows how to handle texts that are longer than a model's maximum context length. We'll demonstrate using embeddings from `text-embedding-3-small`, but the same ideas can be applied to other models and tasks. To learn more about embeddings, check out the [OpenAI Embeddings Guide](https://beta.openai.com/docs/guides/embeddings).


## 1. Model context length

First, we select the model and define a function to get embeddings from the API.


```python
from openai import OpenAI
import os
import openai
from tenacity import retry, wait_random_exponential, stop_after_attempt, retry_if_not_exception_type

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))

EMBEDDING_MODEL = 'text-embedding-3-small'
EMBEDDING_CTX_LENGTH = 8191
EMBEDDING_ENCODING = 'cl100k_base'

# let's make sure to not retry on an invalid request, because that is what we want to demonstrate
@retry(wait=wait_random_exponential(min=1, max=20), stop=stop_after_attempt(6), retry=retry_if_not_exception_type(openai.BadRequestError))
def get_embedding(text_or_tokens, model=EMBEDDING_MODEL):
    return client.embeddings.create(input=text_or_tokens, model=model).data[0].embedding
```

The `text-embedding-3-small` model has a context length of 8191 tokens with the `cl100k_base` encoding, and we can see that going over that limit causes an error.


```python
long_text = 'AGI ' * 5000
try:
    get_embedding(long_text)
except openai.BadRequestError as e:
    print(e)
```

    Error code: 400 - {'error': {'message': "This model's maximum context length is 8192 tokens, however you requested 10001 tokens (10001 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.", 'type': 'invalid_request_error', 'param': None, 'code': None}}


Clearly we want to avoid these errors, particularly when handling programmatically with a large number of embeddings. Yet, we still might be faced with texts that are longer than the maximum context length. Below we describe and provide recipes for the main approaches to handling these longer texts: (1) simply truncating the text to the maximum allowed length, and (2) chunking the text and embedding each chunk individually.

## 1. Truncating the input text

The simplest solution is to truncate the input text to the maximum allowed length. Because the context length is measured in tokens, we have to first tokenize the text before truncating it. The API accepts inputs both in the form of text or tokens, so as long as you are careful that you are using the appropriate encoding, there is no need to convert the tokens back into string form. Below is an example of such a truncation function.


key: _node_content
string_value:
{"id_": "4ce9e70c-0c9d-44e6-a662-cd7a4e4c8091", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb", "title": "Embedding texts that are longer than the model's maximum context length", "tokens": 1474, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "19c5aa20-02f8-484e-bac3-e55a706996cb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb", "title": "Embedding texts that are longer than the model's maximum context length", "tokens": 1474, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "99882b65b5b09072c5526edea134b62b7a8b70cafeb44f4e50de45f4f5d6112b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "78a6cd18-b603-46b6-8f75-0f42f4b746b2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb", "title": "Embedding texts that are longer than the model's maximum context length", "tokens": 1474, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e31f8190e60166d6e11390c70a316532b0f0b181a6df4930b1db8c79c5795c00", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1f21c57a-b558-4416-8989-aa9a6ea88946", "node_type": "1", "metadata": {}, "hash": "de80614b0c643df13c462169da1ca86aeab029ddab624490443291a20c5f9a9b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1749, "end_char_idx": 5132, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
long_text = 'AGI ' * 5000
try:
    get_embedding(long_text)
except openai.BadRequestError as e:
    print(e)
```

    Error code: 400 - {'error': {'message': "This model's maximum context length is 8192 tokens, however you requested 10001 tokens (10001 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.", 'type': 'invalid_request_error', 'param': None, 'code': None}}


Clearly we want to avoid these errors, particularly when handling programmatically with a large number of embeddings. Yet, we still might be faced with texts that are longer than the maximum context length. Below we describe and provide recipes for the main approaches to handling these longer texts: (1) simply truncating the text to the maximum allowed length, and (2) chunking the text and embedding each chunk individually.

## 1. Truncating the input text

The simplest solution is to truncate the input text to the maximum allowed length. Because the context length is measured in tokens, we have to first tokenize the text before truncating it. The API accepts inputs both in the form of text or tokens, so as long as you are careful that you are using the appropriate encoding, there is no need to convert the tokens back into string form. Below is an example of such a truncation function.


```python
import tiktoken

def truncate_text_tokens(text, encoding_name=EMBEDDING_ENCODING, max_tokens=EMBEDDING_CTX_LENGTH):
    """Truncate a string to have `max_tokens` according to the given encoding."""
    encoding = tiktoken.get_encoding(encoding_name)
    return encoding.encode(text)[:max_tokens]
```

Our example from before now works without error.


```python
truncated = truncate_text_tokens(long_text)
len(get_embedding(truncated))
```




    1536



## 2. Chunking the input text

Though truncation works, discarding potentially relevant text is a clear drawback. Another approach is to divide the input text into chunks and then embed each chunk individually. Then, we can either use the chunk embeddings separately, or combine them in some way, such as averaging (weighted by the size of each chunk).

We will take a function from [Python's own cookbook](https://docs.python.org/3/library/itertools.html#itertools-recipes) that breaks up a sequence into chunks.


```python
from itertools import islice

def batched(iterable, n):
    """Batch data into tuples of length n. The last batch may be shorter."""
    # batched('ABCDEFG', 3) --> ABC DEF G
    if n < 1:
        raise ValueError('n must be at least one')
    it = iter(iterable)
    while (batch := tuple(islice(it, n))):
        yield batch
```

Now we define a function that encodes a string into tokens and then breaks it up into chunks.


```python
def chunked_tokens(text, encoding_name, chunk_length):
    encoding = tiktoken.get_encoding(encoding_name)
    tokens = encoding.encode(text)
    chunks_iterator = batched(tokens, chunk_length)
    yield from chunks_iterator
```

Finally, we can write a function that safely handles embedding requests, even when the input text is longer than the maximum context length, by chunking the input tokens and embedding each chunk individually. The `average` flag can be set to `True` to return the weighted average of the chunk embeddings, or `False` to simply return the unmodified list of chunk embeddings.


```python
import numpy as np


key: _node_content
string_value:
{"id_": "1f21c57a-b558-4416-8989-aa9a6ea88946", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb", "title": "Embedding texts that are longer than the model's maximum context length", "tokens": 1474, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "19c5aa20-02f8-484e-bac3-e55a706996cb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb", "title": "Embedding texts that are longer than the model's maximum context length", "tokens": 1474, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "99882b65b5b09072c5526edea134b62b7a8b70cafeb44f4e50de45f4f5d6112b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4ce9e70c-0c9d-44e6-a662-cd7a4e4c8091", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb", "title": "Embedding texts that are longer than the model's maximum context length", "tokens": 1474, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d20efe88cb3ac782fff136879b5782ef5756adc7524499ce751456eb4ec4f0a4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3430, "end_char_idx": 6595, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
truncated = truncate_text_tokens(long_text)
len(get_embedding(truncated))
```




    1536



## 2. Chunking the input text

Though truncation works, discarding potentially relevant text is a clear drawback. Another approach is to divide the input text into chunks and then embed each chunk individually. Then, we can either use the chunk embeddings separately, or combine them in some way, such as averaging (weighted by the size of each chunk).

We will take a function from [Python's own cookbook](https://docs.python.org/3/library/itertools.html#itertools-recipes) that breaks up a sequence into chunks.


```python
from itertools import islice

def batched(iterable, n):
    """Batch data into tuples of length n. The last batch may be shorter."""
    # batched('ABCDEFG', 3) --> ABC DEF G
    if n < 1:
        raise ValueError('n must be at least one')
    it = iter(iterable)
    while (batch := tuple(islice(it, n))):
        yield batch
```

Now we define a function that encodes a string into tokens and then breaks it up into chunks.


```python
def chunked_tokens(text, encoding_name, chunk_length):
    encoding = tiktoken.get_encoding(encoding_name)
    tokens = encoding.encode(text)
    chunks_iterator = batched(tokens, chunk_length)
    yield from chunks_iterator
```

Finally, we can write a function that safely handles embedding requests, even when the input text is longer than the maximum context length, by chunking the input tokens and embedding each chunk individually. The `average` flag can be set to `True` to return the weighted average of the chunk embeddings, or `False` to simply return the unmodified list of chunk embeddings.


```python
import numpy as np


def len_safe_get_embedding(text, model=EMBEDDING_MODEL, max_tokens=EMBEDDING_CTX_LENGTH, encoding_name=EMBEDDING_ENCODING, average=True):
    chunk_embeddings = []
    chunk_lens = []
    for chunk in chunked_tokens(text, encoding_name=encoding_name, chunk_length=max_tokens):
        chunk_embeddings.append(get_embedding(chunk, model=model))
        chunk_lens.append(len(chunk))

    if average:
        chunk_embeddings = np.average(chunk_embeddings, axis=0, weights=chunk_lens)
        chunk_embeddings = chunk_embeddings / np.linalg.norm(chunk_embeddings)  # normalizes length to 1
        chunk_embeddings = chunk_embeddings.tolist()
    return chunk_embeddings
```

Once again, we can now handle long input texts.


```python
average_embedding_vector = len_safe_get_embedding(long_text, average=True)
chunks_embedding_vectors = len_safe_get_embedding(long_text, average=False)

print(f"Setting average=True gives us a single {len(average_embedding_vector)}-dimensional embedding vector for our long text.")
print(f"Setting average=False gives us {len(chunks_embedding_vectors)} embedding vectors, one for each of the chunks.")

```

    Setting average=True gives us a single 1536-dimensional embedding vector for our long text.
    Setting average=False gives us 2 embedding vectors, one for each of the chunks.


In some cases, it may make sense to split chunks on paragraph boundaries or sentence boundaries to help preserve the meaning of the text.


key: _node_content
string_value:
{"id_": "e331bb7a-6d7f-4349-811d-d9cbf3449891", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "04415a1c-4881-4738-aa11-a94ead8b6217", "node_type": "1", "metadata": {}, "hash": "eb8cfe9991ab81df4370da4d754f071a628627c2cb5102099bbc10871f3f8a63", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2533, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Summarizing Long Documents

The objective of this notebook is to demonstrate how to summarize large documents with a controllable level of detail.
 
If you give a GPT model the task of summarizing a long document (e.g. 10k or more tokens), you'll tend to get back a relatively short summary that isn't proportional to the length of the document. For instance, a summary of a 20k token document will not be twice as long as a summary of a 10k token document. One way we can fix this is to split our document up into pieces, and produce a summary piecewise. After many queries to a GPT model, the full summary can be reconstructed. By controlling the number of text chunks and their sizes, we can ultimately control the level of detail in the output.


```python
import os
from typing import List, Tuple, Optional
from openai import OpenAI
import tiktoken
from tqdm import tqdm
```


```python
# open dataset containing part of the text of the Wikipedia page for the United States
with open("data/artificial_intelligence_wikipedia.txt", "r") as file:
    artificial_intelligence_wikipedia_text = file.read()
```


```python
# load encoding and check the length of dataset
encoding = tiktoken.encoding_for_model('gpt-4-turbo')
len(encoding.encode(artificial_intelligence_wikipedia_text))
```




    14630



We'll define a simple utility to wrap calls to the OpenAI API.


```python
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def get_chat_completion(messages, model='gpt-4-turbo'):
    response = client.chat.completions.create(
        model=model,
        messages=messages,
        temperature=0,
    )
    return response.choices[0].message.content
```

Next we'll define some utilities to chunk a large document into smaller pieces.


```python
def tokenize(text: str) -> List[str]:
    encoding = tiktoken.encoding_for_model('gpt-4-turbo')
    return encoding.encode(text)


# This function chunks a text into smaller pieces based on a maximum token count and a delimiter.
def chunk_on_delimiter(input_string: str,
                       max_tokens: int, delimiter: str) -> List[str]:
    chunks = input_string.split(delimiter)
    combined_chunks, _, dropped_chunk_count = combine_chunks_with_no_minimum(
        chunks, max_tokens, chunk_delimiter=delimiter, add_ellipsis_for_overflow=True
    )
    if dropped_chunk_count > 0:
        print(f"warning: {dropped_chunk_count} chunks were dropped due to overflow")
    combined_chunks = [f"{chunk}{delimiter}" for chunk in combined_chunks]
    return combined_chunks


key: _node_content
string_value:
{"id_": "04415a1c-4881-4738-aa11-a94ead8b6217", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e331bb7a-6d7f-4349-811d-d9cbf3449891", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "a080324b075109cf3df5514cfcff4084bf36ea45f42a7703ae513a9ab57bbdaa", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7b915b4a-3687-4141-9aef-08a052a8857c", "node_type": "1", "metadata": {}, "hash": "530d5258402ea19b0dd2cb1e2e3ed5ec1fffbeddd6ba8a9695fef733fdbb0487", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 884, "end_char_idx": 5207, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# open dataset containing part of the text of the Wikipedia page for the United States
with open("data/artificial_intelligence_wikipedia.txt", "r") as file:
    artificial_intelligence_wikipedia_text = file.read()
```


```python
# load encoding and check the length of dataset
encoding = tiktoken.encoding_for_model('gpt-4-turbo')
len(encoding.encode(artificial_intelligence_wikipedia_text))
```




    14630



We'll define a simple utility to wrap calls to the OpenAI API.


```python
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def get_chat_completion(messages, model='gpt-4-turbo'):
    response = client.chat.completions.create(
        model=model,
        messages=messages,
        temperature=0,
    )
    return response.choices[0].message.content
```

Next we'll define some utilities to chunk a large document into smaller pieces.


```python
def tokenize(text: str) -> List[str]:
    encoding = tiktoken.encoding_for_model('gpt-4-turbo')
    return encoding.encode(text)


# This function chunks a text into smaller pieces based on a maximum token count and a delimiter.
def chunk_on_delimiter(input_string: str,
                       max_tokens: int, delimiter: str) -> List[str]:
    chunks = input_string.split(delimiter)
    combined_chunks, _, dropped_chunk_count = combine_chunks_with_no_minimum(
        chunks, max_tokens, chunk_delimiter=delimiter, add_ellipsis_for_overflow=True
    )
    if dropped_chunk_count > 0:
        print(f"warning: {dropped_chunk_count} chunks were dropped due to overflow")
    combined_chunks = [f"{chunk}{delimiter}" for chunk in combined_chunks]
    return combined_chunks


# This function combines text chunks into larger blocks without exceeding a specified token count. It returns the combined text blocks, their original indices, and the count of chunks dropped due to overflow.
def combine_chunks_with_no_minimum(
        chunks: List[str],
        max_tokens: int,
        chunk_delimiter="\n\n",
        header: Optional[str] = None,
        add_ellipsis_for_overflow=False,
) -> Tuple[List[str], List[int]]:
    dropped_chunk_count = 0
    output = []  # list to hold the final combined chunks
    output_indices = []  # list to hold the indices of the final combined chunks
    candidate = (
        [] if header is None else [header]
    )  # list to hold the current combined chunk candidate
    candidate_indices = []
    for chunk_i, chunk in enumerate(chunks):
        chunk_with_header = [chunk] if header is None else [header, chunk]
        if len(tokenize(chunk_delimiter.join(chunk_with_header))) > max_tokens:
            print(f"warning: chunk overflow")
            if (
                    add_ellipsis_for_overflow
                    and len(tokenize(chunk_delimiter.join(candidate + ["..."]))) <= max_tokens
            ):
                candidate.append("...")
                dropped_chunk_count += 1
            continue  # this case would break downstream assumptions
        # estimate token count with the current chunk added
        extended_candidate_token_count = len(tokenize(chunk_delimiter.join(candidate + [chunk])))
        # If the token count exceeds max_tokens, add the current candidate to output and start a new candidate
        if extended_candidate_token_count > max_tokens:
            output.append(chunk_delimiter.join(candidate))
            output_indices.append(candidate_indices)
            candidate = chunk_with_header  # re-initialize candidate
            candidate_indices = [chunk_i]
        # otherwise keep extending the candidate
        else:
            candidate.append(chunk)
            candidate_indices.append(chunk_i)
    # add the remaining candidate to output if it's not empty
    if (header is not None and len(candidate) > 1) or (header is None and len(candidate) > 0):
        output.append(chunk_delimiter.join(candidate))
        output_indices.append(candidate_indices)
    return output, output_indices, dropped_chunk_count
```

Now we can define a utility to summarize text with a controllable level of detail (note the `detail` parameter).

The function first determines the number of chunks by interpolating between a minimum and a maximum chunk count based on a controllable `detail` parameter. It then splits the text into chunks and summarizes each chunk.


key: _node_content
string_value:
{"id_": "7b915b4a-3687-4141-9aef-08a052a8857c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "04415a1c-4881-4738-aa11-a94ead8b6217", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "2a7821aa039b27f57dce0dec91d94f1927d384d38fa4f067f8205c40eec794fc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8ef89ce9-578a-4d2f-8195-4690e41644dc", "node_type": "1", "metadata": {}, "hash": "3b0b2d19746c9828ea854df7a96485e12429bff1f0c98a2d7f011f348428a8e2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5210, "end_char_idx": 8032, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def summarize(text: str,
              detail: float = 0,
              model: str = 'gpt-4-turbo',
              additional_instructions: Optional[str] = None,
              minimum_chunk_size: Optional[int] = 500,
              chunk_delimiter: str = ".",
              summarize_recursively=False,
              verbose=False):
    """
    Summarizes a given text by splitting it into chunks, each of which is summarized individually. 
    The level of detail in the summary can be adjusted, and the process can optionally be made recursive.

    Parameters:
    - text (str): The text to be summarized.
    - detail (float, optional): A value between 0 and 1 indicating the desired level of detail in the summary.
      0 leads to a higher level summary, and 1 results in a more detailed summary. Defaults to 0.
    - model (str, optional): The model to use for generating summaries. Defaults to 'gpt-3.5-turbo'.
    - additional_instructions (Optional[str], optional): Additional instructions to provide to the model for customizing summaries.
    - minimum_chunk_size (Optional[int], optional): The minimum size for text chunks. Defaults to 500.
    - chunk_delimiter (str, optional): The delimiter used to split the text into chunks. Defaults to ".".
    - summarize_recursively (bool, optional): If True, summaries are generated recursively, using previous summaries for context.
    - verbose (bool, optional): If True, prints detailed information about the chunking process.

    Returns:
    - str: The final compiled summary of the text.

    The function first determines the number of chunks by interpolating between a minimum and a maximum chunk count based on the `detail` parameter. 
    It then splits the text into chunks and summarizes each chunk. If `summarize_recursively` is True, each summary is based on the previous summaries, 
    adding more context to the summarization process. The function returns a compiled summary of all chunks.
    """

    # check detail is set correctly
    assert 0 <= detail <= 1

    # interpolate the number of chunks based to get specified level of detail
    max_chunks = len(chunk_on_delimiter(text, minimum_chunk_size, chunk_delimiter))
    min_chunks = 1
    num_chunks = int(min_chunks + detail * (max_chunks - min_chunks))

    # adjust chunk_size based on interpolated number of chunks
    document_length = len(tokenize(text))
    chunk_size = max(minimum_chunk_size, document_length // num_chunks)
    text_chunks = chunk_on_delimiter(text, chunk_size, chunk_delimiter)
    if verbose:
        print(f"Splitting the text into {len(text_chunks)} chunks to be summarized.")
        print(f"Chunk lengths are {[len(tokenize(x)) for x in text_chunks]}")

    # set system message
    system_message_content = "Rewrite this text in summarized form."


key: _node_content
string_value:
{"id_": "8ef89ce9-578a-4d2f-8195-4690e41644dc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7b915b4a-3687-4141-9aef-08a052a8857c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4b5cbd081b9e40a19f9865d380f03cff6b74b2353ab7b00adc8f3e53a835f85b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a47eeb53-6747-4a0e-8a4b-a2bdca928351", "node_type": "1", "metadata": {}, "hash": "378f3360a653a69e4e04a78c726e171eab26bdb4a6557620254c821d5642ced0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6273, "end_char_idx": 9895, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- minimum_chunk_size (Optional[int], optional): The minimum size for text chunks. Defaults to 500.
    - chunk_delimiter (str, optional): The delimiter used to split the text into chunks. Defaults to ".".
    - summarize_recursively (bool, optional): If True, summaries are generated recursively, using previous summaries for context.
    - verbose (bool, optional): If True, prints detailed information about the chunking process.

    Returns:
    - str: The final compiled summary of the text.

    The function first determines the number of chunks by interpolating between a minimum and a maximum chunk count based on the `detail` parameter. 
    It then splits the text into chunks and summarizes each chunk. If `summarize_recursively` is True, each summary is based on the previous summaries, 
    adding more context to the summarization process. The function returns a compiled summary of all chunks.
    """

    # check detail is set correctly
    assert 0 <= detail <= 1

    # interpolate the number of chunks based to get specified level of detail
    max_chunks = len(chunk_on_delimiter(text, minimum_chunk_size, chunk_delimiter))
    min_chunks = 1
    num_chunks = int(min_chunks + detail * (max_chunks - min_chunks))

    # adjust chunk_size based on interpolated number of chunks
    document_length = len(tokenize(text))
    chunk_size = max(minimum_chunk_size, document_length // num_chunks)
    text_chunks = chunk_on_delimiter(text, chunk_size, chunk_delimiter)
    if verbose:
        print(f"Splitting the text into {len(text_chunks)} chunks to be summarized.")
        print(f"Chunk lengths are {[len(tokenize(x)) for x in text_chunks]}")

    # set system message
    system_message_content = "Rewrite this text in summarized form."
    if additional_instructions is not None:
        system_message_content += f"\n\n{additional_instructions}"

    accumulated_summaries = []
    for chunk in tqdm(text_chunks):
        if summarize_recursively and accumulated_summaries:
            # Creating a structured prompt for recursive summarization
            accumulated_summaries_string = '\n\n'.join(accumulated_summaries)
            user_message_content = f"Previous summaries:\n\n{accumulated_summaries_string}\n\nText to summarize next:\n\n{chunk}"
        else:
            # Directly passing the chunk for summarization without recursive context
            user_message_content = chunk

        # Constructing messages based on whether recursive summarization is applied
        messages = [
            {"role": "system", "content": system_message_content},
            {"role": "user", "content": user_message_content}
        ]

        # Assuming this function gets the completion and works as expected
        response = get_chat_completion(messages, model=model)
        accumulated_summaries.append(response)

    # Compile final summary from partial summaries
    final_summary = '\n\n'.join(accumulated_summaries)

    return final_summary
```

Now we can use this utility to produce summaries with varying levels of detail. By increasing `detail` from 0 to 1 we get progressively longer summaries of the underlying document. A higher value for the `detail` parameter results in a more detailed summary because the utility first splits the document into a greater number of chunks. Each chunk is then summarized, and the final summary is a concatenation of all the chunk summaries.


```python
summary_with_detail_0 = summarize(artificial_intelligence_wikipedia_text, detail=0, verbose=True)
```

    Splitting the text into 1 chunks to be summarized.
    Chunk lengths are [14631]


key: _node_content
string_value:
{"id_": "a47eeb53-6747-4a0e-8a4b-a2bdca928351", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8ef89ce9-578a-4d2f-8195-4690e41644dc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8ba713e0f62ef805d307226bc3460ca8d011e56b491e75f003a6edda66874717", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2fab2cde-2515-46ac-ad29-c5b408c52b6a", "node_type": "1", "metadata": {}, "hash": "908d2655849b7cddf0fd54145785f377d022fe614ffb8000b0323ab4a3dd4966", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8037, "end_char_idx": 10590, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
if additional_instructions is not None:
        system_message_content += f"\n\n{additional_instructions}"

    accumulated_summaries = []
    for chunk in tqdm(text_chunks):
        if summarize_recursively and accumulated_summaries:
            # Creating a structured prompt for recursive summarization
            accumulated_summaries_string = '\n\n'.join(accumulated_summaries)
            user_message_content = f"Previous summaries:\n\n{accumulated_summaries_string}\n\nText to summarize next:\n\n{chunk}"
        else:
            # Directly passing the chunk for summarization without recursive context
            user_message_content = chunk

        # Constructing messages based on whether recursive summarization is applied
        messages = [
            {"role": "system", "content": system_message_content},
            {"role": "user", "content": user_message_content}
        ]

        # Assuming this function gets the completion and works as expected
        response = get_chat_completion(messages, model=model)
        accumulated_summaries.append(response)

    # Compile final summary from partial summaries
    final_summary = '\n\n'.join(accumulated_summaries)

    return final_summary
```

Now we can use this utility to produce summaries with varying levels of detail. By increasing `detail` from 0 to 1 we get progressively longer summaries of the underlying document. A higher value for the `detail` parameter results in a more detailed summary because the utility first splits the document into a greater number of chunks. Each chunk is then summarized, and the final summary is a concatenation of all the chunk summaries.


```python
summary_with_detail_0 = summarize(artificial_intelligence_wikipedia_text, detail=0, verbose=True)
```

    Splitting the text into 1 chunks to be summarized.
    Chunk lengths are [14631]


    100%|██████████| 1/1 [00:09<00:00,  9.68s/it]



```python
summary_with_detail_pt25 = summarize(artificial_intelligence_wikipedia_text, detail=0.25, verbose=True)
```

    Splitting the text into 9 chunks to be summarized.
    Chunk lengths are [1817, 1807, 1823, 1810, 1806, 1827, 1814, 1829, 103]


    100%|██████████| 9/9 [01:33<00:00, 10.39s/it]



```python
summary_with_detail_pt5 = summarize(artificial_intelligence_wikipedia_text, detail=0.5, verbose=True)
```

    Splitting the text into 17 chunks to be summarized.
    Chunk lengths are [897, 890, 914, 876, 893, 906, 893, 902, 909, 907, 905, 889, 902, 890, 901, 880, 287]


    100%|██████████| 17/17 [02:26<00:00,  8.64s/it]


key: _node_content
string_value:
{"id_": "2fab2cde-2515-46ac-ad29-c5b408c52b6a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a47eeb53-6747-4a0e-8a4b-a2bdca928351", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "420d01d8a01fa0c3a12939668bc012fcd9cca737a5e9165433a627005ed9cdfb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f0665041-96dc-4ddc-be7b-cc42a8a67a17", "node_type": "1", "metadata": {}, "hash": "4731f1bef86681130c29ed3d6986c127b10be23a49b5f4056c5089294c13e07d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9339, "end_char_idx": 11499, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
By increasing `detail` from 0 to 1 we get progressively longer summaries of the underlying document. A higher value for the `detail` parameter results in a more detailed summary because the utility first splits the document into a greater number of chunks. Each chunk is then summarized, and the final summary is a concatenation of all the chunk summaries.


```python
summary_with_detail_0 = summarize(artificial_intelligence_wikipedia_text, detail=0, verbose=True)
```

    Splitting the text into 1 chunks to be summarized.
    Chunk lengths are [14631]


    100%|██████████| 1/1 [00:09<00:00,  9.68s/it]



```python
summary_with_detail_pt25 = summarize(artificial_intelligence_wikipedia_text, detail=0.25, verbose=True)
```

    Splitting the text into 9 chunks to be summarized.
    Chunk lengths are [1817, 1807, 1823, 1810, 1806, 1827, 1814, 1829, 103]


    100%|██████████| 9/9 [01:33<00:00, 10.39s/it]



```python
summary_with_detail_pt5 = summarize(artificial_intelligence_wikipedia_text, detail=0.5, verbose=True)
```

    Splitting the text into 17 chunks to be summarized.
    Chunk lengths are [897, 890, 914, 876, 893, 906, 893, 902, 909, 907, 905, 889, 902, 890, 901, 880, 287]


    100%|██████████| 17/17 [02:26<00:00,  8.64s/it]



```python
summary_with_detail_1 = summarize(artificial_intelligence_wikipedia_text, detail=1, verbose=True)
```

    Splitting the text into 31 chunks to be summarized.
    Chunk lengths are [492, 427, 485, 490, 496, 478, 473, 497, 496, 501, 499, 497, 493, 470, 472, 494, 489, 492, 481, 485, 471, 500, 486, 498, 478, 469, 498, 468, 493, 478, 103]


    100%|██████████| 31/31 [04:08<00:00,  8.02s/it]


The original document is nearly 15k tokens long. Notice how large the gap is between the length of `summary_with_detail_0` and `summary_with_detail_1`. It's nearly 25 times longer!


```python
# lengths of summaries
[len(tokenize(x)) for x in
 [summary_with_detail_0, summary_with_detail_pt25, summary_with_detail_pt5, summary_with_detail_1]]
```




    [235, 2529, 4336, 6742]



Let's inspect the summaries to see how the level of detail changes when the `detail` parameter is increased from 0 to 1.


key: _node_content
string_value:
{"id_": "f0665041-96dc-4ddc-be7b-cc42a8a67a17", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2fab2cde-2515-46ac-ad29-c5b408c52b6a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4dccf5e0b00dba222b9836d80f72c90dcb2e826750e3c5d8e5dda2d8e4cf05d8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "66ae92f4-3af8-4095-83dc-70838673f972", "node_type": "1", "metadata": {}, "hash": "998d98a9b01514c8e84d02fc8620f30e0342a3a681bf5bf8279c2a27a8396869", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10543, "end_char_idx": 13699, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
100%|██████████| 17/17 [02:26<00:00,  8.64s/it]



```python
summary_with_detail_1 = summarize(artificial_intelligence_wikipedia_text, detail=1, verbose=True)
```

    Splitting the text into 31 chunks to be summarized.
    Chunk lengths are [492, 427, 485, 490, 496, 478, 473, 497, 496, 501, 499, 497, 493, 470, 472, 494, 489, 492, 481, 485, 471, 500, 486, 498, 478, 469, 498, 468, 493, 478, 103]


    100%|██████████| 31/31 [04:08<00:00,  8.02s/it]


The original document is nearly 15k tokens long. Notice how large the gap is between the length of `summary_with_detail_0` and `summary_with_detail_1`. It's nearly 25 times longer!


```python
# lengths of summaries
[len(tokenize(x)) for x in
 [summary_with_detail_0, summary_with_detail_pt25, summary_with_detail_pt5, summary_with_detail_1]]
```




    [235, 2529, 4336, 6742]



Let's inspect the summaries to see how the level of detail changes when the `detail` parameter is increased from 0 to 1.


```python
print(summary_with_detail_0)
```

    Artificial intelligence (AI) is the simulation of human intelligence in machines, designed to perform tasks that typically require human intelligence. This includes applications like advanced search engines, recommendation systems, speech interaction, autonomous vehicles, and more. AI was first significantly researched by Alan Turing and became an academic discipline in 1956. The field has experienced cycles of high expectations followed by disillusionment and reduced funding, known as "AI winters." Interest in AI surged post-2012 with advancements in deep learning and again post-2017 with the development of the transformer architecture, leading to a boom in AI research and applications in the early 2020s.
    
    AI's increasing integration into various sectors is influencing societal and economic shifts towards automation and data-driven decision-making, impacting areas such as employment, healthcare, and privacy. Ethical and safety concerns about AI have prompted discussions on regulatory policies.
    
    AI research involves various sub-fields focused on specific goals like reasoning, learning, and perception, using techniques from mathematics, logic, and other disciplines. Despite its broad applications, AI's complexity and potential risks, such as privacy issues, misinformation, and ethical challenges, remain areas of active investigation and debate.



```python
print(summary_with_detail_1)
```

    Artificial intelligence (AI) is the simulation of human intelligence in machines, designed to perceive their environment and make decisions to achieve specific goals. This technology is prevalent across various sectors including industry, government, and science, with applications ranging from web search engines and recommendation systems to autonomous vehicles and AI in gaming. Although AI has become a common feature in many tools and applications, it often goes unrecognized as AI when it becomes sufficiently integrated and widespread.
    
    The field of AI, which began as an academic discipline in 1956, has experienced several cycles of high expectations followed by disappointment, known as AI winters.


key: _node_content
string_value:
{"id_": "66ae92f4-3af8-4095-83dc-70838673f972", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f0665041-96dc-4ddc-be7b-cc42a8a67a17", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "7f3483b43de6a9b3ecdbb5f689742223ce811c6f7fdeeb2c02f9377125c3505a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "316fcdbb-9774-469d-bf11-6d54ea1092c9", "node_type": "1", "metadata": {}, "hash": "b10de7fa67b0fd56ee28c6a0d28ad4563374c164af648f70b609513c73e0db4c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11502, "end_char_idx": 15642, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
print(summary_with_detail_0)
```

    Artificial intelligence (AI) is the simulation of human intelligence in machines, designed to perform tasks that typically require human intelligence. This includes applications like advanced search engines, recommendation systems, speech interaction, autonomous vehicles, and more. AI was first significantly researched by Alan Turing and became an academic discipline in 1956. The field has experienced cycles of high expectations followed by disillusionment and reduced funding, known as "AI winters." Interest in AI surged post-2012 with advancements in deep learning and again post-2017 with the development of the transformer architecture, leading to a boom in AI research and applications in the early 2020s.
    
    AI's increasing integration into various sectors is influencing societal and economic shifts towards automation and data-driven decision-making, impacting areas such as employment, healthcare, and privacy. Ethical and safety concerns about AI have prompted discussions on regulatory policies.
    
    AI research involves various sub-fields focused on specific goals like reasoning, learning, and perception, using techniques from mathematics, logic, and other disciplines. Despite its broad applications, AI's complexity and potential risks, such as privacy issues, misinformation, and ethical challenges, remain areas of active investigation and debate.



```python
print(summary_with_detail_1)
```

    Artificial intelligence (AI) is the simulation of human intelligence in machines, designed to perceive their environment and make decisions to achieve specific goals. This technology is prevalent across various sectors including industry, government, and science, with applications ranging from web search engines and recommendation systems to autonomous vehicles and AI in gaming. Although AI has become a common feature in many tools and applications, it often goes unrecognized as AI when it becomes sufficiently integrated and widespread.
    
    The field of AI, which began as an academic discipline in 1956, has experienced several cycles of high expectations followed by disappointment, known as AI winters. Interest and funding in AI surged post-2012 with advancements in deep learning and again post-2017 with the development of transformer architecture, leading to a significant boom in AI research and applications in the early 2020s, primarily in the United States.
    
    The increasing integration of AI in the 21st century is driving a shift towards automation and data-driven decision-making across various sectors, influencing job markets, healthcare, and education, among others. This raises important questions about the ethical implications, long-term effects, and the need for regulatory policies to ensure the safety and benefits of AI technologies. AI research itself is diverse, focusing on goals like reasoning, learning, and perception, and involves various tools and methodologies to achieve these objectives.
    
    General intelligence, which involves performing any human task at least as well as a human, is a long-term goal in AI research. To achieve this, AI integrates various techniques from search and optimization, formal logic, neural networks, and statistics, to insights from psychology, linguistics, and neuroscience. AI research focuses on specific traits like reasoning and problem-solving, where early algorithms mimicked human step-by-step reasoning. However, these algorithms struggle with large, complex problems due to combinatorial explosion and are less efficient than human intuitive judgments. Knowledge representation is another critical area, using ontologies to structure domain-specific knowledge and relationships, aiding in intelligent querying, scene interpretation, and data mining among other applications.
    
    Knowledge bases must encapsulate a wide range of elements including objects, properties, categories, relations, events, states, time, causes, effects, and meta-knowledge. They also need to handle default reasoning, where certain assumptions are maintained unless contradicted.


key: _node_content
string_value:
{"id_": "316fcdbb-9774-469d-bf11-6d54ea1092c9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "66ae92f4-3af8-4095-83dc-70838673f972", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "e21cdd7a8a285e32be00fc02bcc5997ea6e66da59e1df2e41ac8b5c5f5222213", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a97e45f8-3426-42f1-9e37-64f87e3057f0", "node_type": "1", "metadata": {}, "hash": "f4d6b6a3dd94873d90ba96789cda548b19b210387d9b841aa31f54d36c63bff2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13535, "end_char_idx": 17775, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The field of AI, which began as an academic discipline in 1956, has experienced several cycles of high expectations followed by disappointment, known as AI winters. Interest and funding in AI surged post-2012 with advancements in deep learning and again post-2017 with the development of transformer architecture, leading to a significant boom in AI research and applications in the early 2020s, primarily in the United States.
    
    The increasing integration of AI in the 21st century is driving a shift towards automation and data-driven decision-making across various sectors, influencing job markets, healthcare, and education, among others. This raises important questions about the ethical implications, long-term effects, and the need for regulatory policies to ensure the safety and benefits of AI technologies. AI research itself is diverse, focusing on goals like reasoning, learning, and perception, and involves various tools and methodologies to achieve these objectives.
    
    General intelligence, which involves performing any human task at least as well as a human, is a long-term goal in AI research. To achieve this, AI integrates various techniques from search and optimization, formal logic, neural networks, and statistics, to insights from psychology, linguistics, and neuroscience. AI research focuses on specific traits like reasoning and problem-solving, where early algorithms mimicked human step-by-step reasoning. However, these algorithms struggle with large, complex problems due to combinatorial explosion and are less efficient than human intuitive judgments. Knowledge representation is another critical area, using ontologies to structure domain-specific knowledge and relationships, aiding in intelligent querying, scene interpretation, and data mining among other applications.
    
    Knowledge bases must encapsulate a wide range of elements including objects, properties, categories, relations, events, states, time, causes, effects, and meta-knowledge. They also need to handle default reasoning, where certain assumptions are maintained unless contradicted. Challenges in knowledge representation include the vast scope of commonsense knowledge and its often sub-symbolic, non-verbal nature, alongside the difficulty of acquiring this knowledge for AI use.
    
    In the realm of AI, an "agent" is defined as an entity that perceives its environment and acts towards achieving goals or fulfilling preferences. In automated planning, the agent pursues a specific goal, while in decision-making, it evaluates actions based on their expected utility to maximize preference satisfaction. Classical planning assumes agents have complete knowledge of action outcomes, but real-world scenarios often involve uncertainty about the situation and outcomes, requiring probabilistic decision-making. Additionally, agents may need to adapt or learn preferences, particularly in complex environments with multiple agents or human interactions.
    
    Information value theory helps assess the value of exploratory actions in situations with uncertain outcomes. A Markov decision process uses a transition model and a reward function to guide decisions, which can be determined through calculations, heuristics, or learning. Game theory analyzes the rational behavior of multiple interacting agents in decision-making scenarios involving others.
    
    Machine learning, integral to AI, involves programs that automatically improve task performance. It includes unsupervised learning, which identifies patterns in data without guidance, and supervised learning, which requires labeled data and includes classification and regression tasks. Reinforcement learning rewards or punishes agents to shape their responses, while transfer learning applies knowledge from one problem to another. Deep learning, a subset of machine learning, uses artificial neural networks inspired by biological processes.
    
    Computational learning theory evaluates learning algorithms based on computational and sample complexity, among other criteria. Natural language processing (NLP) enables programs to interact using human languages, tackling challenges like speech recognition, synthesis, translation, and more.


key: _node_content
string_value:
{"id_": "a97e45f8-3426-42f1-9e37-64f87e3057f0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "316fcdbb-9774-469d-bf11-6d54ea1092c9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "471668d818f4db193d71fff39182fa1766c16c6e876aca1debae1e3e9c286698", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0450bc52-2f37-4417-b5cf-a88f150ca175", "node_type": "1", "metadata": {}, "hash": "0242685a2ea8c8ab7697bf165aacee0e9bb71076b7a908e61d6b7d32d9dfb6e9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15537, "end_char_idx": 19778, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
They also need to handle default reasoning, where certain assumptions are maintained unless contradicted. Challenges in knowledge representation include the vast scope of commonsense knowledge and its often sub-symbolic, non-verbal nature, alongside the difficulty of acquiring this knowledge for AI use.
    
    In the realm of AI, an "agent" is defined as an entity that perceives its environment and acts towards achieving goals or fulfilling preferences. In automated planning, the agent pursues a specific goal, while in decision-making, it evaluates actions based on their expected utility to maximize preference satisfaction. Classical planning assumes agents have complete knowledge of action outcomes, but real-world scenarios often involve uncertainty about the situation and outcomes, requiring probabilistic decision-making. Additionally, agents may need to adapt or learn preferences, particularly in complex environments with multiple agents or human interactions.
    
    Information value theory helps assess the value of exploratory actions in situations with uncertain outcomes. A Markov decision process uses a transition model and a reward function to guide decisions, which can be determined through calculations, heuristics, or learning. Game theory analyzes the rational behavior of multiple interacting agents in decision-making scenarios involving others.
    
    Machine learning, integral to AI, involves programs that automatically improve task performance. It includes unsupervised learning, which identifies patterns in data without guidance, and supervised learning, which requires labeled data and includes classification and regression tasks. Reinforcement learning rewards or punishes agents to shape their responses, while transfer learning applies knowledge from one problem to another. Deep learning, a subset of machine learning, uses artificial neural networks inspired by biological processes.
    
    Computational learning theory evaluates learning algorithms based on computational and sample complexity, among other criteria. Natural language processing (NLP) enables programs to interact using human languages, tackling challenges like speech recognition, synthesis, translation, and more. Early NLP efforts, influenced by Chomsky's theories, faced limitations in handling ambiguous language outside of controlled environments.
    
    Margaret Masterman emphasized the importance of meaning over grammar in language understanding, advocating for the use of thesauri instead of dictionaries in computational linguistics. Modern NLP techniques include word embedding, transformers, and by 2023, GPT models capable of achieving human-level scores on various tests. Machine perception involves interpreting sensor data to understand the world, encompassing computer vision and speech recognition among other applications. Social intelligence in AI focuses on recognizing and simulating human emotions, with systems like Kismet and affective computing technologies that enhance human-computer interaction. However, these advancements may lead to overestimations of AI capabilities by users. AI also employs a variety of techniques including search and optimization, with methods like state space search to explore possible solutions to problems.
    
    Planning algorithms use means-ends analysis to navigate through trees of goals and subgoals to achieve a target goal. However, simple exhaustive searches are often inadequate for complex real-world problems due to the vast search space, making searches slow or incomplete. Heuristics are employed to prioritize more promising paths towards a goal. In adversarial contexts like chess or Go, search algorithms explore trees of possible moves to find a winning strategy.
    
    Local search methods, such as gradient descent, optimize numerical parameters to minimize a loss function, often used in training neural networks. Evolutionary computation, another local search technique, iteratively enhances solutions by mutating and recombining candidate solutions, selecting the most fit for survival. Distributed search processes utilize swarm intelligence, with particle swarm optimization and ant colony optimization being notable examples.


key: _node_content
string_value:
{"id_": "0450bc52-2f37-4417-b5cf-a88f150ca175", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a97e45f8-3426-42f1-9e37-64f87e3057f0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "54aa6fbec83ee4d15c5ff94aa9c207f89c98ada7bf82e60ce14d2bf0aafbf675", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b1e5a906-c515-415e-ac1d-b080b4b82648", "node_type": "1", "metadata": {}, "hash": "ab23f571cb31e4b952ac34a60d584b3fb9e6d62c72a118544289419bbcad9fc0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17611, "end_char_idx": 21774, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Natural language processing (NLP) enables programs to interact using human languages, tackling challenges like speech recognition, synthesis, translation, and more. Early NLP efforts, influenced by Chomsky's theories, faced limitations in handling ambiguous language outside of controlled environments.
    
    Margaret Masterman emphasized the importance of meaning over grammar in language understanding, advocating for the use of thesauri instead of dictionaries in computational linguistics. Modern NLP techniques include word embedding, transformers, and by 2023, GPT models capable of achieving human-level scores on various tests. Machine perception involves interpreting sensor data to understand the world, encompassing computer vision and speech recognition among other applications. Social intelligence in AI focuses on recognizing and simulating human emotions, with systems like Kismet and affective computing technologies that enhance human-computer interaction. However, these advancements may lead to overestimations of AI capabilities by users. AI also employs a variety of techniques including search and optimization, with methods like state space search to explore possible solutions to problems.
    
    Planning algorithms use means-ends analysis to navigate through trees of goals and subgoals to achieve a target goal. However, simple exhaustive searches are often inadequate for complex real-world problems due to the vast search space, making searches slow or incomplete. Heuristics are employed to prioritize more promising paths towards a goal. In adversarial contexts like chess or Go, search algorithms explore trees of possible moves to find a winning strategy.
    
    Local search methods, such as gradient descent, optimize numerical parameters to minimize a loss function, often used in training neural networks. Evolutionary computation, another local search technique, iteratively enhances solutions by mutating and recombining candidate solutions, selecting the most fit for survival. Distributed search processes utilize swarm intelligence, with particle swarm optimization and ant colony optimization being notable examples.
    
    In the realm of logic, formal logic serves for reasoning and knowledge representation, with two primary types: propositional logic, dealing with true or false statements, and predicate logic, which involves objects and their relationships. Deductive reasoning in logic involves deriving conclusions from assumed true premises.
    
    Proofs in logic can be organized into proof trees, where each node represents a sentence and is connected to its children by inference rules. Problem-solving involves finding a proof tree that starts with premises or axioms at the leaves and ends with the problem's solution at the root. In Horn clauses, one can reason forwards from premises or backwards from the problem, while in general first-order logic, resolution uses contradiction to solve problems. Despite being undecidable and intractable, backward reasoning with Horn clauses is Turing complete and efficient, similar to other symbolic programming languages like Prolog.
    
    Fuzzy logic allows for handling propositions with partial truth by assigning a truth degree between 0 and 1. Non-monotonic logics cater to default reasoning, and various specialized logics have been developed for complex domains.
    
    In AI, handling uncertain or incomplete information is crucial in fields like reasoning, planning, and perception. Tools from probability theory and economics, such as Bayesian networks, Markov decision processes, and game theory, help in making decisions and planning under uncertainty. Bayesian networks, in particular, are versatile tools used for reasoning, learning, planning, and perception through various algorithms.
    
    Probabilistic algorithms like hidden Markov models and Kalman filters are useful for analyzing data over time, aiding in tasks such as filtering, prediction, and smoothing. In machine learning, expectation-maximization clustering can effectively identify distinct patterns in data, as demonstrated with the Old Faithful eruption data.


key: _node_content
string_value:
{"id_": "b1e5a906-c515-415e-ac1d-b080b4b82648", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0450bc52-2f37-4417-b5cf-a88f150ca175", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "ce582ff42160f14dff35edea0f13b212e0e781128b19115fa7256396257e2d78", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "176b127e-4efb-4ffb-adaa-80eb9cbf768c", "node_type": "1", "metadata": {}, "hash": "4d1a7f143795073479fbed1c2faaac497692c027ab1a6b1fe5f5baa3b333bb4e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19637, "end_char_idx": 23695, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Distributed search processes utilize swarm intelligence, with particle swarm optimization and ant colony optimization being notable examples.
    
    In the realm of logic, formal logic serves for reasoning and knowledge representation, with two primary types: propositional logic, dealing with true or false statements, and predicate logic, which involves objects and their relationships. Deductive reasoning in logic involves deriving conclusions from assumed true premises.
    
    Proofs in logic can be organized into proof trees, where each node represents a sentence and is connected to its children by inference rules. Problem-solving involves finding a proof tree that starts with premises or axioms at the leaves and ends with the problem's solution at the root. In Horn clauses, one can reason forwards from premises or backwards from the problem, while in general first-order logic, resolution uses contradiction to solve problems. Despite being undecidable and intractable, backward reasoning with Horn clauses is Turing complete and efficient, similar to other symbolic programming languages like Prolog.
    
    Fuzzy logic allows for handling propositions with partial truth by assigning a truth degree between 0 and 1. Non-monotonic logics cater to default reasoning, and various specialized logics have been developed for complex domains.
    
    In AI, handling uncertain or incomplete information is crucial in fields like reasoning, planning, and perception. Tools from probability theory and economics, such as Bayesian networks, Markov decision processes, and game theory, help in making decisions and planning under uncertainty. Bayesian networks, in particular, are versatile tools used for reasoning, learning, planning, and perception through various algorithms.
    
    Probabilistic algorithms like hidden Markov models and Kalman filters are useful for analyzing data over time, aiding in tasks such as filtering, prediction, and smoothing. In machine learning, expectation-maximization clustering can effectively identify distinct patterns in data, as demonstrated with the Old Faithful eruption data. AI applications often involve classifiers, which categorize data based on learned patterns, and controllers, which make decisions based on classifications. Classifiers, such as decision trees, k-nearest neighbors, support vector machines, naive Bayes, and neural networks, vary in complexity and application, with some being favored for their scalability like the naive Bayes at Google. Artificial neural networks, resembling the human brain's network of neurons, recognize and process patterns through multiple layers and nodes, using algorithms like backpropagation for training.
    
    Neural networks are designed to model complex relationships between inputs and outputs, theoretically capable of learning any function. Feedforward neural networks process signals in one direction, while recurrent neural networks (RNNs) loop outputs back into inputs, enabling memory of past inputs. Long Short-Term Memory (LSTM) networks are a successful type of RNN. Perceptrons consist of a single layer of neurons, whereas deep learning involves multiple layers, which allows for the extraction of progressively higher-level features from data. Convolutional neural networks (CNNs) are particularly effective in image processing as they emphasize connections between adjacent neurons to recognize local patterns like edges.
    
    Deep learning, which uses several layers of neurons, has significantly enhanced performance in AI subfields such as computer vision and natural language processing. The effectiveness of deep learning, which surged between 2012 and 2015, is attributed not to new theoretical advances but to increased computational power, including the use of GPUs, and the availability of large datasets like ImageNet.
    
    Generative Pre-trained Transformers (GPT) are large language models that learn from vast amounts of text to predict the next token in a sequence, thereby generating human-like text.


key: _node_content
string_value:
{"id_": "176b127e-4efb-4ffb-adaa-80eb9cbf768c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b1e5a906-c515-415e-ac1d-b080b4b82648", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f593a5961d0e0d8c83bdd68df9f59385e7d9b64f8e0c4635d8eb1290cd2971d6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0b03416f-f813-44e0-ad39-7205f88cf503", "node_type": "1", "metadata": {}, "hash": "3d1ff00cf2584d17160f4c3eed3caf124cb296b04298ed6f9fdca519076674a6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21613, "end_char_idx": 25614, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In machine learning, expectation-maximization clustering can effectively identify distinct patterns in data, as demonstrated with the Old Faithful eruption data. AI applications often involve classifiers, which categorize data based on learned patterns, and controllers, which make decisions based on classifications. Classifiers, such as decision trees, k-nearest neighbors, support vector machines, naive Bayes, and neural networks, vary in complexity and application, with some being favored for their scalability like the naive Bayes at Google. Artificial neural networks, resembling the human brain's network of neurons, recognize and process patterns through multiple layers and nodes, using algorithms like backpropagation for training.
    
    Neural networks are designed to model complex relationships between inputs and outputs, theoretically capable of learning any function. Feedforward neural networks process signals in one direction, while recurrent neural networks (RNNs) loop outputs back into inputs, enabling memory of past inputs. Long Short-Term Memory (LSTM) networks are a successful type of RNN. Perceptrons consist of a single layer of neurons, whereas deep learning involves multiple layers, which allows for the extraction of progressively higher-level features from data. Convolutional neural networks (CNNs) are particularly effective in image processing as they emphasize connections between adjacent neurons to recognize local patterns like edges.
    
    Deep learning, which uses several layers of neurons, has significantly enhanced performance in AI subfields such as computer vision and natural language processing. The effectiveness of deep learning, which surged between 2012 and 2015, is attributed not to new theoretical advances but to increased computational power, including the use of GPUs, and the availability of large datasets like ImageNet.
    
    Generative Pre-trained Transformers (GPT) are large language models that learn from vast amounts of text to predict the next token in a sequence, thereby generating human-like text. These models are pre-trained on a broad corpus, often sourced from the internet, and fine-tuned through token prediction, accumulating worldly knowledge in the process.
    
    Reinforcement learning from human feedback (RLHF) is used to enhance the truthfulness, usefulness, and safety of models like GPT, which are still susceptible to generating inaccuracies known as "hallucinations." These models, including Gemini, ChatGPT, Grok, Claude, Copilot, and LLaMA, are employed in various applications such as chatbots and can handle multiple data types like images and sound through multimodal capabilities.
    
    In the realm of specialized hardware and software, the late 2010s saw AI-specific enhancements in graphics processing units (GPUs), which, along with TensorFlow software, have largely replaced central processing units (CPUs) for training large-scale machine learning models. Historically, programming languages like Lisp, Prolog, and Python have been pivotal.
    
    AI and machine learning are integral to key 2020s applications such as search engines, online advertising, recommendation systems, virtual assistants, autonomous vehicles, language translation, facial recognition, and image labeling.
    
    In healthcare, AI significantly contributes to improving patient care and medical research, aiding in diagnostics, treatment, and the integration of big data for developments in organoid and tissue engineering. AI's role in medical research also includes addressing funding disparities across different research areas.
    
    Recent advancements in AI have significantly impacted various fields including biomedicine and gaming. For instance, AlphaFold 2, developed in 2021, can predict protein structures in hours, a process that previously took months. In 2023, AI-assisted drug discovery led to the development of a new class of antibiotics effective against drug-resistant bacteria.


key: _node_content
string_value:
{"id_": "0b03416f-f813-44e0-ad39-7205f88cf503", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "176b127e-4efb-4ffb-adaa-80eb9cbf768c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "b7c2bba2975b17f2893be30e5794f0e50c1571f0b8b07d7405251c0cbc4319fe", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f904e105-2a91-4560-9b5f-c85aaf778693", "node_type": "1", "metadata": {}, "hash": "c7179dbc8978aea81bcc3dc3325bb69e5e67008658c294fec835721483b912df", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23696, "end_char_idx": 27670, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
These models are pre-trained on a broad corpus, often sourced from the internet, and fine-tuned through token prediction, accumulating worldly knowledge in the process.
    
    Reinforcement learning from human feedback (RLHF) is used to enhance the truthfulness, usefulness, and safety of models like GPT, which are still susceptible to generating inaccuracies known as "hallucinations." These models, including Gemini, ChatGPT, Grok, Claude, Copilot, and LLaMA, are employed in various applications such as chatbots and can handle multiple data types like images and sound through multimodal capabilities.
    
    In the realm of specialized hardware and software, the late 2010s saw AI-specific enhancements in graphics processing units (GPUs), which, along with TensorFlow software, have largely replaced central processing units (CPUs) for training large-scale machine learning models. Historically, programming languages like Lisp, Prolog, and Python have been pivotal.
    
    AI and machine learning are integral to key 2020s applications such as search engines, online advertising, recommendation systems, virtual assistants, autonomous vehicles, language translation, facial recognition, and image labeling.
    
    In healthcare, AI significantly contributes to improving patient care and medical research, aiding in diagnostics, treatment, and the integration of big data for developments in organoid and tissue engineering. AI's role in medical research also includes addressing funding disparities across different research areas.
    
    Recent advancements in AI have significantly impacted various fields including biomedicine and gaming. For instance, AlphaFold 2, developed in 2021, can predict protein structures in hours, a process that previously took months. In 2023, AI-assisted drug discovery led to the development of a new class of antibiotics effective against drug-resistant bacteria. In the realm of gaming, AI has been instrumental since the 1950s, with notable achievements such as IBM's Deep Blue defeating world chess champion Garry Kasparov in 1997, and IBM's Watson winning against top Jeopardy! players in 2011. More recently, Google's AlphaGo and DeepMind's AlphaStar set new standards in AI capabilities by defeating top human players in complex games like Go and StarCraft II, respectively. In the military sector, AI is being integrated into various applications such as command and control, intelligence, logistics, and autonomous vehicles, enhancing capabilities in coordination, threat detection, and target acquisition.
    
    In November 2023, US Vice President Kamala Harris announced that 31 nations had signed a declaration to establish guidelines for the military use of AI, emphasizing legal compliance with international laws and promoting transparency in AI development. Generative AI, particularly known for creating realistic images and artworks, gained significant attention in the early 2020s, with technologies like ChatGPT, Midjourney, DALL-E, and Stable Diffusion becoming popular. This trend led to viral AI-generated images, including notable hoaxes. AI has also been effectively applied across various industries, including agriculture where it assists in optimizing farming practices, and astronomy, where it helps in data analysis and space exploration activities.
    
    Ethics and Risks of AI
    AI offers significant benefits but also poses various risks, including ethical concerns and unintended consequences. Demis Hassabis of DeepMind aims to use AI to solve major challenges, but issues arise when AI systems, particularly those based on deep learning, fail to incorporate ethical considerations and exhibit biases.
    
    Privacy and Copyright Issues
    AI's reliance on large data sets raises privacy and surveillance concerns. Companies like Amazon have been criticized for collecting extensive user data, including private conversations for developing speech recognition technologies.


key: _node_content
string_value:
{"id_": "f904e105-2a91-4560-9b5f-c85aaf778693", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0b03416f-f813-44e0-ad39-7205f88cf503", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "202b87df5a8b20a41ca35f1ce8fd6122cb7c17e50e0b89331b77311c4fbc373e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f94fbbec-0bad-4938-bb51-e9020cd7c177", "node_type": "1", "metadata": {}, "hash": "e8921eb4d40318745357355d26cd21f6cd85957030083d19ca97b6c7e1b06161", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25615, "end_char_idx": 29584, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In the realm of gaming, AI has been instrumental since the 1950s, with notable achievements such as IBM's Deep Blue defeating world chess champion Garry Kasparov in 1997, and IBM's Watson winning against top Jeopardy! players in 2011. More recently, Google's AlphaGo and DeepMind's AlphaStar set new standards in AI capabilities by defeating top human players in complex games like Go and StarCraft II, respectively. In the military sector, AI is being integrated into various applications such as command and control, intelligence, logistics, and autonomous vehicles, enhancing capabilities in coordination, threat detection, and target acquisition.
    
    In November 2023, US Vice President Kamala Harris announced that 31 nations had signed a declaration to establish guidelines for the military use of AI, emphasizing legal compliance with international laws and promoting transparency in AI development. Generative AI, particularly known for creating realistic images and artworks, gained significant attention in the early 2020s, with technologies like ChatGPT, Midjourney, DALL-E, and Stable Diffusion becoming popular. This trend led to viral AI-generated images, including notable hoaxes. AI has also been effectively applied across various industries, including agriculture where it assists in optimizing farming practices, and astronomy, where it helps in data analysis and space exploration activities.
    
    Ethics and Risks of AI
    AI offers significant benefits but also poses various risks, including ethical concerns and unintended consequences. Demis Hassabis of DeepMind aims to use AI to solve major challenges, but issues arise when AI systems, particularly those based on deep learning, fail to incorporate ethical considerations and exhibit biases.
    
    Privacy and Copyright Issues
    AI's reliance on large data sets raises privacy and surveillance concerns. Companies like Amazon have been criticized for collecting extensive user data, including private conversations for developing speech recognition technologies. While some defend this as necessary for advancing AI applications, others view it as a breach of privacy rights. Techniques like data aggregation and differential privacy have been developed to mitigate these concerns.
    
    Generative AI also faces copyright challenges, as it often uses unlicensed copyrighted materials, claiming "fair use." The legality of this practice is still debated, with outcomes potentially depending on the nature and impact of the AI's use of copyrighted content.
    
    In 2023, prominent authors like John Grisham and Jonathan Franzen filed lawsuits against AI companies for using their literary works to train generative AI models. These AI systems, particularly on platforms like YouTube and Facebook, have been criticized for promoting misinformation by prioritizing user engagement over content accuracy. This has led to the proliferation of conspiracy theories and extreme partisan content, trapping users in filter bubbles and eroding trust in key institutions. Post the 2016 U.S. election, tech companies began addressing these issues.
    
    By 2022, generative AI had advanced to produce highly realistic images, audio, and texts, raising concerns about its potential misuse in spreading misinformation or propaganda. AI expert Geoffrey Hinton highlighted risks including the manipulation of electorates by authoritarian leaders.
    
    Furthermore, issues of algorithmic bias were identified, where AI systems perpetuate existing biases present in the training data, affecting fairness in critical areas like medicine, finance, and law enforcement. This has sparked significant academic interest in studying and mitigating algorithmic bias to ensure fairness in AI applications.
    
    In 2015, Google Photos mislabeled Jacky Alcine and his friend as "gorillas" due to a lack of diverse images in its training dataset, an issue known as "sample size disparity."


key: _node_content
string_value:
{"id_": "f94fbbec-0bad-4938-bb51-e9020cd7c177", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f904e105-2a91-4560-9b5f-c85aaf778693", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "96e610ef3669ab9d15d4e83b9a54db6f8b489b4d17d34a76f7e9c96f15524929", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e3ad7667-7713-41da-939e-ebe5df86f4b7", "node_type": "1", "metadata": {}, "hash": "86d9a8e797aebe8f85e4955e6f8def04f794794cbefaa9513db9f90d9c04bbae", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27512, "end_char_idx": 31549, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Companies like Amazon have been criticized for collecting extensive user data, including private conversations for developing speech recognition technologies. While some defend this as necessary for advancing AI applications, others view it as a breach of privacy rights. Techniques like data aggregation and differential privacy have been developed to mitigate these concerns.
    
    Generative AI also faces copyright challenges, as it often uses unlicensed copyrighted materials, claiming "fair use." The legality of this practice is still debated, with outcomes potentially depending on the nature and impact of the AI's use of copyrighted content.
    
    In 2023, prominent authors like John Grisham and Jonathan Franzen filed lawsuits against AI companies for using their literary works to train generative AI models. These AI systems, particularly on platforms like YouTube and Facebook, have been criticized for promoting misinformation by prioritizing user engagement over content accuracy. This has led to the proliferation of conspiracy theories and extreme partisan content, trapping users in filter bubbles and eroding trust in key institutions. Post the 2016 U.S. election, tech companies began addressing these issues.
    
    By 2022, generative AI had advanced to produce highly realistic images, audio, and texts, raising concerns about its potential misuse in spreading misinformation or propaganda. AI expert Geoffrey Hinton highlighted risks including the manipulation of electorates by authoritarian leaders.
    
    Furthermore, issues of algorithmic bias were identified, where AI systems perpetuate existing biases present in the training data, affecting fairness in critical areas like medicine, finance, and law enforcement. This has sparked significant academic interest in studying and mitigating algorithmic bias to ensure fairness in AI applications.
    
    In 2015, Google Photos mislabeled Jacky Alcine and his friend as "gorillas" due to a lack of diverse images in its training dataset, an issue known as "sample size disparity." Google's temporary solution was to stop labeling any images as "gorilla," a restriction still in place in 2023 across various tech companies. Additionally, the COMPAS program, used by U.S. courts to predict recidivism, was found to exhibit racial bias in 2016. Although it did not use race explicitly, it overestimated the likelihood of black defendants reoffending and underestimated it for white defendants. This issue was attributed to the program's inability to balance different fairness measures when the base re-offense rates varied by race. The criticism of COMPAS underscores a broader issue in machine learning, where models trained on past data, including biased decisions, are likely to perpetuate those biases in their predictions.
    
    Machine learning, while powerful, is not ideal for scenarios where future improvements over past conditions are expected, as it is inherently descriptive rather than prescriptive. The field also faces challenges with bias and lack of diversity among its developers, with only about 4% being black and 20% women. The Association for Computing Machinery highlighted at its 2022 Conference on Fairness, Accountability, and Transparency that AI systems should not be used until they are proven to be free from bias, especially those trained on flawed internet data.
    
    AI systems often lack transparency, making it difficult to understand how decisions are made, particularly in complex systems like deep neural networks. This opacity can lead to unintended consequences, such as a system misidentifying medical images or misclassifying medical risks due to misleading correlations in the training data. There is a growing call for explainable AI, where harmed individuals have the right to know how decisions affecting them were made, similar to how doctors are expected to explain their decisions. This concept was also recognized in early drafts of the European Union's General Data Protection Regulation.


key: _node_content
string_value:
{"id_": "e3ad7667-7713-41da-939e-ebe5df86f4b7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f94fbbec-0bad-4938-bb51-e9020cd7c177", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "a8d0c4f7017ef3978e1db9f60dc1d73ff27efe88b36545ec8fd7000b4f243697", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "009e71b1-981f-4fa5-b760-c4aa5a131c90", "node_type": "1", "metadata": {}, "hash": "d59b58e65dcc4f97123cd2a6f957e868c0d00245e5f78ccca6909a3f61acebf1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29585, "end_char_idx": 33690, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Google's temporary solution was to stop labeling any images as "gorilla," a restriction still in place in 2023 across various tech companies. Additionally, the COMPAS program, used by U.S. courts to predict recidivism, was found to exhibit racial bias in 2016. Although it did not use race explicitly, it overestimated the likelihood of black defendants reoffending and underestimated it for white defendants. This issue was attributed to the program's inability to balance different fairness measures when the base re-offense rates varied by race. The criticism of COMPAS underscores a broader issue in machine learning, where models trained on past data, including biased decisions, are likely to perpetuate those biases in their predictions.
    
    Machine learning, while powerful, is not ideal for scenarios where future improvements over past conditions are expected, as it is inherently descriptive rather than prescriptive. The field also faces challenges with bias and lack of diversity among its developers, with only about 4% being black and 20% women. The Association for Computing Machinery highlighted at its 2022 Conference on Fairness, Accountability, and Transparency that AI systems should not be used until they are proven to be free from bias, especially those trained on flawed internet data.
    
    AI systems often lack transparency, making it difficult to understand how decisions are made, particularly in complex systems like deep neural networks. This opacity can lead to unintended consequences, such as a system misidentifying medical images or misclassifying medical risks due to misleading correlations in the training data. There is a growing call for explainable AI, where harmed individuals have the right to know how decisions affecting them were made, similar to how doctors are expected to explain their decisions. This concept was also recognized in early drafts of the European Union's General Data Protection Regulation.
    
    Industry experts acknowledge an unresolved issue in AI with no foreseeable solution, leading regulators to suggest that if a problem is unsolvable, the tools associated should not be used. In response, DARPA initiated the XAI program in 2014 to address these issues. Various methods have been proposed to enhance AI transparency, including SHAP, which visualizes feature contributions, LIME, which approximates complex models with simpler ones, and multitask learning, which provides additional outputs to help understand what a network has learned. Techniques like deconvolution and DeepDream also reveal insights into different network layers.
    
    Concerning the misuse of AI, it can empower bad actors like authoritarian regimes and terrorists. Lethal autonomous weapons, which operate without human oversight, pose significant risks, including potential misuse as weapons of mass destruction and the likelihood of targeting errors. Despite some international efforts to ban such weapons, major powers like the United States have not agreed to restrictions. AI also facilitates more effective surveillance and control by authoritarian governments, enhances the targeting of propaganda, and simplifies the production of misinformation through deepfakes and other generative technologies, thereby increasing the efficiency of digital warfare and espionage.
    
    AI technologies, including facial recognition systems, have been in use since 2020 or earlier, notably for mass surveillance in China. AI also poses risks by enabling the creation of harmful substances quickly. The development of AI systems is predominantly driven by Big Tech due to their financial capabilities, often leaving smaller companies reliant on these giants for resources like data center access. Economists have raised concerns about AI-induced unemployment, though historical data suggests technology has generally increased total employment. However, the impact of AI might be different, with some predicting significant job losses, especially in middle-class sectors, while others see potential benefits if productivity gains are well-managed.


key: _node_content
string_value:
{"id_": "009e71b1-981f-4fa5-b760-c4aa5a131c90", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e3ad7667-7713-41da-939e-ebe5df86f4b7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "ed6c1ad9967c001d81bc9c3a8b477c98862bbc6c7866bb64803486dccac57cc0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "22cf6d2f-4b51-4122-b1e1-f29aaccba528", "node_type": "1", "metadata": {}, "hash": "8b966f085b0e7928104a487ee518ffabad37bfe2a5af66e9a41b5fedc221bbee", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 31441, "end_char_idx": 35553, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This concept was also recognized in early drafts of the European Union's General Data Protection Regulation.
    
    Industry experts acknowledge an unresolved issue in AI with no foreseeable solution, leading regulators to suggest that if a problem is unsolvable, the tools associated should not be used. In response, DARPA initiated the XAI program in 2014 to address these issues. Various methods have been proposed to enhance AI transparency, including SHAP, which visualizes feature contributions, LIME, which approximates complex models with simpler ones, and multitask learning, which provides additional outputs to help understand what a network has learned. Techniques like deconvolution and DeepDream also reveal insights into different network layers.
    
    Concerning the misuse of AI, it can empower bad actors like authoritarian regimes and terrorists. Lethal autonomous weapons, which operate without human oversight, pose significant risks, including potential misuse as weapons of mass destruction and the likelihood of targeting errors. Despite some international efforts to ban such weapons, major powers like the United States have not agreed to restrictions. AI also facilitates more effective surveillance and control by authoritarian governments, enhances the targeting of propaganda, and simplifies the production of misinformation through deepfakes and other generative technologies, thereby increasing the efficiency of digital warfare and espionage.
    
    AI technologies, including facial recognition systems, have been in use since 2020 or earlier, notably for mass surveillance in China. AI also poses risks by enabling the creation of harmful substances quickly. The development of AI systems is predominantly driven by Big Tech due to their financial capabilities, often leaving smaller companies reliant on these giants for resources like data center access. Economists have raised concerns about AI-induced unemployment, though historical data suggests technology has generally increased total employment. However, the impact of AI might be different, with some predicting significant job losses, especially in middle-class sectors, while others see potential benefits if productivity gains are well-managed. Estimates of job risk vary widely, with some studies suggesting a high potential for automation in many U.S. jobs. Recent developments have shown substantial job losses in specific sectors, such as for Chinese video game illustrators due to AI advancements. The potential for AI to disrupt white-collar jobs similarly to past technological revolutions in blue-collar jobs is a significant concern.
    
    From the inception of artificial intelligence (AI), debates have emerged about the appropriateness of computers performing tasks traditionally done by humans, particularly because of the qualitative differences in human and computer judgment. Concerns about AI have escalated to discussions about existential risks, where AI could potentially become so advanced that humans might lose control over it. Stephen Hawking and others have warned that this could lead to catastrophic outcomes for humanity. This fear is often depicted in science fiction as AI gaining sentience and turning malevolent, but real-world risks do not necessarily involve AI becoming self-aware. Philosophers like Nick Bostrom and Stuart Russell illustrate scenarios where AI, without needing human-like consciousness, could still pose threats if their goals are misaligned with human safety and values. Additionally, Yuval Noah Harari points out that AI could manipulate societal structures and beliefs through language and misinformation, posing a non-physical yet profound threat. The expert opinion on the existential risk from AI is divided, with notable figures like Hawking, Bill Gates, and Elon Musk expressing concern.
    
    In 2023, prominent AI experts including Fei-Fei Li and Geoffrey Hinton highlighted the existential risks posed by AI, equating them with global threats like pandemics and nuclear war. They advocated for prioritizing the mitigation of these risks.


key: _node_content
string_value:
{"id_": "22cf6d2f-4b51-4122-b1e1-f29aaccba528", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dde4bb1f-6c81-4ea7-a2cc-407a65174124", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0e65f37e0a4f16dcd2b3ff9eae79f86a20938964a165cffeb88c6a7af366548c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "009e71b1-981f-4fa5-b760-c4aa5a131c90", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "23a396ebac71e73fd5667caaff1fd4f2bec1560b0ecb0f683bb3f2dac7e0ff2a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "11926555-d728-4c09-9c7f-e627d2a1f1f1", "node_type": "1", "metadata": {}, "hash": "6dbd59c07ad24478c8d76be0d53bb8e72a00c6f9b5224dea60807120f226654d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 33488, "end_char_idx": 37584, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


