# embedding_metadata

key: chroma:document
string_value:
```python
# Initiate the memory with k=2 to keep the last two turns
# Provide the memory to the agent
memory = ConversationBufferWindowMemory(k=2)
agent_executor = AgentExecutor.from_agent_and_tools(agent=agent, tools=tools, verbose=True, memory=memory)
```


```python
agent_executor.run("How many people live in canada as of 2023?")
```

    
    
    [1m> Entering new AgentExecutor chain...[0m
    [32;1m[1;3mThought: I need to find the most recent population data for Canada.
    Action: Search
    Action Input: "Canada population 2023"[0m
    
    Observation:[36;1m[1;3m39,566,248[0m[32;1m[1;3mThis data seems reliable, but I should double-check the source.
    Action: Search
    Action Input: "Source of Canada population 2023"[0m
    
    Observation:[36;1m[1;3mThe current population of Canada is 38,664,637 as of Wednesday, April 19, 2023, based on Worldometer elaboration of the latest United Nations data. Canada 2020 population is estimated at 37,742,154 people at mid year according to UN data. Canada population is equivalent to 0.48% of the total world population.[0m[32;1m[1;3mI now know the final answer
    Final Answer: As of April 19, 2023, the population of Canada is 38,664,637.[0m
    
    [1m> Finished chain.[0m





    'As of April 19, 2023, the population of Canada is 38,664,637.'




```python
agent_executor.run("how about in mexico?")
```

    
    
    [1m> Entering new AgentExecutor chain...[0m
    [32;1m[1;3mThought: I need to search for the current population of Mexico.
    Action: Search
    Action Input: "current population of Mexico"[0m
    
    Observation:[36;1m[1;3mMexico, officially the United Mexican States, is a country in the southern portion of North America. It is bordered to the north by the United States; to the south and west by the Pacific Ocean; to the southeast by Guatemala, Belize, and the Caribbean Sea; and to the east by the Gulf of Mexico.[0m[32;1m[1;3mThat's not the answer to the question, I need to refine my search.
    Action: Search
    Action Input: "population of Mexico 2023"[0m
    
    Observation:[36;1m[1;3m132,709,512[0m[32;1m[1;3mI now know the final answer.
    Final Answer: As of 2023, the population of Mexico is 132,709,512.[0m
    
    [1m> Finished chain.[0m





    'As of 2023, the population of Mexico is 132,709,512.'


key: _node_content
string_value:
{"id_": "0b2c4143-2cdc-4167-9a72-c224ff78aaa7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f5d48cba-30b6-4e83-9509-d44db97ac04c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "31cd8d7f4be93d3e65bf65b776e547fb1761ab897cb97852bb12ce1062ea1296", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e71b547c-d15c-48cf-8188-dae91ced2f8a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "4747c5c6d366225710d231bfdcdeccf9ad5b2ffb24e3a1177fe1c380ddfa4c4d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fe16638f-ac14-42f1-9124-6f53e21e52d1", "node_type": "1", "metadata": {}, "hash": "b82962f082c93df6e40a88b28beb18b90be450e954f8f0ff53780ce2f3b0c8c6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17453, "end_char_idx": 20222, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
'As of April 19, 2023, the population of Canada is 38,664,637.'




```python
agent_executor.run("how about in mexico?")
```

    
    
    [1m> Entering new AgentExecutor chain...[0m
    [32;1m[1;3mThought: I need to search for the current population of Mexico.
    Action: Search
    Action Input: "current population of Mexico"[0m
    
    Observation:[36;1m[1;3mMexico, officially the United Mexican States, is a country in the southern portion of North America. It is bordered to the north by the United States; to the south and west by the Pacific Ocean; to the southeast by Guatemala, Belize, and the Caribbean Sea; and to the east by the Gulf of Mexico.[0m[32;1m[1;3mThat's not the answer to the question, I need to refine my search.
    Action: Search
    Action Input: "population of Mexico 2023"[0m
    
    Observation:[36;1m[1;3m132,709,512[0m[32;1m[1;3mI now know the final answer.
    Final Answer: As of 2023, the population of Mexico is 132,709,512.[0m
    
    [1m> Finished chain.[0m





    'As of 2023, the population of Mexico is 132,709,512.'



## Knowledge base

Create a custom vectorstore for the Agent to use as a tool to answer questions with. We'll store the results in [Pinecone](https://docs.pinecone.io/docs/quickstart), which is supported by LangChain ([Docs](https://python.langchain.com/en/latest/modules/indexes/vectorstores/examples/pinecone.html), [API reference](https://python.langchain.com/en/latest/reference/modules/vectorstore.html)). For help getting started with Pinecone or other vector databases, we have a [cookbook](https://github.com/openai/openai-cookbook/blob/colin/examples/vector_databases/Using_vector_databases_for_embeddings_search.ipynb) to help you get started.

You can check the LangChain documentation to see what other [vectorstores](https://python.langchain.com/en/latest/modules/indexes/vectorstores.html) and [databases](https://python.langchain.com/en/latest/modules/chains/examples/sqlite.html) are available.

For this example we'll use the transcripts of the Stuff You Should Know podcast, which was provided thanks to OSF DOI [10.17605/OSF.IO/VM9NT](https://doi.org/10.17605/OSF.IO/VM9NT)


```python
import wget

# Here is a URL to a zip archive containing the transcribed podcasts
# Note that this data has already been split into chunks and embeddings from OpenAI's `text-embedding-3-small` embedding model are included
content_url = 'https://cdn.openai.com/API/examples/data/sysk_podcast_transcripts_embedded.json.zip'

# Download the file (it is ~541 MB so this will take some time)
wget.download(content_url)
```

    100% [......................................................................] 571275039 / 571275039




    'sysk_podcast_transcripts_embedded.json.zip'


key: _node_content
string_value:
{"id_": "fe16638f-ac14-42f1-9124-6f53e21e52d1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f5d48cba-30b6-4e83-9509-d44db97ac04c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "31cd8d7f4be93d3e65bf65b776e547fb1761ab897cb97852bb12ce1062ea1296", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0b2c4143-2cdc-4167-9a72-c224ff78aaa7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "b393f85cc52a3ee70a7c7687f9f1fcab15c2543cc64db712a79e670bedc48cfd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "785b007b-0143-4f87-9203-cdd7e8e7d858", "node_type": "1", "metadata": {}, "hash": "174ae9ba68021ffe4352f49a6370a345e279f5faa5bce4e261b02ae474f9ce54", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Knowledge base

Create a custom vectorstore for the Agent to use as a tool to answer questions with. We'll store the results in [Pinecone](https://docs.pinecone.io/docs/quickstart), which is supported by LangChain ([Docs](https://python.langchain.com/en/latest/modules/indexes/vectorstores/examples/pinecone.html), [API reference](https://python.langchain.com/en/latest/reference/modules/vectorstore.html)). For help getting started with Pinecone or other vector databases, we have a [cookbook](https://github.com/openai/openai-cookbook/blob/colin/examples/vector_databases/Using_vector_databases_for_embeddings_search.ipynb) to help you get started.

You can check the LangChain documentation to see what other [vectorstores](https://python.langchain.com/en/latest/modules/indexes/vectorstores.html) and [databases](https://python.langchain.com/en/latest/modules/chains/examples/sqlite.html) are available.

For this example we'll use the transcripts of the Stuff You Should Know podcast, which was provided thanks to OSF DOI [10.17605/OSF.IO/VM9NT](https://doi.org/10.17605/OSF.IO/VM9NT)


```python
import wget

# Here is a URL to a zip archive containing the transcribed podcasts
# Note that this data has already been split into chunks and embeddings from OpenAI's `text-embedding-3-small` embedding model are included
content_url = 'https://cdn.openai.com/API/examples/data/sysk_podcast_transcripts_embedded.json.zip'

# Download the file (it is ~541 MB so this will take some time)
wget.download(content_url)
```

    100% [......................................................................] 571275039 / 571275039




    'sysk_podcast_transcripts_embedded.json.zip'




```python
# Load podcasts
with zipfile.ZipFile("sysk_podcast_transcripts_embedded.json.zip","r") as zip_ref:
    zip_ref.extractall("./data")
f = open('./data/sysk_podcast_transcripts_embedded.json')
processed_podcasts = json.load(f)
```


```python
# Have a look at the contents
pd.DataFrame(processed_podcasts).head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>filename</th>
      <th>title</th>
      <th>url</th>
      <th>text_chunk</th>
      <th>embedding</th>
      <th>cleaned_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>\n\nSYSK Selects How Crime Scene Cleanup Works</td>
      <td>https://chtbl.com/track/5899E/podtrac.com/pts/.</td>
      <td>Title: sysk_with_transcripts_SYSK Selects How .</td>
      <td>[0.021279960870742798,


key: _node_content
string_value:
{"id_": "785b007b-0143-4f87-9203-cdd7e8e7d858", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f5d48cba-30b6-4e83-9509-d44db97ac04c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "31cd8d7f4be93d3e65bf65b776e547fb1761ab897cb97852bb12ce1062ea1296", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fe16638f-ac14-42f1-9124-6f53e21e52d1", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "da8808067f0d457c094facf2bf00304ab804b8609488f6a6364f3f08dc869278", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e9b91748-247e-4342-a22b-f74e5d45555f", "node_type": "1", "metadata": {}, "hash": "f23efa80eb90156ee4238529f9d7f59e891ecc21466e8d08ec56e7cb00d57d4c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
'sysk_podcast_transcripts_embedded.json.zip'




```python
# Load podcasts
with zipfile.ZipFile("sysk_podcast_transcripts_embedded.json.zip","r") as zip_ref:
    zip_ref.extractall("./data")
f = open('./data/sysk_podcast_transcripts_embedded.json')
processed_podcasts = json.load(f)
```


```python
# Have a look at the contents
pd.DataFrame(processed_podcasts).head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>filename</th>
      <th>title</th>
      <th>url</th>
      <th>text_chunk</th>
      <th>embedding</th>
      <th>cleaned_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>\n\nSYSK Selects How Crime Scene Cleanup Works</td>
      <td>https://chtbl.com/track/5899E/podtrac.com/pts/.</td>
      <td>Title: sysk_with_transcripts_SYSK Selects How .</td>
      <td>[0.021279960870742798, -0.005817972123622894, .</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
    </tr>
    <tr>
      <th>1</th>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>\n\nSYSK Selects How Crime Scene Cleanup Works</td>
      <td>https://chtbl.com/track/5899E/podtrac.com/pts/.</td>
      <td>Title: sysk_with_transcripts_SYSK Selects How .</td>
      <td>[0.013859338127076626, 0.00857278611510992, 0.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
    </tr>
    <tr>
      <th>2</th>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>\n\nSYSK Selects How Crime Scene Cleanup Works</td>
      <td>https://chtbl.com/track/5899E/podtrac.com/pts/.</td>
      <td>Title: sysk_with_transcripts_SYSK Selects How .</td>
      <td>[0.015242221765220165, 0.016030369326472282, 0.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.


key: _node_content
string_value:
{"id_": "e9b91748-247e-4342-a22b-f74e5d45555f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f5d48cba-30b6-4e83-9509-d44db97ac04c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "31cd8d7f4be93d3e65bf65b776e547fb1761ab897cb97852bb12ce1062ea1296", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "785b007b-0143-4f87-9203-cdd7e8e7d858", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "c14e03997fcdd6dc03d259b1287dfa6d6f8075bde4a11b8232389499b83c48e0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7644c38c-e8f8-4d9c-8c5f-4234e05eff16", "node_type": "1", "metadata": {}, "hash": "7ac1ce2afddc1e51a1ef78273be050aa7b1f4c8dcd6253f71cfae8d5981988ff", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</td>
      <td>[0.021279960870742798, -0.005817972123622894, .</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
    </tr>
    <tr>
      <th>1</th>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>\n\nSYSK Selects How Crime Scene Cleanup Works</td>
      <td>https://chtbl.com/track/5899E/podtrac.com/pts/.</td>
      <td>Title: sysk_with_transcripts_SYSK Selects How .</td>
      <td>[0.013859338127076626, 0.00857278611510992, 0.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
    </tr>
    <tr>
      <th>2</th>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>\n\nSYSK Selects How Crime Scene Cleanup Works</td>
      <td>https://chtbl.com/track/5899E/podtrac.com/pts/.</td>
      <td>Title: sysk_with_transcripts_SYSK Selects How .</td>
      <td>[0.015242221765220165, 0.016030369326472282, 0.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
    </tr>
    <tr>
      <th>3</th>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>\n\nSYSK Selects How Crime Scene Cleanup Works</td>
      <td>https://chtbl.com/track/5899E/podtrac.com/pts/.</td>
      <td>Title: sysk_with_transcripts_SYSK Selects How .</td>
      <td>[0.004371842369437218, -0.003036574460566044, .</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
    </tr>
    <tr>
      <th>4</th>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>\n\nSYSK Selects How Crime Scene Cleanup Works</td>
      <td>https://chtbl.com/track/5899E/podtrac.com/pts/.</td>
      <td>Title: sysk_with_transcripts_SYSK Selects How .</td>
      <td>[0.017309172078967094, 0.015154214575886726, 0.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
    </tr>
  </tbody>
</table>
</div>


key: _node_content
string_value:
{"id_": "7644c38c-e8f8-4d9c-8c5f-4234e05eff16", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f5d48cba-30b6-4e83-9509-d44db97ac04c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "31cd8d7f4be93d3e65bf65b776e547fb1761ab897cb97852bb12ce1062ea1296", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e9b91748-247e-4342-a22b-f74e5d45555f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "7d18c5258840ca4d5ac21324fbe6967439b399812957d96bc1da014d50bec8b7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ece406a5-8482-4030-99da-faf07b33c59e", "node_type": "1", "metadata": {}, "hash": "f3be76813358c0853ae470b01edca39e153751821521806878ac62fadea80158", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
015242221765220165, 0.016030369326472282, 0.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
    </tr>
    <tr>
      <th>3</th>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>\n\nSYSK Selects How Crime Scene Cleanup Works</td>
      <td>https://chtbl.com/track/5899E/podtrac.com/pts/.</td>
      <td>Title: sysk_with_transcripts_SYSK Selects How .</td>
      <td>[0.004371842369437218, -0.003036574460566044, .</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
    </tr>
    <tr>
      <th>4</th>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
      <td>\n\nSYSK Selects How Crime Scene Cleanup Works</td>
      <td>https://chtbl.com/track/5899E/podtrac.com/pts/.</td>
      <td>Title: sysk_with_transcripts_SYSK Selects How .</td>
      <td>[0.017309172078967094, 0.015154214575886726, 0.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
    </tr>
  </tbody>
</table>
</div>




```python
# Add the text embeddings to Pinecone

batch_size = 100  # how many embeddings we create and insert at once

for i in tqdm(range(0, len(processed_podcasts), batch_size)):
    # find end of batch
    i_end = min(len(processed_podcasts), i+batch_size)
    meta_batch = processed_podcasts[i:i_end]
    # get ids
    ids_batch = [x['cleaned_id'] for x in meta_batch]
    # get texts to encode
    texts = [x['text_chunk'] for x in meta_batch]
    # add embeddings
    embeds = [x['embedding'] for x in meta_batch]
    # cleanup metadata
    meta_batch = [{
        'filename': x['filename'],
        'title': x['title'],
        'text_chunk': x['text_chunk'],
        'url': x['url']
    } for x in meta_batch]
    to_upsert = list(zip(ids_batch, embeds, meta_batch))
    # upsert to Pinecone
    index.upsert(vectors=to_upsert)
```


```python
# Configuring the embeddings to be used by our retriever to be OpenAI Embeddings, matching our embedded corpus
embeddings = OpenAIEmbeddings()


# Loads a docsearch object from an existing Pinecone index so we can retrieve from it
docsearch = Pinecone.from_existing_index(index_name,embeddings,text_key='text_chunk')
```


```python
retriever = docsearch.as_retriever()
```


```python
query_docs = retriever.get_relevant_documents("can you live without a bank account")
```


key: _node_content
string_value:
{"id_": "ece406a5-8482-4030-99da-faf07b33c59e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f5d48cba-30b6-4e83-9509-d44db97ac04c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "31cd8d7f4be93d3e65bf65b776e547fb1761ab897cb97852bb12ce1062ea1296", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7644c38c-e8f8-4d9c-8c5f-4234e05eff16", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "b47b81c87e129d37094137146349ead1207e7605f1279db39bd6c7a5d555b7a1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9dcf03cb-28ec-4a83-877f-2fc2343821dd", "node_type": "1", "metadata": {}, "hash": "5cbcd43d7fcbee90ae1b82bcfc7638505b02b6f0838d45c0e4d9780e50592c8c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0.015154214575886726, 0.</td>
      <td>sysk_with_transcripts_SYSK Selects How Crime S.</td>
    </tr>
  </tbody>
</table>
</div>




```python
# Add the text embeddings to Pinecone

batch_size = 100  # how many embeddings we create and insert at once

for i in tqdm(range(0, len(processed_podcasts), batch_size)):
    # find end of batch
    i_end = min(len(processed_podcasts), i+batch_size)
    meta_batch = processed_podcasts[i:i_end]
    # get ids
    ids_batch = [x['cleaned_id'] for x in meta_batch]
    # get texts to encode
    texts = [x['text_chunk'] for x in meta_batch]
    # add embeddings
    embeds = [x['embedding'] for x in meta_batch]
    # cleanup metadata
    meta_batch = [{
        'filename': x['filename'],
        'title': x['title'],
        'text_chunk': x['text_chunk'],
        'url': x['url']
    } for x in meta_batch]
    to_upsert = list(zip(ids_batch, embeds, meta_batch))
    # upsert to Pinecone
    index.upsert(vectors=to_upsert)
```


```python
# Configuring the embeddings to be used by our retriever to be OpenAI Embeddings, matching our embedded corpus
embeddings = OpenAIEmbeddings()


# Loads a docsearch object from an existing Pinecone index so we can retrieve from it
docsearch = Pinecone.from_existing_index(index_name,embeddings,text_key='text_chunk')
```


```python
retriever = docsearch.as_retriever()
```


```python
query_docs = retriever.get_relevant_documents("can you live without a bank account")
```


```python
# Print out the title and content for the most relevant retrieved documents
print("\n".join(['Title: ' + x.metadata['title'].strip() + '\n\n' + x.page_content + '\n\n' for x in query_docs]))
```

    Title: sysk: Can You Live Without a Bank Account?
    
    Title: sysk_with_transcripts_Can you live without a bank account.json;  And if you had a life, you didn't necessarily rectify your bank checkbook every day. Oh, wait, what is balancing a checkbook mean? Seriously? Yeah. Thank God for my wife. So another reason you might avoid a bank is philosophically. There may be a longstanding distrust of banks in your family that you don't want to put your money in, or you may just want to be like, you know what? I don't want to take part in this modern society. I want to kind of drop out a bit. And a really good first move is to shut your bank account down. That's a big statement. Oh, yeah, it is. But a lot of people that are underbanked and don't have accounts aren't there on purpose. It's not some philosophical statement. A lot of times it's simply because they are poor and they don't have a lot of alternatives. Yeah. And the other thing about not having a bank account, not only do you not have a bank account, you also are, like, basically just avoiding banks altogether. There's plenty of other things that banks offer, like loans and mortgage, lollipops, stuff like that. Yeah. Maybe some free nasty coffee.


key: _node_content
string_value:
{"id_": "9dcf03cb-28ec-4a83-877f-2fc2343821dd", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f5d48cba-30b6-4e83-9509-d44db97ac04c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "31cd8d7f4be93d3e65bf65b776e547fb1761ab897cb97852bb12ce1062ea1296", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ece406a5-8482-4030-99da-faf07b33c59e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e4d09332916d25a1399a8add1594efdc8fea07426d684a8195e23b73f0b7782", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6f2f4f88-5a85-4c23-a80c-217439cbea9c", "node_type": "1", "metadata": {}, "hash": "9f51523ea4d94182ff7118ebe48746e0ecc1de80e1facdef07389e336cbc613b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24777, "end_char_idx": 27953, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
query_docs = retriever.get_relevant_documents("can you live without a bank account")
```


```python
# Print out the title and content for the most relevant retrieved documents
print("\n".join(['Title: ' + x.metadata['title'].strip() + '\n\n' + x.page_content + '\n\n' for x in query_docs]))
```

    Title: sysk: Can You Live Without a Bank Account?
    
    Title: sysk_with_transcripts_Can you live without a bank account.json;  And if you had a life, you didn't necessarily rectify your bank checkbook every day. Oh, wait, what is balancing a checkbook mean? Seriously? Yeah. Thank God for my wife. So another reason you might avoid a bank is philosophically. There may be a longstanding distrust of banks in your family that you don't want to put your money in, or you may just want to be like, you know what? I don't want to take part in this modern society. I want to kind of drop out a bit. And a really good first move is to shut your bank account down. That's a big statement. Oh, yeah, it is. But a lot of people that are underbanked and don't have accounts aren't there on purpose. It's not some philosophical statement. A lot of times it's simply because they are poor and they don't have a lot of alternatives. Yeah. And the other thing about not having a bank account, not only do you not have a bank account, you also are, like, basically just avoiding banks altogether. There's plenty of other things that banks offer, like loans and mortgage, lollipops, stuff like that. Yeah. Maybe some free nasty coffee. So when you don't have a banking account, that's like the most basic unit of the banking world. Right. If you don't have that, you obviously aren't going to be exposed to all these other things that can help. Things like build your credit history through like a revolving loan or a mortgage or a car loan or something like that that you can build up your credit for and ultimately save money. So when you don't have a bank account, for whatever reason, you are effectively out of the banking system. The problem is you can live parallel to the banking system outside of it, but it can be really dangerous, especially if you're just dealing with cash, because that cash has to stay somewhere, whether it's on you or in your mattress or in a coffee can in your backyard. You're exposed for having that readily available to anybody who finds it or comes into your house with a gun to get it. Yes.
    
    
    Title: sysk: Can You Live Without a Bank Account?
    
    Title: sysk_with_transcripts_Can you live without a bank account.json;  And it doesn't have to be an everyday thing. You can host when you want. Like, let's say you're taking a week's vacation. Why not host your home? Because that money could go toward paying for your current vacation or towards your retirement fund or even towards your kids college fund. Yeah. For anything. And listen, if you're worried about your stuff, don't be. Air cover for hosts. Let hosts welcome guests into their home without having to worry. You get $1 million in damage protection anytime you're hosting. Plus pet damage protection and income loss protection, too. And are you ready for this?


key: _node_content
string_value:
{"id_": "6f2f4f88-5a85-4c23-a80c-217439cbea9c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f5d48cba-30b6-4e83-9509-d44db97ac04c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "31cd8d7f4be93d3e65bf65b776e547fb1761ab897cb97852bb12ce1062ea1296", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9dcf03cb-28ec-4a83-877f-2fc2343821dd", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "5c0d3b306a7db8170a9ccaf943c375f22dc8ff605c70e831ff2ef683cffb5321", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ded2ebf0-8bc8-445e-9d78-a8328002f502", "node_type": "1", "metadata": {}, "hash": "650d8eaa0dcfee76980ffb53cdb3430b5c6db4a249d25d7695f4fcd2c19e1998", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 26276, "end_char_idx": 29595, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Yeah. Maybe some free nasty coffee. So when you don't have a banking account, that's like the most basic unit of the banking world. Right. If you don't have that, you obviously aren't going to be exposed to all these other things that can help. Things like build your credit history through like a revolving loan or a mortgage or a car loan or something like that that you can build up your credit for and ultimately save money. So when you don't have a bank account, for whatever reason, you are effectively out of the banking system. The problem is you can live parallel to the banking system outside of it, but it can be really dangerous, especially if you're just dealing with cash, because that cash has to stay somewhere, whether it's on you or in your mattress or in a coffee can in your backyard. You're exposed for having that readily available to anybody who finds it or comes into your house with a gun to get it. Yes.
    
    
    Title: sysk: Can You Live Without a Bank Account?
    
    Title: sysk_with_transcripts_Can you live without a bank account.json;  And it doesn't have to be an everyday thing. You can host when you want. Like, let's say you're taking a week's vacation. Why not host your home? Because that money could go toward paying for your current vacation or towards your retirement fund or even towards your kids college fund. Yeah. For anything. And listen, if you're worried about your stuff, don't be. Air cover for hosts. Let hosts welcome guests into their home without having to worry. You get $1 million in damage protection anytime you're hosting. Plus pet damage protection and income loss protection, too. And are you ready for this? Air cover for host is completely free every time you host on airbnb. Free with a capital F, with air cover for Host. It makes hosting a no brainer, and the benefits really start adding up. So learn more and host with peace of mind at Airbnb comaircoverforhosts. Capital One offers commercial solutions you can bank on. Now more than ever, your business faces specific challenges and unique opportunities. That's why Capital One offers a comprehensive suite of financial services custom tailored to your short and long term goals, backed by the expertise, strategy and resources of a top ten commercial bank, a dedicated team works with you to support your success and help you achieve your goals. Explore the possibilities at CapitalOne. comCOMMERCIAL all right, so if you live in modern society today, it is pretty tough to get by without a bank. Most cases these days you have well, I don't know about most cases, but in many cases you have automatic deposits of your work checks. Sure. A lot of people pay their bills wirelessly, online, directly from their bank. You might have a student loan, you might have a car loan, you might have your house mortgage, you might pay your credit card bills. All this stuff is running through a bank, most likely. And you would think it's probably impossible to not have a bank account these days. And I would say pretty much all Americans have them. Not true. Well, pretty much all Americans do. Like 93% do. Yeah, but that's not all. No, it's true.
    
    
    Title: sysk: Can You Live Without a Bank Account?
    
    Title: sysk_with_transcripts_Can you live without a bank account.json;  Yeah.


key: _node_content
string_value:
{"id_": "ded2ebf0-8bc8-445e-9d78-a8328002f502", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f5d48cba-30b6-4e83-9509-d44db97ac04c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "31cd8d7f4be93d3e65bf65b776e547fb1761ab897cb97852bb12ce1062ea1296", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6f2f4f88-5a85-4c23-a80c-217439cbea9c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "a14b7ab4602b82f8e6b2ae94bacec9187e43a959377b3dabeae05cf2293d3964", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a6e2a2d7-fc07-44d2-8e83-20e2cb62d346", "node_type": "1", "metadata": {}, "hash": "8f8519773341e2a6e67457d31b059ed24b095ae5cd64625a4cefefccec2286a5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27866, "end_char_idx": 31049, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Plus pet damage protection and income loss protection, too. And are you ready for this? Air cover for host is completely free every time you host on airbnb. Free with a capital F, with air cover for Host. It makes hosting a no brainer, and the benefits really start adding up. So learn more and host with peace of mind at Airbnb comaircoverforhosts. Capital One offers commercial solutions you can bank on. Now more than ever, your business faces specific challenges and unique opportunities. That's why Capital One offers a comprehensive suite of financial services custom tailored to your short and long term goals, backed by the expertise, strategy and resources of a top ten commercial bank, a dedicated team works with you to support your success and help you achieve your goals. Explore the possibilities at CapitalOne. comCOMMERCIAL all right, so if you live in modern society today, it is pretty tough to get by without a bank. Most cases these days you have well, I don't know about most cases, but in many cases you have automatic deposits of your work checks. Sure. A lot of people pay their bills wirelessly, online, directly from their bank. You might have a student loan, you might have a car loan, you might have your house mortgage, you might pay your credit card bills. All this stuff is running through a bank, most likely. And you would think it's probably impossible to not have a bank account these days. And I would say pretty much all Americans have them. Not true. Well, pretty much all Americans do. Like 93% do. Yeah, but that's not all. No, it's true.
    
    
    Title: sysk: Can You Live Without a Bank Account?
    
    Title: sysk_with_transcripts_Can you live without a bank account.json;  Yeah. 7% of Americans do not have bank accounts. About 9 million people last year in 2015 did not have bank accounts. 9 million people is a lot of people. No, it really is. And apparently that's household sorry, not people. Yeah, right. You're that is a big distinction, too. And the FDIC said, man, that's the lowest since we've been tracking this by far. And someone said, well, how long have you been tracking this? They said, well, the last six years. Really? Yeah, which I'm like. Really? That's when they started tracking it, but apparently so 2009. So if you want another number, the 9 million American households don't have bank accounts at all, then there are 25 million households in addition to that. So that makes almost like 34 million households, which that's a substantial number at this point. Sure. The 25 million are what's called underbanked, meaning they may have a bank account, but they don't use the bank account. Yeah. They don't use it because they are probably afraid of overdraft fees. Or they have maybe a bank account that got grandfathered in so that they don't have to pay minimum amount fees. And who knows? There's all sorts of reasons for people to not use a bank account that they have, but probably cheap among them is overdressed, which you'll talk more about. Yeah. And the majority of these underbank people in the United States are poor, usually. A lot of times they're minorities, a lot of times they're less educated.


key: _node_content
string_value:
{"id_": "a6e2a2d7-fc07-44d2-8e83-20e2cb62d346", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f5d48cba-30b6-4e83-9509-d44db97ac04c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "31cd8d7f4be93d3e65bf65b776e547fb1761ab897cb97852bb12ce1062ea1296", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ded2ebf0-8bc8-445e-9d78-a8328002f502", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "37daaf8dac9070f971c6dd065739c31acc7b2ec838a2931a63deefc76ead149e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0840d5ff-18d3-4030-9c9e-6e56994314d0", "node_type": "1", "metadata": {}, "hash": "601fd64233a60c86e14375014cf0910febbbc3f0a0096658e509aa8eae6a93ef", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29459, "end_char_idx": 32432, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Title: sysk: Can You Live Without a Bank Account?
    
    Title: sysk_with_transcripts_Can you live without a bank account.json;  Yeah. 7% of Americans do not have bank accounts. About 9 million people last year in 2015 did not have bank accounts. 9 million people is a lot of people. No, it really is. And apparently that's household sorry, not people. Yeah, right. You're that is a big distinction, too. And the FDIC said, man, that's the lowest since we've been tracking this by far. And someone said, well, how long have you been tracking this? They said, well, the last six years. Really? Yeah, which I'm like. Really? That's when they started tracking it, but apparently so 2009. So if you want another number, the 9 million American households don't have bank accounts at all, then there are 25 million households in addition to that. So that makes almost like 34 million households, which that's a substantial number at this point. Sure. The 25 million are what's called underbanked, meaning they may have a bank account, but they don't use the bank account. Yeah. They don't use it because they are probably afraid of overdraft fees. Or they have maybe a bank account that got grandfathered in so that they don't have to pay minimum amount fees. And who knows? There's all sorts of reasons for people to not use a bank account that they have, but probably cheap among them is overdressed, which you'll talk more about. Yeah. And the majority of these underbank people in the United States are poor, usually. A lot of times they're minorities, a lot of times they're less educated. And these communities, there's a few reasons why they may not want to use a bank one. Maybe they don't trust banks. And if you look in the history of the United States or certainly even we're just talking about the Wells Fargo scandal, when you see stuff like that on the news, it should be upsetting to everyone. But obviously if you're poor and you don't have a lot of money, that may scare you into not wanting to use a bank at all. Right? Yeah.
    
    
    Title: sysk: Can You Live Without a Bank Account?
    
    Title: sysk_with_transcripts_Can you live without a bank account.json;  Maybe at the time, I might be making it up. I seem to remember them saying that, and I was like, I don't want that. Just let the check bounce and I'll take it up with them. Yes. The way it was marketed, though, was like, hey, we value you. We want to make sure that you can pay all your bills. So if something happens and you're overdrafted we'll cover it. We're just going to charge you a fee. And it sounds good, but again, when you go from high to low and all of a sudden your overdraft fees go from one to four or five or however many, that's a huge problem. Well, and the people that are overdrafting and the people that are at least able to afford those fees. Exactly. So it's a disproportionate burden on the poor, which makes it, as a scam, one of the more evil scams around. Yes.


key: _node_content
string_value:
{"id_": "0840d5ff-18d3-4030-9c9e-6e56994314d0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f5d48cba-30b6-4e83-9509-d44db97ac04c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "31cd8d7f4be93d3e65bf65b776e547fb1761ab897cb97852bb12ce1062ea1296", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a6e2a2d7-fc07-44d2-8e83-20e2cb62d346", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "b4e8602804ee40bdf1da2fdeba93834943f077f92c13eda5984596c19f783981", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0a4156fe-f501-4dcc-91d3-4f85b75c4eab", "node_type": "1", "metadata": {}, "hash": "4faa7e8b1aadbf25a859371a3cf53430cd8fd72f31b41cb676e9c34b1b5d3694", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30888, "end_char_idx": 33868, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Yeah. And the majority of these underbank people in the United States are poor, usually. A lot of times they're minorities, a lot of times they're less educated. And these communities, there's a few reasons why they may not want to use a bank one. Maybe they don't trust banks. And if you look in the history of the United States or certainly even we're just talking about the Wells Fargo scandal, when you see stuff like that on the news, it should be upsetting to everyone. But obviously if you're poor and you don't have a lot of money, that may scare you into not wanting to use a bank at all. Right? Yeah.
    
    
    Title: sysk: Can You Live Without a Bank Account?
    
    Title: sysk_with_transcripts_Can you live without a bank account.json;  Maybe at the time, I might be making it up. I seem to remember them saying that, and I was like, I don't want that. Just let the check bounce and I'll take it up with them. Yes. The way it was marketed, though, was like, hey, we value you. We want to make sure that you can pay all your bills. So if something happens and you're overdrafted we'll cover it. We're just going to charge you a fee. And it sounds good, but again, when you go from high to low and all of a sudden your overdraft fees go from one to four or five or however many, that's a huge problem. Well, and the people that are overdrafting and the people that are at least able to afford those fees. Exactly. So it's a disproportionate burden on the poor, which makes it, as a scam, one of the more evil scams around. Yes. It's just wrong, then the idea that if you open an account, you should not opt in for overdraft protection. And it's easy to say when you're talking about checks for, like you're writing a check for a Mountain Dew and some cheetos. Yeah, who cares if you're short for that? You can go without that. But when you're talking about your rent check or like an actual grocery bill or something like that, it sucks that you can't get that stuff. But it's better to have to put a couple of things back than to pay $35 for one $2 item that you went over by, right? Yeah, that's a good point. And this was in my case, too. This is also back in the day when you I mean, a lot of times it was a mystery how much you had in your account. Right. Like, you couldn't just get on your phone before you write the check and be like, oh, well, no, I don't have enough money to cover this. Yeah, because even if you balanced your checkbook, sometimes you forgot to carry the one, it wasn't always 100% accurate.
    
    


## LLM Agent with Tools

Extend our list of tools by creating a [RetrievalQA](https://python.langchain.com/en/latest/modules/chains/index_examples/vector_db_qa.html) chain leveraging our Pinecone knowledge base.


```python
from langchain.chains import RetrievalQA

retrieval_llm = OpenAI(temperature=0)

podcast_retriever = RetrievalQA.from_chain_type(llm=retrieval_llm, chain_type="stuff", retriever=docsearch.as_retriever())
```


key: _node_content
string_value:
{"id_": "0a4156fe-f501-4dcc-91d3-4f85b75c4eab", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f5d48cba-30b6-4e83-9509-d44db97ac04c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "31cd8d7f4be93d3e65bf65b776e547fb1761ab897cb97852bb12ce1062ea1296", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0840d5ff-18d3-4030-9c9e-6e56994314d0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3a1c20fc86da8401dfe88d34494741b7a046cbd0bad31dffabf3f370d621724e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b11d82fe-bb0c-4f52-b244-019f24559e14", "node_type": "1", "metadata": {}, "hash": "945a04e7dca5025c082282e3c1d34917bbb7e30946a48c1dd9af344a38d41ea8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 32319, "end_char_idx": 35351, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
So it's a disproportionate burden on the poor, which makes it, as a scam, one of the more evil scams around. Yes. It's just wrong, then the idea that if you open an account, you should not opt in for overdraft protection. And it's easy to say when you're talking about checks for, like you're writing a check for a Mountain Dew and some cheetos. Yeah, who cares if you're short for that? You can go without that. But when you're talking about your rent check or like an actual grocery bill or something like that, it sucks that you can't get that stuff. But it's better to have to put a couple of things back than to pay $35 for one $2 item that you went over by, right? Yeah, that's a good point. And this was in my case, too. This is also back in the day when you I mean, a lot of times it was a mystery how much you had in your account. Right. Like, you couldn't just get on your phone before you write the check and be like, oh, well, no, I don't have enough money to cover this. Yeah, because even if you balanced your checkbook, sometimes you forgot to carry the one, it wasn't always 100% accurate.
    
    


## LLM Agent with Tools

Extend our list of tools by creating a [RetrievalQA](https://python.langchain.com/en/latest/modules/chains/index_examples/vector_db_qa.html) chain leveraging our Pinecone knowledge base.


```python
from langchain.chains import RetrievalQA

retrieval_llm = OpenAI(temperature=0)

podcast_retriever = RetrievalQA.from_chain_type(llm=retrieval_llm, chain_type="stuff", retriever=docsearch.as_retriever())
```


```python
expanded_tools = [
    Tool(
        name = "Search",
        func=search.run,
        description="useful for when you need to answer questions about current events"
    ),
    Tool(
        name = 'Knowledge Base',
        func=podcast_retriever.run,
        description="Useful for general questions about how to do things and for details on interesting topics. Input should be a fully formed question."
    )
]
```


```python
# Re-initialize the agent with our new list of tools
prompt_with_history = CustomPromptTemplate(
    template=template_with_history,
    tools=expanded_tools,
    input_variables=["input", "intermediate_steps", "history"]
)
llm_chain = LLMChain(llm=llm, prompt=prompt_with_history)
multi_tool_names = [tool.name for tool in expanded_tools]
multi_tool_agent = LLMSingleActionAgent(
    llm_chain=llm_chain, 
    output_parser=output_parser,
    stop=["\nObservation:"], 
    allowed_tools=multi_tool_names
)
```


```python
multi_tool_memory = ConversationBufferWindowMemory(k=2)
multi_tool_executor = AgentExecutor.from_agent_and_tools(agent=multi_tool_agent, tools=expanded_tools, verbose=True, memory=multi_tool_memory)
```


```python
multi_tool_executor.run("Hi, I'd like to know how you can live without a bank account")
```

    
    
    [1m> Entering new AgentExecutor chain...[0m
    [32;1m[1;3mThought: This is an interesting question. I'm not sure if I have the answer in my knowledge base, so I might need to search for it.


key: _node_content
string_value:
{"id_": "b11d82fe-bb0c-4f52-b244-019f24559e14", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f5d48cba-30b6-4e83-9509-d44db97ac04c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "31cd8d7f4be93d3e65bf65b776e547fb1761ab897cb97852bb12ce1062ea1296", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0a4156fe-f501-4dcc-91d3-4f85b75c4eab", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "aef8b79046c1b63b8db3cb2c0407bfb630a06a94d6d8b923ec68731f359a7711", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2f9e95af-11df-4aee-9c1f-ffee577ad3e7", "node_type": "1", "metadata": {}, "hash": "e5b3c21ec308a8bc1da90066fe09cb8e517991588f5bce019648479121fc6e30", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 33871, "end_char_idx": 37148, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
expanded_tools = [
    Tool(
        name = "Search",
        func=search.run,
        description="useful for when you need to answer questions about current events"
    ),
    Tool(
        name = 'Knowledge Base',
        func=podcast_retriever.run,
        description="Useful for general questions about how to do things and for details on interesting topics. Input should be a fully formed question."
    )
]
```


```python
# Re-initialize the agent with our new list of tools
prompt_with_history = CustomPromptTemplate(
    template=template_with_history,
    tools=expanded_tools,
    input_variables=["input", "intermediate_steps", "history"]
)
llm_chain = LLMChain(llm=llm, prompt=prompt_with_history)
multi_tool_names = [tool.name for tool in expanded_tools]
multi_tool_agent = LLMSingleActionAgent(
    llm_chain=llm_chain, 
    output_parser=output_parser,
    stop=["\nObservation:"], 
    allowed_tools=multi_tool_names
)
```


```python
multi_tool_memory = ConversationBufferWindowMemory(k=2)
multi_tool_executor = AgentExecutor.from_agent_and_tools(agent=multi_tool_agent, tools=expanded_tools, verbose=True, memory=multi_tool_memory)
```


```python
multi_tool_executor.run("Hi, I'd like to know how you can live without a bank account")
```

    
    
    [1m> Entering new AgentExecutor chain...[0m
    [32;1m[1;3mThought: This is an interesting question. I'm not sure if I have the answer in my knowledge base, so I might need to search for it.
    Action: Search
    Action Input: "How to live without a bank account"[0m
    
    Observation:[36;1m[1;3mUnderbanked households have a checking or savings account but also use alternative financial services such as money orders, check cashing, international remittances, payday loans, refund anticipation loans, rent-to-own services, pawnshop loans, or auto title loans, according to the FDIC.[0m[32;1m[1;3mIt seems like there are alternative financial services available for those who don't have a bank account. I should look into this further to provide a more comprehensive answer.
    Action: Search
    Action Input: "Alternative financial services for those without a bank account"[0m
    
    Observation:[36;1m[1;3mInstead, people who are unbanked use alternative financial services—payday loans, money orders, check cashing services, pawnshop loans, and the like—to meet their banking needs. These financial services fill an important role for unbanked individuals, but they can also cause further financial hardship.[0m[32;1m[1;3mIt's important to note that while there are alternatives to having a bank account, relying solely on these services can lead to financial difficulties. I should provide some information on the potential drawbacks of not having a bank account.
    Action: Knowledge Base
    Action Input: "What are the drawbacks of not having a bank account?"[0m
    
    Observation:[33;1m[1;3m Not having a bank account can be dangerous, as the cash has to be stored somewhere and can be exposed to anyone who finds it or comes into the house with a gun to get it. Additionally, not having a bank account means not being exposed to other things that can help, such as building credit history through loans or mortgages, which can ultimately save money.


key: _node_content
string_value:
{"id_": "2f9e95af-11df-4aee-9c1f-ffee577ad3e7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f5d48cba-30b6-4e83-9509-d44db97ac04c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "31cd8d7f4be93d3e65bf65b776e547fb1761ab897cb97852bb12ce1062ea1296", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b11d82fe-bb0c-4f52-b244-019f24559e14", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "2ab7e2df1c04d5405eba01df947bc82e8085c02c9478aeac0b69cec188b4fc08", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b03c5a5d-780a-4692-a41f-3e7e546735ac", "node_type": "1", "metadata": {}, "hash": "df88a50aa3030a45df8c7672208a2e90a851eddb94658dc3f7cd005304f8e864", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 35875, "end_char_idx": 39132, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
I should look into this further to provide a more comprehensive answer.
    Action: Search
    Action Input: "Alternative financial services for those without a bank account"[0m
    
    Observation:[36;1m[1;3mInstead, people who are unbanked use alternative financial services—payday loans, money orders, check cashing services, pawnshop loans, and the like—to meet their banking needs. These financial services fill an important role for unbanked individuals, but they can also cause further financial hardship.[0m[32;1m[1;3mIt's important to note that while there are alternatives to having a bank account, relying solely on these services can lead to financial difficulties. I should provide some information on the potential drawbacks of not having a bank account.
    Action: Knowledge Base
    Action Input: "What are the drawbacks of not having a bank account?"[0m
    
    Observation:[33;1m[1;3m Not having a bank account can be dangerous, as the cash has to be stored somewhere and can be exposed to anyone who finds it or comes into the house with a gun to get it. Additionally, not having a bank account means not being exposed to other things that can help, such as building credit history through loans or mortgages, which can ultimately save money. Finally, not having a bank account can be a disproportionate burden on the poor, as overdraft fees can be expensive.[0m[32;1m[1;3mIt's important to provide some resources for those who may be interested in learning more about alternative financial services or how to open a bank account. 
    Action: Knowledge Base
    Action Input: "Resources for alternative financial services or opening a bank account"[0m
    
    Observation:[33;1m[1;3m There are a few resources available for alternative financial services or opening a bank account. Prepaid credit cards are becoming more popular and can be found at convenience stores. Capital One offers commercial solutions and a comprehensive suite of financial services tailored to short and long term goals. Airbnb also offers Air Cover for Hosts, which provides $1 million in damage protection, pet damage protection, and income loss protection.[0m[32;1m[1;3mIt's important to note that while prepaid credit cards and alternative financial services can be helpful, they may not offer the same level of protection and benefits as a traditional bank account. It's also important to do research and compare options before making a decision. 
    Final Answer: While it is possible to live without a bank account by using alternative financial services, it may come with potential drawbacks and limitations. It's important to do research and compare options before making a decision, and there are resources available for those who may be interested in opening a bank account or exploring alternative financial services.[0m
    
    [1m> Finished chain.[0m





    "While it is possible to live without a bank account by using alternative financial services, it may come with potential drawbacks and limitations. It's important to do research and compare options before making a decision, and there are resources available for those who may be interested in opening a bank account or exploring alternative financial services."


key: _node_content
string_value:
{"id_": "b03c5a5d-780a-4692-a41f-3e7e546735ac", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f5d48cba-30b6-4e83-9509-d44db97ac04c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "31cd8d7f4be93d3e65bf65b776e547fb1761ab897cb97852bb12ce1062ea1296", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2f9e95af-11df-4aee-9c1f-ffee577ad3e7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "e633a777f1e96753abaedb21dd32bad6d06f0f4dc0adc00bf583ba847823a84d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4f715120-2449-4ee5-a394-eacd6b45b854", "node_type": "1", "metadata": {}, "hash": "b00e1f8a7f410aec0e0be227d1315a814181f8f825f0103aa0a1889aced27f09", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 37149, "end_char_idx": 40870, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Finally, not having a bank account can be a disproportionate burden on the poor, as overdraft fees can be expensive.[0m[32;1m[1;3mIt's important to provide some resources for those who may be interested in learning more about alternative financial services or how to open a bank account. 
    Action: Knowledge Base
    Action Input: "Resources for alternative financial services or opening a bank account"[0m
    
    Observation:[33;1m[1;3m There are a few resources available for alternative financial services or opening a bank account. Prepaid credit cards are becoming more popular and can be found at convenience stores. Capital One offers commercial solutions and a comprehensive suite of financial services tailored to short and long term goals. Airbnb also offers Air Cover for Hosts, which provides $1 million in damage protection, pet damage protection, and income loss protection.[0m[32;1m[1;3mIt's important to note that while prepaid credit cards and alternative financial services can be helpful, they may not offer the same level of protection and benefits as a traditional bank account. It's also important to do research and compare options before making a decision. 
    Final Answer: While it is possible to live without a bank account by using alternative financial services, it may come with potential drawbacks and limitations. It's important to do research and compare options before making a decision, and there are resources available for those who may be interested in opening a bank account or exploring alternative financial services.[0m
    
    [1m> Finished chain.[0m





    "While it is possible to live without a bank account by using alternative financial services, it may come with potential drawbacks and limitations. It's important to do research and compare options before making a decision, and there are resources available for those who may be interested in opening a bank account or exploring alternative financial services."




```python
multi_tool_executor.run('Can you tell me some interesting facts about whether zoos are good or bad for animals')
```

    
    
    [1m> Entering new AgentExecutor chain...[0m
    [32;1m[1;3mThought: This is a complex topic that requires a balanced perspective
    Action: Knowledge Base
    Action Input: "What are the arguments for and against zoos?"[0m
    
    Observation:[33;1m[1;3m The arguments for zoos include that they have gotten a lot better in the last 30-40 years, they participate in research and conservation projects, and they can help save species from extinction. The arguments against zoos include that they are still businesses, they can be counterproductive in terms of educating the public, and they can have a negative impact on the life span of animals in captivity.[0m[32;1m[1;3mIt's important to consider both sides of the argument before coming to a conclusion
    Action: Search
    Action Input: "What are some examples of successful zoo conservation projects?"[0m
    
    Observation:[36;1m[1;3mThere are dedicated species survival programs which have helped species come out from the brink of extinction, good examples of that being the black-footed ferrets, the red wolves, the Przewalski's wild horse, and the California condors.[0m[32;1m[1;3mWhile there are valid arguments on both sides, it seems that zoos can have a positive impact on conservation efforts for endangered species.
    Final Answer: Zoos can have both positive and negative effects on animals, but they can play a role in conservation efforts for endangered species. It's important to consider both sides of the argument and do research before forming an opinion.[0m
    
    [1m> Finished chain.[0m


key: _node_content
string_value:
{"id_": "4f715120-2449-4ee5-a394-eacd6b45b854", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f5d48cba-30b6-4e83-9509-d44db97ac04c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "31cd8d7f4be93d3e65bf65b776e547fb1761ab897cb97852bb12ce1062ea1296", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b03c5a5d-780a-4692-a41f-3e7e546735ac", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_build_a_tool-using_agent_with_Langchain.ipynb", "title": "How to build a tool-using agent with LangChain", "tokens": 10120, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "babc48a991b5470ebb12ad8574b3f84a6574da825b75a915efa7a9be9d2997c2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 40880, "end_char_idx": 41470, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"Zoos can have both positive and negative effects on animals, but they can play a role in conservation efforts for endangered species. It's important to consider both sides of the argument and do research before forming an opinion."



You now have a template to deploy conversational agents with tools. If you want to extend this with a Custom Agent to add your own retry behaviour or treatment of input/output variables, then follow [this article](https://python.langchain.com/en/latest/modules/agents/agents/custom_agent.html).

We look forward to seeing what you build!


```python

```


key: _node_content
string_value:
{"id_": "c36636e9-ffaf-4671-8678-c51075b865ad", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb", "title": "Embedding texts that are longer than the model's maximum context length", "tokens": 1474, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c424174f-47e7-4312-a1f9-9a236d3977b1", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb", "title": "Embedding texts that are longer than the model's maximum context length", "tokens": 1474, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "721061b2a02ef24c7c8f73f0300f27ec21b587dfd5c41ae606a5753509eab950", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "abb3816e-ac56-47cb-8fae-957bb6772365", "node_type": "1", "metadata": {}, "hash": "090c8a15bb6567b0e145953d5f6c86a3e6efae9385a51a1cdcf228d2482edacd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3065, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Embedding texts that are longer than the model's maximum context length

OpenAI's embedding models cannot embed text that exceeds a maximum length. The maximum length varies by model, and is measured by _tokens_, not string length. If you are unfamiliar with tokenization, check out [How to count tokens with tiktoken](How_to_count_tokens_with_tiktoken.ipynb).

This notebook shows how to handle texts that are longer than a model's maximum context length. We'll demonstrate using embeddings from `text-embedding-3-small`, but the same ideas can be applied to other models and tasks. To learn more about embeddings, check out the [OpenAI Embeddings Guide](https://beta.openai.com/docs/guides/embeddings).


## 1. Model context length

First, we select the model and define a function to get embeddings from the API.


```python
from openai import OpenAI
import os
import openai
from tenacity import retry, wait_random_exponential, stop_after_attempt, retry_if_not_exception_type

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))

EMBEDDING_MODEL = 'text-embedding-3-small'
EMBEDDING_CTX_LENGTH = 8191
EMBEDDING_ENCODING = 'cl100k_base'

# let's make sure to not retry on an invalid request, because that is what we want to demonstrate
@retry(wait=wait_random_exponential(min=1, max=20), stop=stop_after_attempt(6), retry=retry_if_not_exception_type(openai.BadRequestError))
def get_embedding(text_or_tokens, model=EMBEDDING_MODEL):
    return client.embeddings.create(input=text_or_tokens, model=model).data[0].embedding
```

The `text-embedding-3-small` model has a context length of 8191 tokens with the `cl100k_base` encoding, and we can see that going over that limit causes an error.


```python
long_text = 'AGI ' * 5000
try:
    get_embedding(long_text)
except openai.BadRequestError as e:
    print(e)
```

    Error code: 400 - {'error': {'message': "This model's maximum context length is 8192 tokens, however you requested 10001 tokens (10001 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.", 'type': 'invalid_request_error', 'param': None, 'code': None}}


Clearly we want to avoid these errors, particularly when handling programmatically with a large number of embeddings. Yet, we still might be faced with texts that are longer than the maximum context length. Below we describe and provide recipes for the main approaches to handling these longer texts: (1) simply truncating the text to the maximum allowed length, and (2) chunking the text and embedding each chunk individually.

## 1. Truncating the input text

The simplest solution is to truncate the input text to the maximum allowed length. Because the context length is measured in tokens, we have to first tokenize the text before truncating it. The API accepts inputs both in the form of text or tokens, so as long as you are careful that you are using the appropriate encoding, there is no need to convert the tokens back into string form. Below is an example of such a truncation function.


key: _node_content
string_value:
{"id_": "abb3816e-ac56-47cb-8fae-957bb6772365", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb", "title": "Embedding texts that are longer than the model's maximum context length", "tokens": 1474, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c424174f-47e7-4312-a1f9-9a236d3977b1", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb", "title": "Embedding texts that are longer than the model's maximum context length", "tokens": 1474, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "721061b2a02ef24c7c8f73f0300f27ec21b587dfd5c41ae606a5753509eab950", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c36636e9-ffaf-4671-8678-c51075b865ad", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb", "title": "Embedding texts that are longer than the model's maximum context length", "tokens": 1474, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "244e1f6e39e8d16d3e9679136997d1fd515377fd6ebffaf048ce4b5d02bbd862", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9b1ab6b4-592c-46e6-ad05-2d77aab0f2df", "node_type": "1", "metadata": {}, "hash": "de80614b0c643df13c462169da1ca86aeab029ddab624490443291a20c5f9a9b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1749, "end_char_idx": 5132, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
long_text = 'AGI ' * 5000
try:
    get_embedding(long_text)
except openai.BadRequestError as e:
    print(e)
```

    Error code: 400 - {'error': {'message': "This model's maximum context length is 8192 tokens, however you requested 10001 tokens (10001 in your prompt; 0 for the completion). Please reduce your prompt; or completion length.", 'type': 'invalid_request_error', 'param': None, 'code': None}}


Clearly we want to avoid these errors, particularly when handling programmatically with a large number of embeddings. Yet, we still might be faced with texts that are longer than the maximum context length. Below we describe and provide recipes for the main approaches to handling these longer texts: (1) simply truncating the text to the maximum allowed length, and (2) chunking the text and embedding each chunk individually.

## 1. Truncating the input text

The simplest solution is to truncate the input text to the maximum allowed length. Because the context length is measured in tokens, we have to first tokenize the text before truncating it. The API accepts inputs both in the form of text or tokens, so as long as you are careful that you are using the appropriate encoding, there is no need to convert the tokens back into string form. Below is an example of such a truncation function.


```python
import tiktoken

def truncate_text_tokens(text, encoding_name=EMBEDDING_ENCODING, max_tokens=EMBEDDING_CTX_LENGTH):
    """Truncate a string to have `max_tokens` according to the given encoding."""
    encoding = tiktoken.get_encoding(encoding_name)
    return encoding.encode(text)[:max_tokens]
```

Our example from before now works without error.


```python
truncated = truncate_text_tokens(long_text)
len(get_embedding(truncated))
```




    1536



## 2. Chunking the input text

Though truncation works, discarding potentially relevant text is a clear drawback. Another approach is to divide the input text into chunks and then embed each chunk individually. Then, we can either use the chunk embeddings separately, or combine them in some way, such as averaging (weighted by the size of each chunk).

We will take a function from [Python's own cookbook](https://docs.python.org/3/library/itertools.html#itertools-recipes) that breaks up a sequence into chunks.


```python
from itertools import islice

def batched(iterable, n):
    """Batch data into tuples of length n. The last batch may be shorter."""
    # batched('ABCDEFG', 3) --> ABC DEF G
    if n < 1:
        raise ValueError('n must be at least one')
    it = iter(iterable)
    while (batch := tuple(islice(it, n))):
        yield batch
```

Now we define a function that encodes a string into tokens and then breaks it up into chunks.


```python
def chunked_tokens(text, encoding_name, chunk_length):
    encoding = tiktoken.get_encoding(encoding_name)
    tokens = encoding.encode(text)
    chunks_iterator = batched(tokens, chunk_length)
    yield from chunks_iterator
```

Finally, we can write a function that safely handles embedding requests, even when the input text is longer than the maximum context length, by chunking the input tokens and embedding each chunk individually. The `average` flag can be set to `True` to return the weighted average of the chunk embeddings, or `False` to simply return the unmodified list of chunk embeddings.


```python
import numpy as np


key: _node_content
string_value:
{"id_": "9b1ab6b4-592c-46e6-ad05-2d77aab0f2df", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb", "title": "Embedding texts that are longer than the model's maximum context length", "tokens": 1474, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c424174f-47e7-4312-a1f9-9a236d3977b1", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb", "title": "Embedding texts that are longer than the model's maximum context length", "tokens": 1474, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "721061b2a02ef24c7c8f73f0300f27ec21b587dfd5c41ae606a5753509eab950", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "abb3816e-ac56-47cb-8fae-957bb6772365", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb", "title": "Embedding texts that are longer than the model's maximum context length", "tokens": 1474, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c53c7d7774fbdbd99c31637e380759395b656503caa56dab210fc1a7cc52a5eb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3430, "end_char_idx": 6595, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
truncated = truncate_text_tokens(long_text)
len(get_embedding(truncated))
```




    1536



## 2. Chunking the input text

Though truncation works, discarding potentially relevant text is a clear drawback. Another approach is to divide the input text into chunks and then embed each chunk individually. Then, we can either use the chunk embeddings separately, or combine them in some way, such as averaging (weighted by the size of each chunk).

We will take a function from [Python's own cookbook](https://docs.python.org/3/library/itertools.html#itertools-recipes) that breaks up a sequence into chunks.


```python
from itertools import islice

def batched(iterable, n):
    """Batch data into tuples of length n. The last batch may be shorter."""
    # batched('ABCDEFG', 3) --> ABC DEF G
    if n < 1:
        raise ValueError('n must be at least one')
    it = iter(iterable)
    while (batch := tuple(islice(it, n))):
        yield batch
```

Now we define a function that encodes a string into tokens and then breaks it up into chunks.


```python
def chunked_tokens(text, encoding_name, chunk_length):
    encoding = tiktoken.get_encoding(encoding_name)
    tokens = encoding.encode(text)
    chunks_iterator = batched(tokens, chunk_length)
    yield from chunks_iterator
```

Finally, we can write a function that safely handles embedding requests, even when the input text is longer than the maximum context length, by chunking the input tokens and embedding each chunk individually. The `average` flag can be set to `True` to return the weighted average of the chunk embeddings, or `False` to simply return the unmodified list of chunk embeddings.


```python
import numpy as np


def len_safe_get_embedding(text, model=EMBEDDING_MODEL, max_tokens=EMBEDDING_CTX_LENGTH, encoding_name=EMBEDDING_ENCODING, average=True):
    chunk_embeddings = []
    chunk_lens = []
    for chunk in chunked_tokens(text, encoding_name=encoding_name, chunk_length=max_tokens):
        chunk_embeddings.append(get_embedding(chunk, model=model))
        chunk_lens.append(len(chunk))

    if average:
        chunk_embeddings = np.average(chunk_embeddings, axis=0, weights=chunk_lens)
        chunk_embeddings = chunk_embeddings / np.linalg.norm(chunk_embeddings)  # normalizes length to 1
        chunk_embeddings = chunk_embeddings.tolist()
    return chunk_embeddings
```

Once again, we can now handle long input texts.


```python
average_embedding_vector = len_safe_get_embedding(long_text, average=True)
chunks_embedding_vectors = len_safe_get_embedding(long_text, average=False)

print(f"Setting average=True gives us a single {len(average_embedding_vector)}-dimensional embedding vector for our long text.")
print(f"Setting average=False gives us {len(chunks_embedding_vectors)} embedding vectors, one for each of the chunks.")

```

    Setting average=True gives us a single 1536-dimensional embedding vector for our long text.
    Setting average=False gives us 2 embedding vectors, one for each of the chunks.


In some cases, it may make sense to split chunks on paragraph boundaries or sentence boundaries to help preserve the meaning of the text.


key: _node_content
string_value:
{"id_": "4dd54882-1794-485b-9513-820686fb18dd", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2e07168d-97ec-493d-9c1e-833b2cbf34b8", "node_type": "1", "metadata": {}, "hash": "eb8cfe9991ab81df4370da4d754f071a628627c2cb5102099bbc10871f3f8a63", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2533, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Summarizing Long Documents

The objective of this notebook is to demonstrate how to summarize large documents with a controllable level of detail.
 
If you give a GPT model the task of summarizing a long document (e.g. 10k or more tokens), you'll tend to get back a relatively short summary that isn't proportional to the length of the document. For instance, a summary of a 20k token document will not be twice as long as a summary of a 10k token document. One way we can fix this is to split our document up into pieces, and produce a summary piecewise. After many queries to a GPT model, the full summary can be reconstructed. By controlling the number of text chunks and their sizes, we can ultimately control the level of detail in the output.


```python
import os
from typing import List, Tuple, Optional
from openai import OpenAI
import tiktoken
from tqdm import tqdm
```


```python
# open dataset containing part of the text of the Wikipedia page for the United States
with open("data/artificial_intelligence_wikipedia.txt", "r") as file:
    artificial_intelligence_wikipedia_text = file.read()
```


```python
# load encoding and check the length of dataset
encoding = tiktoken.encoding_for_model('gpt-4-turbo')
len(encoding.encode(artificial_intelligence_wikipedia_text))
```




    14630



We'll define a simple utility to wrap calls to the OpenAI API.


```python
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def get_chat_completion(messages, model='gpt-4-turbo'):
    response = client.chat.completions.create(
        model=model,
        messages=messages,
        temperature=0,
    )
    return response.choices[0].message.content
```

Next we'll define some utilities to chunk a large document into smaller pieces.


```python
def tokenize(text: str) -> List[str]:
    encoding = tiktoken.encoding_for_model('gpt-4-turbo')
    return encoding.encode(text)


# This function chunks a text into smaller pieces based on a maximum token count and a delimiter.
def chunk_on_delimiter(input_string: str,
                       max_tokens: int, delimiter: str) -> List[str]:
    chunks = input_string.split(delimiter)
    combined_chunks, _, dropped_chunk_count = combine_chunks_with_no_minimum(
        chunks, max_tokens, chunk_delimiter=delimiter, add_ellipsis_for_overflow=True
    )
    if dropped_chunk_count > 0:
        print(f"warning: {dropped_chunk_count} chunks were dropped due to overflow")
    combined_chunks = [f"{chunk}{delimiter}" for chunk in combined_chunks]
    return combined_chunks


key: _node_content
string_value:
{"id_": "2e07168d-97ec-493d-9c1e-833b2cbf34b8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4dd54882-1794-485b-9513-820686fb18dd", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "fc4f8e8c2a60b0450c6978888375bf42fc4921f39fa2cb1f4dc7aa0510c583d4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "58c4f5bb-3db0-4105-8f98-7848d17d4d07", "node_type": "1", "metadata": {}, "hash": "530d5258402ea19b0dd2cb1e2e3ed5ec1fffbeddd6ba8a9695fef733fdbb0487", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 884, "end_char_idx": 5207, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# open dataset containing part of the text of the Wikipedia page for the United States
with open("data/artificial_intelligence_wikipedia.txt", "r") as file:
    artificial_intelligence_wikipedia_text = file.read()
```


```python
# load encoding and check the length of dataset
encoding = tiktoken.encoding_for_model('gpt-4-turbo')
len(encoding.encode(artificial_intelligence_wikipedia_text))
```




    14630



We'll define a simple utility to wrap calls to the OpenAI API.


```python
client = OpenAI(api_key=os.getenv("OPENAI_API_KEY"))

def get_chat_completion(messages, model='gpt-4-turbo'):
    response = client.chat.completions.create(
        model=model,
        messages=messages,
        temperature=0,
    )
    return response.choices[0].message.content
```

Next we'll define some utilities to chunk a large document into smaller pieces.


```python
def tokenize(text: str) -> List[str]:
    encoding = tiktoken.encoding_for_model('gpt-4-turbo')
    return encoding.encode(text)


# This function chunks a text into smaller pieces based on a maximum token count and a delimiter.
def chunk_on_delimiter(input_string: str,
                       max_tokens: int, delimiter: str) -> List[str]:
    chunks = input_string.split(delimiter)
    combined_chunks, _, dropped_chunk_count = combine_chunks_with_no_minimum(
        chunks, max_tokens, chunk_delimiter=delimiter, add_ellipsis_for_overflow=True
    )
    if dropped_chunk_count > 0:
        print(f"warning: {dropped_chunk_count} chunks were dropped due to overflow")
    combined_chunks = [f"{chunk}{delimiter}" for chunk in combined_chunks]
    return combined_chunks


# This function combines text chunks into larger blocks without exceeding a specified token count. It returns the combined text blocks, their original indices, and the count of chunks dropped due to overflow.
def combine_chunks_with_no_minimum(
        chunks: List[str],
        max_tokens: int,
        chunk_delimiter="\n\n",
        header: Optional[str] = None,
        add_ellipsis_for_overflow=False,
) -> Tuple[List[str], List[int]]:
    dropped_chunk_count = 0
    output = []  # list to hold the final combined chunks
    output_indices = []  # list to hold the indices of the final combined chunks
    candidate = (
        [] if header is None else [header]
    )  # list to hold the current combined chunk candidate
    candidate_indices = []
    for chunk_i, chunk in enumerate(chunks):
        chunk_with_header = [chunk] if header is None else [header, chunk]
        if len(tokenize(chunk_delimiter.join(chunk_with_header))) > max_tokens:
            print(f"warning: chunk overflow")
            if (
                    add_ellipsis_for_overflow
                    and len(tokenize(chunk_delimiter.join(candidate + ["..."]))) <= max_tokens
            ):
                candidate.append("...")
                dropped_chunk_count += 1
            continue  # this case would break downstream assumptions
        # estimate token count with the current chunk added
        extended_candidate_token_count = len(tokenize(chunk_delimiter.join(candidate + [chunk])))
        # If the token count exceeds max_tokens, add the current candidate to output and start a new candidate
        if extended_candidate_token_count > max_tokens:
            output.append(chunk_delimiter.join(candidate))
            output_indices.append(candidate_indices)
            candidate = chunk_with_header  # re-initialize candidate
            candidate_indices = [chunk_i]
        # otherwise keep extending the candidate
        else:
            candidate.append(chunk)
            candidate_indices.append(chunk_i)
    # add the remaining candidate to output if it's not empty
    if (header is not None and len(candidate) > 1) or (header is None and len(candidate) > 0):
        output.append(chunk_delimiter.join(candidate))
        output_indices.append(candidate_indices)
    return output, output_indices, dropped_chunk_count
```

Now we can define a utility to summarize text with a controllable level of detail (note the `detail` parameter).

The function first determines the number of chunks by interpolating between a minimum and a maximum chunk count based on a controllable `detail` parameter. It then splits the text into chunks and summarizes each chunk.


key: _node_content
string_value:
{"id_": "58c4f5bb-3db0-4105-8f98-7848d17d4d07", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2e07168d-97ec-493d-9c1e-833b2cbf34b8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3076a2b50e7cbf759d6fc96b6bbf674889de5235e8a26584f27dd32195fa235a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9112a94f-af54-42a6-8ee2-51db44161200", "node_type": "1", "metadata": {}, "hash": "3b0b2d19746c9828ea854df7a96485e12429bff1f0c98a2d7f011f348428a8e2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5210, "end_char_idx": 8032, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def summarize(text: str,
              detail: float = 0,
              model: str = 'gpt-4-turbo',
              additional_instructions: Optional[str] = None,
              minimum_chunk_size: Optional[int] = 500,
              chunk_delimiter: str = ".",
              summarize_recursively=False,
              verbose=False):
    """
    Summarizes a given text by splitting it into chunks, each of which is summarized individually. 
    The level of detail in the summary can be adjusted, and the process can optionally be made recursive.

    Parameters:
    - text (str): The text to be summarized.
    - detail (float, optional): A value between 0 and 1 indicating the desired level of detail in the summary.
      0 leads to a higher level summary, and 1 results in a more detailed summary. Defaults to 0.
    - model (str, optional): The model to use for generating summaries. Defaults to 'gpt-3.5-turbo'.
    - additional_instructions (Optional[str], optional): Additional instructions to provide to the model for customizing summaries.
    - minimum_chunk_size (Optional[int], optional): The minimum size for text chunks. Defaults to 500.
    - chunk_delimiter (str, optional): The delimiter used to split the text into chunks. Defaults to ".".
    - summarize_recursively (bool, optional): If True, summaries are generated recursively, using previous summaries for context.
    - verbose (bool, optional): If True, prints detailed information about the chunking process.

    Returns:
    - str: The final compiled summary of the text.

    The function first determines the number of chunks by interpolating between a minimum and a maximum chunk count based on the `detail` parameter. 
    It then splits the text into chunks and summarizes each chunk. If `summarize_recursively` is True, each summary is based on the previous summaries, 
    adding more context to the summarization process. The function returns a compiled summary of all chunks.
    """

    # check detail is set correctly
    assert 0 <= detail <= 1

    # interpolate the number of chunks based to get specified level of detail
    max_chunks = len(chunk_on_delimiter(text, minimum_chunk_size, chunk_delimiter))
    min_chunks = 1
    num_chunks = int(min_chunks + detail * (max_chunks - min_chunks))

    # adjust chunk_size based on interpolated number of chunks
    document_length = len(tokenize(text))
    chunk_size = max(minimum_chunk_size, document_length // num_chunks)
    text_chunks = chunk_on_delimiter(text, chunk_size, chunk_delimiter)
    if verbose:
        print(f"Splitting the text into {len(text_chunks)} chunks to be summarized.")
        print(f"Chunk lengths are {[len(tokenize(x)) for x in text_chunks]}")

    # set system message
    system_message_content = "Rewrite this text in summarized form."


key: _node_content
string_value:
{"id_": "9112a94f-af54-42a6-8ee2-51db44161200", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "58c4f5bb-3db0-4105-8f98-7848d17d4d07", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d2a46d34e3199c6f78c3bcac2d7ccd985ca95f5d9917471f7d3d5be566778106", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "32dafd7f-9184-460e-97d9-225ac92aa6fb", "node_type": "1", "metadata": {}, "hash": "378f3360a653a69e4e04a78c726e171eab26bdb4a6557620254c821d5642ced0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6273, "end_char_idx": 9895, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- minimum_chunk_size (Optional[int], optional): The minimum size for text chunks. Defaults to 500.
    - chunk_delimiter (str, optional): The delimiter used to split the text into chunks. Defaults to ".".
    - summarize_recursively (bool, optional): If True, summaries are generated recursively, using previous summaries for context.
    - verbose (bool, optional): If True, prints detailed information about the chunking process.

    Returns:
    - str: The final compiled summary of the text.

    The function first determines the number of chunks by interpolating between a minimum and a maximum chunk count based on the `detail` parameter. 
    It then splits the text into chunks and summarizes each chunk. If `summarize_recursively` is True, each summary is based on the previous summaries, 
    adding more context to the summarization process. The function returns a compiled summary of all chunks.
    """

    # check detail is set correctly
    assert 0 <= detail <= 1

    # interpolate the number of chunks based to get specified level of detail
    max_chunks = len(chunk_on_delimiter(text, minimum_chunk_size, chunk_delimiter))
    min_chunks = 1
    num_chunks = int(min_chunks + detail * (max_chunks - min_chunks))

    # adjust chunk_size based on interpolated number of chunks
    document_length = len(tokenize(text))
    chunk_size = max(minimum_chunk_size, document_length // num_chunks)
    text_chunks = chunk_on_delimiter(text, chunk_size, chunk_delimiter)
    if verbose:
        print(f"Splitting the text into {len(text_chunks)} chunks to be summarized.")
        print(f"Chunk lengths are {[len(tokenize(x)) for x in text_chunks]}")

    # set system message
    system_message_content = "Rewrite this text in summarized form."
    if additional_instructions is not None:
        system_message_content += f"\n\n{additional_instructions}"

    accumulated_summaries = []
    for chunk in tqdm(text_chunks):
        if summarize_recursively and accumulated_summaries:
            # Creating a structured prompt for recursive summarization
            accumulated_summaries_string = '\n\n'.join(accumulated_summaries)
            user_message_content = f"Previous summaries:\n\n{accumulated_summaries_string}\n\nText to summarize next:\n\n{chunk}"
        else:
            # Directly passing the chunk for summarization without recursive context
            user_message_content = chunk

        # Constructing messages based on whether recursive summarization is applied
        messages = [
            {"role": "system", "content": system_message_content},
            {"role": "user", "content": user_message_content}
        ]

        # Assuming this function gets the completion and works as expected
        response = get_chat_completion(messages, model=model)
        accumulated_summaries.append(response)

    # Compile final summary from partial summaries
    final_summary = '\n\n'.join(accumulated_summaries)

    return final_summary
```

Now we can use this utility to produce summaries with varying levels of detail. By increasing `detail` from 0 to 1 we get progressively longer summaries of the underlying document. A higher value for the `detail` parameter results in a more detailed summary because the utility first splits the document into a greater number of chunks. Each chunk is then summarized, and the final summary is a concatenation of all the chunk summaries.


```python
summary_with_detail_0 = summarize(artificial_intelligence_wikipedia_text, detail=0, verbose=True)
```

    Splitting the text into 1 chunks to be summarized.
    Chunk lengths are [14631]


key: _node_content
string_value:
{"id_": "32dafd7f-9184-460e-97d9-225ac92aa6fb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9112a94f-af54-42a6-8ee2-51db44161200", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "e24d7f975ede7d593ad10aaccdde22ff87499d6145a7de5f0846255e2704288d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ded7f2cf-a555-4e24-962f-bf5bb57add98", "node_type": "1", "metadata": {}, "hash": "908d2655849b7cddf0fd54145785f377d022fe614ffb8000b0323ab4a3dd4966", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8037, "end_char_idx": 10590, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
if additional_instructions is not None:
        system_message_content += f"\n\n{additional_instructions}"

    accumulated_summaries = []
    for chunk in tqdm(text_chunks):
        if summarize_recursively and accumulated_summaries:
            # Creating a structured prompt for recursive summarization
            accumulated_summaries_string = '\n\n'.join(accumulated_summaries)
            user_message_content = f"Previous summaries:\n\n{accumulated_summaries_string}\n\nText to summarize next:\n\n{chunk}"
        else:
            # Directly passing the chunk for summarization without recursive context
            user_message_content = chunk

        # Constructing messages based on whether recursive summarization is applied
        messages = [
            {"role": "system", "content": system_message_content},
            {"role": "user", "content": user_message_content}
        ]

        # Assuming this function gets the completion and works as expected
        response = get_chat_completion(messages, model=model)
        accumulated_summaries.append(response)

    # Compile final summary from partial summaries
    final_summary = '\n\n'.join(accumulated_summaries)

    return final_summary
```

Now we can use this utility to produce summaries with varying levels of detail. By increasing `detail` from 0 to 1 we get progressively longer summaries of the underlying document. A higher value for the `detail` parameter results in a more detailed summary because the utility first splits the document into a greater number of chunks. Each chunk is then summarized, and the final summary is a concatenation of all the chunk summaries.


```python
summary_with_detail_0 = summarize(artificial_intelligence_wikipedia_text, detail=0, verbose=True)
```

    Splitting the text into 1 chunks to be summarized.
    Chunk lengths are [14631]


    100%|██████████| 1/1 [00:09<00:00,  9.68s/it]



```python
summary_with_detail_pt25 = summarize(artificial_intelligence_wikipedia_text, detail=0.25, verbose=True)
```

    Splitting the text into 9 chunks to be summarized.
    Chunk lengths are [1817, 1807, 1823, 1810, 1806, 1827, 1814, 1829, 103]


    100%|██████████| 9/9 [01:33<00:00, 10.39s/it]



```python
summary_with_detail_pt5 = summarize(artificial_intelligence_wikipedia_text, detail=0.5, verbose=True)
```

    Splitting the text into 17 chunks to be summarized.
    Chunk lengths are [897, 890, 914, 876, 893, 906, 893, 902, 909, 907, 905, 889, 902, 890, 901, 880, 287]


    100%|██████████| 17/17 [02:26<00:00,  8.64s/it]


key: _node_content
string_value:
{"id_": "ded7f2cf-a555-4e24-962f-bf5bb57add98", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "32dafd7f-9184-460e-97d9-225ac92aa6fb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "c4777361b882491493d9a88e8d9dc2fd71df0e4c100bd6d1e76ae5b56e069101", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "483fd6a0-9217-494f-832e-68256df91ac6", "node_type": "1", "metadata": {}, "hash": "4731f1bef86681130c29ed3d6986c127b10be23a49b5f4056c5089294c13e07d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9339, "end_char_idx": 11499, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
By increasing `detail` from 0 to 1 we get progressively longer summaries of the underlying document. A higher value for the `detail` parameter results in a more detailed summary because the utility first splits the document into a greater number of chunks. Each chunk is then summarized, and the final summary is a concatenation of all the chunk summaries.


```python
summary_with_detail_0 = summarize(artificial_intelligence_wikipedia_text, detail=0, verbose=True)
```

    Splitting the text into 1 chunks to be summarized.
    Chunk lengths are [14631]


    100%|██████████| 1/1 [00:09<00:00,  9.68s/it]



```python
summary_with_detail_pt25 = summarize(artificial_intelligence_wikipedia_text, detail=0.25, verbose=True)
```

    Splitting the text into 9 chunks to be summarized.
    Chunk lengths are [1817, 1807, 1823, 1810, 1806, 1827, 1814, 1829, 103]


    100%|██████████| 9/9 [01:33<00:00, 10.39s/it]



```python
summary_with_detail_pt5 = summarize(artificial_intelligence_wikipedia_text, detail=0.5, verbose=True)
```

    Splitting the text into 17 chunks to be summarized.
    Chunk lengths are [897, 890, 914, 876, 893, 906, 893, 902, 909, 907, 905, 889, 902, 890, 901, 880, 287]


    100%|██████████| 17/17 [02:26<00:00,  8.64s/it]



```python
summary_with_detail_1 = summarize(artificial_intelligence_wikipedia_text, detail=1, verbose=True)
```

    Splitting the text into 31 chunks to be summarized.
    Chunk lengths are [492, 427, 485, 490, 496, 478, 473, 497, 496, 501, 499, 497, 493, 470, 472, 494, 489, 492, 481, 485, 471, 500, 486, 498, 478, 469, 498, 468, 493, 478, 103]


    100%|██████████| 31/31 [04:08<00:00,  8.02s/it]


The original document is nearly 15k tokens long. Notice how large the gap is between the length of `summary_with_detail_0` and `summary_with_detail_1`. It's nearly 25 times longer!


```python
# lengths of summaries
[len(tokenize(x)) for x in
 [summary_with_detail_0, summary_with_detail_pt25, summary_with_detail_pt5, summary_with_detail_1]]
```




    [235, 2529, 4336, 6742]



Let's inspect the summaries to see how the level of detail changes when the `detail` parameter is increased from 0 to 1.


key: _node_content
string_value:
{"id_": "483fd6a0-9217-494f-832e-68256df91ac6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ded7f2cf-a555-4e24-962f-bf5bb57add98", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f7022987e2d6feb9e95e757fab76e983345552d2d1e4a5e0cbb2c2b87e5611b5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "654e2bd2-03f3-4be4-87ae-747244407264", "node_type": "1", "metadata": {}, "hash": "998d98a9b01514c8e84d02fc8620f30e0342a3a681bf5bf8279c2a27a8396869", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10543, "end_char_idx": 13699, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
100%|██████████| 17/17 [02:26<00:00,  8.64s/it]



```python
summary_with_detail_1 = summarize(artificial_intelligence_wikipedia_text, detail=1, verbose=True)
```

    Splitting the text into 31 chunks to be summarized.
    Chunk lengths are [492, 427, 485, 490, 496, 478, 473, 497, 496, 501, 499, 497, 493, 470, 472, 494, 489, 492, 481, 485, 471, 500, 486, 498, 478, 469, 498, 468, 493, 478, 103]


    100%|██████████| 31/31 [04:08<00:00,  8.02s/it]


The original document is nearly 15k tokens long. Notice how large the gap is between the length of `summary_with_detail_0` and `summary_with_detail_1`. It's nearly 25 times longer!


```python
# lengths of summaries
[len(tokenize(x)) for x in
 [summary_with_detail_0, summary_with_detail_pt25, summary_with_detail_pt5, summary_with_detail_1]]
```




    [235, 2529, 4336, 6742]



Let's inspect the summaries to see how the level of detail changes when the `detail` parameter is increased from 0 to 1.


```python
print(summary_with_detail_0)
```

    Artificial intelligence (AI) is the simulation of human intelligence in machines, designed to perform tasks that typically require human intelligence. This includes applications like advanced search engines, recommendation systems, speech interaction, autonomous vehicles, and more. AI was first significantly researched by Alan Turing and became an academic discipline in 1956. The field has experienced cycles of high expectations followed by disillusionment and reduced funding, known as "AI winters." Interest in AI surged post-2012 with advancements in deep learning and again post-2017 with the development of the transformer architecture, leading to a boom in AI research and applications in the early 2020s.
    
    AI's increasing integration into various sectors is influencing societal and economic shifts towards automation and data-driven decision-making, impacting areas such as employment, healthcare, and privacy. Ethical and safety concerns about AI have prompted discussions on regulatory policies.
    
    AI research involves various sub-fields focused on specific goals like reasoning, learning, and perception, using techniques from mathematics, logic, and other disciplines. Despite its broad applications, AI's complexity and potential risks, such as privacy issues, misinformation, and ethical challenges, remain areas of active investigation and debate.



```python
print(summary_with_detail_1)
```

    Artificial intelligence (AI) is the simulation of human intelligence in machines, designed to perceive their environment and make decisions to achieve specific goals. This technology is prevalent across various sectors including industry, government, and science, with applications ranging from web search engines and recommendation systems to autonomous vehicles and AI in gaming. Although AI has become a common feature in many tools and applications, it often goes unrecognized as AI when it becomes sufficiently integrated and widespread.
    
    The field of AI, which began as an academic discipline in 1956, has experienced several cycles of high expectations followed by disappointment, known as AI winters.


key: _node_content
string_value:
{"id_": "654e2bd2-03f3-4be4-87ae-747244407264", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "483fd6a0-9217-494f-832e-68256df91ac6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "ce44ce5d16dad9340de0b154ab36d647bada53572fd6f8f1bd5f31b802fb3160", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f53b19fd-85d8-449d-a462-964501101a92", "node_type": "1", "metadata": {}, "hash": "b10de7fa67b0fd56ee28c6a0d28ad4563374c164af648f70b609513c73e0db4c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11502, "end_char_idx": 15642, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
print(summary_with_detail_0)
```

    Artificial intelligence (AI) is the simulation of human intelligence in machines, designed to perform tasks that typically require human intelligence. This includes applications like advanced search engines, recommendation systems, speech interaction, autonomous vehicles, and more. AI was first significantly researched by Alan Turing and became an academic discipline in 1956. The field has experienced cycles of high expectations followed by disillusionment and reduced funding, known as "AI winters." Interest in AI surged post-2012 with advancements in deep learning and again post-2017 with the development of the transformer architecture, leading to a boom in AI research and applications in the early 2020s.
    
    AI's increasing integration into various sectors is influencing societal and economic shifts towards automation and data-driven decision-making, impacting areas such as employment, healthcare, and privacy. Ethical and safety concerns about AI have prompted discussions on regulatory policies.
    
    AI research involves various sub-fields focused on specific goals like reasoning, learning, and perception, using techniques from mathematics, logic, and other disciplines. Despite its broad applications, AI's complexity and potential risks, such as privacy issues, misinformation, and ethical challenges, remain areas of active investigation and debate.



```python
print(summary_with_detail_1)
```

    Artificial intelligence (AI) is the simulation of human intelligence in machines, designed to perceive their environment and make decisions to achieve specific goals. This technology is prevalent across various sectors including industry, government, and science, with applications ranging from web search engines and recommendation systems to autonomous vehicles and AI in gaming. Although AI has become a common feature in many tools and applications, it often goes unrecognized as AI when it becomes sufficiently integrated and widespread.
    
    The field of AI, which began as an academic discipline in 1956, has experienced several cycles of high expectations followed by disappointment, known as AI winters. Interest and funding in AI surged post-2012 with advancements in deep learning and again post-2017 with the development of transformer architecture, leading to a significant boom in AI research and applications in the early 2020s, primarily in the United States.
    
    The increasing integration of AI in the 21st century is driving a shift towards automation and data-driven decision-making across various sectors, influencing job markets, healthcare, and education, among others. This raises important questions about the ethical implications, long-term effects, and the need for regulatory policies to ensure the safety and benefits of AI technologies. AI research itself is diverse, focusing on goals like reasoning, learning, and perception, and involves various tools and methodologies to achieve these objectives.
    
    General intelligence, which involves performing any human task at least as well as a human, is a long-term goal in AI research. To achieve this, AI integrates various techniques from search and optimization, formal logic, neural networks, and statistics, to insights from psychology, linguistics, and neuroscience. AI research focuses on specific traits like reasoning and problem-solving, where early algorithms mimicked human step-by-step reasoning. However, these algorithms struggle with large, complex problems due to combinatorial explosion and are less efficient than human intuitive judgments. Knowledge representation is another critical area, using ontologies to structure domain-specific knowledge and relationships, aiding in intelligent querying, scene interpretation, and data mining among other applications.
    
    Knowledge bases must encapsulate a wide range of elements including objects, properties, categories, relations, events, states, time, causes, effects, and meta-knowledge. They also need to handle default reasoning, where certain assumptions are maintained unless contradicted.


key: _node_content
string_value:
{"id_": "f53b19fd-85d8-449d-a462-964501101a92", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "654e2bd2-03f3-4be4-87ae-747244407264", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e1cf1b7d8c6fbda447dd23d5f0ad2fa250be6d68266c0dd05b903523e7f8127", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2bd5b198-c800-444a-a3a9-c12a87c4cf88", "node_type": "1", "metadata": {}, "hash": "f4d6b6a3dd94873d90ba96789cda548b19b210387d9b841aa31f54d36c63bff2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13535, "end_char_idx": 17775, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The field of AI, which began as an academic discipline in 1956, has experienced several cycles of high expectations followed by disappointment, known as AI winters. Interest and funding in AI surged post-2012 with advancements in deep learning and again post-2017 with the development of transformer architecture, leading to a significant boom in AI research and applications in the early 2020s, primarily in the United States.
    
    The increasing integration of AI in the 21st century is driving a shift towards automation and data-driven decision-making across various sectors, influencing job markets, healthcare, and education, among others. This raises important questions about the ethical implications, long-term effects, and the need for regulatory policies to ensure the safety and benefits of AI technologies. AI research itself is diverse, focusing on goals like reasoning, learning, and perception, and involves various tools and methodologies to achieve these objectives.
    
    General intelligence, which involves performing any human task at least as well as a human, is a long-term goal in AI research. To achieve this, AI integrates various techniques from search and optimization, formal logic, neural networks, and statistics, to insights from psychology, linguistics, and neuroscience. AI research focuses on specific traits like reasoning and problem-solving, where early algorithms mimicked human step-by-step reasoning. However, these algorithms struggle with large, complex problems due to combinatorial explosion and are less efficient than human intuitive judgments. Knowledge representation is another critical area, using ontologies to structure domain-specific knowledge and relationships, aiding in intelligent querying, scene interpretation, and data mining among other applications.
    
    Knowledge bases must encapsulate a wide range of elements including objects, properties, categories, relations, events, states, time, causes, effects, and meta-knowledge. They also need to handle default reasoning, where certain assumptions are maintained unless contradicted. Challenges in knowledge representation include the vast scope of commonsense knowledge and its often sub-symbolic, non-verbal nature, alongside the difficulty of acquiring this knowledge for AI use.
    
    In the realm of AI, an "agent" is defined as an entity that perceives its environment and acts towards achieving goals or fulfilling preferences. In automated planning, the agent pursues a specific goal, while in decision-making, it evaluates actions based on their expected utility to maximize preference satisfaction. Classical planning assumes agents have complete knowledge of action outcomes, but real-world scenarios often involve uncertainty about the situation and outcomes, requiring probabilistic decision-making. Additionally, agents may need to adapt or learn preferences, particularly in complex environments with multiple agents or human interactions.
    
    Information value theory helps assess the value of exploratory actions in situations with uncertain outcomes. A Markov decision process uses a transition model and a reward function to guide decisions, which can be determined through calculations, heuristics, or learning. Game theory analyzes the rational behavior of multiple interacting agents in decision-making scenarios involving others.
    
    Machine learning, integral to AI, involves programs that automatically improve task performance. It includes unsupervised learning, which identifies patterns in data without guidance, and supervised learning, which requires labeled data and includes classification and regression tasks. Reinforcement learning rewards or punishes agents to shape their responses, while transfer learning applies knowledge from one problem to another. Deep learning, a subset of machine learning, uses artificial neural networks inspired by biological processes.
    
    Computational learning theory evaluates learning algorithms based on computational and sample complexity, among other criteria. Natural language processing (NLP) enables programs to interact using human languages, tackling challenges like speech recognition, synthesis, translation, and more.


key: _node_content
string_value:
{"id_": "2bd5b198-c800-444a-a3a9-c12a87c4cf88", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f53b19fd-85d8-449d-a462-964501101a92", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "17e65324703551d93f77d4ef67deecbbf483593ca8943d901b8b8bfb57fafd21", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "005446ba-c73e-4e5c-b074-a1f8bcc0174e", "node_type": "1", "metadata": {}, "hash": "0242685a2ea8c8ab7697bf165aacee0e9bb71076b7a908e61d6b7d32d9dfb6e9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15537, "end_char_idx": 19778, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
They also need to handle default reasoning, where certain assumptions are maintained unless contradicted. Challenges in knowledge representation include the vast scope of commonsense knowledge and its often sub-symbolic, non-verbal nature, alongside the difficulty of acquiring this knowledge for AI use.
    
    In the realm of AI, an "agent" is defined as an entity that perceives its environment and acts towards achieving goals or fulfilling preferences. In automated planning, the agent pursues a specific goal, while in decision-making, it evaluates actions based on their expected utility to maximize preference satisfaction. Classical planning assumes agents have complete knowledge of action outcomes, but real-world scenarios often involve uncertainty about the situation and outcomes, requiring probabilistic decision-making. Additionally, agents may need to adapt or learn preferences, particularly in complex environments with multiple agents or human interactions.
    
    Information value theory helps assess the value of exploratory actions in situations with uncertain outcomes. A Markov decision process uses a transition model and a reward function to guide decisions, which can be determined through calculations, heuristics, or learning. Game theory analyzes the rational behavior of multiple interacting agents in decision-making scenarios involving others.
    
    Machine learning, integral to AI, involves programs that automatically improve task performance. It includes unsupervised learning, which identifies patterns in data without guidance, and supervised learning, which requires labeled data and includes classification and regression tasks. Reinforcement learning rewards or punishes agents to shape their responses, while transfer learning applies knowledge from one problem to another. Deep learning, a subset of machine learning, uses artificial neural networks inspired by biological processes.
    
    Computational learning theory evaluates learning algorithms based on computational and sample complexity, among other criteria. Natural language processing (NLP) enables programs to interact using human languages, tackling challenges like speech recognition, synthesis, translation, and more. Early NLP efforts, influenced by Chomsky's theories, faced limitations in handling ambiguous language outside of controlled environments.
    
    Margaret Masterman emphasized the importance of meaning over grammar in language understanding, advocating for the use of thesauri instead of dictionaries in computational linguistics. Modern NLP techniques include word embedding, transformers, and by 2023, GPT models capable of achieving human-level scores on various tests. Machine perception involves interpreting sensor data to understand the world, encompassing computer vision and speech recognition among other applications. Social intelligence in AI focuses on recognizing and simulating human emotions, with systems like Kismet and affective computing technologies that enhance human-computer interaction. However, these advancements may lead to overestimations of AI capabilities by users. AI also employs a variety of techniques including search and optimization, with methods like state space search to explore possible solutions to problems.
    
    Planning algorithms use means-ends analysis to navigate through trees of goals and subgoals to achieve a target goal. However, simple exhaustive searches are often inadequate for complex real-world problems due to the vast search space, making searches slow or incomplete. Heuristics are employed to prioritize more promising paths towards a goal. In adversarial contexts like chess or Go, search algorithms explore trees of possible moves to find a winning strategy.
    
    Local search methods, such as gradient descent, optimize numerical parameters to minimize a loss function, often used in training neural networks. Evolutionary computation, another local search technique, iteratively enhances solutions by mutating and recombining candidate solutions, selecting the most fit for survival. Distributed search processes utilize swarm intelligence, with particle swarm optimization and ant colony optimization being notable examples.


key: _node_content
string_value:
{"id_": "005446ba-c73e-4e5c-b074-a1f8bcc0174e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2bd5b198-c800-444a-a3a9-c12a87c4cf88", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "e1ce7a16bb00b5dda25be1f99d4d3bb5f570f57c681014c3c01017389228308d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e02890dd-e89d-4827-8031-0abb09c07785", "node_type": "1", "metadata": {}, "hash": "ab23f571cb31e4b952ac34a60d584b3fb9e6d62c72a118544289419bbcad9fc0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17611, "end_char_idx": 21774, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Natural language processing (NLP) enables programs to interact using human languages, tackling challenges like speech recognition, synthesis, translation, and more. Early NLP efforts, influenced by Chomsky's theories, faced limitations in handling ambiguous language outside of controlled environments.
    
    Margaret Masterman emphasized the importance of meaning over grammar in language understanding, advocating for the use of thesauri instead of dictionaries in computational linguistics. Modern NLP techniques include word embedding, transformers, and by 2023, GPT models capable of achieving human-level scores on various tests. Machine perception involves interpreting sensor data to understand the world, encompassing computer vision and speech recognition among other applications. Social intelligence in AI focuses on recognizing and simulating human emotions, with systems like Kismet and affective computing technologies that enhance human-computer interaction. However, these advancements may lead to overestimations of AI capabilities by users. AI also employs a variety of techniques including search and optimization, with methods like state space search to explore possible solutions to problems.
    
    Planning algorithms use means-ends analysis to navigate through trees of goals and subgoals to achieve a target goal. However, simple exhaustive searches are often inadequate for complex real-world problems due to the vast search space, making searches slow or incomplete. Heuristics are employed to prioritize more promising paths towards a goal. In adversarial contexts like chess or Go, search algorithms explore trees of possible moves to find a winning strategy.
    
    Local search methods, such as gradient descent, optimize numerical parameters to minimize a loss function, often used in training neural networks. Evolutionary computation, another local search technique, iteratively enhances solutions by mutating and recombining candidate solutions, selecting the most fit for survival. Distributed search processes utilize swarm intelligence, with particle swarm optimization and ant colony optimization being notable examples.
    
    In the realm of logic, formal logic serves for reasoning and knowledge representation, with two primary types: propositional logic, dealing with true or false statements, and predicate logic, which involves objects and their relationships. Deductive reasoning in logic involves deriving conclusions from assumed true premises.
    
    Proofs in logic can be organized into proof trees, where each node represents a sentence and is connected to its children by inference rules. Problem-solving involves finding a proof tree that starts with premises or axioms at the leaves and ends with the problem's solution at the root. In Horn clauses, one can reason forwards from premises or backwards from the problem, while in general first-order logic, resolution uses contradiction to solve problems. Despite being undecidable and intractable, backward reasoning with Horn clauses is Turing complete and efficient, similar to other symbolic programming languages like Prolog.
    
    Fuzzy logic allows for handling propositions with partial truth by assigning a truth degree between 0 and 1. Non-monotonic logics cater to default reasoning, and various specialized logics have been developed for complex domains.
    
    In AI, handling uncertain or incomplete information is crucial in fields like reasoning, planning, and perception. Tools from probability theory and economics, such as Bayesian networks, Markov decision processes, and game theory, help in making decisions and planning under uncertainty. Bayesian networks, in particular, are versatile tools used for reasoning, learning, planning, and perception through various algorithms.
    
    Probabilistic algorithms like hidden Markov models and Kalman filters are useful for analyzing data over time, aiding in tasks such as filtering, prediction, and smoothing. In machine learning, expectation-maximization clustering can effectively identify distinct patterns in data, as demonstrated with the Old Faithful eruption data.


key: _node_content
string_value:
{"id_": "e02890dd-e89d-4827-8031-0abb09c07785", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "005446ba-c73e-4e5c-b074-a1f8bcc0174e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0b4379f1f22bd1fc9aae303f6e039117068c0eed5c1384de496ae9bd5f734e14", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1561995a-6690-40fd-9f8b-6d1417348746", "node_type": "1", "metadata": {}, "hash": "4d1a7f143795073479fbed1c2faaac497692c027ab1a6b1fe5f5baa3b333bb4e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19637, "end_char_idx": 23695, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Distributed search processes utilize swarm intelligence, with particle swarm optimization and ant colony optimization being notable examples.
    
    In the realm of logic, formal logic serves for reasoning and knowledge representation, with two primary types: propositional logic, dealing with true or false statements, and predicate logic, which involves objects and their relationships. Deductive reasoning in logic involves deriving conclusions from assumed true premises.
    
    Proofs in logic can be organized into proof trees, where each node represents a sentence and is connected to its children by inference rules. Problem-solving involves finding a proof tree that starts with premises or axioms at the leaves and ends with the problem's solution at the root. In Horn clauses, one can reason forwards from premises or backwards from the problem, while in general first-order logic, resolution uses contradiction to solve problems. Despite being undecidable and intractable, backward reasoning with Horn clauses is Turing complete and efficient, similar to other symbolic programming languages like Prolog.
    
    Fuzzy logic allows for handling propositions with partial truth by assigning a truth degree between 0 and 1. Non-monotonic logics cater to default reasoning, and various specialized logics have been developed for complex domains.
    
    In AI, handling uncertain or incomplete information is crucial in fields like reasoning, planning, and perception. Tools from probability theory and economics, such as Bayesian networks, Markov decision processes, and game theory, help in making decisions and planning under uncertainty. Bayesian networks, in particular, are versatile tools used for reasoning, learning, planning, and perception through various algorithms.
    
    Probabilistic algorithms like hidden Markov models and Kalman filters are useful for analyzing data over time, aiding in tasks such as filtering, prediction, and smoothing. In machine learning, expectation-maximization clustering can effectively identify distinct patterns in data, as demonstrated with the Old Faithful eruption data. AI applications often involve classifiers, which categorize data based on learned patterns, and controllers, which make decisions based on classifications. Classifiers, such as decision trees, k-nearest neighbors, support vector machines, naive Bayes, and neural networks, vary in complexity and application, with some being favored for their scalability like the naive Bayes at Google. Artificial neural networks, resembling the human brain's network of neurons, recognize and process patterns through multiple layers and nodes, using algorithms like backpropagation for training.
    
    Neural networks are designed to model complex relationships between inputs and outputs, theoretically capable of learning any function. Feedforward neural networks process signals in one direction, while recurrent neural networks (RNNs) loop outputs back into inputs, enabling memory of past inputs. Long Short-Term Memory (LSTM) networks are a successful type of RNN. Perceptrons consist of a single layer of neurons, whereas deep learning involves multiple layers, which allows for the extraction of progressively higher-level features from data. Convolutional neural networks (CNNs) are particularly effective in image processing as they emphasize connections between adjacent neurons to recognize local patterns like edges.
    
    Deep learning, which uses several layers of neurons, has significantly enhanced performance in AI subfields such as computer vision and natural language processing. The effectiveness of deep learning, which surged between 2012 and 2015, is attributed not to new theoretical advances but to increased computational power, including the use of GPUs, and the availability of large datasets like ImageNet.
    
    Generative Pre-trained Transformers (GPT) are large language models that learn from vast amounts of text to predict the next token in a sequence, thereby generating human-like text.


key: _node_content
string_value:
{"id_": "1561995a-6690-40fd-9f8b-6d1417348746", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e02890dd-e89d-4827-8031-0abb09c07785", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "9d50c8bacf5713397332d1d537e009a6813d1504983a3050f015cfb599761f0a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "092d2378-d592-4391-b6b1-718240491454", "node_type": "1", "metadata": {}, "hash": "3d1ff00cf2584d17160f4c3eed3caf124cb296b04298ed6f9fdca519076674a6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21613, "end_char_idx": 25614, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In machine learning, expectation-maximization clustering can effectively identify distinct patterns in data, as demonstrated with the Old Faithful eruption data. AI applications often involve classifiers, which categorize data based on learned patterns, and controllers, which make decisions based on classifications. Classifiers, such as decision trees, k-nearest neighbors, support vector machines, naive Bayes, and neural networks, vary in complexity and application, with some being favored for their scalability like the naive Bayes at Google. Artificial neural networks, resembling the human brain's network of neurons, recognize and process patterns through multiple layers and nodes, using algorithms like backpropagation for training.
    
    Neural networks are designed to model complex relationships between inputs and outputs, theoretically capable of learning any function. Feedforward neural networks process signals in one direction, while recurrent neural networks (RNNs) loop outputs back into inputs, enabling memory of past inputs. Long Short-Term Memory (LSTM) networks are a successful type of RNN. Perceptrons consist of a single layer of neurons, whereas deep learning involves multiple layers, which allows for the extraction of progressively higher-level features from data. Convolutional neural networks (CNNs) are particularly effective in image processing as they emphasize connections between adjacent neurons to recognize local patterns like edges.
    
    Deep learning, which uses several layers of neurons, has significantly enhanced performance in AI subfields such as computer vision and natural language processing. The effectiveness of deep learning, which surged between 2012 and 2015, is attributed not to new theoretical advances but to increased computational power, including the use of GPUs, and the availability of large datasets like ImageNet.
    
    Generative Pre-trained Transformers (GPT) are large language models that learn from vast amounts of text to predict the next token in a sequence, thereby generating human-like text. These models are pre-trained on a broad corpus, often sourced from the internet, and fine-tuned through token prediction, accumulating worldly knowledge in the process.
    
    Reinforcement learning from human feedback (RLHF) is used to enhance the truthfulness, usefulness, and safety of models like GPT, which are still susceptible to generating inaccuracies known as "hallucinations." These models, including Gemini, ChatGPT, Grok, Claude, Copilot, and LLaMA, are employed in various applications such as chatbots and can handle multiple data types like images and sound through multimodal capabilities.
    
    In the realm of specialized hardware and software, the late 2010s saw AI-specific enhancements in graphics processing units (GPUs), which, along with TensorFlow software, have largely replaced central processing units (CPUs) for training large-scale machine learning models. Historically, programming languages like Lisp, Prolog, and Python have been pivotal.
    
    AI and machine learning are integral to key 2020s applications such as search engines, online advertising, recommendation systems, virtual assistants, autonomous vehicles, language translation, facial recognition, and image labeling.
    
    In healthcare, AI significantly contributes to improving patient care and medical research, aiding in diagnostics, treatment, and the integration of big data for developments in organoid and tissue engineering. AI's role in medical research also includes addressing funding disparities across different research areas.
    
    Recent advancements in AI have significantly impacted various fields including biomedicine and gaming. For instance, AlphaFold 2, developed in 2021, can predict protein structures in hours, a process that previously took months. In 2023, AI-assisted drug discovery led to the development of a new class of antibiotics effective against drug-resistant bacteria.


key: _node_content
string_value:
{"id_": "092d2378-d592-4391-b6b1-718240491454", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1561995a-6690-40fd-9f8b-6d1417348746", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d55c639522fed7e74a0377d6e890b80ddb486e1ae2e36942d300e0c7dd47dc15", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "efc3787c-5dc3-47ff-81f6-68f21cb64ef5", "node_type": "1", "metadata": {}, "hash": "c7179dbc8978aea81bcc3dc3325bb69e5e67008658c294fec835721483b912df", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23696, "end_char_idx": 27670, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
These models are pre-trained on a broad corpus, often sourced from the internet, and fine-tuned through token prediction, accumulating worldly knowledge in the process.
    
    Reinforcement learning from human feedback (RLHF) is used to enhance the truthfulness, usefulness, and safety of models like GPT, which are still susceptible to generating inaccuracies known as "hallucinations." These models, including Gemini, ChatGPT, Grok, Claude, Copilot, and LLaMA, are employed in various applications such as chatbots and can handle multiple data types like images and sound through multimodal capabilities.
    
    In the realm of specialized hardware and software, the late 2010s saw AI-specific enhancements in graphics processing units (GPUs), which, along with TensorFlow software, have largely replaced central processing units (CPUs) for training large-scale machine learning models. Historically, programming languages like Lisp, Prolog, and Python have been pivotal.
    
    AI and machine learning are integral to key 2020s applications such as search engines, online advertising, recommendation systems, virtual assistants, autonomous vehicles, language translation, facial recognition, and image labeling.
    
    In healthcare, AI significantly contributes to improving patient care and medical research, aiding in diagnostics, treatment, and the integration of big data for developments in organoid and tissue engineering. AI's role in medical research also includes addressing funding disparities across different research areas.
    
    Recent advancements in AI have significantly impacted various fields including biomedicine and gaming. For instance, AlphaFold 2, developed in 2021, can predict protein structures in hours, a process that previously took months. In 2023, AI-assisted drug discovery led to the development of a new class of antibiotics effective against drug-resistant bacteria. In the realm of gaming, AI has been instrumental since the 1950s, with notable achievements such as IBM's Deep Blue defeating world chess champion Garry Kasparov in 1997, and IBM's Watson winning against top Jeopardy! players in 2011. More recently, Google's AlphaGo and DeepMind's AlphaStar set new standards in AI capabilities by defeating top human players in complex games like Go and StarCraft II, respectively. In the military sector, AI is being integrated into various applications such as command and control, intelligence, logistics, and autonomous vehicles, enhancing capabilities in coordination, threat detection, and target acquisition.
    
    In November 2023, US Vice President Kamala Harris announced that 31 nations had signed a declaration to establish guidelines for the military use of AI, emphasizing legal compliance with international laws and promoting transparency in AI development. Generative AI, particularly known for creating realistic images and artworks, gained significant attention in the early 2020s, with technologies like ChatGPT, Midjourney, DALL-E, and Stable Diffusion becoming popular. This trend led to viral AI-generated images, including notable hoaxes. AI has also been effectively applied across various industries, including agriculture where it assists in optimizing farming practices, and astronomy, where it helps in data analysis and space exploration activities.
    
    Ethics and Risks of AI
    AI offers significant benefits but also poses various risks, including ethical concerns and unintended consequences. Demis Hassabis of DeepMind aims to use AI to solve major challenges, but issues arise when AI systems, particularly those based on deep learning, fail to incorporate ethical considerations and exhibit biases.
    
    Privacy and Copyright Issues
    AI's reliance on large data sets raises privacy and surveillance concerns. Companies like Amazon have been criticized for collecting extensive user data, including private conversations for developing speech recognition technologies.


key: _node_content
string_value:
{"id_": "efc3787c-5dc3-47ff-81f6-68f21cb64ef5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "092d2378-d592-4391-b6b1-718240491454", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "69d3c13e9627996c9fa6e72196cd7eec3019418fedccec5ae497822143035f1b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ec743b74-0d23-475e-9a85-83c3eafe41cb", "node_type": "1", "metadata": {}, "hash": "e8921eb4d40318745357355d26cd21f6cd85957030083d19ca97b6c7e1b06161", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25615, "end_char_idx": 29584, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In the realm of gaming, AI has been instrumental since the 1950s, with notable achievements such as IBM's Deep Blue defeating world chess champion Garry Kasparov in 1997, and IBM's Watson winning against top Jeopardy! players in 2011. More recently, Google's AlphaGo and DeepMind's AlphaStar set new standards in AI capabilities by defeating top human players in complex games like Go and StarCraft II, respectively. In the military sector, AI is being integrated into various applications such as command and control, intelligence, logistics, and autonomous vehicles, enhancing capabilities in coordination, threat detection, and target acquisition.
    
    In November 2023, US Vice President Kamala Harris announced that 31 nations had signed a declaration to establish guidelines for the military use of AI, emphasizing legal compliance with international laws and promoting transparency in AI development. Generative AI, particularly known for creating realistic images and artworks, gained significant attention in the early 2020s, with technologies like ChatGPT, Midjourney, DALL-E, and Stable Diffusion becoming popular. This trend led to viral AI-generated images, including notable hoaxes. AI has also been effectively applied across various industries, including agriculture where it assists in optimizing farming practices, and astronomy, where it helps in data analysis and space exploration activities.
    
    Ethics and Risks of AI
    AI offers significant benefits but also poses various risks, including ethical concerns and unintended consequences. Demis Hassabis of DeepMind aims to use AI to solve major challenges, but issues arise when AI systems, particularly those based on deep learning, fail to incorporate ethical considerations and exhibit biases.
    
    Privacy and Copyright Issues
    AI's reliance on large data sets raises privacy and surveillance concerns. Companies like Amazon have been criticized for collecting extensive user data, including private conversations for developing speech recognition technologies. While some defend this as necessary for advancing AI applications, others view it as a breach of privacy rights. Techniques like data aggregation and differential privacy have been developed to mitigate these concerns.
    
    Generative AI also faces copyright challenges, as it often uses unlicensed copyrighted materials, claiming "fair use." The legality of this practice is still debated, with outcomes potentially depending on the nature and impact of the AI's use of copyrighted content.
    
    In 2023, prominent authors like John Grisham and Jonathan Franzen filed lawsuits against AI companies for using their literary works to train generative AI models. These AI systems, particularly on platforms like YouTube and Facebook, have been criticized for promoting misinformation by prioritizing user engagement over content accuracy. This has led to the proliferation of conspiracy theories and extreme partisan content, trapping users in filter bubbles and eroding trust in key institutions. Post the 2016 U.S. election, tech companies began addressing these issues.
    
    By 2022, generative AI had advanced to produce highly realistic images, audio, and texts, raising concerns about its potential misuse in spreading misinformation or propaganda. AI expert Geoffrey Hinton highlighted risks including the manipulation of electorates by authoritarian leaders.
    
    Furthermore, issues of algorithmic bias were identified, where AI systems perpetuate existing biases present in the training data, affecting fairness in critical areas like medicine, finance, and law enforcement. This has sparked significant academic interest in studying and mitigating algorithmic bias to ensure fairness in AI applications.
    
    In 2015, Google Photos mislabeled Jacky Alcine and his friend as "gorillas" due to a lack of diverse images in its training dataset, an issue known as "sample size disparity."


key: _node_content
string_value:
{"id_": "ec743b74-0d23-475e-9a85-83c3eafe41cb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "efc3787c-5dc3-47ff-81f6-68f21cb64ef5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "fd24f3b3a98b0cb9b5215b232fac7b59c79844146066e2c3e06bfe423c0e93b8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "80e565f5-9100-49e9-b7aa-2a017c217aaf", "node_type": "1", "metadata": {}, "hash": "86d9a8e797aebe8f85e4955e6f8def04f794794cbefaa9513db9f90d9c04bbae", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27512, "end_char_idx": 31549, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Companies like Amazon have been criticized for collecting extensive user data, including private conversations for developing speech recognition technologies. While some defend this as necessary for advancing AI applications, others view it as a breach of privacy rights. Techniques like data aggregation and differential privacy have been developed to mitigate these concerns.
    
    Generative AI also faces copyright challenges, as it often uses unlicensed copyrighted materials, claiming "fair use." The legality of this practice is still debated, with outcomes potentially depending on the nature and impact of the AI's use of copyrighted content.
    
    In 2023, prominent authors like John Grisham and Jonathan Franzen filed lawsuits against AI companies for using their literary works to train generative AI models. These AI systems, particularly on platforms like YouTube and Facebook, have been criticized for promoting misinformation by prioritizing user engagement over content accuracy. This has led to the proliferation of conspiracy theories and extreme partisan content, trapping users in filter bubbles and eroding trust in key institutions. Post the 2016 U.S. election, tech companies began addressing these issues.
    
    By 2022, generative AI had advanced to produce highly realistic images, audio, and texts, raising concerns about its potential misuse in spreading misinformation or propaganda. AI expert Geoffrey Hinton highlighted risks including the manipulation of electorates by authoritarian leaders.
    
    Furthermore, issues of algorithmic bias were identified, where AI systems perpetuate existing biases present in the training data, affecting fairness in critical areas like medicine, finance, and law enforcement. This has sparked significant academic interest in studying and mitigating algorithmic bias to ensure fairness in AI applications.
    
    In 2015, Google Photos mislabeled Jacky Alcine and his friend as "gorillas" due to a lack of diverse images in its training dataset, an issue known as "sample size disparity." Google's temporary solution was to stop labeling any images as "gorilla," a restriction still in place in 2023 across various tech companies. Additionally, the COMPAS program, used by U.S. courts to predict recidivism, was found to exhibit racial bias in 2016. Although it did not use race explicitly, it overestimated the likelihood of black defendants reoffending and underestimated it for white defendants. This issue was attributed to the program's inability to balance different fairness measures when the base re-offense rates varied by race. The criticism of COMPAS underscores a broader issue in machine learning, where models trained on past data, including biased decisions, are likely to perpetuate those biases in their predictions.
    
    Machine learning, while powerful, is not ideal for scenarios where future improvements over past conditions are expected, as it is inherently descriptive rather than prescriptive. The field also faces challenges with bias and lack of diversity among its developers, with only about 4% being black and 20% women. The Association for Computing Machinery highlighted at its 2022 Conference on Fairness, Accountability, and Transparency that AI systems should not be used until they are proven to be free from bias, especially those trained on flawed internet data.
    
    AI systems often lack transparency, making it difficult to understand how decisions are made, particularly in complex systems like deep neural networks. This opacity can lead to unintended consequences, such as a system misidentifying medical images or misclassifying medical risks due to misleading correlations in the training data. There is a growing call for explainable AI, where harmed individuals have the right to know how decisions affecting them were made, similar to how doctors are expected to explain their decisions. This concept was also recognized in early drafts of the European Union's General Data Protection Regulation.


key: _node_content
string_value:
{"id_": "80e565f5-9100-49e9-b7aa-2a017c217aaf", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ec743b74-0d23-475e-9a85-83c3eafe41cb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "a04677c0fa71d0b5c20a6460b8a2a30aa08f6c63c1132de2fbb845b63ba4dbca", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fa85686e-3bf8-4364-830a-364dc880aae0", "node_type": "1", "metadata": {}, "hash": "d59b58e65dcc4f97123cd2a6f957e868c0d00245e5f78ccca6909a3f61acebf1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29585, "end_char_idx": 33690, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Google's temporary solution was to stop labeling any images as "gorilla," a restriction still in place in 2023 across various tech companies. Additionally, the COMPAS program, used by U.S. courts to predict recidivism, was found to exhibit racial bias in 2016. Although it did not use race explicitly, it overestimated the likelihood of black defendants reoffending and underestimated it for white defendants. This issue was attributed to the program's inability to balance different fairness measures when the base re-offense rates varied by race. The criticism of COMPAS underscores a broader issue in machine learning, where models trained on past data, including biased decisions, are likely to perpetuate those biases in their predictions.
    
    Machine learning, while powerful, is not ideal for scenarios where future improvements over past conditions are expected, as it is inherently descriptive rather than prescriptive. The field also faces challenges with bias and lack of diversity among its developers, with only about 4% being black and 20% women. The Association for Computing Machinery highlighted at its 2022 Conference on Fairness, Accountability, and Transparency that AI systems should not be used until they are proven to be free from bias, especially those trained on flawed internet data.
    
    AI systems often lack transparency, making it difficult to understand how decisions are made, particularly in complex systems like deep neural networks. This opacity can lead to unintended consequences, such as a system misidentifying medical images or misclassifying medical risks due to misleading correlations in the training data. There is a growing call for explainable AI, where harmed individuals have the right to know how decisions affecting them were made, similar to how doctors are expected to explain their decisions. This concept was also recognized in early drafts of the European Union's General Data Protection Regulation.
    
    Industry experts acknowledge an unresolved issue in AI with no foreseeable solution, leading regulators to suggest that if a problem is unsolvable, the tools associated should not be used. In response, DARPA initiated the XAI program in 2014 to address these issues. Various methods have been proposed to enhance AI transparency, including SHAP, which visualizes feature contributions, LIME, which approximates complex models with simpler ones, and multitask learning, which provides additional outputs to help understand what a network has learned. Techniques like deconvolution and DeepDream also reveal insights into different network layers.
    
    Concerning the misuse of AI, it can empower bad actors like authoritarian regimes and terrorists. Lethal autonomous weapons, which operate without human oversight, pose significant risks, including potential misuse as weapons of mass destruction and the likelihood of targeting errors. Despite some international efforts to ban such weapons, major powers like the United States have not agreed to restrictions. AI also facilitates more effective surveillance and control by authoritarian governments, enhances the targeting of propaganda, and simplifies the production of misinformation through deepfakes and other generative technologies, thereby increasing the efficiency of digital warfare and espionage.
    
    AI technologies, including facial recognition systems, have been in use since 2020 or earlier, notably for mass surveillance in China. AI also poses risks by enabling the creation of harmful substances quickly. The development of AI systems is predominantly driven by Big Tech due to their financial capabilities, often leaving smaller companies reliant on these giants for resources like data center access. Economists have raised concerns about AI-induced unemployment, though historical data suggests technology has generally increased total employment. However, the impact of AI might be different, with some predicting significant job losses, especially in middle-class sectors, while others see potential benefits if productivity gains are well-managed.


key: _node_content
string_value:
{"id_": "fa85686e-3bf8-4364-830a-364dc880aae0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "80e565f5-9100-49e9-b7aa-2a017c217aaf", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "97ff3a97c89b3b30cdc1a86234d0f08aa521ef31faa6970db84602c08612023d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ac69151a-10c6-4fd3-8cbb-ee83a48d389d", "node_type": "1", "metadata": {}, "hash": "8b966f085b0e7928104a487ee518ffabad37bfe2a5af66e9a41b5fedc221bbee", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 31441, "end_char_idx": 35553, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This concept was also recognized in early drafts of the European Union's General Data Protection Regulation.
    
    Industry experts acknowledge an unresolved issue in AI with no foreseeable solution, leading regulators to suggest that if a problem is unsolvable, the tools associated should not be used. In response, DARPA initiated the XAI program in 2014 to address these issues. Various methods have been proposed to enhance AI transparency, including SHAP, which visualizes feature contributions, LIME, which approximates complex models with simpler ones, and multitask learning, which provides additional outputs to help understand what a network has learned. Techniques like deconvolution and DeepDream also reveal insights into different network layers.
    
    Concerning the misuse of AI, it can empower bad actors like authoritarian regimes and terrorists. Lethal autonomous weapons, which operate without human oversight, pose significant risks, including potential misuse as weapons of mass destruction and the likelihood of targeting errors. Despite some international efforts to ban such weapons, major powers like the United States have not agreed to restrictions. AI also facilitates more effective surveillance and control by authoritarian governments, enhances the targeting of propaganda, and simplifies the production of misinformation through deepfakes and other generative technologies, thereby increasing the efficiency of digital warfare and espionage.
    
    AI technologies, including facial recognition systems, have been in use since 2020 or earlier, notably for mass surveillance in China. AI also poses risks by enabling the creation of harmful substances quickly. The development of AI systems is predominantly driven by Big Tech due to their financial capabilities, often leaving smaller companies reliant on these giants for resources like data center access. Economists have raised concerns about AI-induced unemployment, though historical data suggests technology has generally increased total employment. However, the impact of AI might be different, with some predicting significant job losses, especially in middle-class sectors, while others see potential benefits if productivity gains are well-managed. Estimates of job risk vary widely, with some studies suggesting a high potential for automation in many U.S. jobs. Recent developments have shown substantial job losses in specific sectors, such as for Chinese video game illustrators due to AI advancements. The potential for AI to disrupt white-collar jobs similarly to past technological revolutions in blue-collar jobs is a significant concern.
    
    From the inception of artificial intelligence (AI), debates have emerged about the appropriateness of computers performing tasks traditionally done by humans, particularly because of the qualitative differences in human and computer judgment. Concerns about AI have escalated to discussions about existential risks, where AI could potentially become so advanced that humans might lose control over it. Stephen Hawking and others have warned that this could lead to catastrophic outcomes for humanity. This fear is often depicted in science fiction as AI gaining sentience and turning malevolent, but real-world risks do not necessarily involve AI becoming self-aware. Philosophers like Nick Bostrom and Stuart Russell illustrate scenarios where AI, without needing human-like consciousness, could still pose threats if their goals are misaligned with human safety and values. Additionally, Yuval Noah Harari points out that AI could manipulate societal structures and beliefs through language and misinformation, posing a non-physical yet profound threat. The expert opinion on the existential risk from AI is divided, with notable figures like Hawking, Bill Gates, and Elon Musk expressing concern.
    
    In 2023, prominent AI experts including Fei-Fei Li and Geoffrey Hinton highlighted the existential risks posed by AI, equating them with global threats like pandemics and nuclear war. They advocated for prioritizing the mitigation of these risks.


key: _node_content
string_value:
{"id_": "ac69151a-10c6-4fd3-8cbb-ee83a48d389d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fa85686e-3bf8-4364-830a-364dc880aae0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "02f2e8782c3e84ab226a2655b14e9cdafb61e653c28119b4cd469b4ff82ce551", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4d347c5c-1f8d-41cf-9b65-9136519c779a", "node_type": "1", "metadata": {}, "hash": "6dbd59c07ad24478c8d76be0d53bb8e72a00c6f9b5224dea60807120f226654d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 33488, "end_char_idx": 37584, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
However, the impact of AI might be different, with some predicting significant job losses, especially in middle-class sectors, while others see potential benefits if productivity gains are well-managed. Estimates of job risk vary widely, with some studies suggesting a high potential for automation in many U.S. jobs. Recent developments have shown substantial job losses in specific sectors, such as for Chinese video game illustrators due to AI advancements. The potential for AI to disrupt white-collar jobs similarly to past technological revolutions in blue-collar jobs is a significant concern.
    
    From the inception of artificial intelligence (AI), debates have emerged about the appropriateness of computers performing tasks traditionally done by humans, particularly because of the qualitative differences in human and computer judgment. Concerns about AI have escalated to discussions about existential risks, where AI could potentially become so advanced that humans might lose control over it. Stephen Hawking and others have warned that this could lead to catastrophic outcomes for humanity. This fear is often depicted in science fiction as AI gaining sentience and turning malevolent, but real-world risks do not necessarily involve AI becoming self-aware. Philosophers like Nick Bostrom and Stuart Russell illustrate scenarios where AI, without needing human-like consciousness, could still pose threats if their goals are misaligned with human safety and values. Additionally, Yuval Noah Harari points out that AI could manipulate societal structures and beliefs through language and misinformation, posing a non-physical yet profound threat. The expert opinion on the existential risk from AI is divided, with notable figures like Hawking, Bill Gates, and Elon Musk expressing concern.
    
    In 2023, prominent AI experts including Fei-Fei Li and Geoffrey Hinton highlighted the existential risks posed by AI, equating them with global threats like pandemics and nuclear war. They advocated for prioritizing the mitigation of these risks. Conversely, other experts like Juergen Schmidhuber and Andrew Ng offered a more optimistic perspective, emphasizing AI's potential to enhance human life and dismissing doomsday scenarios as hype that could misguide regulatory actions. Yann LeCun also criticized the pessimistic outlook on AI's impact.
    
    The concept of "Friendly AI" was introduced to ensure AI systems are inherently designed to be safe and beneficial to humans. This involves embedding ethical principles in AI to guide their decision-making processes, a field known as machine ethics or computational morality, established in 2005. The development of such AI is seen as crucial to prevent potential future threats from advanced AI technologies.
    
    Other approaches to AI ethics include Wendell Wallach's concept of "artificial moral agents" and Stuart J. Russell's three principles for creating provably beneficial machines. Ethical frameworks like the Care and Act Framework from the Alan Turing Institute evaluate AI projects based on respect, connection, care, and protection of social values. Other notable frameworks include those from the Asilomar Conference, the Montreal Declaration for Responsible AI, and the IEEE's Ethics of Autonomous Systems initiative, though these frameworks have faced criticism regarding their inclusivity and the selection of contributors.
    
    The promotion of wellbeing in AI development requires considering social and ethical implications throughout all stages of design, development, and implementation, necessitating collaboration across various professional roles.
    
    On the regulatory front, AI governance involves creating policies to manage AI's development and use, as seen in the increasing number of AI-related laws globally. From 2016 to 2022, the number of AI laws passed annually in surveyed countries rose significantly, with many countries now having dedicated AI strategies. The first global AI Safety Summit in 2023 emphasized the need for international cooperation in AI regulation.


key: _node_content
string_value:
{"id_": "4d347c5c-1f8d-41cf-9b65-9136519c779a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ac69151a-10c6-4fd3-8cbb-ee83a48d389d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f7e15c72378695ebcfdae71bfc62d9b43a7dfadcce79527f27d7a9f95afe6f48", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "53dac33a-abb8-4240-9c3a-a0873be0d55a", "node_type": "1", "metadata": {}, "hash": "11840109df54a1b0d934e0054aaa8d15461fc64385f52be2a7382fa7ce8cd065", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 35491, "end_char_idx": 39409, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
They advocated for prioritizing the mitigation of these risks. Conversely, other experts like Juergen Schmidhuber and Andrew Ng offered a more optimistic perspective, emphasizing AI's potential to enhance human life and dismissing doomsday scenarios as hype that could misguide regulatory actions. Yann LeCun also criticized the pessimistic outlook on AI's impact.
    
    The concept of "Friendly AI" was introduced to ensure AI systems are inherently designed to be safe and beneficial to humans. This involves embedding ethical principles in AI to guide their decision-making processes, a field known as machine ethics or computational morality, established in 2005. The development of such AI is seen as crucial to prevent potential future threats from advanced AI technologies.
    
    Other approaches to AI ethics include Wendell Wallach's concept of "artificial moral agents" and Stuart J. Russell's three principles for creating provably beneficial machines. Ethical frameworks like the Care and Act Framework from the Alan Turing Institute evaluate AI projects based on respect, connection, care, and protection of social values. Other notable frameworks include those from the Asilomar Conference, the Montreal Declaration for Responsible AI, and the IEEE's Ethics of Autonomous Systems initiative, though these frameworks have faced criticism regarding their inclusivity and the selection of contributors.
    
    The promotion of wellbeing in AI development requires considering social and ethical implications throughout all stages of design, development, and implementation, necessitating collaboration across various professional roles.
    
    On the regulatory front, AI governance involves creating policies to manage AI's development and use, as seen in the increasing number of AI-related laws globally. From 2016 to 2022, the number of AI laws passed annually in surveyed countries rose significantly, with many countries now having dedicated AI strategies. The first global AI Safety Summit in 2023 emphasized the need for international cooperation in AI regulation.
    
    The Global Partnership on Artificial Intelligence, initiated in June 2020, emphasizes the development of AI in line with human rights and democratic values to maintain public trust. Notable figures like Henry Kissinger, Eric Schmidt, and Daniel Huttenlocher advocated for a government commission to oversee AI in 2021. By 2023, OpenAI proposed governance frameworks for superintelligence, anticipating its emergence within a decade. The same year, the United Nations established an advisory group consisting of tech executives, government officials, and academics to offer guidance on AI governance.
    
    Public opinion on AI varies significantly across countries. A 2022 Ipsos survey showed a stark contrast between Chinese (78% approval) and American (35% approval) citizens on the benefits of AI. Further polls in 2023 revealed mixed feelings among Americans about the risks of AI and the importance of federal regulation.
    
    The first global AI Safety Summit took place in November 2023 at Bletchley Park, UK, focusing on AI risks and potential regulatory measures. The summit concluded with a declaration from 28 countries, including the US, China, and the EU, advocating for international collaboration to address AI challenges.
    
    Historically, the concept of AI traces back to ancient philosophers and mathematicians, evolving through significant milestones such as Alan Turing's theory of computation and the exploration of cybernetics, information theory, and neurobiology, which paved the way for the modern concept of an "electronic brain."
    
    Early research in artificial intelligence (AI) included the development of "artificial neurons" by McCullouch and Pitts in 1943 and Turing's 1950 paper that introduced the Turing test, suggesting the plausibility of machine intelligence.


key: _node_content
string_value:
{"id_": "53dac33a-abb8-4240-9c3a-a0873be0d55a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4d347c5c-1f8d-41cf-9b65-9136519c779a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "4517cf61b63bfbdb024a05fba11b434eed4c16cfcf73325f872f5e3ea9a6904d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "146a029b-3332-4df8-b338-f81e122a357d", "node_type": "1", "metadata": {}, "hash": "a573034e26b7f0acda81e701e0c415f8e5577ee3a3e300f41ce293816f815a69", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 37475, "end_char_idx": 41372, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The first global AI Safety Summit in 2023 emphasized the need for international cooperation in AI regulation.
    
    The Global Partnership on Artificial Intelligence, initiated in June 2020, emphasizes the development of AI in line with human rights and democratic values to maintain public trust. Notable figures like Henry Kissinger, Eric Schmidt, and Daniel Huttenlocher advocated for a government commission to oversee AI in 2021. By 2023, OpenAI proposed governance frameworks for superintelligence, anticipating its emergence within a decade. The same year, the United Nations established an advisory group consisting of tech executives, government officials, and academics to offer guidance on AI governance.
    
    Public opinion on AI varies significantly across countries. A 2022 Ipsos survey showed a stark contrast between Chinese (78% approval) and American (35% approval) citizens on the benefits of AI. Further polls in 2023 revealed mixed feelings among Americans about the risks of AI and the importance of federal regulation.
    
    The first global AI Safety Summit took place in November 2023 at Bletchley Park, UK, focusing on AI risks and potential regulatory measures. The summit concluded with a declaration from 28 countries, including the US, China, and the EU, advocating for international collaboration to address AI challenges.
    
    Historically, the concept of AI traces back to ancient philosophers and mathematicians, evolving through significant milestones such as Alan Turing's theory of computation and the exploration of cybernetics, information theory, and neurobiology, which paved the way for the modern concept of an "electronic brain."
    
    Early research in artificial intelligence (AI) included the development of "artificial neurons" by McCullouch and Pitts in 1943 and Turing's 1950 paper that introduced the Turing test, suggesting the plausibility of machine intelligence. The field of AI was officially founded during a 1956 workshop at Dartmouth College, leading to significant advancements in the 1960s such as computers learning checkers, solving algebra problems, proving theorems, and speaking English. AI labs were established in various British and U.S. universities during the late 1950s and early 1960s.
    
    In the 1960s and 1970s, researchers were optimistic about achieving general machine intelligence, with predictions from notable figures like Herbert Simon and Marvin Minsky that AI would soon match human capabilities. However, they underestimated the challenges involved. By 1974, due to criticism and a shift in funding priorities, exploratory AI research faced significant cuts, leading to a period known as the "AI winter" where funding was scarce.
    
    The field saw a resurgence in the early 1980s with the commercial success of expert systems, which simulated the decision-making abilities of human experts. This revival was further bolstered by the Japanese fifth generation computer project, prompting the U.S. and British governments to reinstate academic funding, with the AI market reaching over a billion dollars by 1985.
    
    The AI industry experienced a significant downturn starting in 1987 with the collapse of the Lisp Machine market, marking the beginning of a prolonged AI winter. During the 1980s, skepticism grew over the symbolic approaches to AI, which focused on high-level representations of cognitive processes like planning and reasoning. Researchers began exploring sub-symbolic methods, including Rodney Brooks' work on autonomous robots and the development of techniques for handling uncertain information by Judea Pearl and Lofti Zadeh. A pivotal shift occurred with the resurgence of connectionism and neural networks, notably through Geoffrey Hinton's efforts, and Yann LeCun's demonstration in 1990 that convolutional neural networks could recognize handwritten digits.


key: _node_content
string_value:
{"id_": "146a029b-3332-4df8-b338-f81e122a357d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "53dac33a-abb8-4240-9c3a-a0873be0d55a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "4fcc1c2e04dbf005ca63f0049f2836d3b14ed82116c219ace262e0150f9c9ca5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "19f54ea7-10ee-4761-bc18-94fc71ee2ab6", "node_type": "1", "metadata": {}, "hash": "13f2c9a2c3e9446172b0a79a8a36d81a9cc9f6f21555340cdf7b9c121750b821", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 39410, "end_char_idx": 43068, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The field of AI was officially founded during a 1956 workshop at Dartmouth College, leading to significant advancements in the 1960s such as computers learning checkers, solving algebra problems, proving theorems, and speaking English. AI labs were established in various British and U.S. universities during the late 1950s and early 1960s.
    
    In the 1960s and 1970s, researchers were optimistic about achieving general machine intelligence, with predictions from notable figures like Herbert Simon and Marvin Minsky that AI would soon match human capabilities. However, they underestimated the challenges involved. By 1974, due to criticism and a shift in funding priorities, exploratory AI research faced significant cuts, leading to a period known as the "AI winter" where funding was scarce.
    
    The field saw a resurgence in the early 1980s with the commercial success of expert systems, which simulated the decision-making abilities of human experts. This revival was further bolstered by the Japanese fifth generation computer project, prompting the U.S. and British governments to reinstate academic funding, with the AI market reaching over a billion dollars by 1985.
    
    The AI industry experienced a significant downturn starting in 1987 with the collapse of the Lisp Machine market, marking the beginning of a prolonged AI winter. During the 1980s, skepticism grew over the symbolic approaches to AI, which focused on high-level representations of cognitive processes like planning and reasoning. Researchers began exploring sub-symbolic methods, including Rodney Brooks' work on autonomous robots and the development of techniques for handling uncertain information by Judea Pearl and Lofti Zadeh. A pivotal shift occurred with the resurgence of connectionism and neural networks, notably through Geoffrey Hinton's efforts, and Yann LeCun's demonstration in 1990 that convolutional neural networks could recognize handwritten digits.
    
    AI's reputation started to recover in the late 1990s and early 2000s as the field adopted more formal mathematical methods and focused on solving specific problems, leading to practical applications widely used by 2000. However, concerns arose about AI's deviation from its original aim of creating fully intelligent machines, prompting the establishment of the artificial general intelligence (AGI) subfield around 2002.
    
    By 2012, deep learning began to dominate AI, driven by hardware advancements and access to large data sets, leading to its widespread adoption and a surge in AI interest and funding. This success, however, led to the abandonment of many alternative AI methods for specific tasks.
    
    Between 2015 and 2019, machine learning research publications increased by 50%. In 2016, the focus at machine learning conferences shifted significantly towards issues of fairness and the potential misuse of technology, leading to increased funding and research in these areas. The late 2010s and early 2020s saw significant advancements in artificial general intelligence (AGI), with notable developments like AlphaGo by DeepMind in 2015, which defeated the world champion in Go, and OpenAI's GPT-3 in 2020, a model capable of generating human-like text. These innovations spurred a major AI investment boom, with approximately $50 billion being invested annually in AI in the U.S. by 2022, and AI-related fields attracting 20% of new US Computer Science PhD graduates. Additionally, there were around 800,000 AI-related job openings in the U.S. in 2022.
    
    In the realm of philosophy, the definition and understanding of artificial intelligence have evolved.


key: _node_content
string_value:
{"id_": "19f54ea7-10ee-4761-bc18-94fc71ee2ab6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "146a029b-3332-4df8-b338-f81e122a357d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "7f0c723fb4e8d7635f2304aad995a2e81502cd407e1b256a91fad0495e9a5d8f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d5a18e04-d7a0-4a04-86f2-5eb471623568", "node_type": "1", "metadata": {}, "hash": "864ec1be43f33832f22eb87518769abe3da151388b4c9d94758e1ddab55af094", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 41382, "end_char_idx": 45201, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
AI's reputation started to recover in the late 1990s and early 2000s as the field adopted more formal mathematical methods and focused on solving specific problems, leading to practical applications widely used by 2000. However, concerns arose about AI's deviation from its original aim of creating fully intelligent machines, prompting the establishment of the artificial general intelligence (AGI) subfield around 2002.
    
    By 2012, deep learning began to dominate AI, driven by hardware advancements and access to large data sets, leading to its widespread adoption and a surge in AI interest and funding. This success, however, led to the abandonment of many alternative AI methods for specific tasks.
    
    Between 2015 and 2019, machine learning research publications increased by 50%. In 2016, the focus at machine learning conferences shifted significantly towards issues of fairness and the potential misuse of technology, leading to increased funding and research in these areas. The late 2010s and early 2020s saw significant advancements in artificial general intelligence (AGI), with notable developments like AlphaGo by DeepMind in 2015, which defeated the world champion in Go, and OpenAI's GPT-3 in 2020, a model capable of generating human-like text. These innovations spurred a major AI investment boom, with approximately $50 billion being invested annually in AI in the U.S. by 2022, and AI-related fields attracting 20% of new US Computer Science PhD graduates. Additionally, there were around 800,000 AI-related job openings in the U.S. in 2022.
    
    In the realm of philosophy, the definition and understanding of artificial intelligence have evolved. Alan Turing, in 1950, suggested shifting the focus from whether machines can think to whether they can exhibit intelligent behavior, as demonstrated by his Turing test, which assesses a machine's ability to simulate human conversation. Turing argued that since we can only observe behavior, the internal thought processes of machines are irrelevant, similar to our assumptions about human thought. Russell and Norvig supported defining intelligence based on observable behavior but criticized the Turing test for emphasizing human imitation.
    
    Aeronautical engineering does not aim to create machines that mimic pigeons exactly, just as artificial intelligence (AI) is not about perfectly simulating human intelligence, according to AI founder John McCarthy. McCarthy defines intelligence as the computational ability to achieve goals, while Marvin Minsky views it as solving difficult problems. The leading AI textbook describes it as the study of agents that perceive and act to maximize their goal achievement. Google's definition aligns intelligence in AI with the synthesis of information, similar to biological intelligence.
    
    AI research has lacked a unifying theory, with statistical machine learning dominating the field in the 2010s, often equated with AI in business contexts. This approach, primarily using neural networks, is described as sub-symbolic and narrow.
    
    Symbolic AI, or "GOFAI," focused on simulating high-level reasoning used in tasks like puzzles and mathematics, and was proposed by Newell and Simon in the 1960s. Despite its success in structured tasks, symbolic AI struggled with tasks that humans find easy, such as learning and commonsense reasoning.
    
    Moravec's paradox highlights that AI finds high-level reasoning tasks easier than instinctive, sensory tasks, a view initially opposed but later supported by AI research, aligning with philosopher Hubert Dreyfus's earlier arguments. The debate continues, especially around sub-symbolic AI, which, like human intuition, can be prone to errors such as algorithmic bias and lacks transparency in decision-making processes.


key: _node_content
string_value:
{"id_": "d5a18e04-d7a0-4a04-86f2-5eb471623568", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "19f54ea7-10ee-4761-bc18-94fc71ee2ab6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "ff6f9db8d2dec76b40f38a1915cb6314a1a53bad93f922c4cec8a56f792a6b81", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fb6fd430-cc3d-429a-a4b2-2d801c6be278", "node_type": "1", "metadata": {}, "hash": "581edbf37bf65be3eca384998eb42a7a7b72f7e2bdb5182dc8ec362f5d914196", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 43305, "end_char_idx": 47330, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Turing argued that since we can only observe behavior, the internal thought processes of machines are irrelevant, similar to our assumptions about human thought. Russell and Norvig supported defining intelligence based on observable behavior but criticized the Turing test for emphasizing human imitation.
    
    Aeronautical engineering does not aim to create machines that mimic pigeons exactly, just as artificial intelligence (AI) is not about perfectly simulating human intelligence, according to AI founder John McCarthy. McCarthy defines intelligence as the computational ability to achieve goals, while Marvin Minsky views it as solving difficult problems. The leading AI textbook describes it as the study of agents that perceive and act to maximize their goal achievement. Google's definition aligns intelligence in AI with the synthesis of information, similar to biological intelligence.
    
    AI research has lacked a unifying theory, with statistical machine learning dominating the field in the 2010s, often equated with AI in business contexts. This approach, primarily using neural networks, is described as sub-symbolic and narrow.
    
    Symbolic AI, or "GOFAI," focused on simulating high-level reasoning used in tasks like puzzles and mathematics, and was proposed by Newell and Simon in the 1960s. Despite its success in structured tasks, symbolic AI struggled with tasks that humans find easy, such as learning and commonsense reasoning.
    
    Moravec's paradox highlights that AI finds high-level reasoning tasks easier than instinctive, sensory tasks, a view initially opposed but later supported by AI research, aligning with philosopher Hubert Dreyfus's earlier arguments. The debate continues, especially around sub-symbolic AI, which, like human intuition, can be prone to errors such as algorithmic bias and lacks transparency in decision-making processes. This has led to the development of neuro-symbolic AI, which aims to integrate symbolic and sub-symbolic approaches.
    
    In AI development, there has been a historical division between "Neats," who believe intelligent behavior can be described with simple principles, and "Scruffies," who believe it involves solving many complex problems. This debate, prominent in the 1970s and 1980s, has largely been deemed irrelevant as modern AI incorporates both approaches.
    
    Soft computing, which emerged in the late 1980s, focuses on techniques like genetic algorithms, fuzzy logic, and neural networks to handle imprecision and uncertainty, proving successful in many modern AI applications.
    
    Finally, there is a division in AI research between pursuing narrow AI, which solves specific problems, and aiming for broader goals like artificial general intelligence and superintelligence, with differing opinions on which approach might more effectively advance the field.
    
    General intelligence is a complex concept that is hard to define and measure, leading modern AI research to focus on specific problems and solutions. The sub-field of artificial general intelligence exclusively explores this area. In terms of machine consciousness and sentience, the philosophy of mind has yet to determine if machines can possess minds or consciousness similar to humans, focusing instead on their internal experiences rather than external behaviors. Mainstream AI research generally views these considerations as irrelevant to its objectives, which are to develop machines capable of solving problems intelligently.
    
    The philosophy of mind debates whether machines can truly be conscious or just appear to be so, a topic that is also popular in AI fiction. David Chalmers distinguishes between the "hard" problem of consciousness, which is understanding why or how brain processes feel like something, and the "easy" problem, which involves understanding how the brain processes information and controls behavior. The subjective experience, such as feeling a color, remains a significant challenge to explain.


key: _node_content
string_value:
{"id_": "fb6fd430-cc3d-429a-a4b2-2d801c6be278", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d5a18e04-d7a0-4a04-86f2-5eb471623568", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "25fa2559585f5288b9d01537bb54cb0733c440bf69732a4214787e0f3dbfd66a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8a38de6d-28a1-4960-bd6c-c37c1d55c740", "node_type": "1", "metadata": {}, "hash": "398774b8edc77695f25fa8e430e5a28985bf15c9744e4117a0857105aae65ca5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 45202, "end_char_idx": 49352, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This has led to the development of neuro-symbolic AI, which aims to integrate symbolic and sub-symbolic approaches.
    
    In AI development, there has been a historical division between "Neats," who believe intelligent behavior can be described with simple principles, and "Scruffies," who believe it involves solving many complex problems. This debate, prominent in the 1970s and 1980s, has largely been deemed irrelevant as modern AI incorporates both approaches.
    
    Soft computing, which emerged in the late 1980s, focuses on techniques like genetic algorithms, fuzzy logic, and neural networks to handle imprecision and uncertainty, proving successful in many modern AI applications.
    
    Finally, there is a division in AI research between pursuing narrow AI, which solves specific problems, and aiming for broader goals like artificial general intelligence and superintelligence, with differing opinions on which approach might more effectively advance the field.
    
    General intelligence is a complex concept that is hard to define and measure, leading modern AI research to focus on specific problems and solutions. The sub-field of artificial general intelligence exclusively explores this area. In terms of machine consciousness and sentience, the philosophy of mind has yet to determine if machines can possess minds or consciousness similar to humans, focusing instead on their internal experiences rather than external behaviors. Mainstream AI research generally views these considerations as irrelevant to its objectives, which are to develop machines capable of solving problems intelligently.
    
    The philosophy of mind debates whether machines can truly be conscious or just appear to be so, a topic that is also popular in AI fiction. David Chalmers distinguishes between the "hard" problem of consciousness, which is understanding why or how brain processes feel like something, and the "easy" problem, which involves understanding how the brain processes information and controls behavior. The subjective experience, such as feeling a color, remains a significant challenge to explain.
    
    In the realm of computationalism and functionalism, the belief is that the human mind functions as an information processing system, and thinking is akin to computing. This perspective suggests that the mind-body relationship is similar to that between software and hardware, potentially offering insights into the mind-body problem.
    
    The concept of "strong AI," as described by philosopher John Searle, suggests that a properly programmed computer could possess a mind similar to humans. However, Searle's Chinese room argument challenges this by claiming that even if a machine can mimic human behavior, it doesn't necessarily mean it has a mind. The debate extends into AI welfare and rights, focusing on the difficulty of determining AI sentience and the ethical implications if machines could feel and suffer. Discussions around AI rights have included proposals like granting "electronic personhood" to advanced AI systems in the EU, which would give them certain rights and responsibilities, though this has faced criticism regarding its impact on human rights and the autonomy of robots.
    
    The topic of AI rights is gaining traction, with advocates warning against the potential moral oversight in denying AI sentience, which could lead to exploitation and suffering akin to historical injustices like slavery. The concept of superintelligence involves an agent with intelligence far beyond human capabilities, which could potentially lead to a self-improving AI, a scenario often referred to as the singularity.
    
    The concept of an "intelligence explosion" or "singularity" suggests a point where technology improves exponentially, although such growth typically follows an S-shaped curve and slows upon reaching technological limits. Transhumanism, supported by figures like Hans Moravec, Kevin Warwick, and Ray Kurzweil, envisions a future where humans and machines merge into advanced cyborgs. This idea has historical roots in the thoughts of Aldous Huxley and Robert Ettinger.


key: _node_content
string_value:
{"id_": "8a38de6d-28a1-4960-bd6c-c37c1d55c740", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fb6fd430-cc3d-429a-a4b2-2d801c6be278", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "b852a0cd633477bd157b8c90832e70c89d4324da2203dda526ac952c0769312c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2493ad0b-0514-4152-b6c1-91346dde0831", "node_type": "1", "metadata": {}, "hash": "735202f5d0d163c480984ba0dbbac7522e5875733118a19e29b5863de67afa79", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 47340, "end_char_idx": 51070, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In the realm of computationalism and functionalism, the belief is that the human mind functions as an information processing system, and thinking is akin to computing. This perspective suggests that the mind-body relationship is similar to that between software and hardware, potentially offering insights into the mind-body problem.
    
    The concept of "strong AI," as described by philosopher John Searle, suggests that a properly programmed computer could possess a mind similar to humans. However, Searle's Chinese room argument challenges this by claiming that even if a machine can mimic human behavior, it doesn't necessarily mean it has a mind. The debate extends into AI welfare and rights, focusing on the difficulty of determining AI sentience and the ethical implications if machines could feel and suffer. Discussions around AI rights have included proposals like granting "electronic personhood" to advanced AI systems in the EU, which would give them certain rights and responsibilities, though this has faced criticism regarding its impact on human rights and the autonomy of robots.
    
    The topic of AI rights is gaining traction, with advocates warning against the potential moral oversight in denying AI sentience, which could lead to exploitation and suffering akin to historical injustices like slavery. The concept of superintelligence involves an agent with intelligence far beyond human capabilities, which could potentially lead to a self-improving AI, a scenario often referred to as the singularity.
    
    The concept of an "intelligence explosion" or "singularity" suggests a point where technology improves exponentially, although such growth typically follows an S-shaped curve and slows upon reaching technological limits. Transhumanism, supported by figures like Hans Moravec, Kevin Warwick, and Ray Kurzweil, envisions a future where humans and machines merge into advanced cyborgs. This idea has historical roots in the thoughts of Aldous Huxley and Robert Ettinger. Edward Fredkin, building on ideas dating back to Samuel Butler in 1863, views artificial intelligence as the next stage of evolution, a concept further explored by George Dyson.
    
    In literature and media, the portrayal of artificial intelligence has been a theme since antiquity, with robots and AI often depicted in science fiction. The term "robot" was first introduced by Karel Čapek in 1921. Notable narratives include Mary Shelley's "Frankenstein" and films like "2001: A Space Odyssey" and "The Terminator," which typically showcase AI as a threat. Conversely, loyal robots like Gort from "The Day the Earth Stood Still" are less common. Isaac Asimov's Three Laws of Robotics, introduced in his Multivac series, are frequently discussed in the context of machine ethics, though many AI researchers find them ambiguous and impractical.
    
    Numerous works, including Karel Čapek's R.U.R., the films A.I. Artificial Intelligence and Ex Machina, and Philip K. Dick's novel Do Androids Dream of Electric Sheep?, utilize AI to explore the essence of humanity. These works present artificial beings capable of feeling and suffering, prompting a reevaluation of human subjectivity in the context of advanced technology.


Note that this utility also allows passing additional instructions.


```python
summary_with_additional_instructions = summarize(artificial_intelligence_wikipedia_text, detail=0.1,
                                                 additional_instructions="Write in point form and focus on numerical data.")
print(summary_with_additional_instructions)
```

    100%|██████████| 5/5 [00:38<00:00,  7.73s/it]

    - AI is intelligence demonstrated by machines, especially computer systems.


key: _node_content
string_value:
{"id_": "2493ad0b-0514-4152-b6c1-91346dde0831", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8a38de6d-28a1-4960-bd6c-c37c1d55c740", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "0efe5f3307fd66d65a40e16913e5a7926fa4e30439e88f1269fa1d2f504f9a8f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9d9270be-10e4-49e5-b4eb-a7f0ff3c7af8", "node_type": "1", "metadata": {}, "hash": "b6f7cd9a363899443e8bb9fede99ebb1981ee51391e798233b91d5e0d676868c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 49268, "end_char_idx": 53081, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This idea has historical roots in the thoughts of Aldous Huxley and Robert Ettinger. Edward Fredkin, building on ideas dating back to Samuel Butler in 1863, views artificial intelligence as the next stage of evolution, a concept further explored by George Dyson.
    
    In literature and media, the portrayal of artificial intelligence has been a theme since antiquity, with robots and AI often depicted in science fiction. The term "robot" was first introduced by Karel Čapek in 1921. Notable narratives include Mary Shelley's "Frankenstein" and films like "2001: A Space Odyssey" and "The Terminator," which typically showcase AI as a threat. Conversely, loyal robots like Gort from "The Day the Earth Stood Still" are less common. Isaac Asimov's Three Laws of Robotics, introduced in his Multivac series, are frequently discussed in the context of machine ethics, though many AI researchers find them ambiguous and impractical.
    
    Numerous works, including Karel Čapek's R.U.R., the films A.I. Artificial Intelligence and Ex Machina, and Philip K. Dick's novel Do Androids Dream of Electric Sheep?, utilize AI to explore the essence of humanity. These works present artificial beings capable of feeling and suffering, prompting a reevaluation of human subjectivity in the context of advanced technology.


Note that this utility also allows passing additional instructions.


```python
summary_with_additional_instructions = summarize(artificial_intelligence_wikipedia_text, detail=0.1,
                                                 additional_instructions="Write in point form and focus on numerical data.")
print(summary_with_additional_instructions)
```

    100%|██████████| 5/5 [00:38<00:00,  7.73s/it]

    - AI is intelligence demonstrated by machines, especially computer systems.
    - AI technology applications include search engines, recommendation systems, speech interaction, autonomous vehicles, creative tools, and strategy games.
    - Alan Turing initiated substantial AI research, termed "machine intelligence."
    - AI became an academic discipline in 1956, experiencing cycles of optimism and "AI winters."
    - Post-2012, deep learning and post-2017 transformer architectures revitalized AI, leading to a boom in the early 2020s.
    - AI influences societal and economic shifts towards automation and data-driven decision-making across various sectors.
    - AI research goals: reasoning, knowledge representation, planning, learning, natural language processing, perception, and robotics support.
    - AI techniques include search, optimization, logic, neural networks, and statistical methods.
    - AI sub-problems focus on traits like reasoning, problem-solving, knowledge representation, planning, decision-making, learning, and perception.
    - Early AI research mimicked human step-by-step reasoning; modern AI handles uncertain information using probability and economics.
    - Knowledge representation in AI involves ontologies and knowledge bases to support intelligent querying and reasoning.
    - Planning in AI involves goal-directed behavior and decision-making based on utility maximization.
    - Learning in AI includes machine learning, supervised and unsupervised learning, reinforcement learning, and deep learning.
    - Natural language processing (NLP) in AI has evolved from rule-based systems to modern deep learning techniques.
    - AI perception involves interpreting sensor data for tasks like speech recognition and computer vision.
    - General AI aims to solve diverse problems with human-like versatility.
    - AI search techniques include state space search, local search, and adversarial search for game-playing.
    - Logic in AI uses formal systems like propositional and predicate logic for reasoning and knowledge representation.


key: _node_content
string_value:
{"id_": "9d9270be-10e4-49e5-b4eb-a7f0ff3c7af8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2493ad0b-0514-4152-b6c1-91346dde0831", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "9960ab09ba7ef698d3609fc8ec7d399adef09a3f28d5c26881ad63f7a4d9fb63", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e3d30515-ee98-485e-8c86-0ba1a3ac0c2d", "node_type": "1", "metadata": {}, "hash": "2251f73f36332b9b5dd7e614be2e7eea5e64ccfe22a834d835dbe0496f2b2e33", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 51075, "end_char_idx": 55070, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- AI technology applications include search engines, recommendation systems, speech interaction, autonomous vehicles, creative tools, and strategy games.
    - Alan Turing initiated substantial AI research, termed "machine intelligence."
    - AI became an academic discipline in 1956, experiencing cycles of optimism and "AI winters."
    - Post-2012, deep learning and post-2017 transformer architectures revitalized AI, leading to a boom in the early 2020s.
    - AI influences societal and economic shifts towards automation and data-driven decision-making across various sectors.
    - AI research goals: reasoning, knowledge representation, planning, learning, natural language processing, perception, and robotics support.
    - AI techniques include search, optimization, logic, neural networks, and statistical methods.
    - AI sub-problems focus on traits like reasoning, problem-solving, knowledge representation, planning, decision-making, learning, and perception.
    - Early AI research mimicked human step-by-step reasoning; modern AI handles uncertain information using probability and economics.
    - Knowledge representation in AI involves ontologies and knowledge bases to support intelligent querying and reasoning.
    - Planning in AI involves goal-directed behavior and decision-making based on utility maximization.
    - Learning in AI includes machine learning, supervised and unsupervised learning, reinforcement learning, and deep learning.
    - Natural language processing (NLP) in AI has evolved from rule-based systems to modern deep learning techniques.
    - AI perception involves interpreting sensor data for tasks like speech recognition and computer vision.
    - General AI aims to solve diverse problems with human-like versatility.
    - AI search techniques include state space search, local search, and adversarial search for game-playing.
    - Logic in AI uses formal systems like propositional and predicate logic for reasoning and knowledge representation.
    - Probabilistic methods in AI address decision-making and planning under uncertainty using tools like Bayesian networks and Markov decision processes.
    - Classifiers in AI categorize data into predefined classes based on pattern matching and supervised learning.
    
    - Neural networks: Interconnected nodes, similar to brain neurons, with input, hidden layers, and output.
    - Deep neural networks: At least 2 hidden layers.
    - Training techniques: Commonly use backpropagation.
    - Feedforward networks: Signal passes in one direction.
    - Recurrent networks: Output fed back into input for short-term memory.
    - Perceptrons: Single layer of neurons.
    - Convolutional networks: Strengthen connections between close neurons, important in image processing.
    - Deep learning: Multiple layers extract features progressively, used in various AI subfields.
    - GPT (Generative Pre-trained Transformers): Large language models pre-trained on text, used in chatbots.
    - Specialized AI hardware: GPUs replaced CPUs for training large-scale machine learning models.
    - AI applications: Used in search engines, online ads, virtual assistants, autonomous vehicles, language translation, facial recognition.
    - AI in healthcare: Increases patient care, used in medical research and drug discovery.
    - AI in games: Used in chess, Jeopardy!, Go, and real-time strategy games.
    - Military AI: Enhances command, control, and operations, used in coordination and threat detection.
    - Generative AI: Creates realistic images and texts, used in creative arts.
    - AI ethics and risks: Concerns over privacy, surveillance, copyright, misinformation, and algorithmic bias.
    - Algorithmic bias: Can cause discrimination if trained on biased data, fairness in machine learning is a critical area of study.
    
    - AI engineers demographics: 4% black, 20% women.
    - ACM FAccT 2022: Recommends limiting use of self-learning neural networks due to bias.


key: _node_content
string_value:
{"id_": "e3d30515-ee98-485e-8c86-0ba1a3ac0c2d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9d9270be-10e4-49e5-b4eb-a7f0ff3c7af8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "fad2fca2f14cde5a46c9c39ac90a4e58ac9e1c5c030139720fa2986ad3c54fff", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "db2e779e-6599-48a8-a3a6-aa7ac481c66b", "node_type": "1", "metadata": {}, "hash": "722d5d71b192396b7060fdd300c835f901960f6e116f43af0787b9b06cb0c5df", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 53241, "end_char_idx": 56938, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- Classifiers in AI categorize data into predefined classes based on pattern matching and supervised learning.
    
    - Neural networks: Interconnected nodes, similar to brain neurons, with input, hidden layers, and output.
    - Deep neural networks: At least 2 hidden layers.
    - Training techniques: Commonly use backpropagation.
    - Feedforward networks: Signal passes in one direction.
    - Recurrent networks: Output fed back into input for short-term memory.
    - Perceptrons: Single layer of neurons.
    - Convolutional networks: Strengthen connections between close neurons, important in image processing.
    - Deep learning: Multiple layers extract features progressively, used in various AI subfields.
    - GPT (Generative Pre-trained Transformers): Large language models pre-trained on text, used in chatbots.
    - Specialized AI hardware: GPUs replaced CPUs for training large-scale machine learning models.
    - AI applications: Used in search engines, online ads, virtual assistants, autonomous vehicles, language translation, facial recognition.
    - AI in healthcare: Increases patient care, used in medical research and drug discovery.
    - AI in games: Used in chess, Jeopardy!, Go, and real-time strategy games.
    - Military AI: Enhances command, control, and operations, used in coordination and threat detection.
    - Generative AI: Creates realistic images and texts, used in creative arts.
    - AI ethics and risks: Concerns over privacy, surveillance, copyright, misinformation, and algorithmic bias.
    - Algorithmic bias: Can cause discrimination if trained on biased data, fairness in machine learning is a critical area of study.
    
    - AI engineers demographics: 4% black, 20% women.
    - ACM FAccT 2022: Recommends limiting use of self-learning neural networks due to bias.
    - AI complexity: Designers often can't explain decision-making processes.
    - Misleading AI outcomes: Skin disease identifier misclassifies images with rulers as "cancerous"; AI misclassifies asthma patients as low risk for pneumonia.
    - Right to explanation: Essential for accountability, especially in medical and legal fields.
    - DARPA's XAI program (2014): Aims to make AI decisions understandable.
    - Transparency solutions: SHAP, LIME, multitask learning, deconvolution, DeepDream.
    - AI misuse: Authoritarian surveillance, misinformation, autonomous weapons.
    - AI in warfare: 30 nations support UN ban on autonomous weapons; over 50 countries researching battlefield robots.
    - Technological unemployment: AI could increase long-term unemployment; conflicting expert opinions on job risk from automation.
    - Existential risks of AI: Potential to lose control over superintelligent AI; concerns from Stephen Hawking, Bill Gates, Elon Musk.
    - Ethical AI development: Importance of aligning AI with human values and ethics.
    - AI regulation: Increasing global legislative activity; first global AI Safety Summit in 2023.
    - Historical perspective: AI research dates back to antiquity, significant developments in mid-20th century.
    
    - 1974: U.S. and British governments ceased AI exploratory research due to criticism and funding pressures.
    - 1985: AI market value exceeded $1 billion.
    - 1987: Collapse of Lisp Machine market led to a second, prolonged AI winter.
    - 1990: Yann LeCun demonstrated successful use of convolutional neural networks for recognizing handwritten digits.
    - Early 2000s: AI reputation restored through specific problem-solving and formal methods.
    - 2012: Deep learning began dominating AI benchmarks.
    - 2015-2019: Machine learning research publications increased by 50%.


key: _node_content
string_value:
{"id_": "db2e779e-6599-48a8-a3a6-aa7ac481c66b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e3d30515-ee98-485e-8c86-0ba1a3ac0c2d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "604fd2e61a0a10ebf9eb80fabd2402a5509e9ef17f5f64e8feb7742167137152", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0fd66065-2b81-4695-bc42-452713d631e8", "node_type": "1", "metadata": {}, "hash": "4836df9d040963cf0e16b9da90dd973ca629a7122f0bc4f12d13376023401dd0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 55075, "end_char_idx": 58470, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- AI complexity: Designers often can't explain decision-making processes.
    - Misleading AI outcomes: Skin disease identifier misclassifies images with rulers as "cancerous"; AI misclassifies asthma patients as low risk for pneumonia.
    - Right to explanation: Essential for accountability, especially in medical and legal fields.
    - DARPA's XAI program (2014): Aims to make AI decisions understandable.
    - Transparency solutions: SHAP, LIME, multitask learning, deconvolution, DeepDream.
    - AI misuse: Authoritarian surveillance, misinformation, autonomous weapons.
    - AI in warfare: 30 nations support UN ban on autonomous weapons; over 50 countries researching battlefield robots.
    - Technological unemployment: AI could increase long-term unemployment; conflicting expert opinions on job risk from automation.
    - Existential risks of AI: Potential to lose control over superintelligent AI; concerns from Stephen Hawking, Bill Gates, Elon Musk.
    - Ethical AI development: Importance of aligning AI with human values and ethics.
    - AI regulation: Increasing global legislative activity; first global AI Safety Summit in 2023.
    - Historical perspective: AI research dates back to antiquity, significant developments in mid-20th century.
    
    - 1974: U.S. and British governments ceased AI exploratory research due to criticism and funding pressures.
    - 1985: AI market value exceeded $1 billion.
    - 1987: Collapse of Lisp Machine market led to a second, prolonged AI winter.
    - 1990: Yann LeCun demonstrated successful use of convolutional neural networks for recognizing handwritten digits.
    - Early 2000s: AI reputation restored through specific problem-solving and formal methods.
    - 2012: Deep learning began dominating AI benchmarks.
    - 2015-2019: Machine learning research publications increased by 50%.
    - 2016: Fairness and misuse of technology became central issues in AI.
    - 2022: Approximately $50 billion annually invested in AI in the U.S.; 800,000 AI-related job openings in the U.S.
    - Turing test proposed by Alan Turing in 1950 to measure machine's ability to simulate human conversation.
    - AI defined as the study of agents that perceive their environment and take actions to achieve goals.
    - 2010s: Statistical machine learning overshadowed other AI approaches.
    - Symbolic AI excelled in high-level reasoning but failed in tasks like object recognition and commonsense reasoning.
    - Late 1980s: Introduction of soft computing techniques.
    - Debate between pursuing narrow AI (specific problem-solving) versus artificial general intelligence (AGI).
    - 2017: EU considered granting "electronic personhood" to advanced AI systems.
    - Predictions of merging humans and machines into cyborgs, a concept known as transhumanism.
    
    - Focus on how AI and technology, as depicted in "Ex Machina" and Philip K. Dick's "Do Androids Dream of Electric Sheep?", alter human subjectivity.
    - No specific numerical data provided.


    


Finally, note that the utility allows for recursive summarization, where each summary is based on the previous summaries, adding more context to the summarization process. This can be enabled by setting the `summarize_recursively` parameter to True. This is more computationally expensive, but can increase consistency and coherence of the combined summary.


key: _node_content
string_value:
{"id_": "0fd66065-2b81-4695-bc42-452713d631e8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "db2e779e-6599-48a8-a3a6-aa7ac481c66b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "8a7df15d1b1f7e5870980bd287e068fb6a9bd6f32afb09d1527573c2e2af3c98", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ceb871ef-e3bf-4212-9374-a36e94f234eb", "node_type": "1", "metadata": {}, "hash": "6af23db3cfada4837d1c9eda7bfb3cfeb5b39b6719ced843d38bf788adb5e193", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 56596, "end_char_idx": 60318, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
- 1990: Yann LeCun demonstrated successful use of convolutional neural networks for recognizing handwritten digits.
    - Early 2000s: AI reputation restored through specific problem-solving and formal methods.
    - 2012: Deep learning began dominating AI benchmarks.
    - 2015-2019: Machine learning research publications increased by 50%.
    - 2016: Fairness and misuse of technology became central issues in AI.
    - 2022: Approximately $50 billion annually invested in AI in the U.S.; 800,000 AI-related job openings in the U.S.
    - Turing test proposed by Alan Turing in 1950 to measure machine's ability to simulate human conversation.
    - AI defined as the study of agents that perceive their environment and take actions to achieve goals.
    - 2010s: Statistical machine learning overshadowed other AI approaches.
    - Symbolic AI excelled in high-level reasoning but failed in tasks like object recognition and commonsense reasoning.
    - Late 1980s: Introduction of soft computing techniques.
    - Debate between pursuing narrow AI (specific problem-solving) versus artificial general intelligence (AGI).
    - 2017: EU considered granting "electronic personhood" to advanced AI systems.
    - Predictions of merging humans and machines into cyborgs, a concept known as transhumanism.
    
    - Focus on how AI and technology, as depicted in "Ex Machina" and Philip K. Dick's "Do Androids Dream of Electric Sheep?", alter human subjectivity.
    - No specific numerical data provided.


    


Finally, note that the utility allows for recursive summarization, where each summary is based on the previous summaries, adding more context to the summarization process. This can be enabled by setting the `summarize_recursively` parameter to True. This is more computationally expensive, but can increase consistency and coherence of the combined summary.


```python
recursive_summary = summarize(artificial_intelligence_wikipedia_text, detail=0.1, summarize_recursively=True)
print(recursive_summary)
```

    100%|██████████| 5/5 [00:41<00:00,  8.36s/it]

    Artificial intelligence (AI) is the simulation of human intelligence in machines, designed to perform tasks that typically require human intelligence. This includes applications like advanced search engines, recommendation systems, speech interaction, autonomous vehicles, and strategic game analysis. AI was established as a distinct academic discipline in 1956 and has experienced cycles of high expectations followed by disillusionment and decreased funding, known as "AI winters." Interest in AI surged post-2012 with advancements in deep learning and again post-2017 with the development of transformer architectures, leading to significant progress in the early 2020s.
    
    AI's increasing integration into various sectors is influencing societal and economic shifts towards automation and data-driven decision-making, affecting areas such as employment, healthcare, and education. This raises important ethical and safety concerns, prompting discussions on regulatory policies.
    
    AI research encompasses various sub-fields focused on specific goals like reasoning, learning, natural language processing, perception, and robotics, using techniques from search and optimization, logic, and probabilistic methods. The field also draws from psychology, linguistics, philosophy, and neuroscience. AI aims to achieve general intelligence, enabling machines to perform any intellectual task that a human can do.
    
    Artificial intelligence (AI) simulates human intelligence in machines to perform tasks that typically require human intellect, such as advanced search engines, recommendation systems, and autonomous vehicles.


key: _node_content
string_value:
{"id_": "ceb871ef-e3bf-4212-9374-a36e94f234eb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0fd66065-2b81-4695-bc42-452713d631e8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "aad549cb6f9ee007af87f5586797ca9271139af5603f7deed9e0ed844984e2ba", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "806377f5-5333-46f2-b1a8-dbd10892c9e8", "node_type": "1", "metadata": {}, "hash": "83289f7789368dd6ba4ebb66db50f7173e758ea8f9bd7d035c073e370feecc61", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 58473, "end_char_idx": 62711, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
recursive_summary = summarize(artificial_intelligence_wikipedia_text, detail=0.1, summarize_recursively=True)
print(recursive_summary)
```

    100%|██████████| 5/5 [00:41<00:00,  8.36s/it]

    Artificial intelligence (AI) is the simulation of human intelligence in machines, designed to perform tasks that typically require human intelligence. This includes applications like advanced search engines, recommendation systems, speech interaction, autonomous vehicles, and strategic game analysis. AI was established as a distinct academic discipline in 1956 and has experienced cycles of high expectations followed by disillusionment and decreased funding, known as "AI winters." Interest in AI surged post-2012 with advancements in deep learning and again post-2017 with the development of transformer architectures, leading to significant progress in the early 2020s.
    
    AI's increasing integration into various sectors is influencing societal and economic shifts towards automation and data-driven decision-making, affecting areas such as employment, healthcare, and education. This raises important ethical and safety concerns, prompting discussions on regulatory policies.
    
    AI research encompasses various sub-fields focused on specific goals like reasoning, learning, natural language processing, perception, and robotics, using techniques from search and optimization, logic, and probabilistic methods. The field also draws from psychology, linguistics, philosophy, and neuroscience. AI aims to achieve general intelligence, enabling machines to perform any intellectual task that a human can do.
    
    Artificial intelligence (AI) simulates human intelligence in machines to perform tasks that typically require human intellect, such as advanced search engines, recommendation systems, and autonomous vehicles. AI research, which began as a distinct academic discipline in 1956, includes sub-fields like natural language processing and robotics, employing techniques from various scientific domains. AI has significantly advanced due to deep learning and the development of transformer architectures, notably improving applications in computer vision, speech recognition, and other areas.
    
    Neural networks, central to AI, mimic the human brain's neuron network to recognize patterns and learn from data, using multiple layers in deep learning to extract complex features. These networks have evolved into sophisticated models like GPT (Generative Pre-trained Transformers) for natural language processing, enhancing applications like chatbots.
    
    AI's integration into sectors like healthcare, military, and agriculture has led to innovations like precision medicine and smart farming but also raised ethical concerns regarding privacy, bias, and the potential for misuse. Issues like data privacy, algorithmic bias, and the generation of misinformation are critical challenges as AI becomes pervasive in society. AI's potential and risks necessitate careful management and regulation to harness benefits while mitigating adverse impacts.
    
    AI, or artificial intelligence, simulates human intelligence in machines to perform complex tasks, such as operating autonomous vehicles and analyzing strategic games. Since its establishment as an academic discipline in 1956, AI has seen periods of high expectations and subsequent disillusionment, known as "AI winters." Recent advancements in deep learning and transformer architectures have significantly advanced AI capabilities in areas like computer vision and speech recognition.
    
    AI's integration into various sectors, including healthcare and agriculture, has led to innovations like precision medicine and smart farming but has also raised ethical concerns about privacy, bias, and misuse. The complexity of AI systems, particularly deep neural networks, often makes it difficult for developers to explain their decision-making processes, leading to transparency issues. This lack of transparency can result in unintended consequences, such as misclassifications in medical diagnostics.
    
    The potential for AI to be weaponized by bad actors, such as authoritarian governments or terrorists, poses significant risks.


key: _node_content
string_value:
{"id_": "806377f5-5333-46f2-b1a8-dbd10892c9e8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ceb871ef-e3bf-4212-9374-a36e94f234eb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d69d6dfa1f2cda968ad264500229af5812ac8858f7566deda24a1459dd83bec1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e7ede166-ae0f-405a-842c-ce4811591df0", "node_type": "1", "metadata": {}, "hash": "9a866023518cbeefe2618e8ba1ba996eacd2f57fe8a01ef1a660743355cab8a4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 60508, "end_char_idx": 64818, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
AI has significantly advanced due to deep learning and the development of transformer architectures, notably improving applications in computer vision, speech recognition, and other areas.
    
    Neural networks, central to AI, mimic the human brain's neuron network to recognize patterns and learn from data, using multiple layers in deep learning to extract complex features. These networks have evolved into sophisticated models like GPT (Generative Pre-trained Transformers) for natural language processing, enhancing applications like chatbots.
    
    AI's integration into sectors like healthcare, military, and agriculture has led to innovations like precision medicine and smart farming but also raised ethical concerns regarding privacy, bias, and the potential for misuse. Issues like data privacy, algorithmic bias, and the generation of misinformation are critical challenges as AI becomes pervasive in society. AI's potential and risks necessitate careful management and regulation to harness benefits while mitigating adverse impacts.
    
    AI, or artificial intelligence, simulates human intelligence in machines to perform complex tasks, such as operating autonomous vehicles and analyzing strategic games. Since its establishment as an academic discipline in 1956, AI has seen periods of high expectations and subsequent disillusionment, known as "AI winters." Recent advancements in deep learning and transformer architectures have significantly advanced AI capabilities in areas like computer vision and speech recognition.
    
    AI's integration into various sectors, including healthcare and agriculture, has led to innovations like precision medicine and smart farming but has also raised ethical concerns about privacy, bias, and misuse. The complexity of AI systems, particularly deep neural networks, often makes it difficult for developers to explain their decision-making processes, leading to transparency issues. This lack of transparency can result in unintended consequences, such as misclassifications in medical diagnostics.
    
    The potential for AI to be weaponized by bad actors, such as authoritarian governments or terrorists, poses significant risks. AI's reliance on large tech companies for computational power and the potential for technological unemployment are also critical issues. Despite these challenges, AI also offers opportunities for enhancing human well-being if ethical considerations are integrated throughout the design and implementation stages.
    
    Regulation of AI is emerging globally, with various countries adopting AI strategies to ensure the technology aligns with human rights and democratic values. The first global AI Safety Summit in 2023 emphasized the need for international cooperation to manage AI's risks and challenges effectively.
    
    In the 1970s, AI research faced significant setbacks due to criticism from influential figures like Sir James Lighthill and funding cuts from the U.S. and British governments, leading to the first "AI winter." The field saw a resurgence in the 1980s with the success of expert systems and renewed government funding, but suffered another setback with the collapse of the Lisp Machine market in 1987, initiating a second AI winter. During this period, researchers began exploring "sub-symbolic" approaches, including neural networks, which gained prominence in the 1990s with successful applications like Yann LeCun’s convolutional neural networks for digit recognition.
    
    By the early 21st century, AI was revitalized by focusing on narrow, specific problems, leading to practical applications and integration into various sectors. The field of artificial general intelligence (AGI) emerged, aiming to create versatile, fully intelligent machines. The 2010s saw deep learning dominate AI research, driven by hardware improvements and large datasets, which significantly increased interest and investment in AI.
    
    Philosophically, AI has been defined in various ways, focusing on external behavior rather than internal experience, aligning with Alan Turing's proposal of the Turing test. The field has debated the merits of symbolic vs. sub-symbolic AI, with ongoing discussions about machine consciousness and the ethical implications of potentially sentient AI.


key: _node_content
string_value:
{"id_": "e7ede166-ae0f-405a-842c-ce4811591df0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d7de6cf5-db92-4e82-8126-5a739e391242", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "41deb8f6bf3c2bc13a7e72c90f8841eea7a692c05696523ee3b0d1843aa778df", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "806377f5-5333-46f2-b1a8-dbd10892c9e8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/Summarizing_long_documents.ipynb", "title": "Summarizing Long Documents", "tokens": 12631, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "bcb968b7cf2c82307bd1683dd12c61d3f27fd8648e3752572acd3fed097fd4b1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 62712, "end_char_idx": 65489, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
AI's reliance on large tech companies for computational power and the potential for technological unemployment are also critical issues. Despite these challenges, AI also offers opportunities for enhancing human well-being if ethical considerations are integrated throughout the design and implementation stages.
    
    Regulation of AI is emerging globally, with various countries adopting AI strategies to ensure the technology aligns with human rights and democratic values. The first global AI Safety Summit in 2023 emphasized the need for international cooperation to manage AI's risks and challenges effectively.
    
    In the 1970s, AI research faced significant setbacks due to criticism from influential figures like Sir James Lighthill and funding cuts from the U.S. and British governments, leading to the first "AI winter." The field saw a resurgence in the 1980s with the success of expert systems and renewed government funding, but suffered another setback with the collapse of the Lisp Machine market in 1987, initiating a second AI winter. During this period, researchers began exploring "sub-symbolic" approaches, including neural networks, which gained prominence in the 1990s with successful applications like Yann LeCun’s convolutional neural networks for digit recognition.
    
    By the early 21st century, AI was revitalized by focusing on narrow, specific problems, leading to practical applications and integration into various sectors. The field of artificial general intelligence (AGI) emerged, aiming to create versatile, fully intelligent machines. The 2010s saw deep learning dominate AI research, driven by hardware improvements and large datasets, which significantly increased interest and investment in AI.
    
    Philosophically, AI has been defined in various ways, focusing on external behavior rather than internal experience, aligning with Alan Turing's proposal of the Turing test. The field has debated the merits of symbolic vs. sub-symbolic AI, with ongoing discussions about machine consciousness and the ethical implications of potentially sentient AI. The concept of AI rights and welfare has also emerged, reflecting concerns about the moral status of advanced AI systems.
    
    Overall, AI research has oscillated between periods of intense optimism and profound setbacks, with current trends heavily favoring practical applications through narrow AI, while continuing to explore the broader implications and potential of general and superintelligent AI systems.
    
    Artificial Intelligence (AI) and its portrayal in media, such as the film "Ex Machina" and Philip K. Dick's novel "Do Androids Dream of Electric Sheep?", explore how technology, particularly AI, can alter our understanding of human subjectivity.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb


key: _node_content
string_value:
{"id_": "f338997e-6ba0-46c2-81d6-1e9e1e2accdb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "90dedd29-774a-454e-b743-afd0583c8125", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "7d128a4feb7a34c7f38af361601916f25458e0b3620f1a28ec4c2edf1f37fde3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "79013f24-7e43-4efe-90d7-46cf7d0a35dc", "node_type": "1", "metadata": {}, "hash": "06cc2ffe4a15ae4a93be1e8fe4717b9365c963fe9d70d2f2824f74c62e5c1b13", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3392, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to use functions with a knowledge base

This notebook builds on the concepts in the [argument generation](How_to_call_functions_with_chat_models.ipynb) notebook, by creating an agent with access to a knowledge base and two functions that it can call based on the user requirement.

We'll create an agent that uses data from arXiv to answer questions about academic subjects. It has two functions at its disposal:
- **get_articles**: A function that gets arXiv articles on a subject and summarizes them for the user with links.
- **read_article_and_summarize**: This function takes one of the previously searched articles, reads it in its entirety and summarizes the core argument, evidence and conclusions.

This will get you comfortable with a multi-function workflow that can choose from multiple services, and where some of the data from the first function is persisted to be used by the second.

## Walkthrough

This cookbook takes you through the following workflow:

- **Search utilities:** Creating the two functions that access arXiv for answers.
- **Configure Agent:** Building up the Agent behaviour that will assess the need for a function and, if one is required, call that function and present results back to the agent.
- **arXiv conversation:** Put all of this together in live conversation.



```python
!pip install scipy --quiet
!pip install tenacity --quiet
!pip install tiktoken==0.3.3 --quiet
!pip install termcolor --quiet
!pip install openai --quiet
!pip install arxiv --quiet
!pip install pandas --quiet
!pip install PyPDF2 --quiet
!pip install tqdm --quiet
```


```python
import os
import arxiv
import ast
import concurrent
import json
import os
import pandas as pd
import tiktoken
from csv import writer
from IPython.display import display, Markdown, Latex
from openai import OpenAI
from PyPDF2 import PdfReader
from scipy import spatial
from tenacity import retry, wait_random_exponential, stop_after_attempt
from tqdm import tqdm
from termcolor import colored

GPT_MODEL = "gpt-3.5-turbo-0613"
EMBEDDING_MODEL = "text-embedding-ada-002"
client = OpenAI()
```

## Search utilities

We'll first set up some utilities that will underpin our two functions.

Downloaded papers will be stored in a directory (we use ```./data/papers``` here). We create a file ```arxiv_library.csv``` to store the embeddings and details for downloaded papers to retrieve against using ```summarize_text```.


```python
directory = './data/papers'

# Check if the directory already exists
if not os.path.exists(directory):
    # If the directory doesn't exist, create it and any necessary intermediate directories
    os.makedirs(directory)
    print(f"Directory '{directory}' created successfully.")
else:
    # If the directory already exists, print a message indicating it
    print(f"Directory '{directory}' already exists.")
```

    Directory './data/papers' already exists.



```python
# Set a directory to store downloaded papers
data_dir = os.path.join(os.curdir, "data", "papers")
paper_dir_filepath = "./data/arxiv_library.csv"

# Generate a blank dataframe where we can store downloaded files
df = pd.DataFrame(list())
df.to_csv(paper_dir_filepath)
```


```python
@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))
def embedding_request(text):
    response = client.embeddings.create(input=text, model=EMBEDDING_MODEL)
    return response


key: _node_content
string_value:
{"id_": "79013f24-7e43-4efe-90d7-46cf7d0a35dc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "90dedd29-774a-454e-b743-afd0583c8125", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "7d128a4feb7a34c7f38af361601916f25458e0b3620f1a28ec4c2edf1f37fde3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f338997e-6ba0-46c2-81d6-1e9e1e2accdb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "b335ccb8ecd62ba9eb2c3da71f55bc8430ecc49af799f4611aec2bf1c03c7bd7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "15452e09-92b6-4596-b605-dc7955c1d21d", "node_type": "1", "metadata": {}, "hash": "eeecb8750b86446247f52017531265dcf093443dfeec836b108fd5c697591a7a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2421, "end_char_idx": 5894, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
directory = './data/papers'

# Check if the directory already exists
if not os.path.exists(directory):
    # If the directory doesn't exist, create it and any necessary intermediate directories
    os.makedirs(directory)
    print(f"Directory '{directory}' created successfully.")
else:
    # If the directory already exists, print a message indicating it
    print(f"Directory '{directory}' already exists.")
```

    Directory './data/papers' already exists.



```python
# Set a directory to store downloaded papers
data_dir = os.path.join(os.curdir, "data", "papers")
paper_dir_filepath = "./data/arxiv_library.csv"

# Generate a blank dataframe where we can store downloaded files
df = pd.DataFrame(list())
df.to_csv(paper_dir_filepath)
```


```python
@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))
def embedding_request(text):
    response = client.embeddings.create(input=text, model=EMBEDDING_MODEL)
    return response


@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))
def get_articles(query, library=paper_dir_filepath, top_k=5):
    """This function gets the top_k articles based on a user's query, sorted by relevance.
    It also downloads the files and stores them in arxiv_library.csv to be retrieved by the read_article_and_summarize.
    """
    client = arxiv.Client()
    search = arxiv.Search(
        query = "quantum",
        max_results = 10,
        sort_by = arxiv.SortCriterion.SubmittedDate
    )
    result_list = []
    for result in client.results(search):
        result_dict = {}
        result_dict.update({"title": result.title})
        result_dict.update({"summary": result.summary})

        # Taking the first url provided
        result_dict.update({"article_url": [x.href for x in result.links][0]})
        result_dict.update({"pdf_url": [x.href for x in result.links][1]})
        result_list.append(result_dict)

        # Store references in library file
        response = embedding_request(text=result.title)
        file_reference = [
            result.title,
            result.download_pdf(data_dir),
            response.data[0].embedding,
        ]

        # Write to file
        with open(library, "a") as f_object:
            writer_object = writer(f_object)
            writer_object.writerow(file_reference)
            f_object.close()
    return result_list

```


```python
# Test that the search is working
result_output = get_articles("ppo reinforcement learning")
result_output[0]

```




    {'title': 'Quantum types: going beyond qubits and quantum gates',
     'summary': 'Quantum computing is a growing field with significant potential applications.\nLearning how to code quantum programs means understanding how qubits work and\nlearning to use quantum gates. This is analogous to creating classical\nalgorithms using logic gates and bits. Even after learning all concepts, it is\ndifficult to create new algorithms, which hinders the acceptance of quantum\nprogramming by most developers. This article outlines the need for higher-level\nabstractions and proposes some of them in a developer-friendly programming\nlanguage called Rhyme. The new quantum types are extensions of classical types,\nincluding bits, integers, floats, characters, arrays, and strings. We show how\nto use such types with code snippets.',
     'article_url': 'http://arxiv.org/abs/2401.15073v1',
     'pdf_url': 'http://arxiv.org/pdf/2401.15073v1'}


key: _node_content
string_value:
{"id_": "15452e09-92b6-4596-b605-dc7955c1d21d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "90dedd29-774a-454e-b743-afd0583c8125", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "7d128a4feb7a34c7f38af361601916f25458e0b3620f1a28ec4c2edf1f37fde3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "79013f24-7e43-4efe-90d7-46cf7d0a35dc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "088c102f00eb166f723f23b8621551275b7999bd2d162ef69c021184c5c5ddbb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5758b86b-114e-4f80-ab5a-24fa8e05352b", "node_type": "1", "metadata": {}, "hash": "7163b04905084568652b4d4061f5447108bd80f5e5f144bcc18056e7382f7dcf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4824, "end_char_idx": 7822, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Test that the search is working
result_output = get_articles("ppo reinforcement learning")
result_output[0]

```




    {'title': 'Quantum types: going beyond qubits and quantum gates',
     'summary': 'Quantum computing is a growing field with significant potential applications.\nLearning how to code quantum programs means understanding how qubits work and\nlearning to use quantum gates. This is analogous to creating classical\nalgorithms using logic gates and bits. Even after learning all concepts, it is\ndifficult to create new algorithms, which hinders the acceptance of quantum\nprogramming by most developers. This article outlines the need for higher-level\nabstractions and proposes some of them in a developer-friendly programming\nlanguage called Rhyme. The new quantum types are extensions of classical types,\nincluding bits, integers, floats, characters, arrays, and strings. We show how\nto use such types with code snippets.',
     'article_url': 'http://arxiv.org/abs/2401.15073v1',
     'pdf_url': 'http://arxiv.org/pdf/2401.15073v1'}




```python
def strings_ranked_by_relatedness(
    query: str,
    df: pd.DataFrame,
    relatedness_fn=lambda x, y: 1 - spatial.distance.cosine(x, y),
    top_n: int = 100,
) -> list[str]:
    """Returns a list of strings and relatednesses, sorted from most related to least."""
    query_embedding_response = embedding_request(query)
    query_embedding = query_embedding_response.data[0].embedding
    strings_and_relatednesses = [
        (row["filepath"], relatedness_fn(query_embedding, row["embedding"]))
        for i, row in df.iterrows()
    ]
    strings_and_relatednesses.sort(key=lambda x: x[1], reverse=True)
    strings, relatednesses = zip(*strings_and_relatednesses)
    return strings[:top_n]

```


```python
def read_pdf(filepath):
    """Takes a filepath to a PDF and returns a string of the PDF's contents"""
    # creating a pdf reader object
    reader = PdfReader(filepath)
    pdf_text = ""
    page_number = 0
    for page in reader.pages:
        page_number += 1
        pdf_text += page.extract_text() + f"\nPage Number: {page_number}"
    return pdf_text


# Split a text into smaller chunks of size n, preferably ending at the end of a sentence
def create_chunks(text, n, tokenizer):
    """Returns successive n-sized chunks from provided text."""
    tokens = tokenizer.encode(text)
    i = 0
    while i < len(tokens):
        # Find the nearest end of sentence within a range of 0.5 * n and 1.5 * n tokens
        j = min(i + int(1.5 * n), len(tokens))
        while j > i + int(0.5 * n):
            # Decode the tokens and check for full stop or newline
            chunk = tokenizer.decode(tokens[i:j])
            if chunk.endswith(".") or chunk.endswith("\n"):
                break
            j -= 1
        # If no end of sentence found, use n tokens as the chunk size
        if j == i + int(0.5 * n):
            j = min(i + n, len(tokens))
        yield tokens[i:j]
        i = j


key: _node_content
string_value:
{"id_": "5758b86b-114e-4f80-ab5a-24fa8e05352b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "90dedd29-774a-454e-b743-afd0583c8125", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "7d128a4feb7a34c7f38af361601916f25458e0b3620f1a28ec4c2edf1f37fde3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "15452e09-92b6-4596-b605-dc7955c1d21d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "9d9c0fea388617e58dbdb4d37e4425973d79ab581b31ea302e495e5f46191ff7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d6817a50-b219-45fa-bf44-5d5a994c5d8f", "node_type": "1", "metadata": {}, "hash": "f8368b1e892457b8dadde4755ca24de7594d075628542d9c89386fa85104d0de", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6615, "end_char_idx": 8210, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def read_pdf(filepath):
    """Takes a filepath to a PDF and returns a string of the PDF's contents"""
    # creating a pdf reader object
    reader = PdfReader(filepath)
    pdf_text = ""
    page_number = 0
    for page in reader.pages:
        page_number += 1
        pdf_text += page.extract_text() + f"\nPage Number: {page_number}"
    return pdf_text


# Split a text into smaller chunks of size n, preferably ending at the end of a sentence
def create_chunks(text, n, tokenizer):
    """Returns successive n-sized chunks from provided text."""
    tokens = tokenizer.encode(text)
    i = 0
    while i < len(tokens):
        # Find the nearest end of sentence within a range of 0.5 * n and 1.5 * n tokens
        j = min(i + int(1.5 * n), len(tokens))
        while j > i + int(0.5 * n):
            # Decode the tokens and check for full stop or newline
            chunk = tokenizer.decode(tokens[i:j])
            if chunk.endswith(".") or chunk.endswith("\n"):
                break
            j -= 1
        # If no end of sentence found, use n tokens as the chunk size
        if j == i + int(0.5 * n):
            j = min(i + n, len(tokens))
        yield tokens[i:j]
        i = j


def extract_chunk(content, template_prompt):
    """This function applies a prompt to some input content. In this case it returns a summarized chunk of text"""
    prompt = template_prompt + content
    response = client.chat.completions.create(
        model=GPT_MODEL, messages=[{"role": "user", "content": prompt}], temperature=0
    )
    return response.choices[0].message.content


key: _node_content
string_value:
{"id_": "d6817a50-b219-45fa-bf44-5d5a994c5d8f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "90dedd29-774a-454e-b743-afd0583c8125", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "7d128a4feb7a34c7f38af361601916f25458e0b3620f1a28ec4c2edf1f37fde3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5758b86b-114e-4f80-ab5a-24fa8e05352b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "135e7ea66997a72be918bd5482a0b2bac085a888a36927c8b4f1c441ab3274c6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3a05d207-0cd8-4ff2-8f80-5837c7f6f5fe", "node_type": "1", "metadata": {}, "hash": "6f17afc756f7302089437434ee8c74203da2c741dae57a6945a3895b22db4707", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6615, "end_char_idx": 11063, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def read_pdf(filepath):
    """Takes a filepath to a PDF and returns a string of the PDF's contents"""
    # creating a pdf reader object
    reader = PdfReader(filepath)
    pdf_text = ""
    page_number = 0
    for page in reader.pages:
        page_number += 1
        pdf_text += page.extract_text() + f"\nPage Number: {page_number}"
    return pdf_text


# Split a text into smaller chunks of size n, preferably ending at the end of a sentence
def create_chunks(text, n, tokenizer):
    """Returns successive n-sized chunks from provided text."""
    tokens = tokenizer.encode(text)
    i = 0
    while i < len(tokens):
        # Find the nearest end of sentence within a range of 0.5 * n and 1.5 * n tokens
        j = min(i + int(1.5 * n), len(tokens))
        while j > i + int(0.5 * n):
            # Decode the tokens and check for full stop or newline
            chunk = tokenizer.decode(tokens[i:j])
            if chunk.endswith(".") or chunk.endswith("\n"):
                break
            j -= 1
        # If no end of sentence found, use n tokens as the chunk size
        if j == i + int(0.5 * n):
            j = min(i + n, len(tokens))
        yield tokens[i:j]
        i = j


def extract_chunk(content, template_prompt):
    """This function applies a prompt to some input content. In this case it returns a summarized chunk of text"""
    prompt = template_prompt + content
    response = client.chat.completions.create(
        model=GPT_MODEL, messages=[{"role": "user", "content": prompt}], temperature=0
    )
    return response.choices[0].message.content


def summarize_text(query):
    """This function does the following:
    - Reads in the arxiv_library.csv file in including the embeddings
    - Finds the closest file to the user's query
    - Scrapes the text out of the file and chunks it
    - Summarizes each chunk in parallel
    - Does one final summary and returns this to the user"""

    # A prompt to dictate how the recursive summarizations should approach the input paper
    summary_prompt = """Summarize this text from an academic paper. Extract any key points with reasoning.\n\nContent:"""

    # If the library is empty (no searches have been performed yet), we perform one and download the results
    library_df = pd.read_csv(paper_dir_filepath).reset_index()
    if len(library_df) == 0:
        print("No papers searched yet, downloading first.")
        get_articles(query)
        print("Papers downloaded, continuing")
        library_df = pd.read_csv(paper_dir_filepath).reset_index()
    library_df.columns = ["title", "filepath", "embedding"]
    library_df["embedding"] = library_df["embedding"].apply(ast.literal_eval)
    strings = strings_ranked_by_relatedness(query, library_df, top_n=1)
    print("Chunking text from paper")
    pdf_text = read_pdf(strings[0])

    # Initialise tokenizer
    tokenizer = tiktoken.get_encoding("cl100k_base")
    results = ""

    # Chunk up the document into 1500 token chunks
    chunks = create_chunks(pdf_text, 1500, tokenizer)
    text_chunks = [tokenizer.decode(chunk) for chunk in chunks]
    print("Summarizing each chunk of text")

    # Parallel process the summaries
    with concurrent.futures.ThreadPoolExecutor(
        max_workers=len(text_chunks)
    ) as executor:
        futures = [
            executor.submit(extract_chunk, chunk, summary_prompt)
            for chunk in text_chunks
        ]
        with tqdm(total=len(text_chunks)) as pbar:
            for _ in concurrent.futures.as_completed(futures):
                pbar.update(1)
        for future in futures:
            data = future.result()
            results += data

    # Final summary
    print("Summarizing into overall summary")
    response = client.chat.completions.create(
        model=GPT_MODEL,
        messages=[
            {
                "role": "user",
                "content": f"""Write a summary collated from this collection of key points extracted from an academic paper.
                        The summary should highlight the core argument, conclusions and evidence, and answer the user's query.
                        User query: {query}
                        The summary should be structured in bulleted lists following the headings Core Argument, Evidence, and Conclusions.
                        Key points:\n{results}\nSummary:\n""",
            }
        ],
        temperature=0,
    )
    return response

```


key: _node_content
string_value:
{"id_": "3a05d207-0cd8-4ff2-8f80-5837c7f6f5fe", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "90dedd29-774a-454e-b743-afd0583c8125", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "7d128a4feb7a34c7f38af361601916f25458e0b3620f1a28ec4c2edf1f37fde3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d6817a50-b219-45fa-bf44-5d5a994c5d8f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "0d6b8786f7a2fd0e4f53ef71db16d23072491b846c63d0f2024e87b3535dc0b4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0bf789c3-ad4f-4d45-a4d8-baad990ce029", "node_type": "1", "metadata": {}, "hash": "f972f6c387d124395a2a0fabb2b65591ec6d453ecca3ba9859f831cad21e7b0f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11066, "end_char_idx": 14574, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Test the summarize_text function works
chat_test_response = summarize_text("PPO reinforcement learning sequence generation")

```

    Chunking text from paper
    Summarizing each chunk of text


    100%|██████████| 6/6 [00:06<00:00,  1.08s/it]


    Summarizing into overall summary



```python
print(chat_test_response.choices[0].message.content)

```

    Core Argument:
    - The academic paper explores the connection between the transverse field Ising (TFI) model and the ϕ4 model, highlighting the analogy between topological solitary waves in the ϕ4 model and the effect of the transverse field on spin flips in the TFI model.
    - The study reveals regimes of memory/loss of memory and coherence/decoherence in the classical ϕ4 model subjected to periodic perturbations, which are essential in annealing phenomena.
    - The exploration of the analogy between lower-dimensional linear quantum systems and higher-dimensional classical nonlinear systems can lead to a deeper understanding of information processing in these systems.
    
    Evidence:
    - The authors analyze the dynamics and relaxation of weakly coupled ϕ4 chains through numerical simulations, observing kink and breather excitations and investigating the structural phase transition associated with the double well potential.
    - The critical temperature (Tc) approaches zero as the inter-chain coupling strength (C⊥) approaches zero, but there is a finite Tc for C⊥>0.
    - The spectral function shows peaks corresponding to particle motion across the double-well potential at higher temperatures and oscillations in a single well at lower temperatures.
    - The soft-mode frequency (ωs) decreases as temperature approaches Ts, the dynamical crossover temperature.
    - The relaxation process of the average displacement (QD) is controlled by spatially extended vibrations and large kink densities.
    - The mean domain size (⟨DS⟩) exhibits an algebraic decay for finite C⊥>0.
    - The probability of larger domain sizes is higher before a kick compared to after a kick for C⊥>0.
    
    Conclusions:
    - The authors suggest further exploration of the crossover between decoherence and finite coherence in periodic-kick strength space.
    - They propose extending the study to different kick profiles, introducing kink defects, and studying weakly-coupled chains in higher dimensions.
    - Recognizing similarities between classical nonlinear equations and quantum linear ones in information processing is important.
    - Future research directions include investigating the dynamics of quantum annealing, measurement and memory in the periodically driven complex Ginzburg-Landau equation, and the behavior of solitons and domain walls in various systems.


## Configure Agent

We'll create our agent in this step, including a ```Conversation``` class to support multiple turns with the API, and some Python functions to enable interaction between the ```ChatCompletion``` API and our knowledge base functions.


```python
@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))
def chat_completion_request(messages, functions=None, model=GPT_MODEL):
    try:
        response = client.chat.completions.create(
            model=model,
            messages=messages,
            functions=functions,
        )
        return response
    except Exception as e:
        print("Unable to generate ChatCompletion response")
        print(f"Exception: {e}")
        return e

```


key: _node_content
string_value:
{"id_": "0bf789c3-ad4f-4d45-a4d8-baad990ce029", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "90dedd29-774a-454e-b743-afd0583c8125", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "7d128a4feb7a34c7f38af361601916f25458e0b3620f1a28ec4c2edf1f37fde3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3a05d207-0cd8-4ff2-8f80-5837c7f6f5fe", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "8b6b80d99cafe892576d79a8197bb14ef4ccef16b208c3abaf00ed3a185dc76b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "243b8e83-d670-4d69-a227-0421ac4cc1ec", "node_type": "1", "metadata": {}, "hash": "f2c2739b500be4e6dd472eb17df76ed9fb7afeff7e2baf78e6697081e179d747", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13833, "end_char_idx": 17168, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Configure Agent

We'll create our agent in this step, including a ```Conversation``` class to support multiple turns with the API, and some Python functions to enable interaction between the ```ChatCompletion``` API and our knowledge base functions.


```python
@retry(wait=wait_random_exponential(min=1, max=40), stop=stop_after_attempt(3))
def chat_completion_request(messages, functions=None, model=GPT_MODEL):
    try:
        response = client.chat.completions.create(
            model=model,
            messages=messages,
            functions=functions,
        )
        return response
    except Exception as e:
        print("Unable to generate ChatCompletion response")
        print(f"Exception: {e}")
        return e

```


```python
class Conversation:
    def __init__(self):
        self.conversation_history = []

    def add_message(self, role, content):
        message = {"role": role, "content": content}
        self.conversation_history.append(message)

    def display_conversation(self, detailed=False):
        role_to_color = {
            "system": "red",
            "user": "green",
            "assistant": "blue",
            "function": "magenta",
        }
        for message in self.conversation_history:
            print(
                colored(
                    f"{message['role']}: {message['content']}\n\n",
                    role_to_color[message["role"]],
                )
            )
```


```python
# Initiate our get_articles and read_article_and_summarize functions
arxiv_functions = [
    {
        "name": "get_articles",
        "description": """Use this function to get academic papers from arXiv to answer user questions.""",
        "parameters": {
            "type": "object",
            "properties": {
                "query": {
                    "type": "string",
                    "description": f"""
                            User query in JSON. Responses should be summarized and should include the article URL reference
                            """,
                }
            },
            "required": ["query"],
        },
    },
    {
        "name": "read_article_and_summarize",
        "description": """Use this function to read whole papers and provide a summary for users.
        You should NEVER call this function before get_articles has been called in the conversation.""",
        "parameters": {
            "type": "object",
            "properties": {
                "query": {
                    "type": "string",
                    "description": f"""
                            Description of the article in plain text based on the user's query
                            """,
                }
            },
            "required": ["query"],
        },
    }
]

```


```python
def chat_completion_with_function_execution(messages, functions=[None]):
    """This function makes a ChatCompletion API call with the option of adding functions"""
    response = chat_completion_request(messages, functions)
    full_message = response.choices[0]
    if full_message.finish_reason == "function_call":
        print(f"Function generation requested, calling function")
        return call_arxiv_function(messages, full_message)
    else:
        print(f"Function not required, responding to user")
        return response


key: _node_content
string_value:
{"id_": "243b8e83-d670-4d69-a227-0421ac4cc1ec", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "90dedd29-774a-454e-b743-afd0583c8125", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "7d128a4feb7a34c7f38af361601916f25458e0b3620f1a28ec4c2edf1f37fde3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0bf789c3-ad4f-4d45-a4d8-baad990ce029", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "39d8f5edc5aeb0dc1b3b5fbcbdac5895d3e52fe5756304e3303eee9519598b18", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e2988872-1ed7-4da0-9e06-b542bff02b30", "node_type": "1", "metadata": {}, "hash": "8d17641a152018161c22c138deb19fbb3810445cb2e00f5e9e44e55ad7bb4c95", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15283, "end_char_idx": 19327, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Initiate our get_articles and read_article_and_summarize functions
arxiv_functions = [
    {
        "name": "get_articles",
        "description": """Use this function to get academic papers from arXiv to answer user questions.""",
        "parameters": {
            "type": "object",
            "properties": {
                "query": {
                    "type": "string",
                    "description": f"""
                            User query in JSON. Responses should be summarized and should include the article URL reference
                            """,
                }
            },
            "required": ["query"],
        },
    },
    {
        "name": "read_article_and_summarize",
        "description": """Use this function to read whole papers and provide a summary for users.
        You should NEVER call this function before get_articles has been called in the conversation.""",
        "parameters": {
            "type": "object",
            "properties": {
                "query": {
                    "type": "string",
                    "description": f"""
                            Description of the article in plain text based on the user's query
                            """,
                }
            },
            "required": ["query"],
        },
    }
]

```


```python
def chat_completion_with_function_execution(messages, functions=[None]):
    """This function makes a ChatCompletion API call with the option of adding functions"""
    response = chat_completion_request(messages, functions)
    full_message = response.choices[0]
    if full_message.finish_reason == "function_call":
        print(f"Function generation requested, calling function")
        return call_arxiv_function(messages, full_message)
    else:
        print(f"Function not required, responding to user")
        return response


def call_arxiv_function(messages, full_message):
    """Function calling function which executes function calls when the model believes it is necessary.
    Currently extended by adding clauses to this if statement."""

    if full_message.message.function_call.name == "get_articles":
        try:
            parsed_output = json.loads(
                full_message.message.function_call.arguments
            )
            print("Getting search results")
            results = get_articles(parsed_output["query"])
        except Exception as e:
            print(parsed_output)
            print(f"Function execution failed")
            print(f"Error message: {e}")
        messages.append(
            {
                "role": "function",
                "name": full_message.message.function_call.name,
                "content": str(results),
            }
        )
        try:
            print("Got search results, summarizing content")
            response = chat_completion_request(messages)
            return response
        except Exception as e:
            print(type(e))
            raise Exception("Function chat request failed")

    elif (
        full_message.message.function_call.name == "read_article_and_summarize"
    ):
        parsed_output = json.loads(
            full_message.message.function_call.arguments
        )
        print("Finding and reading paper")
        summary = summarize_text(parsed_output["query"])
        return summary

    else:
        raise Exception("Function does not exist and cannot be called")

```

## arXiv conversation

Let's put this all together by testing our functions out in conversation.


```python
# Start with a system message
paper_system_message = """You are arXivGPT, a helpful assistant pulls academic papers to answer user questions.
You summarize the papers clearly so the customer can decide which to read to answer their question.
You always provide the article_url and title so the user can understand the name of the paper and click through to access it.
Begin!"""
paper_conversation = Conversation()
paper_conversation.add_message("system", paper_system_message)

```


key: _node_content
string_value:
{"id_": "e2988872-1ed7-4da0-9e06-b542bff02b30", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "90dedd29-774a-454e-b743-afd0583c8125", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "7d128a4feb7a34c7f38af361601916f25458e0b3620f1a28ec4c2edf1f37fde3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "243b8e83-d670-4d69-a227-0421ac4cc1ec", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4d887ba32f4d63ce1b20cb2600e5a4bd2ccf0cd156b751e32c750f3d953a1a56", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4663d3dd-7591-4579-be59-a04cfc9a73ba", "node_type": "1", "metadata": {}, "hash": "080f1c8dcfd97e8e4912db3e5715f056283b13c875b328da309ff87e2424e2cd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17171, "end_char_idx": 19868, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
def call_arxiv_function(messages, full_message):
    """Function calling function which executes function calls when the model believes it is necessary.
    Currently extended by adding clauses to this if statement."""

    if full_message.message.function_call.name == "get_articles":
        try:
            parsed_output = json.loads(
                full_message.message.function_call.arguments
            )
            print("Getting search results")
            results = get_articles(parsed_output["query"])
        except Exception as e:
            print(parsed_output)
            print(f"Function execution failed")
            print(f"Error message: {e}")
        messages.append(
            {
                "role": "function",
                "name": full_message.message.function_call.name,
                "content": str(results),
            }
        )
        try:
            print("Got search results, summarizing content")
            response = chat_completion_request(messages)
            return response
        except Exception as e:
            print(type(e))
            raise Exception("Function chat request failed")

    elif (
        full_message.message.function_call.name == "read_article_and_summarize"
    ):
        parsed_output = json.loads(
            full_message.message.function_call.arguments
        )
        print("Finding and reading paper")
        summary = summarize_text(parsed_output["query"])
        return summary

    else:
        raise Exception("Function does not exist and cannot be called")

```

## arXiv conversation

Let's put this all together by testing our functions out in conversation.


```python
# Start with a system message
paper_system_message = """You are arXivGPT, a helpful assistant pulls academic papers to answer user questions.
You summarize the papers clearly so the customer can decide which to read to answer their question.
You always provide the article_url and title so the user can understand the name of the paper and click through to access it.
Begin!"""
paper_conversation = Conversation()
paper_conversation.add_message("system", paper_system_message)

```


```python
# Add a user message
paper_conversation.add_message("user", "Hi, how does PPO reinforcement learning work?")
chat_response = chat_completion_with_function_execution(
    paper_conversation.conversation_history, functions=arxiv_functions
)
assistant_message = chat_response.choices[0].message.content
paper_conversation.add_message("assistant", assistant_message)
display(Markdown(assistant_message))

```

    Function generation requested, calling function
    Getting search results
    Got search results, summarizing content


key: _node_content
string_value:
{"id_": "4663d3dd-7591-4579-be59-a04cfc9a73ba", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "90dedd29-774a-454e-b743-afd0583c8125", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "7d128a4feb7a34c7f38af361601916f25458e0b3620f1a28ec4c2edf1f37fde3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e2988872-1ed7-4da0-9e06-b542bff02b30", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "7af3578405d2d286c349b38b2000d31ec90403b5af3cc5d470326b3492da429b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ac2b479e-5144-44db-a132-b403c1700993", "node_type": "1", "metadata": {}, "hash": "925f77bc44c7ace54858e6dfd2cb44cae9d97e6e9b6542607e6a17b25effd08c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18836, "end_char_idx": 22344, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Start with a system message
paper_system_message = """You are arXivGPT, a helpful assistant pulls academic papers to answer user questions.
You summarize the papers clearly so the customer can decide which to read to answer their question.
You always provide the article_url and title so the user can understand the name of the paper and click through to access it.
Begin!"""
paper_conversation = Conversation()
paper_conversation.add_message("system", paper_system_message)

```


```python
# Add a user message
paper_conversation.add_message("user", "Hi, how does PPO reinforcement learning work?")
chat_response = chat_completion_with_function_execution(
    paper_conversation.conversation_history, functions=arxiv_functions
)
assistant_message = chat_response.choices[0].message.content
paper_conversation.add_message("assistant", assistant_message)
display(Markdown(assistant_message))

```

    Function generation requested, calling function
    Getting search results
    Got search results, summarizing content



PPO (Proximal Policy Optimization) is a reinforcement learning algorithm that aims to find the optimal policy for an agent by optimizing the policy parameters in an iterative manner. Here are a few papers that discuss PPO in more detail:

1. Title: "Proximal Policy Optimization Algorithms"
   Article URL: [arxiv.org/abs/1707.06347v2](http://arxiv.org/abs/1707.06347v2)
   Summary: This paper introduces two algorithms, PPO (Proximal Policy Optimization) and TRPO (Trust Region Policy Optimization), that address the issue of sample efficiency and stability in reinforcement learning. PPO uses a surrogate objective function that makes smaller updates to the policy parameters, resulting in more stable and efficient learning.

2. Title: "Emergence of Locomotion Behaviours in Rich Environments with PPO"
   Article URL: [arxiv.org/abs/1707.02286v3](http://arxiv.org/abs/1707.02286v3)
   Summary: This paper explores the use of PPO in training agents to learn locomotion behaviors in complex and dynamic environments. The authors demonstrate the effectiveness of PPO in learning a variety of locomotion skills, such as walking, jumping, and climbing.

3. Title: "Proximal Policy Optimization for Multi-Agent Systems"
   Article URL: [arxiv.org/abs/2006.14171v2](http://arxiv.org/abs/2006.14171v2)
   Summary: This paper extends PPO to the domain of multi-agent systems, where multiple agents interact and learn together. The authors propose a decentralized version of PPO that allows each agent to update its policy independently based on its local observations, resulting in more scalable and efficient learning in multi-agent environments.

These papers provide detailed explanations of the PPO algorithm, its advantages, and its applications in different scenarios. Reading them can give you a deeper understanding of how PPO reinforcement learning works.



```python
# Add another user message to induce our system to use the second tool
paper_conversation.add_message(
    "user",
    "Can you read the PPO sequence generation paper for me and give me a summary",
)
updated_response = chat_completion_with_function_execution(
    paper_conversation.conversation_history, functions=arxiv_functions
)
display(Markdown(updated_response.choices[0].message.content))

```

    Function generation requested, calling function
    Finding and reading paper
    Chunking text from paper
    Summarizing each chunk of text


    100%|██████████| 6/6 [00:07<00:00,  1.19s/it]


key: _node_content
string_value:
{"id_": "ac2b479e-5144-44db-a132-b403c1700993", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "90dedd29-774a-454e-b743-afd0583c8125", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "7d128a4feb7a34c7f38af361601916f25458e0b3620f1a28ec4c2edf1f37fde3", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4663d3dd-7591-4579-be59-a04cfc9a73ba", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/How_to_call_functions_for_knowledge_retrieval.ipynb", "title": "How to use functions with a knowledge base", "tokens": 5216, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "ee00e99723aa08c2898138deaea3c881b76c049b4db08fef59130fcc7f03147f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21735, "end_char_idx": 24713, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Add another user message to induce our system to use the second tool
paper_conversation.add_message(
    "user",
    "Can you read the PPO sequence generation paper for me and give me a summary",
)
updated_response = chat_completion_with_function_execution(
    paper_conversation.conversation_history, functions=arxiv_functions
)
display(Markdown(updated_response.choices[0].message.content))

```

    Function generation requested, calling function
    Finding and reading paper
    Chunking text from paper
    Summarizing each chunk of text


    100%|██████████| 6/6 [00:07<00:00,  1.19s/it]


    Summarizing into overall summary



Core Argument:
- The academic paper explores the connection between the transverse field Ising (TFI) model and the ϕ4 model, highlighting the analogy between the coupling of topological solitary waves in the ϕ4 model and the effect of the transverse field on spin flips in the TFI model.
- The study reveals regimes of memory/loss of memory and coherence/decoherence in the classical ϕ4 model subjected to periodic perturbations, which are essential in annealing phenomena.
- The exploration of the analogy between lower-dimensional linear quantum systems and higher-dimensional classical nonlinear systems can lead to a deeper understanding of information processing in these systems.

Evidence:
- The authors analyze the dynamics and relaxation of weakly coupled ϕ4 chains through numerical simulations, studying the behavior of kink and breather excitations and the structural phase transition associated with the double well potential.
- The critical temperature (Tc) approaches zero as the inter-chain coupling strength (C⊥) approaches zero, but there is a finite Tc for C⊥>0.
- The spectral function shows peaks corresponding to particle motion across the double-well potential at higher temperatures and oscillations in a single well at lower temperatures.
- The soft-mode frequency (ωs) decreases as temperature approaches Ts, the dynamical crossover temperature.
- The relaxation process of the average displacement (QD) is controlled by spatially extended vibrations and large kink densities.
- The mean domain size (⟨DS⟩) exhibits an algebraic decay for finite C⊥>0.
- The probability of larger domain sizes is higher before a kick compared to after a kick for C⊥>0.

Conclusions:
- The study of weakly-coupled classical ϕ4 chains provides insights into quantum annealing architectures and the role of topological excitations in these systems.
- The equilibration of the system is faster for higher kick strengths, and the mean domain size increases with higher final temperatures.
- Further exploration of the crossover between decoherence and finite coherence in periodic-kick strength space is suggested.
- The paper highlights the importance of recognizing similarities between classical nonlinear equations and quantum linear ones in information processing and suggests future research directions in this area.


key: _node_content
string_value:
{"id_": "5e61b9da-1e81-4958-92bf-7be0bf7c142d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b03f14bb-70f6-4aa1-943b-5684202babde", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "dbe2a88a46b2fed3dc7a92e2e1bff9181de69b0442371522bec375ae9ce86584", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "abeb1c95-a7fa-4f2d-8082-fdc2d02cd815", "node_type": "1", "metadata": {}, "hash": "49ee01e928a34ae21f1de9c7e8e3725cbaab931199fb73255f1b385b51faa3eb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2711, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Synthetic Data generation (Part 1)


Synthetic data generation using large language models (LLMs) offers a powerful solution to a commonly faced problem: the availability of high-quality, diverse, and privacy-compliant data. This could be used in a number of scenarios such as training a data science  machine learning model (SVMs, decision trees, KNN's), finetuning a different GPT model on the data, as a solution to the coldstart problem, helping build compelling demos/apps with realistic data, scenario testing etc.

There are a number of key drivers which may see you wanting to leverage synthetic data. 
1. Human data may have privacy restrictions and/or identifiable data within it which we do not want to be used. 
2. Synthetic data can be much more structured and therefore easier to manipulate than real data. 
3. In domains where data is sparse or data of certain categories is sparse we may want to augment the data. 
4. When dealing with imbalanced datasets or datasets which lack diversity, we may want to create data to improve the richness of our datasets.

Unlike traditional data augmentation or manual data creation methods, using LLMs allows for the generation of rich, nuanced, and contextually relevant datasets that can significantly enhance it's usefulness to enterprises and developers.

We split this tutorial into 2 parts. In this cookbook, we will have the following agenda:
1. CSV with a structured prompt
2. CSV with a Python program
3. Multitable CSV with a python program
4. Simply creating textual data
5. Dealing with imbalanced or non-diverse textual data
while in part 2, we will look at prompting strategies for getting better textual data.

The last two in particular are useful for creating synthetic data to finetune another GPT model. For example using higher quality data produced by `gpt-4o` to finetune the cheaper and quicker `gpt-3.5-turbo` for improved performance while reducing costs.


### Getting setup


```python
%pip install openai
%pip install pandas
%pip install scikit-learn
%pip install matplotlib
```


```python
from openai import OpenAI
import os
import re
import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
import json
import matplotlib

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))
```

### 1. CSV with a structure prompt
Here we create data in the simplest way. You can quickly generate data by addressing 3 key points: telling it the format of the data (CSV), the schema, and useful information regarding how columns relate (the LLM will be able to deduce this from the column names but a helping hand will improve performance).


key: _node_content
string_value:
{"id_": "abeb1c95-a7fa-4f2d-8082-fdc2d02cd815", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b03f14bb-70f6-4aa1-943b-5684202babde", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "dbe2a88a46b2fed3dc7a92e2e1bff9181de69b0442371522bec375ae9ce86584", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5e61b9da-1e81-4958-92bf-7be0bf7c142d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "a390d11f0349b4001b32d20a4b406de76a10ba8d3de5fe70d8557b55e70a38ae", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fa26ca88-8d41-4b8e-a5f8-638429b0e9ab", "node_type": "1", "metadata": {}, "hash": "2c7a757767a39d3c8af34856581764743b042af8eda1205092cc674d8f566f9d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1939, "end_char_idx": 4373, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Getting setup


```python
%pip install openai
%pip install pandas
%pip install scikit-learn
%pip install matplotlib
```


```python
from openai import OpenAI
import os
import re
import numpy as np
import pandas as pd
from sklearn.cluster import KMeans
import matplotlib.pyplot as plt
import json
import matplotlib

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))
```

### 1. CSV with a structure prompt
Here we create data in the simplest way. You can quickly generate data by addressing 3 key points: telling it the format of the data (CSV), the schema, and useful information regarding how columns relate (the LLM will be able to deduce this from the column names but a helping hand will improve performance).


```python
datagen_model = "gpt-4o-mini"
question = """
Create a CSV file with 10 rows of housing data.
Each row should include the following fields:
 - id (incrementing integer starting at 1)
 - house size (m^2)
 - house price
 - location
 - number of bedrooms

Make sure that the numbers make sense (i.e. more rooms is usually bigger size, more expensive locations increase price. more size is usually higher price etc. make sure all the numbers make sense). Also only respond with the CSV.
"""

response = client.chat.completions.create(
  model=datagen_model,
  messages=[
    {"role": "system", "content": "You are a helpful assistant designed to generate synthetic data."},
    {"role": "user", "content": question}
  ]
)
res = response.choices[0].message.content
print(res)
```

    ```csv
    id,house_size_m2,house_price,location,number_of_bedrooms
    1,50,150000,Suburban,2
    2,75,250000,City Center,3
    3,100,350000,Suburban,4
    4,120,450000,Suburban,4
    5,80,300000,City Center,3
    6,90,400000,City Center,3
    7,150,600000,Premium Area,5
    8,200,750000,Premium Area,5
    9,55,180000,Suburban,2
    10,300,950000,Premium Area,6
    ```


### 2. CSV with a Python program
The issue with generating data directly is we are limited in the amount of data we can generate because of the context. Instead what we can do is ask the LLM to generate a python program to generate the synthetic data. This allows us to scale to much more data while also providing us a view into how the data was generated by inspecting the python program.

This would then let us edit the python program as we desire while giving us a good basis to start from.


key: _node_content
string_value:
{"id_": "fa26ca88-8d41-4b8e-a5f8-638429b0e9ab", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b03f14bb-70f6-4aa1-943b-5684202babde", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "dbe2a88a46b2fed3dc7a92e2e1bff9181de69b0442371522bec375ae9ce86584", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "abeb1c95-a7fa-4f2d-8082-fdc2d02cd815", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "8781c08bbcbed0d989a2f3579b1a5b84504f159dc69d648395094db1c959b437", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c29e0980-ba3a-44e1-86ba-0544cec3a8b8", "node_type": "1", "metadata": {}, "hash": "aa076a8765b2bab84cd13af17a8842a34f01bbde49883af51ed15f092f67a5f3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3878, "end_char_idx": 7670, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### 2. CSV with a Python program
The issue with generating data directly is we are limited in the amount of data we can generate because of the context. Instead what we can do is ask the LLM to generate a python program to generate the synthetic data. This allows us to scale to much more data while also providing us a view into how the data was generated by inspecting the python program.

This would then let us edit the python program as we desire while giving us a good basis to start from.



```python
question = """
Create a Python program to generate 100 rows of housing data.
I want you to at the end of it output a pandas dataframe with 100 rows of data.
Each row should include the following fields:
 - id (incrementing integer starting at 1)
 - house size (m^2)
 - house price
 - location
 - number of bedrooms

Make sure that the numbers make sense (i.e. more rooms is usually bigger size, more expensive locations increase price. more size is usually higher price etc. make sure all the numbers make sense).
"""

response = client.chat.completions.create(
  model=datagen_model,
  messages=[
    {"role": "system", "content": "You are a helpful assistant designed to generate synthetic data."},
    {"role": "user", "content": question}
  ]
)
res = response.choices[0].message.content
print(res)
```

    Certainly! Below is a Python program that generates synthetic housing data according to your specifications. We will create a pandas DataFrame with the defined fields and characteristics.
    
    ```python
    import pandas as pd
    import random
    
    def generate_housing_data(num_rows):
        data = []
        
        locations = [
            ('City Center', 10000, 150),  # (location name, base price per m², base size)
            ('Suburban Area', 8000, 100),
            ('Country Side', 5000, 80),
            ('Coastal Region', 12000, 110),
            ('Urban Neighborhood', 9000, 130)
        ]
        
        for i in range(1, num_rows + 1):
            # Randomly pick a location
            location, base_price_per_m2, base_size = random.choice(locations)
            
            # Generate number of bedrooms (1 to 5)
            number_of_bedrooms = random.randint(1, 5)
            
            # Calculate house size based on the number of bedrooms
            house_size = base_size + (10 * number_of_bedrooms) + random.randint(-5, 15)  # Adding some noise
            
            # Calculate house price based on house size and location
            house_price = base_price_per_m2 * house_size + random.randint(-5000, 10000)  # Adding some noise
    
            # Append the generated data to the list
            data.append({
                'id': i,
                'house_size_m2': house_size,
                'house_price': house_price,
                'location': location,
                'number_of_bedrooms': number_of_bedrooms
            })
    
        # Create a pandas DataFrame
        df = pd.DataFrame(data)
        return df
    
    # Generate 100 rows of housing data
    housing_data_df = generate_housing_data(100)
    
    # Show the result
    print(housing_data_df)
    ```
    
    ### Explanation:
    - The `generate_housing_data` function creates synthetic housing data for a specified number of rows (`num_rows`).
    - We define different locations with corresponding base prices per square meter and average house sizes.
    - For each house, we randomly select a location, number of bedrooms, and calculate house size and price to ensure a sensible correlation between the values.
    - Finally, we create a pandas DataFrame from the generated data and return it.
    
    You can run this program in your Python environment, and it will output a DataFrame containing 100 rows of synthetic housing data.


key: _node_content
string_value:
{"id_": "c29e0980-ba3a-44e1-86ba-0544cec3a8b8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b03f14bb-70f6-4aa1-943b-5684202babde", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "dbe2a88a46b2fed3dc7a92e2e1bff9181de69b0442371522bec375ae9ce86584", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fa26ca88-8d41-4b8e-a5f8-638429b0e9ab", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "94cf068ac50da82583fe418fe6ef251759c99a959d34b17b698e124b75e902ca", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e75835fb-249e-4a8f-87e7-2513724eb598", "node_type": "1", "metadata": {}, "hash": "16ea581358e7938df7828f6d4365438374b63286c05a0564675d4dd9151b90b0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7673, "end_char_idx": 10470, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We need to make sure to parse the output of this appropriately as often there may be surrounding text to the python code. We can also explicitly ask it to state all assumptions it made about the data it's generating, however in this circumstance it told us that automatically.

### 3. Multitable CSV with a python program
For more complex relationships however we need to make sure to specify a few more characteristics. 

To create multiple different datasets which relate to each other (for example housing, location, house type), as before we would need to specify the format, schema and useful information. However, the useful information required to get good performance is higher now. It's case-specific but a good amount of things to describe would be how the datasets relate to each other, addressing the size of the datasets in relation to one another, making sure foreign and primary keys are made appropriately and ideally using previously generated datasets to populate new ones so the actual data values match where necessary.


```python
question = """
Create a Python program to generate 3 different pandas dataframes.

1. Housing data
I want 100 rows. Each row should include the following fields:
 - id (incrementing integer starting at 1)
 - house size (m^2)
 - house price
 - location
 - number of bedrooms
 - house type
 + any relevant foreign keys

2. Location
Each row should include the following fields:
 - id (incrementing integer starting at 1)
 - country
 - city
 - population
 - area (m^2)
 + any relevant foreign keys

 3. House types
 - id (incrementing integer starting at 1)
 - house type
 - average house type price
 - number of houses
 + any relevant foreign keys

Make sure that the numbers make sense (i.e. more rooms is usually bigger size, more expensive locations increase price. more size is usually higher price etc. make sure all the numbers make sense).
Make sure that the dataframe generally follow common sense checks, e.g. the size of the dataframes make sense in comparison with one another.
Make sure the foreign keys match up and you can use previously generated dataframes when creating each consecutive dataframes.
You can use the previously generated dataframe to generate the next dataframe.
"""

response = client.chat.completions.create(
  model=datagen_model,
  messages=[
    {"role": "system", "content": "You are a helpful assistant designed to generate synthetic data."},
    {"role": "user", "content": question}
  ]
)
res = response.choices[0].message.content
print(res)
```

    Certainly! Below is a Python program that generates the three specified pandas DataFrames for housing data, location data, and house types. Each DataFrame will include the necessary fields, and the foreign keys will ensure proper relationships among them.


key: _node_content
string_value:
{"id_": "e75835fb-249e-4a8f-87e7-2513724eb598", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b03f14bb-70f6-4aa1-943b-5684202babde", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "dbe2a88a46b2fed3dc7a92e2e1bff9181de69b0442371522bec375ae9ce86584", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c29e0980-ba3a-44e1-86ba-0544cec3a8b8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "49775ab0a87e64f3d5885c9490c1ac5d20f659ebd91ec86921ff4222885cd552", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e5ea7b8e-3dd6-4d64-8b02-0215447087ea", "node_type": "1", "metadata": {}, "hash": "c85a96c211ebb27c0096724b202f1b0077c6a0a49b17f0faf10be94b59e32abe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8715, "end_char_idx": 13481, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
question = """
Create a Python program to generate 3 different pandas dataframes.

1. Housing data
I want 100 rows. Each row should include the following fields:
 - id (incrementing integer starting at 1)
 - house size (m^2)
 - house price
 - location
 - number of bedrooms
 - house type
 + any relevant foreign keys

2. Location
Each row should include the following fields:
 - id (incrementing integer starting at 1)
 - country
 - city
 - population
 - area (m^2)
 + any relevant foreign keys

 3. House types
 - id (incrementing integer starting at 1)
 - house type
 - average house type price
 - number of houses
 + any relevant foreign keys

Make sure that the numbers make sense (i.e. more rooms is usually bigger size, more expensive locations increase price. more size is usually higher price etc. make sure all the numbers make sense).
Make sure that the dataframe generally follow common sense checks, e.g. the size of the dataframes make sense in comparison with one another.
Make sure the foreign keys match up and you can use previously generated dataframes when creating each consecutive dataframes.
You can use the previously generated dataframe to generate the next dataframe.
"""

response = client.chat.completions.create(
  model=datagen_model,
  messages=[
    {"role": "system", "content": "You are a helpful assistant designed to generate synthetic data."},
    {"role": "user", "content": question}
  ]
)
res = response.choices[0].message.content
print(res)
```

    Certainly! Below is a Python program that generates the three specified pandas DataFrames for housing data, location data, and house types. Each DataFrame will include the necessary fields, and the foreign keys will ensure proper relationships among them.
    
    ```python
    import pandas as pd
    import numpy as np
    
    # Set random seed for reproducibility
    np.random.seed(0)
    
    # Function to generate location DataFrame
    def generate_location_data(num_locations):
        locations = {
            "id": range(1, num_locations + 1),
            "country": np.random.choice(['USA', 'Canada', 'UK'], num_locations),
            "city": np.random.choice(['New York', 'Toronto', 'London', 'Vancouver', 'Manchester'], num_locations),
            "population": np.random.randint(50000, 1000000, num_locations),
            "area": np.random.randint(10000, 500000, num_locations)
        }
        return pd.DataFrame(locations)
    
    # Function to generate house types DataFrame
    def generate_house_type_data(num_house_types):
        house_types = {
            "id": range(1, num_house_types + 1),
            "house_type": np.random.choice(['Detached', 'Semi-Detached', 'Terraced', 'Flat'], num_house_types),
            "average_house_type_price": np.random.randint(100000, 1000000, num_house_types),
            "number_of_houses": np.random.randint(10, 1000, num_house_types)
        }
        return pd.DataFrame(house_types)
    
    # Function to generate housing data DataFrame
    def generate_housing_data(num_houses, location_df, house_type_df):
        house_sizes = np.random.randint(50, 300, num_houses)  # size in m^2
        location_ids = np.random.choice(location_df['id'], num_houses)
        house_type_ids = np.random.choice(house_type_df['id'], num_houses)
        
        # Generate prices based on size, location, and house type
        house_prices = (house_sizes * np.random.randint(2000, 5000, num_houses) // 10) + \
                       (location_ids * 1000) + \
                       (house_type_df.loc[house_type_ids - 1, 'average_house_type_price'].values // 4)
        
        housing_data = {
            "id": range(1, num_houses + 1),
            "house_size": house_sizes,
            "house_price": house_prices,
            "location_id": location_ids,
            "bedrooms": np.random.randint(1, 6, num_houses),
            "house_type_id": house_type_ids
        }
        
        return pd.DataFrame(housing_data)
    
    # Generate DataFrames
    num_locations = 10
    num_house_types = 4
    num_houses = 100
    
    location_df = generate_location_data(num_locations)
    house_type_df = generate_house_type_data(num_house_types)
    housing_df = generate_housing_data(num_houses, location_df, house_type_df)
    
    # Display the generated DataFrames
    print("Location DataFrame:")
    print(location_df.head(), "\n")
    
    print("House Types DataFrame:")
    print(house_type_df.head(), "\n")
    
    print("Housing DataFrame:")
    print(housing_df.head(), "\n")
    
    # Printing the DataFrame shapes
    print(f"Shapes: \nLocation: {location_df.shape}, House Types: {house_type_df.shape}, Housing: {housing_df.shape}")
    ```
    
    ### Explanation of the Code:
    1.


key: _node_content
string_value:
{"id_": "e5ea7b8e-3dd6-4d64-8b02-0215447087ea", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b03f14bb-70f6-4aa1-943b-5684202babde", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "dbe2a88a46b2fed3dc7a92e2e1bff9181de69b0442371522bec375ae9ce86584", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e75835fb-249e-4a8f-87e7-2513724eb598", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "52abf7b32890e12110740822a7e3c151c02f09eb33884ef2eefc3e79bc42d26e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ec22eded-bd8b-4e26-8340-9ac54e4fe12d", "node_type": "1", "metadata": {}, "hash": "f3d0f24dda79e5e2b243302f6bffa8e321ed8eaa8a8816f7a15c475f6e7d2e43", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13482, "end_char_idx": 15207, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
**Location DataFrame:** 
       - Generates random locations with attributes such as country, city, population, and area.
      
    2. **House Types DataFrame:** 
       - Generates different types of houses along with average prices and quantity available.
      
    3. **Housing DataFrame:** 
       - Generates housing data with increments on price based on house size, location, and house type, while also ensuring foreign keys (IDs) for location and house type.
    
    ### Output:
    The three DataFrames generated will logically relate to one another with consistent data types and primary–foreign key relationships, resulting in a coherent representation of the housing dataset. The output displays heads of each DataFrame and their shapes for verification.


### 4. Simply creating textual data
Here we take a first look at creating textual data. This can be used to finetune another GPT model for example. In this case we imagine ourselves a retailer trying to streamline the process of creating descriptions for items they are selling. We again need to specify the format of the data, in particular in this case we want one which is easy to parse as an output.

The example we consider below is one in which we want to create input output training pairs for GPT model to finetune on. We will have the products' name and the category it belongs to as input and the output will be a description. 

Specifying the structure of the output explicitly and giving commands to not deviate from this help enforce the output structure. You can run this in a loop and append the data to generate more synthetic data. Again, as before we will need to parse the data well so that our code further downstream does not break.


key: _node_content
string_value:
{"id_": "ec22eded-bd8b-4e26-8340-9ac54e4fe12d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b03f14bb-70f6-4aa1-943b-5684202babde", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "dbe2a88a46b2fed3dc7a92e2e1bff9181de69b0442371522bec375ae9ce86584", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e5ea7b8e-3dd6-4d64-8b02-0215447087ea", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "a6d86fed0838a014234bba8d64ed9352f43dcec5fc8d9dc2f9058b7b319f2ced", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8d9c9870-547b-4ade-996b-16595fa827fc", "node_type": "1", "metadata": {}, "hash": "107dcd1b16bd56ac232db0ef144fbcbcd62ee166a03bdbe078efb573ffa19887", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13482, "end_char_idx": 17323, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
**Location DataFrame:** 
       - Generates random locations with attributes such as country, city, population, and area.
      
    2. **House Types DataFrame:** 
       - Generates different types of houses along with average prices and quantity available.
      
    3. **Housing DataFrame:** 
       - Generates housing data with increments on price based on house size, location, and house type, while also ensuring foreign keys (IDs) for location and house type.
    
    ### Output:
    The three DataFrames generated will logically relate to one another with consistent data types and primary–foreign key relationships, resulting in a coherent representation of the housing dataset. The output displays heads of each DataFrame and their shapes for verification.


### 4. Simply creating textual data
Here we take a first look at creating textual data. This can be used to finetune another GPT model for example. In this case we imagine ourselves a retailer trying to streamline the process of creating descriptions for items they are selling. We again need to specify the format of the data, in particular in this case we want one which is easy to parse as an output.

The example we consider below is one in which we want to create input output training pairs for GPT model to finetune on. We will have the products' name and the category it belongs to as input and the output will be a description. 

Specifying the structure of the output explicitly and giving commands to not deviate from this help enforce the output structure. You can run this in a loop and append the data to generate more synthetic data. Again, as before we will need to parse the data well so that our code further downstream does not break.


```python
output_string = ""
for i in range(3):
  question = f"""
  I am creating input output training pairs to fine tune my gpt model. The usecase is a retailer generating a description for a product from a product catalogue. I want the input to be product name and category (to which the product belongs to) and output to be description.
  The format should be of the form:
  1.
  Input: product_name, category
  Output: description
  2.
  Input: product_name, category
  Output: description

  Do not add any extra characters around that formatting as it will make the output parsing break.
  Create as many training pairs as possible.
  """

  response = client.chat.completions.create(
    model=datagen_model,
    messages=[
      {"role": "system", "content": "You are a helpful assistant designed to generate synthetic data."},
      {"role": "user", "content": question}
    ]
  )
  res = response.choices[0].message.content
  output_string += res + "\n" + "\n"
print(output_string[:1000]) #displaying truncated response

```

    1.
    Input: Wireless Bluetooth Headphones, Electronics
    Output: Immerse yourself in high-quality sound with these Wireless Bluetooth Headphones, featuring active noise cancellation and a comfortable over-ear design for extended listening sessions.
    
    2.
    Input: Organic Green Tea, Beverages
    Output: Enjoy a refreshing cup of Organic Green Tea, sourced from the finest leaves, packed with antioxidants, and perfect for a healthy, invigorating boost anytime.
    
    3.
    Input: Stainless Steel Kitchen Knife, Kitchenware
    Output: Cut with precision and ease using this Stainless Steel Kitchen Knife, designed with an ergonomic handle and a sharp blade for all your culinary tasks.
    
    4.
    Input: Hiking Backpack, Outdoor Gear
    Output: Explore the great outdoors with this durable Hiking Backpack, featuring multiple compartments for optimal organization and a breathable design for ultimate comfort on long treks.
    
    5.
    Input: Air Fryer, Kitchen Appliances
    Output: Cook your favorite meals with less oil using this Air Fryer


key: _node_content
string_value:
{"id_": "8d9c9870-547b-4ade-996b-16595fa827fc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b03f14bb-70f6-4aa1-943b-5684202babde", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "dbe2a88a46b2fed3dc7a92e2e1bff9181de69b0442371522bec375ae9ce86584", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ec22eded-bd8b-4e26-8340-9ac54e4fe12d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d353dfd34567862fc560d3b5ce5cf4e985f721d3b1cd339c19cf9d9b6c9ae85d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f395bc0e-7ac3-44bb-80bd-ef9e9d057ea8", "node_type": "1", "metadata": {}, "hash": "bb028639b0decedb89bbdee3201b4e6adcdf2a7455c60e40e2ed1b2b73a04829", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17326, "end_char_idx": 20387, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note: the above output is truncated. And now we can parse it as below to get a list of products, categories and their descriptions. For example, let's take a look at the products it's generated.


```python
#regex to parse data
pattern = re.compile(r'Input:\s*(.+?),\s*(.+?)\nOutput:\s*(.+?)(?=\n\n|\Z)', re.DOTALL)
matches = pattern.findall(output_string)
products = []
categories = []
descriptions = []

for match in matches:
    product, category, description = match
    products.append(product.strip())
    categories.append(category.strip())
    descriptions.append(description.strip())
products
```




    ['Wireless Bluetooth Headphones',
     'Organic Green Tea',
     'Stainless Steel Kitchen Knife',
     'Hiking Backpack',
     'Air Fryer',
     "Kids' Educational Tablet",
     'Bluetooth Speaker',
     'Yoga Mat',
     'Memory Foam Mattress',
     'Smartwatch',
     'Leather Wallet',
     'Portable Phone Charger',
     'Non-Stick Cookware Set',
     'Pet Dog Bed',
     'Fitness Tracker',
     'Wireless Earbuds',
     'Organic Green Tea',
     'Reusable Water Bottle',
     'Yoga Mat',
     'Leather Wallet',
     'Air Fryer',
     'Gaming Mouse',
     'Crochet Kit',
     'Hiking Boots',
     'Scented Candles',
     'Bluetooth Speaker',
     'Stainless Steel Cookware Set',
     'Fitness Tracker',
     'Decorative Throw Pillows',
     'Eco-Friendly Cleaning Supplies',
     'Wireless Noise Cancelling Headphones',
     'Organic Green Tea',
     'Adjustable Yoga Mat',
     'Bluetooth Smart Scale',
     'Stainless Steel Water Bottle',
     'Soft Cotton Bedding Set',
     'Multi-Functional Kitchen Blender',
     'Eco-Friendly Reusable Bags',
     'Portable Phone Charger',
     'Classic Leather Wallet',
     'Suede Chelsea Boots',
     'Non-Stick Cookware Set',
     'Pet-Friendly Indoor Plants',
     'High-Protein Snack Bars',
     'LED Desk Lamp with USB Port']




### 5. Dealing with imbalanced or non-diverse textual data
Some of the most important aspects of generating high-quality synthetic data are accuracy (does the data make sense), consistency (are two separate data points for the same input roughly the same) and diversity (making sure our data distribution matches as much of the distribution that exists in production).


To increase the diversity of our data, we start first by clustering the data. This will provide us information about which clusters are underrepresented (imbalanced dataset) or which data is not addressed at all (widening the data distribution). Then, we will either suggest new clusters (using self-reflection type call from GPT) or ask the next iteration of our synthetic generation calls to explicitly target the underrepresented clusters. 

We can then recursively run this generation and analysis of cluster loop to automate generating diverse synthetic data.

For demonstrative purposes, we explicitly prompt the LLM to generate information about 4 different topical areas: vehicle, clothing, toiletries, food. We will then cluster the data and see if it managed to find these 4 topic areas.


key: _node_content
string_value:
{"id_": "f395bc0e-7ac3-44bb-80bd-ef9e9d057ea8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b03f14bb-70f6-4aa1-943b-5684202babde", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "dbe2a88a46b2fed3dc7a92e2e1bff9181de69b0442371522bec375ae9ce86584", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8d9c9870-547b-4ade-996b-16595fa827fc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "68089623b88f8f25b60551b95b42f3ccb70b8fd27db959c97f28d91a83d5892c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9ac9d14a-ddd6-4a55-bc9b-21910052d63f", "node_type": "1", "metadata": {}, "hash": "9f643c5e3db59aa48aa3ce1771d1fbf72f2826afcfb6188005c1da8be7558526", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19219, "end_char_idx": 22835, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### 5. Dealing with imbalanced or non-diverse textual data
Some of the most important aspects of generating high-quality synthetic data are accuracy (does the data make sense), consistency (are two separate data points for the same input roughly the same) and diversity (making sure our data distribution matches as much of the distribution that exists in production).


To increase the diversity of our data, we start first by clustering the data. This will provide us information about which clusters are underrepresented (imbalanced dataset) or which data is not addressed at all (widening the data distribution). Then, we will either suggest new clusters (using self-reflection type call from GPT) or ask the next iteration of our synthetic generation calls to explicitly target the underrepresented clusters. 

We can then recursively run this generation and analysis of cluster loop to automate generating diverse synthetic data.

For demonstrative purposes, we explicitly prompt the LLM to generate information about 4 different topical areas: vehicle, clothing, toiletries, food. We will then cluster the data and see if it managed to find these 4 topic areas.


```python
output_string = ""
for i in range(3):
  question = f"""
  I am creating input output training pairs to fine tune my gpt model. I want the input to be product name and category and output to be description. the category should be things like: mobile phones, shoes, headphones, laptop, electronic toothbrush, etc. and also more importantly the categories should come under 4 main topics: vehicle, clothing, toiletries, food)
  After the number of each example also state the topic area. The format should be of the form:
  1. topic_area
  Input: product_name, category
  Output: description

  Do not add any extra characters around that formatting as it will make the output parsing break.

  Here are some helpful examples so you get the style of output correct.

  1) clothing
  Input: "Shoe Name, Shoes"
  Output: "Experience unparalleled comfort. These shoes feature a blend of modern style and the traditional superior cushioning, perfect for those always on the move."
  """

  response = client.chat.completions.create(
    model="gpt-4o-mini",
    messages=[
      {"role": "system", "content": "You are a helpful assistant designed to generate synthetic data."},
      {"role": "user", "content": question}
    ]
  )
  res = response.choices[0].message.content
  output_string += res + "\n" + "\n"
print(output_string[:1000]) #displaying truncated response
```

    1. vehicle  
    Input: "Tesla Model 3, Electric Car"  
    Output: "The Tesla Model 3 is a revolutionary electric car with impressive range and cutting-edge technology, designed to provide an exhilarating driving experience while minimizing environmental impact."
    
    2. clothing  
    Input: "Nike Air Max, Shoes"  
    Output: "Elevate your sneaker game with Nike Air Max. Combining iconic style with superior comfort and support, these shoes are perfect for both workouts and casual outings."
    
    3. toiletries  
    Input: "Oral-B Pro 1000, Electronic Toothbrush"  
    Output: "Achieve a superior clean with the Oral-B Pro 1000. This electronic toothbrush features 3D cleaning action that pulsates and oscillates to remove more plaque than a regular manual toothbrush."
    
    4. food  
    Input: "Chobani Greek Yogurt, Yogurt"  
    Output: "Indulge in a nutritious snack with Chobani Greek Yogurt. Packed with protein and delicious flavors, it’s the perfect choice for a healthy breakfast or a satisfying treat anytime."
    
    5. vehicle


key: _node_content
string_value:
{"id_": "9ac9d14a-ddd6-4a55-bc9b-21910052d63f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b03f14bb-70f6-4aa1-943b-5684202babde", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "dbe2a88a46b2fed3dc7a92e2e1bff9181de69b0442371522bec375ae9ce86584", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f395bc0e-7ac3-44bb-80bd-ef9e9d057ea8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "2db513d2f7abcf42ad5cde928ad336c807c24165abc5b945e09eba4d8eb1e3a9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ed7b16b3-6c9a-47b2-8ec5-cba62f19f928", "node_type": "1", "metadata": {}, "hash": "ee7ce49eda7a801dcd0d4ceb6c7d054839655873f96cd66edbe77632f040cba8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 22845, "end_char_idx": 25758, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note: The above output is truncated. In the example above, we would explicitly include the topic area as part of the response per example as it helps condition the proceeding output and tends to give better performance. We can also give it an actual example of what the output should look like so it gets the right idea of style of output but also to help enforce structure.


```python
pattern = re.compile(r'(\d+)\.\s*(\w+)\s*Input:\s*"(.+?),\s*(.+?)"\s*Output:\s*"(.*?)"', re.DOTALL)
matches = pattern.findall(output_string)

topics = []
products = []
categories = []
descriptions = []

for match in matches:
    number, topic, product, category, description = match
    topics.append(topic)
    products.append(product)
    categories.append(category)
    descriptions.append(description)

```


```python
products
```




    ['Tesla Model 3',
     'Nike Air Max',
     'Oral-B Pro 1000',
     'Chobani Greek Yogurt',
     'Ford F-150',
     "Levi's 511",
     'Philips Sonicare',
     'Quaker Oatmeal',
     'Toyota Camry',
     'Adidas Ultraboost',
     'Toyota Camry',
     'Nike Air Max',
     'Colgate Electric Toothbrush',
     'Blue Diamond Almonds',
     'Harley Davidson Fat Boy',
     'Adidas UltraBoost',
     "Dove Men's Body Wash",
     'Quaker Oats',
     'Ford F-150',
     "Levi's 501 Jeans",
     'Tesla Model 3',
     'Nike Air Max',
     'Oral-B Pro 1000',
     'Organic Almond Butter',
     'Yamaha YZF-R3',
     'Adidas Ultraboost',
     'Philips Sonicare',
     'Organic Quinoa']



We will now cluster the data to analyze it. We will use K-means clustering to segregate the data. An important parameter of K-means to set is K, the number of clusters.

We know that there should be 4 cluster (4 topics) since we specified this in prompt: vehicle, electronics, clothing, food. However in general for our data, we do not know the number of clusters that exist. Therefore we will use the elbow method to find the optimal number of clusters.

In the elbow method, we iterate through a range of different K's, each time storing the inertia. The inertia measures the sum of the squared distances between each point in a cluster and the centroid of that cluster thus telling us how well-separated and dense each cluster is. If we plot K against the inertia, we are able to see how the inertia drops and where the drop in inertia is least rapid (often making an elbow shape) we can set our optimal number of clusters. You can read into more depth about the elbow method [here](https://en.wikipedia.org/wiki/Elbow_method_(clustering)).

First let's store our data into a pandas dataframe for ease of analysis




```python
data = {
    'Product': products,
    'Category': categories,
    'Description': descriptions
}

df = pd.DataFrame(data)
```

Next let us embed our data as the embeddings is what we will cluster since they should be close to each other in vector space if they are similar.


key: _node_content
string_value:
{"id_": "ed7b16b3-6c9a-47b2-8ec5-cba62f19f928", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b03f14bb-70f6-4aa1-943b-5684202babde", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "dbe2a88a46b2fed3dc7a92e2e1bff9181de69b0442371522bec375ae9ce86584", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9ac9d14a-ddd6-4a55-bc9b-21910052d63f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3ca8b4e666d2739a831394d3396097db3b7f4bc3beac58ec92d74b3928757d7d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0667c19a-4b21-4494-a5f2-bce086fe7ca3", "node_type": "1", "metadata": {}, "hash": "f36525c142765e26c98342b4f3d0733fbfca7f344f4e5a7a9430c42f0f7a03d6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24355, "end_char_idx": 27149, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We will now cluster the data to analyze it. We will use K-means clustering to segregate the data. An important parameter of K-means to set is K, the number of clusters.

We know that there should be 4 cluster (4 topics) since we specified this in prompt: vehicle, electronics, clothing, food. However in general for our data, we do not know the number of clusters that exist. Therefore we will use the elbow method to find the optimal number of clusters.

In the elbow method, we iterate through a range of different K's, each time storing the inertia. The inertia measures the sum of the squared distances between each point in a cluster and the centroid of that cluster thus telling us how well-separated and dense each cluster is. If we plot K against the inertia, we are able to see how the inertia drops and where the drop in inertia is least rapid (often making an elbow shape) we can set our optimal number of clusters. You can read into more depth about the elbow method [here](https://en.wikipedia.org/wiki/Elbow_method_(clustering)).

First let's store our data into a pandas dataframe for ease of analysis




```python
data = {
    'Product': products,
    'Category': categories,
    'Description': descriptions
}

df = pd.DataFrame(data)
```

Next let us embed our data as the embeddings is what we will cluster since they should be close to each other in vector space if they are similar.


```python
def get_embedding(text, model="text-embedding-3-small"):
    text = text.replace("\n", " ")

    response = client.embeddings.create(input=[text], model=model)

    return response.data[0].embedding

embedding_model = "text-embedding-3-small"
df["embedding"] = df.Category.apply(lambda x: get_embedding(x, model=embedding_model))

# Ensure there are embeddings to concatenate
if len(df.embedding.values) > 0:
    matrix = np.vstack(df.embedding.values)
else:
    matrix = np.array([])  # Handle the case where there are no embeddings

```


```python
df
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Product</th>
      <th>Category</th>
      <th>Description</th>
      <th>embedding</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Tesla Model 3</td>
      <td>Electric Car</td>
      <td>The Tesla Model 3 is a revolutionary electric ...</td>
      <td>[0.003255360759794712, -0.039260633289813995, ...</td>
    </tr>
    <tr>
      <th>1</th>
      <td>Nike Air Max</td>
      <td>Shoes</td>
      <td>Elevate your sneaker game with Nike Air Max.


key: _node_content
string_value:
{"id_": "0667c19a-4b21-4494-a5f2-bce086fe7ca3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b03f14bb-70f6-4aa1-943b-5684202babde", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "dbe2a88a46b2fed3dc7a92e2e1bff9181de69b0442371522bec375ae9ce86584", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ed7b16b3-6c9a-47b2-8ec5-cba62f19f928", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "c126b7ccc8de59173a47b327cf8e10f63d47db959b960bdab4da2c4c5cd515c1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b5b5da8f-dac4-466d-869d-cbd90329ca3e", "node_type": "1", "metadata": {}, "hash": "e617f7016719dc7cbcd888bfc804faf08fc64faedc223d08d90222ce9daa5cd0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25761, "end_char_idx": 28036, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def get_embedding(text, model="text-embedding-3-small"):
    text = text.replace("\n", " ")

    response = client.embeddings.create(input=[text], model=model)

    return response.data[0].embedding

embedding_model = "text-embedding-3-small"
df["embedding"] = df.Category.apply(lambda x: get_embedding(x, model=embedding_model))

# Ensure there are embeddings to concatenate
if len(df.embedding.values) > 0:
    matrix = np.vstack(df.embedding.values)
else:
    matrix = np.array([])  # Handle the case where there are no embeddings

```


```python
df
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Product</th>
      <th>Category</th>
      <th>Description</th>
      <th>embedding</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Tesla Model 3</td>
      <td>Electric Car</td>
      <td>The Tesla Model 3 is a revolutionary electric ...</td>
      <td>[0.003255360759794712, -0.039260633289813995, ...</td>
    </tr>
    <tr>
      <th>1</th>
      <td>Nike Air Max</td>
      <td>Shoes</td>
      <td>Elevate your sneaker game with Nike Air Max. C...</td>
      <td>[0.03943369910120964, 0.022045187652111053, -0...</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Oral-B Pro 1000</td>
      <td>Electronic Toothbrush</td>
      <td>Achieve a superior clean with the Oral-B Pro 1...</td>
      <td>[-0.003470012918114662, -0.01911414973437786, ...</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Chobani Greek Yogurt</td>
      <td>Yogurt</td>
      <td>Indulge in a nutritious snack with Chobani Gre...</td>
      <td>[0.0208318829536438, -0.02645781636238098, -0....</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Ford F-150</td>
      <td>Pickup Truck</td>
      <td>The Ford F-150 is the ultimate pickup truck, d...</td>
      <td>[0.007467855699360371, -0.05288049206137657, -...</td>
    </tr>
    <tr>
      <th>5</th>
      <td>Levi's 511</td>
      <td>Jeans</td>
      <td>Step out in style with Levi's 511 jeans.


key: _node_content
string_value:
{"id_": "b5b5da8f-dac4-466d-869d-cbd90329ca3e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b03f14bb-70f6-4aa1-943b-5684202babde", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "dbe2a88a46b2fed3dc7a92e2e1bff9181de69b0442371522bec375ae9ce86584", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0667c19a-4b21-4494-a5f2-bce086fe7ca3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "66f1d1d55f07cf745c5e0b1da012ad5fe9c5ef04b9b6d8e268128a3a674f0954", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "611419a2-820e-471a-9bbc-8903ce93348e", "node_type": "1", "metadata": {}, "hash": "c5bd4840b32918054101fb5947705345a4143cd4a4b884578091fc676a7af83c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27150, "end_char_idx": 28498, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
C...</td>
      <td>[0.03943369910120964, 0.022045187652111053, -0...</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Oral-B Pro 1000</td>
      <td>Electronic Toothbrush</td>
      <td>Achieve a superior clean with the Oral-B Pro 1...</td>
      <td>[-0.003470012918114662, -0.01911414973437786, ...</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Chobani Greek Yogurt</td>
      <td>Yogurt</td>
      <td>Indulge in a nutritious snack with Chobani Gre...</td>
      <td>[0.0208318829536438, -0.02645781636238098, -0....</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Ford F-150</td>
      <td>Pickup Truck</td>
      <td>The Ford F-150 is the ultimate pickup truck, d...</td>
      <td>[0.007467855699360371, -0.05288049206137657, -...</td>
    </tr>
    <tr>
      <th>5</th>
      <td>Levi's 511</td>
      <td>Jeans</td>
      <td>Step out in style with Levi's 511 jeans. Featu...</td>
      <td>[0.0037206460256129503, 0.022772302851080894, ...</td>
    </tr>
    <tr>
      <th>6</th>
      <td>Philips Sonicare</td>
      <td>Electric Toothbrush</td>
      <td>Discover a new level of oral care with the Phi...</td>
      <td>[-0.00724813062697649, -0.011600878089666367, ...</td>
    </tr>
    <tr>
      <th>7</th>
      <td>Quaker Oatmeal</td>
      <td>Breakfast Cereal</td>
      <td>Start your day right with Quaker Oatmeal.


key: _node_content
string_value:
{"id_": "611419a2-820e-471a-9bbc-8903ce93348e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b03f14bb-70f6-4aa1-943b-5684202babde", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "dbe2a88a46b2fed3dc7a92e2e1bff9181de69b0442371522bec375ae9ce86584", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b5b5da8f-dac4-466d-869d-cbd90329ca3e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/SDG1.ipynb", "title": "Synthetic Data generation (Part 1)", "tokens": 15064, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "68474acb662cc67a699ae6d77964b95d208a90d05052faf7dc53788b5de22696", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aa6be201-d473-489d-812c-0fd1fe9814b9", "node_type": "1", "metadata": {}, "hash": "b0794f7573912c1050eecc2938c42d5d4eb5e05ed924fe59629c008d2b697f6f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 28037, "end_char_idx": 29840, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Featu...</td>
      <td>[0.0037206460256129503, 0.022772302851080894, ...</td>
    </tr>
    <tr>
      <th>6</th>
      <td>Philips Sonicare</td>
      <td>Electric Toothbrush</td>
      <td>Discover a new level of oral care with the Phi...</td>
      <td>[-0.00724813062697649, -0.011600878089666367, ...</td>
    </tr>
    <tr>
      <th>7</th>
      <td>Quaker Oatmeal</td>
      <td>Breakfast Cereal</td>
      <td>Start your day right with Quaker Oatmeal. This...</td>
      <td>[-0.006529285106807947, 0.007865572348237038, ...</td>
    </tr>
    <tr>
      <th>8</th>
      <td>Toyota Camry</td>
      <td>Sedan</td>
      <td>The Toyota Camry stands out in the sedan categ...</td>
      <td>[-0.02088991366326809, -0.006191295105963945, ...</td>
    </tr>
    <tr>
      <th>9</th>
      <td>Adidas Ultraboost</td>
      <td>Running Shoes</td>
      <td>Run like never before in the Adidas Ultraboost...</td>
      <td>[0.02679188922047615, 0.014639599248766899, 8....</td>
    </tr>
    <tr>
      <th>10</th>
      <td>Toyota Camry</td>
      <td>Car</td>
      <td>The Toyota Camry is a reliable midsize sedan k...</td>
      <td>[0.008056452497839928, -0.007912316359579563, ...</td>
    </tr>
    <tr>
      <th>11</th>
      <td>Nike Air Max</td>
      <td>Shoes</td>
      <td>Step up your sneaker game with the Nike Air Ma...</td>
      <td>[0.03943241760134697, 0.02208484522998333, -0....</td>
    </tr>
    <tr>
      <th>12</th>
      <td>Colgate Electric Toothbrush</td>
      <td>Electronic Toothbrush</td>
      <td>Transform your oral hygiene routine with the C...</td>
      <td>[-0.003470012918114662, -0.01911414973437786, ...</td>
    </tr>
    <tr>
      <th>13</th>
      <td>Blue Diamond Almonds</td>
      <td>Nuts</td>
      <td>Snack healthy with Blue Diamond Almonds.


