# embeddings_queue

created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: ba955bcb-b070-4470-ab3c-81d1b429ec15
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"ba955bcb-b070-4470-ab3c-81d1b429ec15\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"47158707-c335-4e76-a61c-3d9ff5dc9613\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b1d4c24c-0db7-4828-a244-9194fb7a4fe0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"51976288180c82096053c76804923ff69c191d4e3c73678c18725e6a1c18f702\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"450f3ccd-c391-4272-b1dc-d7f03b4a18c9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9bec52cb2282aedac099544f087180820d7919e9e5397f42b8b30afc6309fa02\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8516, \"end_char_idx\": 10941, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "doc_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "ref_doc_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "chroma:document": "The dataset contains many small snippets of text data. We will need to merge many snippets from each video to create more substantial chunks of text that contain more information.\n\n\n```python\nfrom tqdm.auto import tqdm\n\nnew_data = []\n\nwindow = 20  # number of sentences to combine\nstride = 4  # number of sentences to 'stride' over, used to create overlap\n\nfor i in tqdm(range(0, len(data), stride)):\n    i_end = min(len(data)-1, i+window)\n    if data[i]['title'] != data[i_end]['title']:\n        # in this case we skip this entry as we have start/end of two videos\n        continue\n    text = ' '.join(data[i:i_end]['text'])\n    # create the new merged dataset\n    new_data.append({\n        'start': data[i]['start'],\n        'end': data[i_end]['end'],\n        'title': data[i]['title'],\n        'text': text,\n        'id': data[i]['id'],\n        'url': data[i]['url'],\n        'published': data[i]['published'],\n        'channel_id': data[i]['channel_id']\n    })\n```\n\n\n      0%|          | 0/52155 [00:00<?, ?it/s]\n\n\n\n```python\nnew_data[0]\n```\n\n\n\n\n    {'start': 0.0,\n     'end': 74.12,\n     'title': 'Training and Testing an Italian BERT - Transformers From Scratch #4',\n     'text': \"Hi, welcome to the video. So this is the fourth video in a Transformers from Scratch mini series. So if you haven't been following along, we've essentially covered what you can see on the screen. So we got some data. We built a tokenizer with it. And then we've set up our input pipeline ready to begin actually training our model, which is what we're going to cover in this video. So let's move over to the code. And we see here that we have essentially everything we've done so far. So we've built our input data, our input pipeline. And we're now at a point where we have a data loader, PyTorch data loader, ready. And we can begin training a model with it. So there are a few things to be aware of. So I mean, first, let's just have a quick look at the structure of our data.\",\n     'id': '35Pdoyi6ZoQ-t0.0',\n     'url': 'https://youtu.be/35Pdoyi6ZoQ',\n     'published': '2021-07-06 13:00:03 UTC',\n     'channel_id': 'UCv83tO5cePwHMt1952IVVHw'}\n\n\n\nNow we need a place to store these embeddings and enable a efficient _vector search_ through them all. To do that we use **`Pinecone`**, we can get a [free API key](https://app.pinecone.io) and enter it below where we will initialize our connection to `Pinecone` and create a new index."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 450f3ccd-c391-4272-b1dc-d7f03b4a18c9
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"450f3ccd-c391-4272-b1dc-d7f03b4a18c9\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"47158707-c335-4e76-a61c-3d9ff5dc9613\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ba955bcb-b070-4470-ab3c-81d1b429ec15\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a19d9e1eab22fdee52fc43a2602787245bf60df53404e2218d623b410c3fcd77\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f0bd62b5-8b6a-4129-8721-750b46684775\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b092832e524d952078343ed675cf6e388505c21a45d340783625e4b7d25166c3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9536, \"end_char_idx\": 11944, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "doc_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "ref_doc_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "chroma:document": "```python\nnew_data[0]\n```\n\n\n\n\n    {'start': 0.0,\n     'end': 74.12,\n     'title': 'Training and Testing an Italian BERT - Transformers From Scratch #4',\n     'text': \"Hi, welcome to the video. So this is the fourth video in a Transformers from Scratch mini series. So if you haven't been following along, we've essentially covered what you can see on the screen. So we got some data. We built a tokenizer with it. And then we've set up our input pipeline ready to begin actually training our model, which is what we're going to cover in this video. So let's move over to the code. And we see here that we have essentially everything we've done so far. So we've built our input data, our input pipeline. And we're now at a point where we have a data loader, PyTorch data loader, ready. And we can begin training a model with it. So there are a few things to be aware of. So I mean, first, let's just have a quick look at the structure of our data.\",\n     'id': '35Pdoyi6ZoQ-t0.0',\n     'url': 'https://youtu.be/35Pdoyi6ZoQ',\n     'published': '2021-07-06 13:00:03 UTC',\n     'channel_id': 'UCv83tO5cePwHMt1952IVVHw'}\n\n\n\nNow we need a place to store these embeddings and enable a efficient _vector search_ through them all. To do that we use **`Pinecone`**, we can get a [free API key](https://app.pinecone.io) and enter it below where we will initialize our connection to `Pinecone` and create a new index.\n\n\n```python\nimport pinecone\n\nindex_name = 'openai-youtube-transcriptions'\n\n# initialize connection to pinecone (get API key at app.pinecone.io)\npinecone.init(\n    api_key=\"PINECONE_API_KEY\",\n    environment=\"us-east1-gcp\"  # may be different, check at app.pinecone.io\n)\n\n# check if index already exists (it shouldn't if this is first time)\nif index_name not in pinecone.list_indexes():\n    # if does not exist, create index\n    pinecone.create_index(\n        index_name,\n        dimension=len(res['data'][0]['embedding']),\n        metric='cosine',\n        metadata_config={'indexed': ['channel_id', 'published']}\n    )\n# connect to index\nindex = pinecone.Index(index_name)\n# view index stats\nindex.describe_index_stats()\n```\n\n\n\n\n    {'dimension': 1536,\n     'index_fullness': 0.0,\n     'namespaces': {},\n     'total_vector_count': 0}\n\n\n\nWe can see the index is currently empty with a `total_vector_count` of `0`. We can begin populating it with OpenAI `text-embedding-3-small` built embeddings like so:"}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: f0bd62b5-8b6a-4129-8721-750b46684775
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"f0bd62b5-8b6a-4129-8721-750b46684775\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"47158707-c335-4e76-a61c-3d9ff5dc9613\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"450f3ccd-c391-4272-b1dc-d7f03b4a18c9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a343269eadb003865388ae311c3463546a1600c73e40fc6db0ce0aee7d702f5b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5c273f12-fe00-4a0a-af90-5c0ee9882d86\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b00234140bd9677416e527d3d90d73e5ada8bf7f13731e0ae3fca0dba8201c7e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10655, \"end_char_idx\": 13467, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "doc_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "ref_doc_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "chroma:document": "Now we need a place to store these embeddings and enable a efficient _vector search_ through them all. To do that we use **`Pinecone`**, we can get a [free API key](https://app.pinecone.io) and enter it below where we will initialize our connection to `Pinecone` and create a new index.\n\n\n```python\nimport pinecone\n\nindex_name = 'openai-youtube-transcriptions'\n\n# initialize connection to pinecone (get API key at app.pinecone.io)\npinecone.init(\n    api_key=\"PINECONE_API_KEY\",\n    environment=\"us-east1-gcp\"  # may be different, check at app.pinecone.io\n)\n\n# check if index already exists (it shouldn't if this is first time)\nif index_name not in pinecone.list_indexes():\n    # if does not exist, create index\n    pinecone.create_index(\n        index_name,\n        dimension=len(res['data'][0]['embedding']),\n        metric='cosine',\n        metadata_config={'indexed': ['channel_id', 'published']}\n    )\n# connect to index\nindex = pinecone.Index(index_name)\n# view index stats\nindex.describe_index_stats()\n```\n\n\n\n\n    {'dimension': 1536,\n     'index_fullness': 0.0,\n     'namespaces': {},\n     'total_vector_count': 0}\n\n\n\nWe can see the index is currently empty with a `total_vector_count` of `0`. We can begin populating it with OpenAI `text-embedding-3-small` built embeddings like so:\n\n\n```python\nfrom tqdm.auto import tqdm\nfrom time import sleep\n\nbatch_size = 100  # how many embeddings we create and insert at once\n\nfor i in tqdm(range(0, len(new_data), batch_size)):\n    # find end of batch\n    i_end = min(len(new_data), i+batch_size)\n    meta_batch = new_data[i:i_end]\n    # get ids\n    ids_batch = [x['id'] for x in meta_batch]\n    # get texts to encode\n    texts = [x['text'] for x in meta_batch]\n    # create embeddings (try-except added to avoid RateLimitError)\n    done = False\n    while not done:\n        try:\n            res = openai.Embedding.create(input=texts, engine=embed_model)\n            done = True\n        except:\n            sleep(5)\n    embeds = [record['embedding'] for record in res['data']]\n    # cleanup metadata\n    meta_batch = [{\n        'start': x['start'],\n        'end': x['end'],\n        'title': x['title'],\n        'text': x['text'],\n        'url': x['url'],\n        'published': x['published'],\n        'channel_id': x['channel_id']\n    } for x in meta_batch]\n    to_upsert = list(zip(ids_batch, embeds, meta_batch))\n    # upsert to Pinecone\n    index.upsert(vectors=to_upsert)\n```\n\n\n      0%|          | 0/487 [00:00<?, ?it/s]\n\n\nNow we search, for this we need to create a _query vector_ `xq`:\n\n\n```python\nres = openai.Embedding.create(\n    input=[query],\n    engine=embed_model\n)\n\n# retrieve from Pinecone\nxq = res['data'][0]['embedding']\n\n# get relevant contexts (including the questions)\nres = index.query(xq, top_k=2, include_metadata=True)\n```\n\n\n```python\nres\n```"}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 5c273f12-fe00-4a0a-af90-5c0ee9882d86
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"5c273f12-fe00-4a0a-af90-5c0ee9882d86\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"47158707-c335-4e76-a61c-3d9ff5dc9613\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f0bd62b5-8b6a-4129-8721-750b46684775\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"681ac22d8d27c5abcd17e3e720a5df6ecd53d22d2a8c8ffa9c68d811843f0a00\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fac1e352-5cde-4f15-8a70-8f09a2f66483\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a83fa20bedec161a7539e949bbb37bd023777514fe06fa0640ba7fd6db2684ad\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13088, \"end_char_idx\": 16901, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "doc_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "ref_doc_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "chroma:document": "0%|          | 0/487 [00:00<?, ?it/s]\n\n\nNow we search, for this we need to create a _query vector_ `xq`:\n\n\n```python\nres = openai.Embedding.create(\n    input=[query],\n    engine=embed_model\n)\n\n# retrieve from Pinecone\nxq = res['data'][0]['embedding']\n\n# get relevant contexts (including the questions)\nres = index.query(xq, top_k=2, include_metadata=True)\n```\n\n\n```python\nres\n```\n\n\n\n\n    {'matches': [{'id': 'pNvujJ1XyeQ-t418.88',\n                  'metadata': {'channel_id': 'UCv83tO5cePwHMt1952IVVHw',\n                               'end': 568.4,\n                               'published': datetime.date(2021, 11, 24),\n                               'start': 418.88,\n                               'text': 'pairs of related sentences you can go '\n                                       'ahead and actually try training or '\n                                       'fine-tuning using NLI with multiple '\n                                       \"negative ranking loss. If you don't have \"\n                                       'that fine. Another option is that you have '\n                                       'a semantic textual similarity data set or '\n                                       'STS and what this is is you have so you '\n                                       'have sentence A here, sentence B here and '\n                                       'then you have a score from from 0 to 1 '\n                                       'that tells you the similarity between '\n                                       'those two scores and you would train this '\n                                       'using something like cosine similarity '\n                                       \"loss. Now if that's not an option and your \"\n                                       'focus or use case is on building a '\n                                       'sentence transformer for another language '\n                                       'where there is no current sentence '\n                                       'transformer you can use multilingual '\n                                       'parallel data. So what I mean by that is '\n                                       'so parallel data just means translation '\n                                       'pairs so if you have for example a English '\n                                       'sentence and then you have another '\n                                       'language here so it can it can be anything '\n                                       \"I'm just going to put XX and that XX is \"\n                                       'your target language you can fine-tune a '\n                                       'model using something called multilingual '\n                                       'knowledge distillation and what that does '\n                                       'is takes a monolingual model for example '\n                                       'in English and using those translation '\n                                       'pairs it distills the knowledge the '\n                                       'semantic similarity knowledge from that '\n                                       'monolingual English model into a '\n                                       'multilingual model which can handle both '\n                                       'English and your target language. So '\n                                       \"they're three options quite popular very \"\n                                       'common that you can go for and as a '\n                                       'supervised methods the chances are that '\n                                       'probably going to outperform anything you '\n                                       'do with unsupervised training at least for '\n                                       'now."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: fac1e352-5cde-4f15-8a70-8f09a2f66483
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"fac1e352-5cde-4f15-8a70-8f09a2f66483\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"47158707-c335-4e76-a61c-3d9ff5dc9613\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5c273f12-fe00-4a0a-af90-5c0ee9882d86\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"230abe3fca201fc905420f118ea2e392e807a7736a2c56cc177d961a5674efc5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"45d2323e-8add-44cf-8b0c-98a8d5db24e9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2cb54677131be85f029cad20fd372c02ae14f928baba44f5c7cc0be67293e37a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14056, \"end_char_idx\": 19068, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "doc_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "ref_doc_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "chroma:document": "If you don't have \"\n                                       'that fine. Another option is that you have '\n                                       'a semantic textual similarity data set or '\n                                       'STS and what this is is you have so you '\n                                       'have sentence A here, sentence B here and '\n                                       'then you have a score from from 0 to 1 '\n                                       'that tells you the similarity between '\n                                       'those two scores and you would train this '\n                                       'using something like cosine similarity '\n                                       \"loss. Now if that's not an option and your \"\n                                       'focus or use case is on building a '\n                                       'sentence transformer for another language '\n                                       'where there is no current sentence '\n                                       'transformer you can use multilingual '\n                                       'parallel data. So what I mean by that is '\n                                       'so parallel data just means translation '\n                                       'pairs so if you have for example a English '\n                                       'sentence and then you have another '\n                                       'language here so it can it can be anything '\n                                       \"I'm just going to put XX and that XX is \"\n                                       'your target language you can fine-tune a '\n                                       'model using something called multilingual '\n                                       'knowledge distillation and what that does '\n                                       'is takes a monolingual model for example '\n                                       'in English and using those translation '\n                                       'pairs it distills the knowledge the '\n                                       'semantic similarity knowledge from that '\n                                       'monolingual English model into a '\n                                       'multilingual model which can handle both '\n                                       'English and your target language. So '\n                                       \"they're three options quite popular very \"\n                                       'common that you can go for and as a '\n                                       'supervised methods the chances are that '\n                                       'probably going to outperform anything you '\n                                       'do with unsupervised training at least for '\n                                       'now. So if none of those sound like '\n                                       'something',\n                               'title': 'Today Unsupervised Sentence Transformers, '\n                                        'Tomorrow Skynet (how TSDAE works)',\n                               'url': 'https://youtu.be/pNvujJ1XyeQ'},\n                  'score': 0.865277052,\n                  'sparseValues': {},\n                  'values': []},\n                 {'id': 'WS1uVMGhlWQ-t737.28',\n                  'metadata': {'channel_id': 'UCv83tO5cePwHMt1952IVVHw',\n                               'end': 900.72,\n                               'published': datetime.date(2021, 10, 20),\n                               'start': 737.28,\n                               'text': \"were actually more accurate. So we can't \"\n                                       \"really do that. We can't use this what is \"\n                                       'called a mean pooling approach. Or we '\n                                       \"can't use it in its current form. Now the \"\n                                       'solution to this problem was introduced by '\n                                       'two people in 2019 Nils Reimers and Irenia '\n                                       'Gurevich. They introduced what is the '\n                                       'first sentence transformer or sentence '\n                                       'BERT. And it was found that sentence BERT '\n                                       'or S BERT outformed all of the previous '\n                                       'Save the Art models on pretty much all '\n                                       'benchmarks. Not all of them but most of '\n                                       'them. And it did it in a very quick time. '\n                                       'So if we compare it to BERT, if we wanted '\n                                       'to find the most similar sentence pair '\n                                       'from 10,000 sentences in that 2019 paper '\n                                       'they found that with BERT that took 65 '\n                                       'hours."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 45d2323e-8add-44cf-8b0c-98a8d5db24e9
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"45d2323e-8add-44cf-8b0c-98a8d5db24e9\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"47158707-c335-4e76-a61c-3d9ff5dc9613\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fac1e352-5cde-4f15-8a70-8f09a2f66483\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"4a9cc008a02ea251492a18d124e9234cd352c05ced1f7dcfbe7adbb299ef178b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a0aefa84-066a-441c-9df0-dbced6a182fc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8d2a0f253254d69b238a99c1386c1b64d40f3ac8ee394d0943fcf5203141fd40\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16902, \"end_char_idx\": 21472, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "doc_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "ref_doc_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "chroma:document": "So if none of those sound like '\n                                       'something',\n                               'title': 'Today Unsupervised Sentence Transformers, '\n                                        'Tomorrow Skynet (how TSDAE works)',\n                               'url': 'https://youtu.be/pNvujJ1XyeQ'},\n                  'score': 0.865277052,\n                  'sparseValues': {},\n                  'values': []},\n                 {'id': 'WS1uVMGhlWQ-t737.28',\n                  'metadata': {'channel_id': 'UCv83tO5cePwHMt1952IVVHw',\n                               'end': 900.72,\n                               'published': datetime.date(2021, 10, 20),\n                               'start': 737.28,\n                               'text': \"were actually more accurate. So we can't \"\n                                       \"really do that. We can't use this what is \"\n                                       'called a mean pooling approach. Or we '\n                                       \"can't use it in its current form. Now the \"\n                                       'solution to this problem was introduced by '\n                                       'two people in 2019 Nils Reimers and Irenia '\n                                       'Gurevich. They introduced what is the '\n                                       'first sentence transformer or sentence '\n                                       'BERT. And it was found that sentence BERT '\n                                       'or S BERT outformed all of the previous '\n                                       'Save the Art models on pretty much all '\n                                       'benchmarks. Not all of them but most of '\n                                       'them. And it did it in a very quick time. '\n                                       'So if we compare it to BERT, if we wanted '\n                                       'to find the most similar sentence pair '\n                                       'from 10,000 sentences in that 2019 paper '\n                                       'they found that with BERT that took 65 '\n                                       'hours. With S BERT embeddings they could '\n                                       'create all the embeddings in just around '\n                                       'five seconds. And then they could compare '\n                                       'all those with cosine similarity in 0.01 '\n                                       \"seconds. So it's a lot faster. We go from \"\n                                       '65 hours to just over five seconds which '\n                                       'is I think pretty incredible. Now I think '\n                                       \"that's pretty much all the context we need \"\n                                       'behind sentence transformers. And what we '\n                                       'do now is dive into a little bit of how '\n                                       'they actually work. Now we said before we '\n                                       'have the core transform models and what S '\n                                       'BERT does is fine tunes on sentence pairs '\n                                       'using what is called a Siamese '\n                                       'architecture or Siamese network. What we '\n                                       'mean by a Siamese network is that we have '\n                                       'what we can see, what can view as two BERT '\n                                       'models that are identical and the weights '\n                                       'between those two models are tied. Now in '\n                                       'reality when implementing this we just use '\n                                       'a single BERT model. And what we do is we '\n                                       'process one sentence, a sentence A through '\n                                       'the model and then we process another '\n                                       'sentence, sentence B through the model. '\n                                       \"And that's the sentence pair. So with our \"\n                                       'cross-linked we were processing the '\n                                       'sentence pair together. We were putting '\n                                       'them both together, processing them all at '\n                                       'once. This time we process them '\n                                       'separately."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: a0aefa84-066a-441c-9df0-dbced6a182fc
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a0aefa84-066a-441c-9df0-dbced6a182fc\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"47158707-c335-4e76-a61c-3d9ff5dc9613\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"45d2323e-8add-44cf-8b0c-98a8d5db24e9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"4acd459784d0569cd6a53761abf54e0ccc798bebca918e2c271913ad1c253491\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"55c27295-36e2-47a6-9ef7-f77908607af7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9642748df13be8897161feb5b6caaba1ed9846066ec02f5fef13e5b9cec0a390\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19069, \"end_char_idx\": 23138, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "doc_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "ref_doc_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "chroma:document": "With S BERT embeddings they could '\n                                       'create all the embeddings in just around '\n                                       'five seconds. And then they could compare '\n                                       'all those with cosine similarity in 0.01 '\n                                       \"seconds. So it's a lot faster. We go from \"\n                                       '65 hours to just over five seconds which '\n                                       'is I think pretty incredible. Now I think '\n                                       \"that's pretty much all the context we need \"\n                                       'behind sentence transformers. And what we '\n                                       'do now is dive into a little bit of how '\n                                       'they actually work. Now we said before we '\n                                       'have the core transform models and what S '\n                                       'BERT does is fine tunes on sentence pairs '\n                                       'using what is called a Siamese '\n                                       'architecture or Siamese network. What we '\n                                       'mean by a Siamese network is that we have '\n                                       'what we can see, what can view as two BERT '\n                                       'models that are identical and the weights '\n                                       'between those two models are tied. Now in '\n                                       'reality when implementing this we just use '\n                                       'a single BERT model. And what we do is we '\n                                       'process one sentence, a sentence A through '\n                                       'the model and then we process another '\n                                       'sentence, sentence B through the model. '\n                                       \"And that's the sentence pair. So with our \"\n                                       'cross-linked we were processing the '\n                                       'sentence pair together. We were putting '\n                                       'them both together, processing them all at '\n                                       'once. This time we process them '\n                                       'separately. And during training what '\n                                       'happens is the weights',\n                               'title': 'Intro to Sentence Embeddings with '\n                                        'Transformers',\n                               'url': 'https://youtu.be/WS1uVMGhlWQ'},\n                  'score': 0.85855335,\n                  'sparseValues': {},\n                  'values': []}],\n     'namespace': ''}\n\n\n\n\n```python\nlimit = 3750\n\ndef retrieve(query):\n    res = openai.Embedding.create(\n        input=[query],\n        engine=embed_model\n    )\n\n    # retrieve from Pinecone\n    xq = res['data'][0]['embedding']\n\n    # get relevant contexts\n    res = index.query(xq, top_k=3, include_metadata=True)\n    contexts = [\n        x['metadata']['text'] for x in res['matches']\n    ]\n\n    # build our prompt with the retrieved contexts included\n    prompt_start = (\n        \"Answer the question based on the context below.\\n\\n\"+\n        \"Context:\\n\"\n    )\n    prompt_end = (\n        f\"\\n\\nQuestion: {query}\\nAnswer:\"\n    )\n    # append contexts until hitting limit\n    for i in range(1, len(contexts)):\n        if len(\"\\n\\n---\\n\\n\".join(contexts[:i])) >= limit:\n            prompt = (\n                prompt_start +\n                \"\\n\\n---\\n\\n\".join(contexts[:i-1]) +\n                prompt_end\n            )\n            break\n        elif i == len(contexts)-1:\n            prompt = (\n                prompt_start +\n                \"\\n\\n---\\n\\n\".join(contexts) +\n                prompt_end\n            )\n    return prompt\n```\n\n\n```python\n# first we retrieve relevant items from Pinecone\nquery_with_contexts = retrieve(query)\nquery_with_contexts\n```"}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 55c27295-36e2-47a6-9ef7-f77908607af7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"55c27295-36e2-47a6-9ef7-f77908607af7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"47158707-c335-4e76-a61c-3d9ff5dc9613\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a0aefa84-066a-441c-9df0-dbced6a182fc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e053cd1e57d1c9e74a03f92a55861b98280f55e0e011ee7d3f342b527e7af8c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6f029da6-6d0b-4cf1-b1d7-4010c03b475b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1ada838368526ceb327e8610d5483bd454bd64380bbaf5f776ab1f796622882a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21393, \"end_char_idx\": 24922, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "doc_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "ref_doc_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "chroma:document": "This time we process them '\n                                       'separately. And during training what '\n                                       'happens is the weights',\n                               'title': 'Intro to Sentence Embeddings with '\n                                        'Transformers',\n                               'url': 'https://youtu.be/WS1uVMGhlWQ'},\n                  'score': 0.85855335,\n                  'sparseValues': {},\n                  'values': []}],\n     'namespace': ''}\n\n\n\n\n```python\nlimit = 3750\n\ndef retrieve(query):\n    res = openai.Embedding.create(\n        input=[query],\n        engine=embed_model\n    )\n\n    # retrieve from Pinecone\n    xq = res['data'][0]['embedding']\n\n    # get relevant contexts\n    res = index.query(xq, top_k=3, include_metadata=True)\n    contexts = [\n        x['metadata']['text'] for x in res['matches']\n    ]\n\n    # build our prompt with the retrieved contexts included\n    prompt_start = (\n        \"Answer the question based on the context below.\\n\\n\"+\n        \"Context:\\n\"\n    )\n    prompt_end = (\n        f\"\\n\\nQuestion: {query}\\nAnswer:\"\n    )\n    # append contexts until hitting limit\n    for i in range(1, len(contexts)):\n        if len(\"\\n\\n---\\n\\n\".join(contexts[:i])) >= limit:\n            prompt = (\n                prompt_start +\n                \"\\n\\n---\\n\\n\".join(contexts[:i-1]) +\n                prompt_end\n            )\n            break\n        elif i == len(contexts)-1:\n            prompt = (\n                prompt_start +\n                \"\\n\\n---\\n\\n\".join(contexts) +\n                prompt_end\n            )\n    return prompt\n```\n\n\n```python\n# first we retrieve relevant items from Pinecone\nquery_with_contexts = retrieve(query)\nquery_with_contexts\n```\n\n\n\n\n    \"Answer the question based on the context below.\\n\\nContext:\\npairs of related sentences you can go ahead and actually try training or fine-tuning using NLI with multiple negative ranking loss. If you don't have that fine. Another option is that you have a semantic textual similarity data set or STS and what this is is you have so you have sentence A here, sentence B here and then you have a score from from 0 to 1 that tells you the similarity between those two scores and you would train this using something like cosine similarity loss. Now if that's not an option and your focus or use case is on building a sentence transformer for another language where there is no current sentence transformer you can use multilingual parallel data. So what I mean by that is so parallel data just means translation pairs so if you have for example a English sentence and then you have another language here so it can it can be anything I'm just going to put XX and that XX is your target language you can fine-tune a model using something called multilingual knowledge distillation and what that does is takes a monolingual model for example in English and using those translation pairs it distills the knowledge the semantic similarity knowledge from that monolingual English model into a multilingual model which can handle both English and your target language. So they're three options quite popular very common that you can go for and as a supervised methods the chances are that probably going to outperform anything you do with unsupervised training at least for now. So if none of those sound like something\\n\\n---\\n\\nwere actually more accurate. So we can't really do that. We can't use this what is called a mean pooling approach. Or we can't use it in its current form."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 6f029da6-6d0b-4cf1-b1d7-4010c03b475b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"6f029da6-6d0b-4cf1-b1d7-4010c03b475b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"47158707-c335-4e76-a61c-3d9ff5dc9613\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"55c27295-36e2-47a6-9ef7-f77908607af7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7c736400978f041b600d7822c32c022845a228b0bf893f560e6fee9e3e69d5d7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6fffadc1-2aa9-4019-bf5c-6150b555d905\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"df5db25bd1aee09d8e65c26743f2013f3c77dc990cf80ad4b5b60c37220b2734\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 23018, \"end_char_idx\": 26475, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "doc_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "ref_doc_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "chroma:document": "```python\n# first we retrieve relevant items from Pinecone\nquery_with_contexts = retrieve(query)\nquery_with_contexts\n```\n\n\n\n\n    \"Answer the question based on the context below.\\n\\nContext:\\npairs of related sentences you can go ahead and actually try training or fine-tuning using NLI with multiple negative ranking loss. If you don't have that fine. Another option is that you have a semantic textual similarity data set or STS and what this is is you have so you have sentence A here, sentence B here and then you have a score from from 0 to 1 that tells you the similarity between those two scores and you would train this using something like cosine similarity loss. Now if that's not an option and your focus or use case is on building a sentence transformer for another language where there is no current sentence transformer you can use multilingual parallel data. So what I mean by that is so parallel data just means translation pairs so if you have for example a English sentence and then you have another language here so it can it can be anything I'm just going to put XX and that XX is your target language you can fine-tune a model using something called multilingual knowledge distillation and what that does is takes a monolingual model for example in English and using those translation pairs it distills the knowledge the semantic similarity knowledge from that monolingual English model into a multilingual model which can handle both English and your target language. So they're three options quite popular very common that you can go for and as a supervised methods the chances are that probably going to outperform anything you do with unsupervised training at least for now. So if none of those sound like something\\n\\n---\\n\\nwere actually more accurate. So we can't really do that. We can't use this what is called a mean pooling approach. Or we can't use it in its current form. Now the solution to this problem was introduced by two people in 2019 Nils Reimers and Irenia Gurevich. They introduced what is the first sentence transformer or sentence BERT. And it was found that sentence BERT or S BERT outformed all of the previous Save the Art models on pretty much all benchmarks. Not all of them but most of them. And it did it in a very quick time. So if we compare it to BERT, if we wanted to find the most similar sentence pair from 10,000 sentences in that 2019 paper they found that with BERT that took 65 hours. With S BERT embeddings they could create all the embeddings in just around five seconds. And then they could compare all those with cosine similarity in 0.01 seconds. So it's a lot faster. We go from 65 hours to just over five seconds which is I think pretty incredible. Now I think that's pretty much all the context we need behind sentence transformers. And what we do now is dive into a little bit of how they actually work. Now we said before we have the core transform models and what S BERT does is fine tunes on sentence pairs using what is called a Siamese architecture or Siamese network. What we mean by a Siamese network is that we have what we can see, what can view as two BERT models that are identical and the weights between those two models are tied. Now in reality when implementing this we just use a single BERT model. And what we do is we process one sentence, a sentence A through the model and then we process another sentence, sentence B through the model. And that's the sentence pair."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 6fffadc1-2aa9-4019-bf5c-6150b555d905
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"6fffadc1-2aa9-4019-bf5c-6150b555d905\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"47158707-c335-4e76-a61c-3d9ff5dc9613\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6f029da6-6d0b-4cf1-b1d7-4010c03b475b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"e02a5a65d348c8771c4b7055d79f29dcf2917709014ae47efa8522110b08969c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"757d0981-1bd9-412e-b4d9-d428f55d76e1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"189755547952daeaed473ce5344dc206add394dadb0d8f96d42ede0a639b6051\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 24825, \"end_char_idx\": 28068, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "doc_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "ref_doc_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "chroma:document": "We can't use this what is called a mean pooling approach. Or we can't use it in its current form. Now the solution to this problem was introduced by two people in 2019 Nils Reimers and Irenia Gurevich. They introduced what is the first sentence transformer or sentence BERT. And it was found that sentence BERT or S BERT outformed all of the previous Save the Art models on pretty much all benchmarks. Not all of them but most of them. And it did it in a very quick time. So if we compare it to BERT, if we wanted to find the most similar sentence pair from 10,000 sentences in that 2019 paper they found that with BERT that took 65 hours. With S BERT embeddings they could create all the embeddings in just around five seconds. And then they could compare all those with cosine similarity in 0.01 seconds. So it's a lot faster. We go from 65 hours to just over five seconds which is I think pretty incredible. Now I think that's pretty much all the context we need behind sentence transformers. And what we do now is dive into a little bit of how they actually work. Now we said before we have the core transform models and what S BERT does is fine tunes on sentence pairs using what is called a Siamese architecture or Siamese network. What we mean by a Siamese network is that we have what we can see, what can view as two BERT models that are identical and the weights between those two models are tied. Now in reality when implementing this we just use a single BERT model. And what we do is we process one sentence, a sentence A through the model and then we process another sentence, sentence B through the model. And that's the sentence pair. So with our cross-linked we were processing the sentence pair together. We were putting them both together, processing them all at once. This time we process them separately. And during training what happens is the weights\\n\\n---\\n\\nTransformer-based Sequential Denoising Autoencoder. So what we'll do is jump straight into it and take a look at where we might want to use this training approach and and how we can actually implement it. So the first question we need to ask is do we really need to resort to unsupervised training? Now what we're going to do here is just have a look at a few of the most popular training approaches and what sort of data we need for that. So the first one we're looking at here is Natural Language Inference or NLI and NLI requires that we have pairs of sentences that are labeled as either contradictory, neutral which means they're not necessarily related or as entailing or as inferring each other. So you have pairs that entail each other so they are both very similar pairs that are neutral and also pairs that are contradictory. And this is the traditional NLI data. Now using another version of fine-tuning with NLI called a multiple negatives ranking loss you can get by with only entailment pairs so pairs that are related to each other or positive pairs and it can also use contradictory pairs to improve the performance of training as well but you don't need it. So if you have positive pairs of related sentences you can go ahead and actually try training or fine-tuning using NLI with multiple negative ranking loss. If you don't have that fine."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 757d0981-1bd9-412e-b4d9-d428f55d76e1
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"757d0981-1bd9-412e-b4d9-d428f55d76e1\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"47158707-c335-4e76-a61c-3d9ff5dc9613\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6fffadc1-2aa9-4019-bf5c-6150b555d905\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ec6b25456bccafc04e1b73847976164c559f7a2252fd7195c2ee806f4e59cd7f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 26233, \"end_char_idx\": 28675, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "doc_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "ref_doc_id": "47158707-c335-4e76-a61c-3d9ff5dc9613", "chroma:document": "Now in reality when implementing this we just use a single BERT model. And what we do is we process one sentence, a sentence A through the model and then we process another sentence, sentence B through the model. And that's the sentence pair. So with our cross-linked we were processing the sentence pair together. We were putting them both together, processing them all at once. This time we process them separately. And during training what happens is the weights\\n\\n---\\n\\nTransformer-based Sequential Denoising Autoencoder. So what we'll do is jump straight into it and take a look at where we might want to use this training approach and and how we can actually implement it. So the first question we need to ask is do we really need to resort to unsupervised training? Now what we're going to do here is just have a look at a few of the most popular training approaches and what sort of data we need for that. So the first one we're looking at here is Natural Language Inference or NLI and NLI requires that we have pairs of sentences that are labeled as either contradictory, neutral which means they're not necessarily related or as entailing or as inferring each other. So you have pairs that entail each other so they are both very similar pairs that are neutral and also pairs that are contradictory. And this is the traditional NLI data. Now using another version of fine-tuning with NLI called a multiple negatives ranking loss you can get by with only entailment pairs so pairs that are related to each other or positive pairs and it can also use contradictory pairs to improve the performance of training as well but you don't need it. So if you have positive pairs of related sentences you can go ahead and actually try training or fine-tuning using NLI with multiple negative ranking loss. If you don't have that fine. Another option is that you have a semantic textual similarity data set or STS and what this is is you have so you have sentence A here, sentence B\\n\\nQuestion: Which training method should I use for sentence transformers when I only have pairs of related sentences?\\nAnswer:\"\n\n\n\n\n```python\n# then we complete the context-infused query\ncomplete(query_with_contexts)\n```\n\n\n\n\n    'You should use Natural Language Inference (NLI) with multiple negative ranking loss.'\n\n\n\nAnd we get a pretty great answer straight away, specifying to use _multiple-rankings loss_ (also called _multiple negatives ranking loss_)."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 32f76b70-1201-4bb8-b370-bbb0e1a8d7bc
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"32f76b70-1201-4bb8-b370-bbb0e1a8d7bc\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"30f57e7b-42e6-4eb5-90c6-cf2474400b23\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"33fa5220faea5bfd8fe4f54ee19a884847c32fa2af84ff1fe447e99a7369ce46\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1673, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "# Retrieval Augmentation for GPT-4 using Pinecone\n\n#### Fixing LLMs that Hallucinate\n\nIn this notebook we will learn how to query relevant contexts to our queries from Pinecone, and pass these to a GPT-4 model to generate an answer backed by real data sources.\n\nGPT-4 is a big step up from previous OpenAI completion models. It also exclusively uses the `ChatCompletion` endpoint, so we must use it in a slightly different way to usual. However, the power of the model makes the change worthwhile, particularly when augmented with an external knowledge base like the Pinecone vector database.\n\nRequired installs for this notebook are:\n\n\n```python\n!pip install -qU bs4 tiktoken openai langchain pinecone-client[grpc]\n```\n\n    \u001b[?25l     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m0.0/1.7 MB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\n\u001b[2K     \u001b[91m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m\u001b[91m\u2578\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m71.4 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m41.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m70.1/70.1 KB\u001b[0m \u001b[31m6.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m396.0/396.0 KB\u001b[0m \u001b[31m28.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m177.2/177.2 KB\u001b[0m \u001b[31m12.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m62.8/62.8 KB\u001b[0m \u001b[31m4.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.0/1.0 MB\u001b[0m \u001b[31m4."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 30f57e7b-42e6-4eb5-90c6-cf2474400b23
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"30f57e7b-42e6-4eb5-90c6-cf2474400b23\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"32f76b70-1201-4bb8-b370-bbb0e1a8d7bc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"8729dacca690457bb82592b5adfe93af37c350dd8cc32e6fb9062ffc0111d68e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d9ca40c2-a012-4716-9f9e-defd857b38fb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6b1f62835839d80a6698b640686e58de5455b9917b873b62e21dec198ebc2fdb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1032, \"end_char_idx\": 2293, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "7/1.7 MB\u001b[0m \u001b[31m41.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m70.1/70.1 KB\u001b[0m \u001b[31m6.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m396.0/396.0 KB\u001b[0m \u001b[31m28.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m177.2/177.2 KB\u001b[0m \u001b[31m12.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m62.8/62.8 KB\u001b[0m \u001b[31m4.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.0/1.0 MB\u001b[0m \u001b[31m4.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m58.3/58.3 KB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m43.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m77.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m158.8/158.8 KB\u001b[0m \u001b[31m19.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m199.2/199.2 KB\u001b[0m \u001b[31m26."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: d9ca40c2-a012-4716-9f9e-defd857b38fb
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"d9ca40c2-a012-4716-9f9e-defd857b38fb\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"30f57e7b-42e6-4eb5-90c6-cf2474400b23\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"85171f2f2dbf462eb9972e6582a16c9eb0dc2eb6ecb795c9b1a07b44b2e215bd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"afeec43f-3038-4b3f-8ba5-11e508a85d02\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7651f0edd1aa663f4a2cf2b3472ca1dc01bf96aacf6f3c0de5d7b76a4619100f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1653, \"end_char_idx\": 2917, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "0/1.0 MB\u001b[0m \u001b[31m4.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m58.3/58.3 KB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m43.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m77.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m158.8/158.8 KB\u001b[0m \u001b[31m19.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m199.2/199.2 KB\u001b[0m \u001b[31m26.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m264.6/264.6 KB\u001b[0m \u001b[31m35.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m114.2/114.2 KB\u001b[0m \u001b[31m15.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m49.1/49.1 KB\u001b[0m \u001b[31m7.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m218.0/218.0 KB\u001b[0m \u001b[31m27.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m218.0/218.0 KB\u001b[0m \u001b[31m28."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: afeec43f-3038-4b3f-8ba5-11e508a85d02
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"afeec43f-3038-4b3f-8ba5-11e508a85d02\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d9ca40c2-a012-4716-9f9e-defd857b38fb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4814286645455af5e261bbab2e281d5845c031285a090d9695b91aa415f017ac\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"23b08a84-4595-4b21-898a-98bba79892e6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4af58b1b57bedcabe1f1f1c8e2d73085660a1fa12e6a30f57541b4da00f9a45a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2270, \"end_char_idx\": 3650, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "2/199.2 KB\u001b[0m \u001b[31m26.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m264.6/264.6 KB\u001b[0m \u001b[31m35.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m114.2/114.2 KB\u001b[0m \u001b[31m15.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m49.1/49.1 KB\u001b[0m \u001b[31m7.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m218.0/218.0 KB\u001b[0m \u001b[31m27.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m218.0/218.0 KB\u001b[0m \u001b[31m28.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m211.7/211.7 KB\u001b[0m \u001b[31m12.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25h\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n    google-cloud-translate 3.8.4 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\n    google-cloud-language 2.6.1 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 23b08a84-4595-4b21-898a-98bba79892e6
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"23b08a84-4595-4b21-898a-98bba79892e6\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"afeec43f-3038-4b3f-8ba5-11e508a85d02\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"939fda61730eecb9c5c28e66f1d8d19044eb384cd432449ad0771beeff89f973\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"709164c1-8c6d-40a8-9c39-dc21ea7a96ed\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5981b35275acee87e1de108f9acff08c84e42b89fb461f0d3e4658537cbc9714\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2792, \"end_char_idx\": 4232, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m218.0/218.0 KB\u001b[0m \u001b[31m28.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m211.7/211.7 KB\u001b[0m \u001b[31m12.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25h\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n    google-cloud-translate 3.8.4 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\n    google-cloud-language 2.6.1 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\n    google-cloud-firestore 2.7.3 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\n    google-cloud-datastore 2.11.1 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\n    google-cloud-bigquery 3.4.2 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 709164c1-8c6d-40a8-9c39-dc21ea7a96ed
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"709164c1-8c6d-40a8-9c39-dc21ea7a96ed\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"23b08a84-4595-4b21-898a-98bba79892e6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"36bb417b286c1fda73e46db556fc31aa09245f69d078e42495b472ec6f8041f7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1860e958-210b-43e0-b3ff-43f31342a144\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3b4df178cd3ed185eb91620d1f33ad7ed08521e4dd9de110d07a265fb46a2112\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3462, \"end_char_idx\": 5119, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "google-cloud-language 2.6.1 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\n    google-cloud-firestore 2.7.3 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\n    google-cloud-datastore 2.11.1 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\n    google-cloud-bigquery 3.4.2 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\n    google-cloud-bigquery-storage 2.19.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\n    google-api-core 2.11.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\u001b[0m\u001b[31m\n    \u001b[0m\n\n## Preparing the Data\n\nIn this example, we will download the LangChain docs from [langchain.readthedocs.io/](https://langchain.readthedocs.io/latest/en/). We get all `.html` files located on the site like so:\n\n\n```python\n!wget -r -A.html -P rtdocs https://python.langchain.com/en/latest/\n```\n\n\n\n\n    <Response [200]>\n\n\n\nThis downloads all HTML into the `rtdocs` directory. Now we can use LangChain itself to process these docs. We do this using the `ReadTheDocsLoader` like so:"}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 1860e958-210b-43e0-b3ff-43f31342a144
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"1860e958-210b-43e0-b3ff-43f31342a144\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"709164c1-8c6d-40a8-9c39-dc21ea7a96ed\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dbd627b97251beb2478a1594283d7109726abce79fd911b433779b9d2f62676b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a48d4cb4-34ba-4501-9fe0-5428f6e050a4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"047fa90d76d86dbb4dc2d7b37b486c2d03bad7b2e6e873c723d50f14d9922768\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4237, \"end_char_idx\": 7142, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "google-cloud-bigquery-storage 2.19.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\n    google-api-core 2.11.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\u001b[0m\u001b[31m\n    \u001b[0m\n\n## Preparing the Data\n\nIn this example, we will download the LangChain docs from [langchain.readthedocs.io/](https://langchain.readthedocs.io/latest/en/). We get all `.html` files located on the site like so:\n\n\n```python\n!wget -r -A.html -P rtdocs https://python.langchain.com/en/latest/\n```\n\n\n\n\n    <Response [200]>\n\n\n\nThis downloads all HTML into the `rtdocs` directory. Now we can use LangChain itself to process these docs. We do this using the `ReadTheDocsLoader` like so:\n\n\n```python\nfrom langchain.document_loaders import ReadTheDocsLoader\n\nloader = ReadTheDocsLoader('rtdocs')\ndocs = loader.load()\nlen(docs)\n```\n\n    .rst .pdf Welcome to LangChain Contents Getting Started Modules Use Cases Reference Docs LangChain Ecosystem Additional Resources Welcome to LangChain# Large language models (LLMs) are emerging as a transformative technology, enabling developers to build applications that they previously could not. But using these LLMs in isolation is often not enough to create a truly powerful app - the real power comes when you are able to combine them with other sources of computation or knowledge. This library is aimed at assisting in the development of those types of applications. Common examples of these types of applications include: \u2753 Question Answering over specific documents Documentation End-to-end Example: Question Answering over Notion Database \ud83d\udcac Chatbots Documentation End-to-end Example: Chat-LangChain \ud83e\udd16 Agents Documentation End-to-end Example: GPT+WolframAlpha Getting Started# Checkout the below guide for a walkthrough of how to get started using LangChain to create an Language Model application. Getting Started Documentation Modules# There are several main modules that LangChain provides support for. For each module we provide some examples to get started, how-to guides, reference docs, and conceptual guides. These modules are, in increasing order of complexity: Prompts: This includes prompt management, prompt optimization, and prompt serialization. LLMs: This includes a generic interface for all LLMs, and common utilities for working with LLMs. Document Loaders: This includes a standard interface for loading documents, as well as specific integrations to all types of text data sources. Utils: Language models are often more powerful when interacting with other sources of knowledge or computation. This can include Python REPLs, embeddings, search engines, and more. LangChain provides a large collection of common utils to use in your application."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: a48d4cb4-34ba-4501-9fe0-5428f6e050a4
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a48d4cb4-34ba-4501-9fe0-5428f6e050a4\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1860e958-210b-43e0-b3ff-43f31342a144\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"7a32b5b2991ca8e0f94078a7e56f53cc416c72fc6672f4b0c2a5f0c9791dc914\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ecf5c6f0-0aba-4a9d-ac9f-aed345f92694\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b5fa00a64712c7c6789a8ff7498ba7e6e7c3be5299febcf5d0f7a7303955d9e9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5567, \"end_char_idx\": 9357, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "But using these LLMs in isolation is often not enough to create a truly powerful app - the real power comes when you are able to combine them with other sources of computation or knowledge. This library is aimed at assisting in the development of those types of applications. Common examples of these types of applications include: \u2753 Question Answering over specific documents Documentation End-to-end Example: Question Answering over Notion Database \ud83d\udcac Chatbots Documentation End-to-end Example: Chat-LangChain \ud83e\udd16 Agents Documentation End-to-end Example: GPT+WolframAlpha Getting Started# Checkout the below guide for a walkthrough of how to get started using LangChain to create an Language Model application. Getting Started Documentation Modules# There are several main modules that LangChain provides support for. For each module we provide some examples to get started, how-to guides, reference docs, and conceptual guides. These modules are, in increasing order of complexity: Prompts: This includes prompt management, prompt optimization, and prompt serialization. LLMs: This includes a generic interface for all LLMs, and common utilities for working with LLMs. Document Loaders: This includes a standard interface for loading documents, as well as specific integrations to all types of text data sources. Utils: Language models are often more powerful when interacting with other sources of knowledge or computation. This can include Python REPLs, embeddings, search engines, and more. LangChain provides a large collection of common utils to use in your application. Chains: Chains go beyond just a single LLM call, and are sequences of calls (whether to an LLM or a different utility). LangChain provides a standard interface for chains, lots of integrations with other tools, and end-to-end chains for common applications. Indexes: Language models are often more powerful when combined with your own text data - this module covers best practices for doing exactly that. Agents: Agents involve an LLM making decisions about which Actions to take, taking that Action, seeing an Observation, and repeating that until done. LangChain provides a standard interface for agents, a selection of agents to choose from, and examples of end to end agents. Memory: Memory is the concept of persisting state between calls of a chain/agent. LangChain provides a standard interface for memory, a collection of memory implementations, and examples of chains/agents that use memory. Chat: Chat models are a variation on Language Models that expose a different API - rather than working with raw text, they work with messages. LangChain provides a standard interface for working with them and doing all the same things as above. Use Cases# The above modules can be used in a variety of ways. LangChain also provides guidance and assistance in this. Below are some of the common use cases LangChain supports. Agents: Agents are systems that use a language model to interact with other tools. These can be used to do more grounded question/answering, interact with APIs, or even take actions. Chatbots: Since language models are good at producing text, that makes them ideal for creating chatbots. Data Augmented Generation: Data Augmented Generation involves specific types of chains that first interact with an external datasource to fetch data to use in the generation step. Examples of this include summarization of long pieces of text and question/answering over specific data sources. Question Answering: Answering questions over specific documents, only utilizing the information in those documents to construct an answer. A type of Data Augmented Generation. Summarization: Summarizing longer documents into shorter, more condensed chunks of information. A type of Data Augmented Generation."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: ecf5c6f0-0aba-4a9d-ac9f-aed345f92694
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"ecf5c6f0-0aba-4a9d-ac9f-aed345f92694\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a48d4cb4-34ba-4501-9fe0-5428f6e050a4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"f6dcb4f6476138af315f53b8c1065777db5e592581960433def9a51924a7d01b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"667f9f42-d17b-4273-bed1-fd0b9e62f4b1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ccd77e967d5990671fee9e6299da6e48b22574b57fe08035a4d356ef69c0da86\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7401, \"end_char_idx\": 11127, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "Indexes: Language models are often more powerful when combined with your own text data - this module covers best practices for doing exactly that. Agents: Agents involve an LLM making decisions about which Actions to take, taking that Action, seeing an Observation, and repeating that until done. LangChain provides a standard interface for agents, a selection of agents to choose from, and examples of end to end agents. Memory: Memory is the concept of persisting state between calls of a chain/agent. LangChain provides a standard interface for memory, a collection of memory implementations, and examples of chains/agents that use memory. Chat: Chat models are a variation on Language Models that expose a different API - rather than working with raw text, they work with messages. LangChain provides a standard interface for working with them and doing all the same things as above. Use Cases# The above modules can be used in a variety of ways. LangChain also provides guidance and assistance in this. Below are some of the common use cases LangChain supports. Agents: Agents are systems that use a language model to interact with other tools. These can be used to do more grounded question/answering, interact with APIs, or even take actions. Chatbots: Since language models are good at producing text, that makes them ideal for creating chatbots. Data Augmented Generation: Data Augmented Generation involves specific types of chains that first interact with an external datasource to fetch data to use in the generation step. Examples of this include summarization of long pieces of text and question/answering over specific data sources. Question Answering: Answering questions over specific documents, only utilizing the information in those documents to construct an answer. A type of Data Augmented Generation. Summarization: Summarizing longer documents into shorter, more condensed chunks of information. A type of Data Augmented Generation. Evaluation: Generative models are notoriously hard to evaluate with traditional metrics. One new way of evaluating them is using language models themselves to do the evaluation. LangChain provides some prompts/chains for assisting in this. Generate similar examples: Generating similar examples to a given input. This is a common use case for many applications, and LangChain provides some prompts/chains for assisting in this. Compare models: Experimenting with different prompts, models, and chains is a big part of developing the best possible application. The ModelLaboratory makes it easy to do so. Reference Docs# All of LangChain\u2019s reference documentation, in one place. Full documentation on all methods, classes, installation methods, and integration setups for LangChain. Reference Documentation LangChain Ecosystem# Guides for how other companies/products can be used with LangChain LangChain Ecosystem Additional Resources# Additional collection of resources we think may be useful as you develop your application! LangChainHub: The LangChainHub is a place to share and explore other prompts, chains, and agents. Glossary: A glossary of all related terms, papers, methods, etc. Whether implemented in LangChain or not! Gallery: A collection of our favorite projects that use LangChain. Useful for finding inspiration or seeing how things were done in other applications. Deployments: A collection of instructions, code snippets, and template repositories for deploying LangChain apps. Discord: Join us on our Discord to discuss all things LangChain! Tracing: A guide on using tracing in LangChain to visualize the execution of chains and agents. Production Support: As you move your LangChains into production, we\u2019d love to offer more comprehensive support."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 667f9f42-d17b-4273-bed1-fd0b9e62f4b1
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"667f9f42-d17b-4273-bed1-fd0b9e62f4b1\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ecf5c6f0-0aba-4a9d-ac9f-aed345f92694\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"3ed80ffea10eccbb63d07abf18c31df265fd1905a4c96924d5bafea00646cac2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"db963e33-3f75-485a-865a-13c96af10994\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"546e6f96602b15542a2a11e010892fbbfe2864bff31dc4751e5575c46ce46016\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9188, \"end_char_idx\": 12529, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "A type of Data Augmented Generation. Summarization: Summarizing longer documents into shorter, more condensed chunks of information. A type of Data Augmented Generation. Evaluation: Generative models are notoriously hard to evaluate with traditional metrics. One new way of evaluating them is using language models themselves to do the evaluation. LangChain provides some prompts/chains for assisting in this. Generate similar examples: Generating similar examples to a given input. This is a common use case for many applications, and LangChain provides some prompts/chains for assisting in this. Compare models: Experimenting with different prompts, models, and chains is a big part of developing the best possible application. The ModelLaboratory makes it easy to do so. Reference Docs# All of LangChain\u2019s reference documentation, in one place. Full documentation on all methods, classes, installation methods, and integration setups for LangChain. Reference Documentation LangChain Ecosystem# Guides for how other companies/products can be used with LangChain LangChain Ecosystem Additional Resources# Additional collection of resources we think may be useful as you develop your application! LangChainHub: The LangChainHub is a place to share and explore other prompts, chains, and agents. Glossary: A glossary of all related terms, papers, methods, etc. Whether implemented in LangChain or not! Gallery: A collection of our favorite projects that use LangChain. Useful for finding inspiration or seeing how things were done in other applications. Deployments: A collection of instructions, code snippets, and template repositories for deploying LangChain apps. Discord: Join us on our Discord to discuss all things LangChain! Tracing: A guide on using tracing in LangChain to visualize the execution of chains and agents. Production Support: As you move your LangChains into production, we\u2019d love to offer more comprehensive support. Please fill out this form and we\u2019ll set up a dedicated support Slack channel. next Quickstart Guide Contents Getting Started Modules Use Cases Reference Docs LangChain Ecosystem Additional Resources By Harrison Chase \u00a9 Copyright 2022, Harrison Chase. Last updated on Mar 15, 2023.\n\n\nThis leaves us with hundreds of processed doc pages. Let's take a look at the format each one contains:\n\n\n```python\ndocs[0]\n```\n\nWe access the plaintext page content like so:\n\n\n```python\nprint(docs[0].page_content)\n```\n\n\n```python\nprint(docs[5].page_content)\n```\n\nWe can also find the source of each document:\n\n\n```python\ndocs[5].metadata['source'].replace('rtdocs/', 'https://')\n```\n\nWe can use these to create our `data` list:\n\n\n```python\ndata = []\n\nfor doc in docs:\n    data.append({\n        'url': doc.metadata['source'].replace('rtdocs/', 'https://'),\n        'text': doc.page_content\n    })\n```\n\n\n```python\ndata[3]\n```\n\n\n\n\n    {'url': 'https://langchain.readthedocs.io/en/latest/modules/memory/types/entity_summary_memory.html',\n     'text': '.ipynb .pdf Entity Memory Contents Using in a chain Inspecting the memory store Entity Memory# This notebook shows how to work with a memory module that remembers things about specific entities. It extracts information on entities (using LLMs) and builds up its knowledge about that entity over time (also using LLMs). Let\u2019s first walk through using this functionality."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: db963e33-3f75-485a-865a-13c96af10994
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"db963e33-3f75-485a-865a-13c96af10994\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"667f9f42-d17b-4273-bed1-fd0b9e62f4b1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e65fb1ac832c8f6252e556c9c1995382cec518a7b911f30e790d5695ee8271db\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"63ab9f0b-b493-4ec0-9f37-9c3380b64692\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"eb0b4c77e92b7376399c30deb3690ca2c3a7c9f91e36ad5e8afbf75e86e174e8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10855, \"end_char_idx\": 13916, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "Discord: Join us on our Discord to discuss all things LangChain! Tracing: A guide on using tracing in LangChain to visualize the execution of chains and agents. Production Support: As you move your LangChains into production, we\u2019d love to offer more comprehensive support. Please fill out this form and we\u2019ll set up a dedicated support Slack channel. next Quickstart Guide Contents Getting Started Modules Use Cases Reference Docs LangChain Ecosystem Additional Resources By Harrison Chase \u00a9 Copyright 2022, Harrison Chase. Last updated on Mar 15, 2023.\n\n\nThis leaves us with hundreds of processed doc pages. Let's take a look at the format each one contains:\n\n\n```python\ndocs[0]\n```\n\nWe access the plaintext page content like so:\n\n\n```python\nprint(docs[0].page_content)\n```\n\n\n```python\nprint(docs[5].page_content)\n```\n\nWe can also find the source of each document:\n\n\n```python\ndocs[5].metadata['source'].replace('rtdocs/', 'https://')\n```\n\nWe can use these to create our `data` list:\n\n\n```python\ndata = []\n\nfor doc in docs:\n    data.append({\n        'url': doc.metadata['source'].replace('rtdocs/', 'https://'),\n        'text': doc.page_content\n    })\n```\n\n\n```python\ndata[3]\n```\n\n\n\n\n    {'url': 'https://langchain.readthedocs.io/en/latest/modules/memory/types/entity_summary_memory.html',\n     'text': '.ipynb .pdf Entity Memory Contents Using in a chain Inspecting the memory store Entity Memory# This notebook shows how to work with a memory module that remembers things about specific entities. It extracts information on entities (using LLMs) and builds up its knowledge about that entity over time (also using LLMs). Let\u2019s first walk through using this functionality. from langchain.llms import OpenAI from langchain.memory import ConversationEntityMemory llm = OpenAI(temperature=0) memory = ConversationEntityMemory(llm=llm) _input = {\"input\": \"Deven & Sam are working on a hackathon project\"} memory.load_memory_variables(_input) memory.save_context( _input, {\"ouput\": \" That sounds like a great project! What kind of project are they working on?\"} ) memory.load_memory_variables({\"input\": \\'who is Sam\\'}) {\\'history\\': \\'Human: Deven & Sam are working on a hackathon project\\\\nAI: That sounds like a great project! What kind of project are they working on?\\', \\'entities\\': {\\'Sam\\': \\'Sam is working on a hackathon project with Deven.\\'}} memory = ConversationEntityMemory(llm=llm, return_messages=True) _input = {\"input\": \"Deven & Sam are working on a hackathon project\"} memory.load_memory_variables(_input) memory.save_context( _input, {\"ouput\": \" That sounds like a great project! What kind of project are they working on?\"} ) memory.load_memory_variables({\"input\": \\'who is Sam\\'}) {\\'history\\': [HumanMessage(content=\\'Deven & Sam are working on a hackathon project\\', additional_kwargs={}), AIMessage(content=\\' That sounds like a great project! What kind of project are they working on?\\', additional_kwargs={})], \\'entities\\': {\\'Sam\\': \\'Sam is working on a hackathon project with Deven.\\'}} Using in a chain# Let\u2019s now use it in a chain!"}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 63ab9f0b-b493-4ec0-9f37-9c3380b64692
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"63ab9f0b-b493-4ec0-9f37-9c3380b64692\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"db963e33-3f75-485a-865a-13c96af10994\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"cecd15a4d5c470a67807b3bb66c2ee5038ec8402bfbdd50fad66031e265c5a8c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7bede923-5d58-4083-9889-5af357af8ca7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"93a94fad140377b4fb75dcedd534cdc40b04b0030773f7bad3fb8d462f608936\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12355, \"end_char_idx\": 15767, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "It extracts information on entities (using LLMs) and builds up its knowledge about that entity over time (also using LLMs). Let\u2019s first walk through using this functionality. from langchain.llms import OpenAI from langchain.memory import ConversationEntityMemory llm = OpenAI(temperature=0) memory = ConversationEntityMemory(llm=llm) _input = {\"input\": \"Deven & Sam are working on a hackathon project\"} memory.load_memory_variables(_input) memory.save_context( _input, {\"ouput\": \" That sounds like a great project! What kind of project are they working on?\"} ) memory.load_memory_variables({\"input\": \\'who is Sam\\'}) {\\'history\\': \\'Human: Deven & Sam are working on a hackathon project\\\\nAI: That sounds like a great project! What kind of project are they working on?\\', \\'entities\\': {\\'Sam\\': \\'Sam is working on a hackathon project with Deven.\\'}} memory = ConversationEntityMemory(llm=llm, return_messages=True) _input = {\"input\": \"Deven & Sam are working on a hackathon project\"} memory.load_memory_variables(_input) memory.save_context( _input, {\"ouput\": \" That sounds like a great project! What kind of project are they working on?\"} ) memory.load_memory_variables({\"input\": \\'who is Sam\\'}) {\\'history\\': [HumanMessage(content=\\'Deven & Sam are working on a hackathon project\\', additional_kwargs={}), AIMessage(content=\\' That sounds like a great project! What kind of project are they working on?\\', additional_kwargs={})], \\'entities\\': {\\'Sam\\': \\'Sam is working on a hackathon project with Deven.\\'}} Using in a chain# Let\u2019s now use it in a chain! from langchain.chains import ConversationChain from langchain.memory import ConversationEntityMemory from langchain.memory.prompt import ENTITY_MEMORY_CONVERSATION_TEMPLATE from pydantic import BaseModel from typing import List, Dict, Any conversation = ConversationChain( llm=llm, verbose=True, prompt=ENTITY_MEMORY_CONVERSATION_TEMPLATE, memory=ConversationEntityMemory(llm=llm) ) conversation.predict(input=\"Deven & Sam are working on a hackathon project\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI. You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 7bede923-5d58-4083-9889-5af357af8ca7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"7bede923-5d58-4083-9889-5af357af8ca7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"63ab9f0b-b493-4ec0-9f37-9c3380b64692\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"89af7ccefa09c4a4e39820edb1492229d5ecd9fa1f934fe5b69d483ec3538e21\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1d7a230c-2644-494f-b1a6-6d7930e3db61\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"03547a629b37fc8603bba0e32de65c7732bf561fa4cbcc427b5f453c13c8d94a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13917, \"end_char_idx\": 17645, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "from langchain.chains import ConversationChain from langchain.memory import ConversationEntityMemory from langchain.memory.prompt import ENTITY_MEMORY_CONVERSATION_TEMPLATE from pydantic import BaseModel from typing import List, Dict, Any conversation = ConversationChain( llm=llm, verbose=True, prompt=ENTITY_MEMORY_CONVERSATION_TEMPLATE, memory=ConversationEntityMemory(llm=llm) ) conversation.predict(input=\"Deven & Sam are working on a hackathon project\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI. You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist. Context: {\\'Deven\\': \\'\\', \\'Sam\\': \\'\\'} Current conversation: Last line: Human: Deven & Sam are working on a hackathon project You: > Finished chain. \\' That sounds like a great project! What kind of project are they working on?\\' conversation.memory.store {\\'Deven\\': \\'Deven is working on a hackathon project with Sam.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven.\\'} conversation.predict(input=\"They are trying to add more complex memory structures to Langchain\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI. You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 1d7a230c-2644-494f-b1a6-6d7930e3db61
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"1d7a230c-2644-494f-b1a6-6d7930e3db61\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7bede923-5d58-4083-9889-5af357af8ca7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"d7488772900bd8b20547485ab55317b653c7a2b0442409710b0099a03223bee3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6f9526d4-767d-415f-89e7-85b42e49ee61\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"73b65920df4df45507208d9e663600963566a119a9a3dbf58ef5150a103d72f6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15768, \"end_char_idx\": 19392, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "Context: {\\'Deven\\': \\'\\', \\'Sam\\': \\'\\'} Current conversation: Last line: Human: Deven & Sam are working on a hackathon project You: > Finished chain. \\' That sounds like a great project! What kind of project are they working on?\\' conversation.memory.store {\\'Deven\\': \\'Deven is working on a hackathon project with Sam.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven.\\'} conversation.predict(input=\"They are trying to add more complex memory structures to Langchain\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI. You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist. Context: {\\'Deven\\': \\'Deven is working on a hackathon project with Sam.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven.\\', \\'Langchain\\': \\'\\'} Current conversation: Human: Deven & Sam are working on a hackathon project AI: That sounds like a great project! What kind of project are they working on? Last line: Human: They are trying to add more complex memory structures to Langchain You: > Finished chain. \\' That sounds like an interesting project! What kind of memory structures are they trying to add?\\' conversation.predict(input=\"They are adding in a key-value store for entities mentioned so far in the conversation.\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI. You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 6f9526d4-767d-415f-89e7-85b42e49ee61
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"6f9526d4-767d-415f-89e7-85b42e49ee61\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1d7a230c-2644-494f-b1a6-6d7930e3db61\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"f092d6c94646f26b16bb6abec0412376672cda66365701d81ea86aaabbe20465\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"864c7d9b-25cc-4d1c-b79e-f32064baf0ce\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4bbc045004cbbbe3e448cd9602c4263ad9580a4dfc81b60eaff21735afcfd15c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17505, \"end_char_idx\": 20991, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist. Context: {\\'Deven\\': \\'Deven is working on a hackathon project with Sam.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven.\\', \\'Langchain\\': \\'\\'} Current conversation: Human: Deven & Sam are working on a hackathon project AI: That sounds like a great project! What kind of project are they working on? Last line: Human: They are trying to add more complex memory structures to Langchain You: > Finished chain. \\' That sounds like an interesting project! What kind of memory structures are they trying to add?\\' conversation.predict(input=\"They are adding in a key-value store for entities mentioned so far in the conversation.\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI. You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist. Context: {\\'Deven\\': \\'Deven is working on a hackathon project with Sam, attempting to add more complex memory structures to Langchain.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven, trying to add more complex memory structures to Langchain.\\', \\'Langchain\\': \\'Langchain is a project that is trying to add more complex memory structures.\\', \\'Key-Value Store\\': \\'\\'} Current conversation: Human: Deven & Sam are working on a hackathon project AI: That sounds like a great project! What kind of project are they working on? Human: They are trying to add more complex memory structures to Langchain AI: That sounds like an interesting project! What kind of memory structures are they trying to add? Last line: Human: They are adding in a key-value store for entities mentioned so far in the conversation. You: > Finished chain. \\' That sounds like a great idea! How will the key-value store work?\\' conversation.predict(input=\"What do you know about Deven & Sam?\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI. You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 864c7d9b-25cc-4d1c-b79e-f32064baf0ce
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"864c7d9b-25cc-4d1c-b79e-f32064baf0ce\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6f9526d4-767d-415f-89e7-85b42e49ee61\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"92e4d306c08f3221a6c1cc4bd99714d9f0ece5faec60093629d381991ecbbd86\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cf663f24-fb1a-4c64-b30b-1015bf32973b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a91c5fe92b5b92b2d19d4dcaa0b5218db6190b093fbcf6073ff67b3e735a8dc1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19201, \"end_char_idx\": 22803, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist. Context: {\\'Deven\\': \\'Deven is working on a hackathon project with Sam, attempting to add more complex memory structures to Langchain.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven, trying to add more complex memory structures to Langchain.\\', \\'Langchain\\': \\'Langchain is a project that is trying to add more complex memory structures.\\', \\'Key-Value Store\\': \\'\\'} Current conversation: Human: Deven & Sam are working on a hackathon project AI: That sounds like a great project! What kind of project are they working on? Human: They are trying to add more complex memory structures to Langchain AI: That sounds like an interesting project! What kind of memory structures are they trying to add? Last line: Human: They are adding in a key-value store for entities mentioned so far in the conversation. You: > Finished chain. \\' That sounds like a great idea! How will the key-value store work?\\' conversation.predict(input=\"What do you know about Deven & Sam?\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI. You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist. Context: {\\'Deven\\': \\'Deven is working on a hackathon project with Sam, attempting to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven, trying to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.\\'} Current conversation: Human: Deven & Sam are working on a hackathon project AI: That sounds like a great project! What kind of project are they working on? Human: They are trying to add more complex memory structures to Langchain AI: That sounds like an interesting project! What kind of memory structures are they trying to add?"}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: cf663f24-fb1a-4c64-b30b-1015bf32973b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"cf663f24-fb1a-4c64-b30b-1015bf32973b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"864c7d9b-25cc-4d1c-b79e-f32064baf0ce\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"8f290fca4ce9be228e62291118fbf5fb7b7b0ba509dcd3668925c0596088c9f9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0b66a5b9-bbe2-4c9d-b23f-d58864e65cdc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"595458a7c2454b396a75fee8e4d50db9cd33a2c6355d1fbf712cedcb45e0945f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 20816, \"end_char_idx\": 24473, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist. Context: {\\'Deven\\': \\'Deven is working on a hackathon project with Sam, attempting to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven, trying to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.\\'} Current conversation: Human: Deven & Sam are working on a hackathon project AI: That sounds like a great project! What kind of project are they working on? Human: They are trying to add more complex memory structures to Langchain AI: That sounds like an interesting project! What kind of memory structures are they trying to add? Human: They are adding in a key-value store for entities mentioned so far in the conversation. AI: That sounds like a great idea! How will the key-value store work? Last line: Human: What do you know about Deven & Sam? You: > Finished chain. \\' Deven and Sam are working on a hackathon project together, attempting to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.\\' Inspecting the memory store# We can also inspect the memory store directly. In the following examaples, we look at it directly, and then go through some examples of adding information and watch how it changes. from pprint import pprint pprint(conversation.memory.store) {\\'Deven\\': \\'Deven is working on a hackathon project with Sam, attempting to add \\' \\'more complex memory structures to Langchain, including a key-value \\' \\'store for entities mentioned so far in the conversation.\\', \\'Key-Value Store\\': \\'A key-value store that stores entities mentioned in the \\' \\'conversation.\\', \\'Langchain\\': \\'Langchain is a project that is trying to add more complex \\' \\'memory structures, including a key-value store for entities \\' \\'mentioned so far in the conversation.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven, attempting to add \\' \\'more complex memory structures to Langchain, including a key-value \\' \\'store for entities mentioned so far in the conversation.\\'} conversation.predict(input=\"Sam is the founder of a company called Daimon.\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 0b66a5b9-bbe2-4c9d-b23f-d58864e65cdc
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0b66a5b9-bbe2-4c9d-b23f-d58864e65cdc\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cf663f24-fb1a-4c64-b30b-1015bf32973b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"8979d6204ba339247e408d696754c052595525a29946c157526b9e37df2ef53e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e116f5ae-41fc-4530-a627-c98fd20e796f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7974fd0b4d1cad29b62feef85eeace12c90f951f9e5629f0395848040e803ba8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 22749, \"end_char_idx\": 26337, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "What kind of memory structures are they trying to add? Human: They are adding in a key-value store for entities mentioned so far in the conversation. AI: That sounds like a great idea! How will the key-value store work? Last line: Human: What do you know about Deven & Sam? You: > Finished chain. \\' Deven and Sam are working on a hackathon project together, attempting to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.\\' Inspecting the memory store# We can also inspect the memory store directly. In the following examaples, we look at it directly, and then go through some examples of adding information and watch how it changes. from pprint import pprint pprint(conversation.memory.store) {\\'Deven\\': \\'Deven is working on a hackathon project with Sam, attempting to add \\' \\'more complex memory structures to Langchain, including a key-value \\' \\'store for entities mentioned so far in the conversation.\\', \\'Key-Value Store\\': \\'A key-value store that stores entities mentioned in the \\' \\'conversation.\\', \\'Langchain\\': \\'Langchain is a project that is trying to add more complex \\' \\'memory structures, including a key-value store for entities \\' \\'mentioned so far in the conversation.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven, attempting to add \\' \\'more complex memory structures to Langchain, including a key-value \\' \\'store for entities mentioned so far in the conversation.\\'} conversation.predict(input=\"Sam is the founder of a company called Daimon.\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI. You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist. Context: {\\'Daimon\\': \\'\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.\\'} Current conversation: Human: They are trying to add more complex memory structures to Langchain AI: That sounds like an interesting project! What kind of memory structures are they trying to add? Human: They are adding in a key-value store for entities mentioned so far in the conversation. AI: That sounds like a great idea! How will the key-value store work? Human: What do you know about Deven & Sam?"}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: e116f5ae-41fc-4530-a627-c98fd20e796f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e116f5ae-41fc-4530-a627-c98fd20e796f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0b66a5b9-bbe2-4c9d-b23f-d58864e65cdc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"f5a7f50dcec839e1fabb745a7b3fe822531125167d3b937c084fe959ef740a3e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2c6f5768-65f6-4c6b-b74f-1d5a4e6ba5b3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"396ad241a19796d845203f794bbdaf4b24872224f53d2d1e0a4537c7ddf19f6b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 24474, \"end_char_idx\": 27912, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist. Context: {\\'Daimon\\': \\'\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.\\'} Current conversation: Human: They are trying to add more complex memory structures to Langchain AI: That sounds like an interesting project! What kind of memory structures are they trying to add? Human: They are adding in a key-value store for entities mentioned so far in the conversation. AI: That sounds like a great idea! How will the key-value store work? Human: What do you know about Deven & Sam? AI: Deven and Sam are working on a hackathon project to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to be very motivated and passionate about their project, and are working hard to make it a success. Last line: Human: Sam is the founder of a company called Daimon. You: > Finished chain. \"\\\\nThat\\'s impressive! It sounds like Sam is a very successful entrepreneur. What kind of company is Daimon?\" from pprint import pprint pprint(conversation.memory.store) {\\'Daimon\\': \\'Daimon is a company founded by Sam.\\', \\'Deven\\': \\'Deven is working on a hackathon project with Sam to add more \\' \\'complex memory structures to Langchain, including a key-value store \\' \\'for entities mentioned so far in the conversation.\\', \\'Key-Value Store\\': \\'Key-Value Store: A data structure that stores values \\' \\'associated with a unique key, allowing for efficient \\' \\'retrieval of values. Deven and Sam are adding a key-value \\' \\'store for entities mentioned so far in the conversation.\\', \\'Langchain\\': \\'Langchain is a project that seeks to add more complex memory \\' \\'structures, including a key-value store for entities mentioned \\' \\'so far in the conversation.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven to add more complex \\' \\'memory structures to Langchain, including a key-value store for \\' \\'entities mentioned so far in the conversation. He is also the founder \\' \\'of a company called Daimon.\\'} conversation.predict(input=\"What do you know about Sam?\")"}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 2c6f5768-65f6-4c6b-b74f-1d5a4e6ba5b3
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"2c6f5768-65f6-4c6b-b74f-1d5a4e6ba5b3\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e116f5ae-41fc-4530-a627-c98fd20e796f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"50d53b8a2ad61fe7e3765829484602531a2c5f1b45a94c7d62cebe680e8fb81e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a3fd656c-500a-4080-ba4b-4a880de18401\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e0cb814d232a103a4a519ed24d8cd7bd263f368a9edac5a3e5315bfd97cfeb62\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 26225, \"end_char_idx\": 29728, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "AI: That sounds like a great idea! How will the key-value store work? Human: What do you know about Deven & Sam? AI: Deven and Sam are working on a hackathon project to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to be very motivated and passionate about their project, and are working hard to make it a success. Last line: Human: Sam is the founder of a company called Daimon. You: > Finished chain. \"\\\\nThat\\'s impressive! It sounds like Sam is a very successful entrepreneur. What kind of company is Daimon?\" from pprint import pprint pprint(conversation.memory.store) {\\'Daimon\\': \\'Daimon is a company founded by Sam.\\', \\'Deven\\': \\'Deven is working on a hackathon project with Sam to add more \\' \\'complex memory structures to Langchain, including a key-value store \\' \\'for entities mentioned so far in the conversation.\\', \\'Key-Value Store\\': \\'Key-Value Store: A data structure that stores values \\' \\'associated with a unique key, allowing for efficient \\' \\'retrieval of values. Deven and Sam are adding a key-value \\' \\'store for entities mentioned so far in the conversation.\\', \\'Langchain\\': \\'Langchain is a project that seeks to add more complex memory \\' \\'structures, including a key-value store for entities mentioned \\' \\'so far in the conversation.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven to add more complex \\' \\'memory structures to Langchain, including a key-value store for \\' \\'entities mentioned so far in the conversation. He is also the founder \\' \\'of a company called Daimon.\\'} conversation.predict(input=\"What do you know about Sam?\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI. You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist. Context: {\\'Sam\\': \\'Sam is working on a hackathon project with Deven to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. He is also the founder of a company called Daimon.\\', \\'Daimon\\': \\'Daimon is a company founded by Sam.\\'} Current conversation: Human: They are adding in a key-value store for entities mentioned so far in the conversation."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: a3fd656c-500a-4080-ba4b-4a880de18401
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a3fd656c-500a-4080-ba4b-4a880de18401\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2c6f5768-65f6-4c6b-b74f-1d5a4e6ba5b3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"9eac9a40892cedf57479eb2e7e26c73ec28161ba7c80c2146a51df89dd0ed474\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3df101ed-78e2-42aa-a652-69d1c0bbb9cc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6c9bc8316944935518f43d74919a37c910648fea6da92b13b4432636fbab4853\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 27796, \"end_char_idx\": 31126, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "He is also the founder \\' \\'of a company called Daimon.\\'} conversation.predict(input=\"What do you know about Sam?\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI. You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist. Context: {\\'Sam\\': \\'Sam is working on a hackathon project with Deven to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. He is also the founder of a company called Daimon.\\', \\'Daimon\\': \\'Daimon is a company founded by Sam.\\'} Current conversation: Human: They are adding in a key-value store for entities mentioned so far in the conversation. AI: That sounds like a great idea! How will the key-value store work? Human: What do you know about Deven & Sam? AI: Deven and Sam are working on a hackathon project to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to be very motivated and passionate about their project, and are working hard to make it a success. Human: Sam is the founder of a company called Daimon. AI: That\\'s impressive! It sounds like Sam is a very successful entrepreneur. What kind of company is Daimon? Last line: Human: What do you know about Sam? You: > Finished chain. \\' Sam is the founder of a company called Daimon. He is also working on a hackathon project with Deven to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. He seems to be very motivated and passionate about his project, and is working hard to make it a success.\\' previous ConversationBufferWindowMemory next Conversation Knowledge Graph Memory Contents Using in a chain Inspecting the memory store By Harrison Chase \u00a9 Copyright 2022, Harrison Chase. Last updated on Mar 15, 2023.'}\n\n\n\nIt's pretty ugly but it's good enough for now. Let's see how we can process all of these. We will chunk everything into ~400 token chunks, we can do this easily with `langchain` and `tiktoken`:"}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 3df101ed-78e2-42aa-a652-69d1c0bbb9cc
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"3df101ed-78e2-42aa-a652-69d1c0bbb9cc\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a3fd656c-500a-4080-ba4b-4a880de18401\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"d69b3db5002183e2d307d6b76ce67f821657f3ee5c3d2e38250aed215ffd3890\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9fae4c70-a13c-49e7-beee-e807bc636f3a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c81be2465c67526e3139ce57ef60dc00f760905647105a886c02041ad22fb64e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 29505, \"end_char_idx\": 32649, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "He is also the founder of a company called Daimon.\\', \\'Daimon\\': \\'Daimon is a company founded by Sam.\\'} Current conversation: Human: They are adding in a key-value store for entities mentioned so far in the conversation. AI: That sounds like a great idea! How will the key-value store work? Human: What do you know about Deven & Sam? AI: Deven and Sam are working on a hackathon project to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to be very motivated and passionate about their project, and are working hard to make it a success. Human: Sam is the founder of a company called Daimon. AI: That\\'s impressive! It sounds like Sam is a very successful entrepreneur. What kind of company is Daimon? Last line: Human: What do you know about Sam? You: > Finished chain. \\' Sam is the founder of a company called Daimon. He is also working on a hackathon project with Deven to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. He seems to be very motivated and passionate about his project, and is working hard to make it a success.\\' previous ConversationBufferWindowMemory next Conversation Knowledge Graph Memory Contents Using in a chain Inspecting the memory store By Harrison Chase \u00a9 Copyright 2022, Harrison Chase. Last updated on Mar 15, 2023.'}\n\n\n\nIt's pretty ugly but it's good enough for now. Let's see how we can process all of these. We will chunk everything into ~400 token chunks, we can do this easily with `langchain` and `tiktoken`:\n\n\n```python\nimport tiktoken\n\ntokenizer = tiktoken.get_encoding('p50k_base')\n\n# create the length function\ndef tiktoken_len(text):\n    tokens = tokenizer.encode(\n        text,\n        disallowed_special=()\n    )\n    return len(tokens)\n```\n\n\n```python\nfrom langchain.text_splitter import RecursiveCharacterTextSplitter\n\ntext_splitter = RecursiveCharacterTextSplitter(\n    chunk_size=400,\n    chunk_overlap=20,\n    length_function=tiktoken_len,\n    separators=[\"\\n\\n\", \"\\n\", \" \", \"\"]\n)\n```\n\nProcess the `data` into more chunks using this approach.\n\n\n```python\nfrom uuid import uuid4\nfrom tqdm.auto import tqdm\n\nchunks = []\n\nfor idx, record in enumerate(tqdm(data)):\n    texts = text_splitter.split_text(record['text'])\n    chunks.extend([{\n        'id': str(uuid4()),\n        'text': texts[i],\n        'chunk': i,\n        'url': record['url']\n    } for i in range(len(texts))])\n```\n\n\n      0%|          | 0/231 [00:00<?, ?it/s]\n\n\nOur chunks are ready so now we move onto embedding and indexing everything.\n\n## Initialize Embedding Model\n\nWe use `text-embedding-3-small` as the embedding model. We can embed text like so:\n\n\n```python\nimport openai\n\n# initialize openai API key\nopenai.api_key = \"sk-...\"\n\nembed_model = \"text-embedding-3-small\"\n\nres = openai.Embedding.create(\n    input=[\n        \"Sample document text goes here\",\n        \"there will be several phrases in each batch\"\n    ], engine=embed_model\n)\n```\n\nIn the response `res` we will find a JSON-like object containing our new embeddings within the `'data'` field."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 9fae4c70-a13c-49e7-beee-e807bc636f3a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"9fae4c70-a13c-49e7-beee-e807bc636f3a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3df101ed-78e2-42aa-a652-69d1c0bbb9cc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"912e4aec2676f9c4ee4e04f93c38ad84bf5352a689dba8143a484c331f547bb5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"589a6fed-239c-4b5d-860f-8d7dde159cf5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"372401475dc1c570393ceb3af2e55d3bbffa5542f795935a80703302d6bcf014\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 31129, \"end_char_idx\": 33560, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "```python\nimport tiktoken\n\ntokenizer = tiktoken.get_encoding('p50k_base')\n\n# create the length function\ndef tiktoken_len(text):\n    tokens = tokenizer.encode(\n        text,\n        disallowed_special=()\n    )\n    return len(tokens)\n```\n\n\n```python\nfrom langchain.text_splitter import RecursiveCharacterTextSplitter\n\ntext_splitter = RecursiveCharacterTextSplitter(\n    chunk_size=400,\n    chunk_overlap=20,\n    length_function=tiktoken_len,\n    separators=[\"\\n\\n\", \"\\n\", \" \", \"\"]\n)\n```\n\nProcess the `data` into more chunks using this approach.\n\n\n```python\nfrom uuid import uuid4\nfrom tqdm.auto import tqdm\n\nchunks = []\n\nfor idx, record in enumerate(tqdm(data)):\n    texts = text_splitter.split_text(record['text'])\n    chunks.extend([{\n        'id': str(uuid4()),\n        'text': texts[i],\n        'chunk': i,\n        'url': record['url']\n    } for i in range(len(texts))])\n```\n\n\n      0%|          | 0/231 [00:00<?, ?it/s]\n\n\nOur chunks are ready so now we move onto embedding and indexing everything.\n\n## Initialize Embedding Model\n\nWe use `text-embedding-3-small` as the embedding model. We can embed text like so:\n\n\n```python\nimport openai\n\n# initialize openai API key\nopenai.api_key = \"sk-...\"\n\nembed_model = \"text-embedding-3-small\"\n\nres = openai.Embedding.create(\n    input=[\n        \"Sample document text goes here\",\n        \"there will be several phrases in each batch\"\n    ], engine=embed_model\n)\n```\n\nIn the response `res` we will find a JSON-like object containing our new embeddings within the `'data'` field.\n\n\n```python\nres.keys()\n```\n\n\n\n\n    dict_keys(['object', 'data', 'model', 'usage'])\n\n\n\nInside `'data'` we will find two records, one for each of the two sentences we just embedded. Each vector embedding contains `1536` dimensions (the output dimensionality of the `text-embedding-3-small` model.\n\n\n```python\nlen(res['data'])\n```\n\n\n\n\n    2\n\n\n\n\n```python\nlen(res['data'][0]['embedding']), len(res['data'][1]['embedding'])\n```\n\n\n\n\n    (1536, 1536)\n\n\n\nWe will apply this same embedding logic to the langchain docs dataset we've just scraped. But before doing so we must create a place to store the embeddings.\n\n## Initializing the Index\n\nNow we need a place to store these embeddings and enable a efficient vector search through them all. To do that we use Pinecone, we can get a [free API key](https://app.pinecone.io/) and enter it below where we will initialize our connection to Pinecone and create a new index."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 589a6fed-239c-4b5d-860f-8d7dde159cf5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"589a6fed-239c-4b5d-860f-8d7dde159cf5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9fae4c70-a13c-49e7-beee-e807bc636f3a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1db5d4b7fc7a983bf814875b1df803e7469a63d8db40dedb512d603658d3d523\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a3301c0c-d82f-452a-9806-88d8e305bd22\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ab6c9a29118bd61a3da4af121a3aa0002fb567936f08c4464280544caa8d3275\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 32014, \"end_char_idx\": 34484, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "0%|          | 0/231 [00:00<?, ?it/s]\n\n\nOur chunks are ready so now we move onto embedding and indexing everything.\n\n## Initialize Embedding Model\n\nWe use `text-embedding-3-small` as the embedding model. We can embed text like so:\n\n\n```python\nimport openai\n\n# initialize openai API key\nopenai.api_key = \"sk-...\"\n\nembed_model = \"text-embedding-3-small\"\n\nres = openai.Embedding.create(\n    input=[\n        \"Sample document text goes here\",\n        \"there will be several phrases in each batch\"\n    ], engine=embed_model\n)\n```\n\nIn the response `res` we will find a JSON-like object containing our new embeddings within the `'data'` field.\n\n\n```python\nres.keys()\n```\n\n\n\n\n    dict_keys(['object', 'data', 'model', 'usage'])\n\n\n\nInside `'data'` we will find two records, one for each of the two sentences we just embedded. Each vector embedding contains `1536` dimensions (the output dimensionality of the `text-embedding-3-small` model.\n\n\n```python\nlen(res['data'])\n```\n\n\n\n\n    2\n\n\n\n\n```python\nlen(res['data'][0]['embedding']), len(res['data'][1]['embedding'])\n```\n\n\n\n\n    (1536, 1536)\n\n\n\nWe will apply this same embedding logic to the langchain docs dataset we've just scraped. But before doing so we must create a place to store the embeddings.\n\n## Initializing the Index\n\nNow we need a place to store these embeddings and enable a efficient vector search through them all. To do that we use Pinecone, we can get a [free API key](https://app.pinecone.io/) and enter it below where we will initialize our connection to Pinecone and create a new index.\n\n\n```python\nimport pinecone\n\nindex_name = 'gpt-4-langchain-docs'\n\n# initialize connection to pinecone\npinecone.init(\n    api_key=\"PINECONE_API_KEY\",  # app.pinecone.io (console)\n    environment=\"PINECONE_ENVIRONMENT\"  # next to API key in console\n)\n\n# check if index already exists (it shouldn't if this is first time)\nif index_name not in pinecone.list_indexes():\n    # if does not exist, create index\n    pinecone.create_index(\n        index_name,\n        dimension=len(res['data'][0]['embedding']),\n        metric='dotproduct'\n    )\n# connect to index\nindex = pinecone.GRPCIndex(index_name)\n# view index stats\nindex.describe_index_stats()\n```\n\n\n\n\n    {'dimension': 1536,\n     'index_fullness': 0.0,\n     'namespaces': {},\n     'total_vector_count': 0}\n\n\n\nWe can see the index is currently empty with a `total_vector_count` of `0`. We can begin populating it with OpenAI `text-embedding-3-small` built embeddings like so:"}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: a3301c0c-d82f-452a-9806-88d8e305bd22
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a3301c0c-d82f-452a-9806-88d8e305bd22\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"589a6fed-239c-4b5d-860f-8d7dde159cf5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"246d28d2b6b03bf6f7d5e495b6f8b8fae23b3eee30c68adb88844d77a8e8aeea\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e335a6bc-7d0f-4702-910f-d00f05d7e6c0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"26dd93ef9645fc2e23b364e6181d28409cd9c492f68aff09f0a39a8b4dc37903\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 32947, \"end_char_idx\": 35681, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "```python\nlen(res['data'])\n```\n\n\n\n\n    2\n\n\n\n\n```python\nlen(res['data'][0]['embedding']), len(res['data'][1]['embedding'])\n```\n\n\n\n\n    (1536, 1536)\n\n\n\nWe will apply this same embedding logic to the langchain docs dataset we've just scraped. But before doing so we must create a place to store the embeddings.\n\n## Initializing the Index\n\nNow we need a place to store these embeddings and enable a efficient vector search through them all. To do that we use Pinecone, we can get a [free API key](https://app.pinecone.io/) and enter it below where we will initialize our connection to Pinecone and create a new index.\n\n\n```python\nimport pinecone\n\nindex_name = 'gpt-4-langchain-docs'\n\n# initialize connection to pinecone\npinecone.init(\n    api_key=\"PINECONE_API_KEY\",  # app.pinecone.io (console)\n    environment=\"PINECONE_ENVIRONMENT\"  # next to API key in console\n)\n\n# check if index already exists (it shouldn't if this is first time)\nif index_name not in pinecone.list_indexes():\n    # if does not exist, create index\n    pinecone.create_index(\n        index_name,\n        dimension=len(res['data'][0]['embedding']),\n        metric='dotproduct'\n    )\n# connect to index\nindex = pinecone.GRPCIndex(index_name)\n# view index stats\nindex.describe_index_stats()\n```\n\n\n\n\n    {'dimension': 1536,\n     'index_fullness': 0.0,\n     'namespaces': {},\n     'total_vector_count': 0}\n\n\n\nWe can see the index is currently empty with a `total_vector_count` of `0`. We can begin populating it with OpenAI `text-embedding-3-small` built embeddings like so:\n\n\n```python\nfrom tqdm.auto import tqdm\nimport datetime\nfrom time import sleep\n\nbatch_size = 100  # how many embeddings we create and insert at once\n\nfor i in tqdm(range(0, len(chunks), batch_size)):\n    # find end of batch\n    i_end = min(len(chunks), i+batch_size)\n    meta_batch = chunks[i:i_end]\n    # get ids\n    ids_batch = [x['id'] for x in meta_batch]\n    # get texts to encode\n    texts = [x['text'] for x in meta_batch]\n    # create embeddings (try-except added to avoid RateLimitError)\n    try:\n        res = openai.Embedding.create(input=texts, engine=embed_model)\n    except:\n        done = False\n        while not done:\n            sleep(5)\n            try:\n                res = openai.Embedding.create(input=texts, engine=embed_model)\n                done = True\n            except:\n                pass\n    embeds = [record['embedding'] for record in res['data']]\n    # cleanup metadata\n    meta_batch = [{\n        'text': x['text'],\n        'chunk': x['chunk'],\n        'url': x['url']\n    } for x in meta_batch]\n    to_upsert = list(zip(ids_batch, embeds, meta_batch))\n    # upsert to Pinecone\n    index.upsert(vectors=to_upsert)\n```\n\n\n      0%|          | 0/12 [00:00<?, ?it/s]"}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: e335a6bc-7d0f-4702-910f-d00f05d7e6c0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e335a6bc-7d0f-4702-910f-d00f05d7e6c0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a3301c0c-d82f-452a-9806-88d8e305bd22\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"d5a718bbe9db768dc878beb3f2bb668d02d198ab09b5a0b796703f5e18a19ee1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4da3be08-9931-44db-af1b-2942631ddd9c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"57703511b8d18ccbc3f90990500aa373c7ede900ce8d0cd64ef167f94f1e03f3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 34215, \"end_char_idx\": 37551, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "{'dimension': 1536,\n     'index_fullness': 0.0,\n     'namespaces': {},\n     'total_vector_count': 0}\n\n\n\nWe can see the index is currently empty with a `total_vector_count` of `0`. We can begin populating it with OpenAI `text-embedding-3-small` built embeddings like so:\n\n\n```python\nfrom tqdm.auto import tqdm\nimport datetime\nfrom time import sleep\n\nbatch_size = 100  # how many embeddings we create and insert at once\n\nfor i in tqdm(range(0, len(chunks), batch_size)):\n    # find end of batch\n    i_end = min(len(chunks), i+batch_size)\n    meta_batch = chunks[i:i_end]\n    # get ids\n    ids_batch = [x['id'] for x in meta_batch]\n    # get texts to encode\n    texts = [x['text'] for x in meta_batch]\n    # create embeddings (try-except added to avoid RateLimitError)\n    try:\n        res = openai.Embedding.create(input=texts, engine=embed_model)\n    except:\n        done = False\n        while not done:\n            sleep(5)\n            try:\n                res = openai.Embedding.create(input=texts, engine=embed_model)\n                done = True\n            except:\n                pass\n    embeds = [record['embedding'] for record in res['data']]\n    # cleanup metadata\n    meta_batch = [{\n        'text': x['text'],\n        'chunk': x['chunk'],\n        'url': x['url']\n    } for x in meta_batch]\n    to_upsert = list(zip(ids_batch, embeds, meta_batch))\n    # upsert to Pinecone\n    index.upsert(vectors=to_upsert)\n```\n\n\n      0%|          | 0/12 [00:00<?, ?it/s]\n\n\nNow we've added all of our langchain docs to the index. With that we can move on to retrieval and then answer generation using GPT-4.\n\n## Retrieval\n\nTo search through our documents we first need to create a query vector `xq`. Using `xq` we will retrieve the most relevant chunks from the LangChain docs, like so:\n\n\n```python\nquery = \"how do I use the LLMChain in LangChain?\"\n\nres = openai.Embedding.create(\n    input=[query],\n    engine=embed_model\n)\n\n# retrieve from Pinecone\nxq = res['data'][0]['embedding']\n\n# get relevant contexts (including the questions)\nres = index.query(xq, top_k=5, include_metadata=True)\n```\n\n\n```python\nres\n```\n\n\n\n\n    {'matches': [{'id': '1fec660b-9937-4f7e-9692-280c8cc7ce0d',\n                  'metadata': {'chunk': 0.0,\n                               'text': '.rst .pdf Chains Chains# Using an LLM in '\n                                       'isolation is fine for some simple '\n                                       'applications, but many more complex ones '\n                                       'require chaining LLMs - either with each '\n                                       'other or with other experts. LangChain '\n                                       'provides a standard interface for Chains, '\n                                       'as well as some common implementations of '\n                                       'chains for ease of use. The following '\n                                       'sections of documentation are provided: '\n                                       'Getting Started: A getting started guide '\n                                       'for chains, to get you up and running '\n                                       'quickly. Key Concepts: A conceptual guide '\n                                       'going over the various concepts related to '\n                                       'chains."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 4da3be08-9931-44db-af1b-2942631ddd9c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"4da3be08-9931-44db-af1b-2942631ddd9c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e335a6bc-7d0f-4702-910f-d00f05d7e6c0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4411630be21542803dc174853a157e685cd3d0584a1eb6168df53eee8104e7f8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a82d4f90-acd7-4c05-96dd-2ed4287ef9d1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7d6e4fb075b39bb9773c9cec5dd2b9d9f95da53633593b12529a1e35fe4a4038\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 35645, \"end_char_idx\": 39783, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "0%|          | 0/12 [00:00<?, ?it/s]\n\n\nNow we've added all of our langchain docs to the index. With that we can move on to retrieval and then answer generation using GPT-4.\n\n## Retrieval\n\nTo search through our documents we first need to create a query vector `xq`. Using `xq` we will retrieve the most relevant chunks from the LangChain docs, like so:\n\n\n```python\nquery = \"how do I use the LLMChain in LangChain?\"\n\nres = openai.Embedding.create(\n    input=[query],\n    engine=embed_model\n)\n\n# retrieve from Pinecone\nxq = res['data'][0]['embedding']\n\n# get relevant contexts (including the questions)\nres = index.query(xq, top_k=5, include_metadata=True)\n```\n\n\n```python\nres\n```\n\n\n\n\n    {'matches': [{'id': '1fec660b-9937-4f7e-9692-280c8cc7ce0d',\n                  'metadata': {'chunk': 0.0,\n                               'text': '.rst .pdf Chains Chains# Using an LLM in '\n                                       'isolation is fine for some simple '\n                                       'applications, but many more complex ones '\n                                       'require chaining LLMs - either with each '\n                                       'other or with other experts. LangChain '\n                                       'provides a standard interface for Chains, '\n                                       'as well as some common implementations of '\n                                       'chains for ease of use. The following '\n                                       'sections of documentation are provided: '\n                                       'Getting Started: A getting started guide '\n                                       'for chains, to get you up and running '\n                                       'quickly. Key Concepts: A conceptual guide '\n                                       'going over the various concepts related to '\n                                       'chains. How-To Guides: A collection of '\n                                       'how-to guides. These highlight how to use '\n                                       'various types of chains. Reference: API '\n                                       'reference documentation for all Chain '\n                                       'classes. previous Vector DB Text '\n                                       'Generation next Getting Started By '\n                                       'Harrison Chase \u00a9 Copyright 2022, Harrison '\n                                       'Chase. Last updated on Mar 15, 2023.',\n                               'url': 'https://langchain.readthedocs.io/en/latest/modules/chains.html'},\n                  'score': 0.8848499,\n                  'sparse_values': {'indices': [], 'values': []},\n                  'values': []},\n                 {'id': 'fe48438d-228a-4e0e-b41e-5cb5c6ba1482',\n                  'metadata': {'chunk': 0.0,\n                               'text': '.rst .pdf LLMs LLMs# Large Language Models '\n                                       '(LLMs) are a core component of LangChain. '\n                                       'LangChain is not a provider of LLMs, but '\n                                       'rather provides a standard interface '\n                                       'through which you can interact with a '\n                                       'variety of LLMs. The following sections of '\n                                       'documentation are provided: Getting '\n                                       'Started: An overview of all the '\n                                       'functionality the LangChain LLM class '\n                                       'provides. Key Concepts: A conceptual guide '\n                                       'going over the various concepts related to '\n                                       'LLMs. How-To Guides: A collection of '\n                                       'how-to guides. These highlight how to '\n                                       'accomplish various objectives with our LLM '\n                                       'class, as well as how to integrate with '\n                                       'various LLM providers."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: a82d4f90-acd7-4c05-96dd-2ed4287ef9d1
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a82d4f90-acd7-4c05-96dd-2ed4287ef9d1\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4da3be08-9931-44db-af1b-2942631ddd9c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"689d56ac6bf36358c07b0501457016dc6b82c0e3960699ced8c5b1066014443a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6cea0cc7-3956-4e9f-b8f5-89c3cb93859a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ad8e6690aa47ff2de0fbf024473e14a0b6c4ff7fc219184db95bb1fd67aa6d38\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 37384, \"end_char_idx\": 41997, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "Key Concepts: A conceptual guide '\n                                       'going over the various concepts related to '\n                                       'chains. How-To Guides: A collection of '\n                                       'how-to guides. These highlight how to use '\n                                       'various types of chains. Reference: API '\n                                       'reference documentation for all Chain '\n                                       'classes. previous Vector DB Text '\n                                       'Generation next Getting Started By '\n                                       'Harrison Chase \u00a9 Copyright 2022, Harrison '\n                                       'Chase. Last updated on Mar 15, 2023.',\n                               'url': 'https://langchain.readthedocs.io/en/latest/modules/chains.html'},\n                  'score': 0.8848499,\n                  'sparse_values': {'indices': [], 'values': []},\n                  'values': []},\n                 {'id': 'fe48438d-228a-4e0e-b41e-5cb5c6ba1482',\n                  'metadata': {'chunk': 0.0,\n                               'text': '.rst .pdf LLMs LLMs# Large Language Models '\n                                       '(LLMs) are a core component of LangChain. '\n                                       'LangChain is not a provider of LLMs, but '\n                                       'rather provides a standard interface '\n                                       'through which you can interact with a '\n                                       'variety of LLMs. The following sections of '\n                                       'documentation are provided: Getting '\n                                       'Started: An overview of all the '\n                                       'functionality the LangChain LLM class '\n                                       'provides. Key Concepts: A conceptual guide '\n                                       'going over the various concepts related to '\n                                       'LLMs. How-To Guides: A collection of '\n                                       'how-to guides. These highlight how to '\n                                       'accomplish various objectives with our LLM '\n                                       'class, as well as how to integrate with '\n                                       'various LLM providers. Reference: API '\n                                       'reference documentation for all LLM '\n                                       'classes. previous Example Selector next '\n                                       'Getting Started By Harrison Chase \u00a9 '\n                                       'Copyright 2022, Harrison Chase. Last '\n                                       'updated on Mar 15, 2023.',\n                               'url': 'https://langchain.readthedocs.io/en/latest/modules/llms.html'},\n                  'score': 0.8595519,\n                  'sparse_values': {'indices': [], 'values': []},\n                  'values': []},\n                 {'id': '60df5bff-5f79-46ee-9456-534d42f6a94e',\n                  'metadata': {'chunk': 0.0,\n                               'text': '.ipynb .pdf Getting Started Contents Why '\n                                       'do we need chains? Query an LLM with the '\n                                       'LLMChain Combine chains with the '\n                                       'SequentialChain Create a custom chain with '\n                                       'the Chain class Getting Started# In this '\n                                       'tutorial, we will learn about creating '\n                                       'simple chains in LangChain. We will learn '\n                                       'how to create a chain, add components to '\n                                       'it, and run it. In this tutorial, we will '\n                                       'cover: Using a simple LLM chain Creating '\n                                       'sequential chains Creating a custom chain '\n                                       'Why do we need chains?# Chains allow us to '\n                                       'combine multiple components together to '\n                                       'create a single, coherent application. For '\n                                       'example, we can create a chain that takes '\n                                       'user input, formats it with a '\n                                       'PromptTemplate, and then passes the '\n                                       'formatted response to an LLM."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 6cea0cc7-3956-4e9f-b8f5-89c3cb93859a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"6cea0cc7-3956-4e9f-b8f5-89c3cb93859a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a82d4f90-acd7-4c05-96dd-2ed4287ef9d1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"231fea718d92d007ef9c1f922b3d69feba58ca80e62fcdd951ffbc61711e4672\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"57a442f7-21db-4cc0-96c6-2de57b2a062c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"88c539957dadbad9293c32fd7cc518ffb82640856baa3c300aff401ca978432a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 39529, \"end_char_idx\": 43942, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "These highlight how to '\n                                       'accomplish various objectives with our LLM '\n                                       'class, as well as how to integrate with '\n                                       'various LLM providers. Reference: API '\n                                       'reference documentation for all LLM '\n                                       'classes. previous Example Selector next '\n                                       'Getting Started By Harrison Chase \u00a9 '\n                                       'Copyright 2022, Harrison Chase. Last '\n                                       'updated on Mar 15, 2023.',\n                               'url': 'https://langchain.readthedocs.io/en/latest/modules/llms.html'},\n                  'score': 0.8595519,\n                  'sparse_values': {'indices': [], 'values': []},\n                  'values': []},\n                 {'id': '60df5bff-5f79-46ee-9456-534d42f6a94e',\n                  'metadata': {'chunk': 0.0,\n                               'text': '.ipynb .pdf Getting Started Contents Why '\n                                       'do we need chains? Query an LLM with the '\n                                       'LLMChain Combine chains with the '\n                                       'SequentialChain Create a custom chain with '\n                                       'the Chain class Getting Started# In this '\n                                       'tutorial, we will learn about creating '\n                                       'simple chains in LangChain. We will learn '\n                                       'how to create a chain, add components to '\n                                       'it, and run it. In this tutorial, we will '\n                                       'cover: Using a simple LLM chain Creating '\n                                       'sequential chains Creating a custom chain '\n                                       'Why do we need chains?# Chains allow us to '\n                                       'combine multiple components together to '\n                                       'create a single, coherent application. For '\n                                       'example, we can create a chain that takes '\n                                       'user input, formats it with a '\n                                       'PromptTemplate, and then passes the '\n                                       'formatted response to an LLM. We can build '\n                                       'more complex chains by combining multiple '\n                                       'chains together, or by combining chains '\n                                       'with other components. Query an LLM with '\n                                       'the LLMChain# The LLMChain is a simple '\n                                       'chain that takes in a prompt template, '\n                                       'formats it with the user input and returns '\n                                       'the response from an LLM. To use the '\n                                       'LLMChain, first create a prompt template. '\n                                       'from langchain.prompts import '\n                                       'PromptTemplate from langchain.llms import '\n                                       'OpenAI llm = OpenAI(temperature=0.9) '\n                                       'prompt = PromptTemplate( '\n                                       'input_variables=[\"product\"], '\n                                       'template=\"What is a good',\n                               'url': 'https://langchain.readthedocs.io/en/latest/modules/chains/getting_started.html'},\n                  'score': 0.8462403,\n                  'sparse_values': {'indices': [], 'values': []},\n                  'values': []},\n                 {'id': '2f11beb1-3935-447e-b565-b20383dc4544',\n                  'metadata': {'chunk': 1.0,\n                               'text': 'chain first uses a LLM to construct the '\n                                       'url to hit, then makes that request with '\n                                       'the Requests wrapper, and finally runs '\n                                       'that result through the language model '\n                                       'again in order to product a natural '\n                                       'language response."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 57a442f7-21db-4cc0-96c6-2de57b2a062c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"57a442f7-21db-4cc0-96c6-2de57b2a062c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6cea0cc7-3956-4e9f-b8f5-89c3cb93859a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"769f79a351cf8d5c2f96033f5c1657be6e0aac8fd7b1698e88b15bb3ed8652ec\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c059a86e-04e2-403d-837e-c96818970663\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"25791e540d6d8b7bd3641afb0ba41d2b68f776c61826ae81d962861fe7414ce3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 41688, \"end_char_idx\": 45748, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "For '\n                                       'example, we can create a chain that takes '\n                                       'user input, formats it with a '\n                                       'PromptTemplate, and then passes the '\n                                       'formatted response to an LLM. We can build '\n                                       'more complex chains by combining multiple '\n                                       'chains together, or by combining chains '\n                                       'with other components. Query an LLM with '\n                                       'the LLMChain# The LLMChain is a simple '\n                                       'chain that takes in a prompt template, '\n                                       'formats it with the user input and returns '\n                                       'the response from an LLM. To use the '\n                                       'LLMChain, first create a prompt template. '\n                                       'from langchain.prompts import '\n                                       'PromptTemplate from langchain.llms import '\n                                       'OpenAI llm = OpenAI(temperature=0.9) '\n                                       'prompt = PromptTemplate( '\n                                       'input_variables=[\"product\"], '\n                                       'template=\"What is a good',\n                               'url': 'https://langchain.readthedocs.io/en/latest/modules/chains/getting_started.html'},\n                  'score': 0.8462403,\n                  'sparse_values': {'indices': [], 'values': []},\n                  'values': []},\n                 {'id': '2f11beb1-3935-447e-b565-b20383dc4544',\n                  'metadata': {'chunk': 1.0,\n                               'text': 'chain first uses a LLM to construct the '\n                                       'url to hit, then makes that request with '\n                                       'the Requests wrapper, and finally runs '\n                                       'that result through the language model '\n                                       'again in order to product a natural '\n                                       'language response. Example Notebook '\n                                       'LLMBash Chain Links Used: BashProcess, '\n                                       'LLMChain Notes: This chain takes user '\n                                       'input (a question), uses an LLM chain to '\n                                       'convert it to a bash command to run in the '\n                                       'terminal, and then returns that as the '\n                                       'result. Example Notebook LLMChecker Chain '\n                                       'Links Used: LLMChain Notes: This chain '\n                                       'takes user input (a question), uses an LLM '\n                                       'chain to answer that question, and then '\n                                       'uses other LLMChains to self-check that '\n                                       'answer. Example Notebook LLMRequests Chain '\n                                       'Links Used: Requests, LLMChain Notes: This '\n                                       'chain takes a URL and other inputs, uses '\n                                       'Requests to get the data at that URL, and '\n                                       'then passes that along with the other '\n                                       'inputs into an LLMChain to generate a '\n                                       'response. The example included shows how '\n                                       'to ask a question to Google - it firsts '\n                                       'constructs a Google url, then fetches the '\n                                       'data there, then passes that data + the '\n                                       'original question into an LLMChain to get '\n                                       'an answer."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: c059a86e-04e2-403d-837e-c96818970663
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c059a86e-04e2-403d-837e-c96818970663\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"57a442f7-21db-4cc0-96c6-2de57b2a062c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"b9d0f41e0621954efa9783cd581e35b427f3635e3b1c5d0ffafdaa6d4349e03e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0417ce69-9be6-459c-abcb-f5dac225691b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c683b285916e74a0c203e83f06cf97e34e052e5679ba964719c322794f0ad425\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 43943, \"end_char_idx\": 48773, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "Example Notebook '\n                                       'LLMBash Chain Links Used: BashProcess, '\n                                       'LLMChain Notes: This chain takes user '\n                                       'input (a question), uses an LLM chain to '\n                                       'convert it to a bash command to run in the '\n                                       'terminal, and then returns that as the '\n                                       'result. Example Notebook LLMChecker Chain '\n                                       'Links Used: LLMChain Notes: This chain '\n                                       'takes user input (a question), uses an LLM '\n                                       'chain to answer that question, and then '\n                                       'uses other LLMChains to self-check that '\n                                       'answer. Example Notebook LLMRequests Chain '\n                                       'Links Used: Requests, LLMChain Notes: This '\n                                       'chain takes a URL and other inputs, uses '\n                                       'Requests to get the data at that URL, and '\n                                       'then passes that along with the other '\n                                       'inputs into an LLMChain to generate a '\n                                       'response. The example included shows how '\n                                       'to ask a question to Google - it firsts '\n                                       'constructs a Google url, then fetches the '\n                                       'data there, then passes that data + the '\n                                       'original question into an LLMChain to get '\n                                       'an answer. Example Notebook Moderation '\n                                       'Chain Links Used: LLMChain, '\n                                       'ModerationChain Notes: This chain shows '\n                                       'how to use OpenAI\u2019s content',\n                               'url': 'https://langchain.readthedocs.io/en/latest/modules/chains/utility_how_to.html'},\n                  'score': 0.8451743,\n                  'sparse_values': {'indices': [], 'values': []},\n                  'values': []},\n                 {'id': 'f3ed41eb-063c-407f-bdaa-706a8c6a2091',\n                  'metadata': {'chunk': 1.0,\n                               'text': 'Prompts: This includes prompt management, '\n                                       'prompt optimization, and prompt '\n                                       'serialization. LLMs: This includes a '\n                                       'generic interface for all LLMs, and common '\n                                       'utilities for working with LLMs. Document '\n                                       'Loaders: This includes a standard '\n                                       'interface for loading documents, as well '\n                                       'as specific integrations to all types of '\n                                       'text data sources. Utils: Language models '\n                                       'are often more powerful when interacting '\n                                       'with other sources of knowledge or '\n                                       'computation. This can include Python '\n                                       'REPLs, embeddings, search engines, and '\n                                       'more. LangChain provides a large '\n                                       'collection of common utils to use in your '\n                                       'application. Chains: Chains go beyond just '\n                                       'a single LLM call, and are sequences of '\n                                       'calls (whether to an LLM or a different '\n                                       'utility). LangChain provides a standard '\n                                       'interface for chains, lots of integrations '\n                                       'with other tools, and end-to-end chains '\n                                       'for common applications. Indexes: Language '\n                                       'models are often more powerful when '\n                                       'combined with your own text data - this '\n                                       'module covers best practices for doing '\n                                       'exactly that. Agents: Agents involve an '\n                                       'LLM making decisions about which Actions '\n                                       'to take, taking that Action, seeing an '\n                                       'Observation, and repeating that until '\n                                       'done."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 0417ce69-9be6-459c-abcb-f5dac225691b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0417ce69-9be6-459c-abcb-f5dac225691b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c059a86e-04e2-403d-837e-c96818970663\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"c040997da5d97b3193ef2bed1b71b983c3090a4afb5b92d0504a30726aff7a95\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"07b59bbc-f603-4839-805e-f48e95908f38\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4f04a6d632c60fbd474f6a50912e97c0d9f6b30b5ff68a975801aeb1d3ef1b60\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 46580, \"end_char_idx\": 50800, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "LLMs: This includes a '\n                                       'generic interface for all LLMs, and common '\n                                       'utilities for working with LLMs. Document '\n                                       'Loaders: This includes a standard '\n                                       'interface for loading documents, as well '\n                                       'as specific integrations to all types of '\n                                       'text data sources. Utils: Language models '\n                                       'are often more powerful when interacting '\n                                       'with other sources of knowledge or '\n                                       'computation. This can include Python '\n                                       'REPLs, embeddings, search engines, and '\n                                       'more. LangChain provides a large '\n                                       'collection of common utils to use in your '\n                                       'application. Chains: Chains go beyond just '\n                                       'a single LLM call, and are sequences of '\n                                       'calls (whether to an LLM or a different '\n                                       'utility). LangChain provides a standard '\n                                       'interface for chains, lots of integrations '\n                                       'with other tools, and end-to-end chains '\n                                       'for common applications. Indexes: Language '\n                                       'models are often more powerful when '\n                                       'combined with your own text data - this '\n                                       'module covers best practices for doing '\n                                       'exactly that. Agents: Agents involve an '\n                                       'LLM making decisions about which Actions '\n                                       'to take, taking that Action, seeing an '\n                                       'Observation, and repeating that until '\n                                       'done. LangChain provides a standard '\n                                       'interface for agents, a selection of '\n                                       'agents to choose from, and examples of end '\n                                       'to end agents. Memory: Memory is the',\n                               'url': 'https://langchain.readthedocs.io/en/latest/'},\n                  'score': 0.84271824,\n                  'sparse_values': {'indices': [], 'values': []},\n                  'values': []}],\n     'namespace': ''}\n\n\n\nWith retrieval complete, we move on to feeding these into GPT-4 to produce answers.\n\n## Retrieval Augmented Generation\n\nGPT-4 is currently accessed via the `ChatCompletions` endpoint of OpenAI. To add the information we retrieved into the model, we need to pass it into our user prompts *alongside* our original query. We can do that like so:\n\n\n```python\n# get list of retrieved text\ncontexts = [item['metadata']['text'] for item in res['matches']]\n\naugmented_query = \"\\n\\n---\\n\\n\".join(contexts)+\"\\n\\n-----\\n\\n\"+query\n```\n\n\n```python\nprint(augmented_query)\n```\n\n    .rst .pdf Chains Chains# Using an LLM in isolation is fine for some simple applications, but many more complex ones require chaining LLMs - either with each other or with other experts. LangChain provides a standard interface for Chains, as well as some common implementations of chains for ease of use. The following sections of documentation are provided: Getting Started: A getting started guide for chains, to get you up and running quickly. Key Concepts: A conceptual guide going over the various concepts related to chains. How-To Guides: A collection of how-to guides. These highlight how to use various types of chains. Reference: API reference documentation for all Chain classes. previous Vector DB Text Generation next Getting Started By Harrison Chase \u00a9 Copyright 2022, Harrison Chase. Last updated on Mar 15, 2023.\n    \n    ---\n    \n    .rst .pdf LLMs LLMs# Large Language Models (LLMs) are a core component of LangChain."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 07b59bbc-f603-4839-805e-f48e95908f38
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"07b59bbc-f603-4839-805e-f48e95908f38\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0417ce69-9be6-459c-abcb-f5dac225691b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"7002cae8f529b7be7a6990aac707767969b0c808e0fa6e8f53057e69ca29011d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fde125f1-49f5-4a06-9d5f-d52747f001a0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"db7b162fa7f1ee7e7b3e69cec0df5b97072d857ce4fae315a496a3d6f6c3818d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 49299, \"end_char_idx\": 52589, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "With retrieval complete, we move on to feeding these into GPT-4 to produce answers.\n\n## Retrieval Augmented Generation\n\nGPT-4 is currently accessed via the `ChatCompletions` endpoint of OpenAI. To add the information we retrieved into the model, we need to pass it into our user prompts *alongside* our original query. We can do that like so:\n\n\n```python\n# get list of retrieved text\ncontexts = [item['metadata']['text'] for item in res['matches']]\n\naugmented_query = \"\\n\\n---\\n\\n\".join(contexts)+\"\\n\\n-----\\n\\n\"+query\n```\n\n\n```python\nprint(augmented_query)\n```\n\n    .rst .pdf Chains Chains# Using an LLM in isolation is fine for some simple applications, but many more complex ones require chaining LLMs - either with each other or with other experts. LangChain provides a standard interface for Chains, as well as some common implementations of chains for ease of use. The following sections of documentation are provided: Getting Started: A getting started guide for chains, to get you up and running quickly. Key Concepts: A conceptual guide going over the various concepts related to chains. How-To Guides: A collection of how-to guides. These highlight how to use various types of chains. Reference: API reference documentation for all Chain classes. previous Vector DB Text Generation next Getting Started By Harrison Chase \u00a9 Copyright 2022, Harrison Chase. Last updated on Mar 15, 2023.\n    \n    ---\n    \n    .rst .pdf LLMs LLMs# Large Language Models (LLMs) are a core component of LangChain. LangChain is not a provider of LLMs, but rather provides a standard interface through which you can interact with a variety of LLMs. The following sections of documentation are provided: Getting Started: An overview of all the functionality the LangChain LLM class provides. Key Concepts: A conceptual guide going over the various concepts related to LLMs. How-To Guides: A collection of how-to guides. These highlight how to accomplish various objectives with our LLM class, as well as how to integrate with various LLM providers. Reference: API reference documentation for all LLM classes. previous Example Selector next Getting Started By Harrison Chase \u00a9 Copyright 2022, Harrison Chase. Last updated on Mar 15, 2023.\n    \n    ---\n    \n    .ipynb .pdf Getting Started Contents Why do we need chains? Query an LLM with the LLMChain Combine chains with the SequentialChain Create a custom chain with the Chain class Getting Started# In this tutorial, we will learn about creating simple chains in LangChain. We will learn how to create a chain, add components to it, and run it. In this tutorial, we will cover: Using a simple LLM chain Creating sequential chains Creating a custom chain Why do we need chains?# Chains allow us to combine multiple components together to create a single, coherent application. For example, we can create a chain that takes user input, formats it with a PromptTemplate, and then passes the formatted response to an LLM. We can build more complex chains by combining multiple chains together, or by combining chains with other components. Query an LLM with the LLMChain# The LLMChain is a simple chain that takes in a prompt template, formats it with the user input and returns the response from an LLM. To use the LLMChain, first create a prompt template."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: fde125f1-49f5-4a06-9d5f-d52747f001a0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"fde125f1-49f5-4a06-9d5f-d52747f001a0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"07b59bbc-f603-4839-805e-f48e95908f38\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"70654a9ae6abf0b18d85d924139a528c87f9af3a68206c8e0150633fb8aad416\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0ae76d7d-449f-48e2-a4f9-8a131963771f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c9f6170af0eaee77d8c905637b70dc9626460b3a075d22e1f706e6297785b007\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 50801, \"end_char_idx\": 54228, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "LangChain is not a provider of LLMs, but rather provides a standard interface through which you can interact with a variety of LLMs. The following sections of documentation are provided: Getting Started: An overview of all the functionality the LangChain LLM class provides. Key Concepts: A conceptual guide going over the various concepts related to LLMs. How-To Guides: A collection of how-to guides. These highlight how to accomplish various objectives with our LLM class, as well as how to integrate with various LLM providers. Reference: API reference documentation for all LLM classes. previous Example Selector next Getting Started By Harrison Chase \u00a9 Copyright 2022, Harrison Chase. Last updated on Mar 15, 2023.\n    \n    ---\n    \n    .ipynb .pdf Getting Started Contents Why do we need chains? Query an LLM with the LLMChain Combine chains with the SequentialChain Create a custom chain with the Chain class Getting Started# In this tutorial, we will learn about creating simple chains in LangChain. We will learn how to create a chain, add components to it, and run it. In this tutorial, we will cover: Using a simple LLM chain Creating sequential chains Creating a custom chain Why do we need chains?# Chains allow us to combine multiple components together to create a single, coherent application. For example, we can create a chain that takes user input, formats it with a PromptTemplate, and then passes the formatted response to an LLM. We can build more complex chains by combining multiple chains together, or by combining chains with other components. Query an LLM with the LLMChain# The LLMChain is a simple chain that takes in a prompt template, formats it with the user input and returns the response from an LLM. To use the LLMChain, first create a prompt template. from langchain.prompts import PromptTemplate from langchain.llms import OpenAI llm = OpenAI(temperature=0.9) prompt = PromptTemplate( input_variables=[\"product\"], template=\"What is a good\n    \n    ---\n    \n    chain first uses a LLM to construct the url to hit, then makes that request with the Requests wrapper, and finally runs that result through the language model again in order to product a natural language response. Example Notebook LLMBash Chain Links Used: BashProcess, LLMChain Notes: This chain takes user input (a question), uses an LLM chain to convert it to a bash command to run in the terminal, and then returns that as the result. Example Notebook LLMChecker Chain Links Used: LLMChain Notes: This chain takes user input (a question), uses an LLM chain to answer that question, and then uses other LLMChains to self-check that answer. Example Notebook LLMRequests Chain Links Used: Requests, LLMChain Notes: This chain takes a URL and other inputs, uses Requests to get the data at that URL, and then passes that along with the other inputs into an LLMChain to generate a response. The example included shows how to ask a question to Google - it firsts constructs a Google url, then fetches the data there, then passes that data + the original question into an LLMChain to get an answer. Example Notebook Moderation Chain Links Used: LLMChain, ModerationChain Notes: This chain shows how to use OpenAI\u2019s content\n    \n    ---\n    \n    Prompts: This includes prompt management, prompt optimization, and prompt serialization. LLMs: This includes a generic interface for all LLMs, and common utilities for working with LLMs."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 0ae76d7d-449f-48e2-a4f9-8a131963771f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0ae76d7d-449f-48e2-a4f9-8a131963771f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fde125f1-49f5-4a06-9d5f-d52747f001a0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"7be14d3a937caed52cba6198905d299ebc14fe0ebd31255a016d88f94f6386ac\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fc0ec6e0-3e91-4880-833f-17607a004a45\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0ea5a0a0365b421b510fde794bf46d94ab27e5befdb44b3c48202f7e84e917fa\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 52537, \"end_char_idx\": 55995, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "To use the LLMChain, first create a prompt template. from langchain.prompts import PromptTemplate from langchain.llms import OpenAI llm = OpenAI(temperature=0.9) prompt = PromptTemplate( input_variables=[\"product\"], template=\"What is a good\n    \n    ---\n    \n    chain first uses a LLM to construct the url to hit, then makes that request with the Requests wrapper, and finally runs that result through the language model again in order to product a natural language response. Example Notebook LLMBash Chain Links Used: BashProcess, LLMChain Notes: This chain takes user input (a question), uses an LLM chain to convert it to a bash command to run in the terminal, and then returns that as the result. Example Notebook LLMChecker Chain Links Used: LLMChain Notes: This chain takes user input (a question), uses an LLM chain to answer that question, and then uses other LLMChains to self-check that answer. Example Notebook LLMRequests Chain Links Used: Requests, LLMChain Notes: This chain takes a URL and other inputs, uses Requests to get the data at that URL, and then passes that along with the other inputs into an LLMChain to generate a response. The example included shows how to ask a question to Google - it firsts constructs a Google url, then fetches the data there, then passes that data + the original question into an LLMChain to get an answer. Example Notebook Moderation Chain Links Used: LLMChain, ModerationChain Notes: This chain shows how to use OpenAI\u2019s content\n    \n    ---\n    \n    Prompts: This includes prompt management, prompt optimization, and prompt serialization. LLMs: This includes a generic interface for all LLMs, and common utilities for working with LLMs. Document Loaders: This includes a standard interface for loading documents, as well as specific integrations to all types of text data sources. Utils: Language models are often more powerful when interacting with other sources of knowledge or computation. This can include Python REPLs, embeddings, search engines, and more. LangChain provides a large collection of common utils to use in your application. Chains: Chains go beyond just a single LLM call, and are sequences of calls (whether to an LLM or a different utility). LangChain provides a standard interface for chains, lots of integrations with other tools, and end-to-end chains for common applications. Indexes: Language models are often more powerful when combined with your own text data - this module covers best practices for doing exactly that. Agents: Agents involve an LLM making decisions about which Actions to take, taking that Action, seeing an Observation, and repeating that until done. LangChain provides a standard interface for agents, a selection of agents to choose from, and examples of end to end agents. Memory: Memory is the\n    \n    -----\n    \n    how do I use the LLMChain in LangChain?\n\n\nNow we ask the question:\n\n\n```python\n# system message to 'prime' the model\nprimer = f\"\"\"You are Q&A bot. A highly intelligent system that answers\nuser questions based on the information provided by the user above\neach question. If the information can not be found in the information\nprovided by the user you truthfully say \"I don't know\".\n\"\"\"\n\nres = openai.ChatCompletion.create(\n    model=\"gpt-4\",\n    messages=[\n        {\"role\": \"system\", \"content\": primer},\n        {\"role\": \"user\", \"content\": augmented_query}\n    ]\n)\n```\n\nTo display this response nicely, we will display it in markdown."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: fc0ec6e0-3e91-4880-833f-17607a004a45
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"fc0ec6e0-3e91-4880-833f-17607a004a45\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0ae76d7d-449f-48e2-a4f9-8a131963771f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"cf49c30205ec3da8523e7893fbf5f3cbdeacd2192daadab2e043a49c6bb2950b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3d0755b8-0a05-441e-bad6-2af13963f35b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"30f9cd2e071e6db21cb6da1f34cbc02c2a9d36734194c203353d0e07de610424\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 54229, \"end_char_idx\": 57341, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "Document Loaders: This includes a standard interface for loading documents, as well as specific integrations to all types of text data sources. Utils: Language models are often more powerful when interacting with other sources of knowledge or computation. This can include Python REPLs, embeddings, search engines, and more. LangChain provides a large collection of common utils to use in your application. Chains: Chains go beyond just a single LLM call, and are sequences of calls (whether to an LLM or a different utility). LangChain provides a standard interface for chains, lots of integrations with other tools, and end-to-end chains for common applications. Indexes: Language models are often more powerful when combined with your own text data - this module covers best practices for doing exactly that. Agents: Agents involve an LLM making decisions about which Actions to take, taking that Action, seeing an Observation, and repeating that until done. LangChain provides a standard interface for agents, a selection of agents to choose from, and examples of end to end agents. Memory: Memory is the\n    \n    -----\n    \n    how do I use the LLMChain in LangChain?\n\n\nNow we ask the question:\n\n\n```python\n# system message to 'prime' the model\nprimer = f\"\"\"You are Q&A bot. A highly intelligent system that answers\nuser questions based on the information provided by the user above\neach question. If the information can not be found in the information\nprovided by the user you truthfully say \"I don't know\".\n\"\"\"\n\nres = openai.ChatCompletion.create(\n    model=\"gpt-4\",\n    messages=[\n        {\"role\": \"system\", \"content\": primer},\n        {\"role\": \"user\", \"content\": augmented_query}\n    ]\n)\n```\n\nTo display this response nicely, we will display it in markdown.\n\n\n```python\nfrom IPython.display import Markdown\n\ndisplay(Markdown(res['choices'][0]['message']['content']))\n```\n\n\nTo use the LLMChain in LangChain, follow these steps:\n\n1. Import the necessary classes:\n```python\nfrom langchain.prompts import PromptTemplate\nfrom langchain.llms import OpenAI\nfrom langchain.chains import LLMChain\n```\n\n2. Create an instance of the LLM and set the configuration options:\n```python\nllm = OpenAI(temperature=0.9)\n```\n\n3. Create a PromptTemplate instance with the input variables and the template:\n```python\nprompt = PromptTemplate(\n    input_variables=[\"product\"],\n    template=\"What is a good product for {product}?\",\n)\n```\n\n4. Create an LLMChain instance by passing the LLM and PromptTemplate instances:\n```python\nllm_chain = LLMChain(llm=llm, prompt_template=prompt)\n```\n\n5. Run the LLMChain with user input:\n```python\nresponse = llm_chain.run({\"product\": \"software development\"})\n```\n\n6. Access the generated response:\n```python\ngenerated_text = response[\"generated_text\"]\n```\n\nIn this example, the LLMChain is used to generate a response by passing through the user input and formatting it using the prompt template. The response is then obtained from the LLM instance (in this case, OpenAI), and the generated text can be accessed from the response dictionary.\n\n\nLet's compare this to a non-augmented query..."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 3d0755b8-0a05-441e-bad6-2af13963f35b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"3d0755b8-0a05-441e-bad6-2af13963f35b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"52f430b5-f26d-45fb-b88f-3e66a3e07543\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fc0ec6e0-3e91-4880-833f-17607a004a45\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"29c29cfea2e8a3ffcdca4197d075a07beeeb856e668c4e849d68721d3006e726\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 55998, \"end_char_idx\": 58764, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "ref_doc_id": "52f430b5-f26d-45fb-b88f-3e66a3e07543", "chroma:document": "```python\nfrom IPython.display import Markdown\n\ndisplay(Markdown(res['choices'][0]['message']['content']))\n```\n\n\nTo use the LLMChain in LangChain, follow these steps:\n\n1. Import the necessary classes:\n```python\nfrom langchain.prompts import PromptTemplate\nfrom langchain.llms import OpenAI\nfrom langchain.chains import LLMChain\n```\n\n2. Create an instance of the LLM and set the configuration options:\n```python\nllm = OpenAI(temperature=0.9)\n```\n\n3. Create a PromptTemplate instance with the input variables and the template:\n```python\nprompt = PromptTemplate(\n    input_variables=[\"product\"],\n    template=\"What is a good product for {product}?\",\n)\n```\n\n4. Create an LLMChain instance by passing the LLM and PromptTemplate instances:\n```python\nllm_chain = LLMChain(llm=llm, prompt_template=prompt)\n```\n\n5. Run the LLMChain with user input:\n```python\nresponse = llm_chain.run({\"product\": \"software development\"})\n```\n\n6. Access the generated response:\n```python\ngenerated_text = response[\"generated_text\"]\n```\n\nIn this example, the LLMChain is used to generate a response by passing through the user input and formatting it using the prompt template. The response is then obtained from the LLM instance (in this case, OpenAI), and the generated text can be accessed from the response dictionary.\n\n\nLet's compare this to a non-augmented query...\n\n\n```python\nres = openai.ChatCompletion.create(\n    model=\"gpt-4\",\n    messages=[\n        {\"role\": \"system\", \"content\": primer},\n        {\"role\": \"user\", \"content\": query}\n    ]\n)\ndisplay(Markdown(res['choices'][0]['message']['content']))\n```\n\n\nI don't know.\n\n\nIf we drop the `\"I don't know\"` part of the `primer`?\n\n\n```python\nres = openai.ChatCompletion.create(\n    model=\"gpt-4\",\n    messages=[\n        {\"role\": \"system\", \"content\": \"You are Q&A bot. A highly intelligent system that answers user questions\"},\n        {\"role\": \"user\", \"content\": query}\n    ]\n)\ndisplay(Markdown(res['choices'][0]['message']['content']))\n```\n\n\nLangChain hasn't provided any public documentation on LLMChain, nor is there a known technology called LLMChain in their library. To better assist you, please provide more information or context about LLMChain and LangChain.\n\nMeanwhile, if you are referring to LangChain, a blockchain-based decentralized AI language model, you can start by visiting their official website (if they have one), exploring their available resources, such as documentation and tutorials, and following any instructions on setting up their technology.\n\nIf you are looking for help with a specific language chain or model in natural language processing, consider rephrasing your question to provide more accurate information or visit relevant resources like GPT-3 or other NLP-related documentation.\n\n\n\n```python\n\n```"}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: bafb82fb-f6f0-4429-9e46-1f31deb53518
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb", "title": "Vector similarity search using Neon Postgres", "tokens": 2947, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"bafb82fb-f6f0-4429-9e46-1f31deb53518\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e8d39b19-a02d-45ef-8d7f-980a3bd2dee3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3be94c682f3a332b91a21c63fd7e215ba9994445c22203fccf81f907a42dce64\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c938a049-8703-44e0-9b96-8ef47fc26912\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"59274334f782b21279e197a473a8d338d5e490e566c81b3a04c4dfd4d69fffd2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3092, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "doc_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "ref_doc_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "chroma:document": "# Vector similarity search using Neon Postgres\n\nThis notebook guides you through using [Neon Serverless Postgres](https://neon.tech/) as a vector database for OpenAI embeddings. It demonstrates how to:\n\n1. Use embeddings created by OpenAI API.\n2. Store embeddings in a Neon Serverless Postgres database.\n3. Convert a raw text query to an embedding with OpenAI API.\n4. Use Neon with the `pgvector` extension to perform vector similarity search.\n\n## Prerequisites\n\nBefore you begin, ensure that you have the following:\n\n1. A Neon Postgres database. You can create an account and set up a project with a ready-to-use `neondb` database in a few simple steps. For instructions, see [Sign up](https://neon.tech/docs/get-started-with-neon/signing-up) and [Create your first project](https://neon.tech/docs/get-started-with-neon/setting-up-a-project).\n2. A connection string for your Neon database. You can copy it from the **Connection Details** widget on the Neon **Dashboard**. See [Connect from any application](https://neon.tech/docs/connect/connect-from-any-app).\n3. The `pgvector` extension. Install the extension in Neon by running `CREATE EXTENSION vector;`. For instructions, see [Enable the pgvector extension](https://neon.tech/docs/extensions/pgvector#enable-the-pgvector-extension). \n4. Your [OpenAI API key](https://platform.openai.com/account/api-keys).\n5. Python and `pip`.\n\n### Install required modules\n\nThis notebook requires the `openai`, `psycopg2`, `pandas`, `wget`, and `python-dotenv` packages. You can install them with `pip`:\n\n\n\n```python\n! pip install openai psycopg2 pandas wget python-dotenv\n```\n\n### Prepare your OpenAI API key\n\nAn OpenAI API key is required to generate vectors for documents and queries.\n\nIf you do not have an OpenAI API key, obtain one from https://platform.openai.com/account/api-keys.\n\nAdd the OpenAI API key as an operating system environment variable or provide it for the session when prompted. If you define an environment variable, name the variable `OPENAI_API_KEY`.\n\nFor information about configuring your OpenAI API key as an environment variable, refer to [Best Practices for API Key Safety](https://help.openai.com/en/articles/5112595-best-practices-for-api-key-safety).\n\n### Test your OpenAPI key\n\n\n```python\n# Test to ensure that your OpenAI API key is defined as an environment variable or provide it when prompted\n# If you run this notebook locally, you may have to reload the terminal and the notebook to make the environment available\n\nimport os\nfrom getpass import getpass\n\n# Check if OPENAI_API_KEY is set as an environment variable\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    print(\"Your OPENAI_API_KEY is ready\")\nelse:\n    # If not, prompt for it\n    api_key = getpass(\"Enter your OPENAI_API_KEY: \")\n    if api_key:\n        print(\"Your OPENAI_API_KEY is now available for this session\")\n        # Optionally, you can set it as an environment variable for the current session\n        os.environ[\"OPENAI_API_KEY\"] = api_key\n    else:\n        print(\"You did not enter your OPENAI_API_KEY\")\n```\n\n    Your OPENAI_API_KEY is ready"}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: c938a049-8703-44e0-9b96-8ef47fc26912
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb", "title": "Vector similarity search using Neon Postgres", "tokens": 2947, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c938a049-8703-44e0-9b96-8ef47fc26912\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e8d39b19-a02d-45ef-8d7f-980a3bd2dee3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3be94c682f3a332b91a21c63fd7e215ba9994445c22203fccf81f907a42dce64\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bafb82fb-f6f0-4429-9e46-1f31deb53518\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8f07d5d2dda35c1e7c51a0317d4d6cef20970e9736d96340bf319529be4795dc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"56a973fe-32fb-43af-a9e3-8caa41d89b94\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b4ba84e4484f66b4b6be27e57e0f02709791c8205cc53d90741d0e567e3a54de\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1547, \"end_char_idx\": 4870, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "doc_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "ref_doc_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "chroma:document": "```python\n! pip install openai psycopg2 pandas wget python-dotenv\n```\n\n### Prepare your OpenAI API key\n\nAn OpenAI API key is required to generate vectors for documents and queries.\n\nIf you do not have an OpenAI API key, obtain one from https://platform.openai.com/account/api-keys.\n\nAdd the OpenAI API key as an operating system environment variable or provide it for the session when prompted. If you define an environment variable, name the variable `OPENAI_API_KEY`.\n\nFor information about configuring your OpenAI API key as an environment variable, refer to [Best Practices for API Key Safety](https://help.openai.com/en/articles/5112595-best-practices-for-api-key-safety).\n\n### Test your OpenAPI key\n\n\n```python\n# Test to ensure that your OpenAI API key is defined as an environment variable or provide it when prompted\n# If you run this notebook locally, you may have to reload the terminal and the notebook to make the environment available\n\nimport os\nfrom getpass import getpass\n\n# Check if OPENAI_API_KEY is set as an environment variable\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    print(\"Your OPENAI_API_KEY is ready\")\nelse:\n    # If not, prompt for it\n    api_key = getpass(\"Enter your OPENAI_API_KEY: \")\n    if api_key:\n        print(\"Your OPENAI_API_KEY is now available for this session\")\n        # Optionally, you can set it as an environment variable for the current session\n        os.environ[\"OPENAI_API_KEY\"] = api_key\n    else:\n        print(\"You did not enter your OPENAI_API_KEY\")\n```\n\n    Your OPENAI_API_KEY is ready\n\n\n## Connect to your Neon database\n\nProvide your Neon database connection string below or define it in an `.env` file using a `DATABASE_URL` variable. For information about obtaining a Neon connection string, see [Connect from any application](https://neon.tech/docs/connect/connect-from-any-app).\n\n\n```python\nimport os\nimport psycopg2\nfrom dotenv import load_dotenv\n\n# Load environment variables from .env file\nload_dotenv()\n\n# The connection string can be provided directly here.\n# Replace the next line with Your Neon connection string.\nconnection_string = \"postgres://<user>:<password>@<hostname>/<dbname>\"\n\n# If connection_string is not directly provided above, \n# then check if DATABASE_URL is set in the environment or .env.\nif not connection_string:\n    connection_string = os.environ.get(\"DATABASE_URL\")\n\n    # If neither method provides a connection string, raise an error.\n    if not connection_string:\n        raise ValueError(\"Please provide a valid connection string either in the code or in the .env file as DATABASE_URL.\")\n\n# Connect using the connection string\nconnection = psycopg2.connect(connection_string)\n\n# Create a new cursor object\ncursor = connection.cursor()\n```\n\nTest the connection to your database:\n\n\n```python\n# Execute this query to test the database connection\ncursor.execute(\"SELECT 1;\")\nresult = cursor.fetchone()\n\n# Check the query result\nif result == (1,):\n    print(\"Your database connection was successful!\")\nelse:\n    print(\"Your connection failed.\")\n```\n\n    Your database connection was successful!\n\n\nThis guide uses pre-computed Wikipedia article embeddings available in the OpenAI Cookbook `examples` directory so that you do not have to compute embeddings with your own OpenAI credits. \n\nImport the pre-computed embeddings zip file:"}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 56a973fe-32fb-43af-a9e3-8caa41d89b94
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb", "title": "Vector similarity search using Neon Postgres", "tokens": 2947, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"56a973fe-32fb-43af-a9e3-8caa41d89b94\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e8d39b19-a02d-45ef-8d7f-980a3bd2dee3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3be94c682f3a332b91a21c63fd7e215ba9994445c22203fccf81f907a42dce64\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c938a049-8703-44e0-9b96-8ef47fc26912\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"d5166e3ab929d2afb8755b10db3fe304982b2cfa139fd3285572cae820da2b0d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a71b2dfb-ff1d-43e7-a274-4f8850c289ad\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1a4c4196141553d253293ffa37a627777c517529d7ab413c91fc7988bba0e329\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3095, \"end_char_idx\": 6311, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "doc_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "ref_doc_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "chroma:document": "## Connect to your Neon database\n\nProvide your Neon database connection string below or define it in an `.env` file using a `DATABASE_URL` variable. For information about obtaining a Neon connection string, see [Connect from any application](https://neon.tech/docs/connect/connect-from-any-app).\n\n\n```python\nimport os\nimport psycopg2\nfrom dotenv import load_dotenv\n\n# Load environment variables from .env file\nload_dotenv()\n\n# The connection string can be provided directly here.\n# Replace the next line with Your Neon connection string.\nconnection_string = \"postgres://<user>:<password>@<hostname>/<dbname>\"\n\n# If connection_string is not directly provided above, \n# then check if DATABASE_URL is set in the environment or .env.\nif not connection_string:\n    connection_string = os.environ.get(\"DATABASE_URL\")\n\n    # If neither method provides a connection string, raise an error.\n    if not connection_string:\n        raise ValueError(\"Please provide a valid connection string either in the code or in the .env file as DATABASE_URL.\")\n\n# Connect using the connection string\nconnection = psycopg2.connect(connection_string)\n\n# Create a new cursor object\ncursor = connection.cursor()\n```\n\nTest the connection to your database:\n\n\n```python\n# Execute this query to test the database connection\ncursor.execute(\"SELECT 1;\")\nresult = cursor.fetchone()\n\n# Check the query result\nif result == (1,):\n    print(\"Your database connection was successful!\")\nelse:\n    print(\"Your connection failed.\")\n```\n\n    Your database connection was successful!\n\n\nThis guide uses pre-computed Wikipedia article embeddings available in the OpenAI Cookbook `examples` directory so that you do not have to compute embeddings with your own OpenAI credits. \n\nImport the pre-computed embeddings zip file:\n\n\n```python\nimport wget\n\nembeddings_url = \"https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip\"\n\n# The file is ~700 MB. Importing it will take several minutes.\nwget.download(embeddings_url)\n```\n\n\n\n\n    'vector_database_wikipedia_articles_embedded.zip'\n\n\n\nExtract the downloaded zip file:\n\n\n```python\nimport zipfile\nimport os\nimport re\nimport tempfile\n\ncurrent_directory = os.getcwd()\nzip_file_path = os.path.join(current_directory, \"vector_database_wikipedia_articles_embedded.zip\")\noutput_directory = os.path.join(current_directory, \"../../data\")\n\nwith zipfile.ZipFile(zip_file_path, \"r\") as zip_ref:\n    zip_ref.extractall(output_directory)\n\n\n# Check to see if the csv file was extracted\nfile_name = \"vector_database_wikipedia_articles_embedded.csv\"\ndata_directory = os.path.join(current_directory, \"../../data\")\nfile_path = os.path.join(data_directory, file_name)\n\n\nif os.path.exists(file_path):\n    print(f\"The csv file {file_name} exists in the data directory.\")\nelse:\n    print(f\"The csv file {file_name} does not exist in the data directory.\")\n```\n\n    The file vector_database_wikipedia_articles_embedded.csv exists in the data directory.\n\n\n## Create a table and add indexes for your vector embeddings\n\nThe vector table created in your database is called **articles**. Each object has **title** and **content** vectors. \n\nAn index is defined on both the **title** and **content** vector columns."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: a71b2dfb-ff1d-43e7-a274-4f8850c289ad
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb", "title": "Vector similarity search using Neon Postgres", "tokens": 2947, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a71b2dfb-ff1d-43e7-a274-4f8850c289ad\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e8d39b19-a02d-45ef-8d7f-980a3bd2dee3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3be94c682f3a332b91a21c63fd7e215ba9994445c22203fccf81f907a42dce64\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"56a973fe-32fb-43af-a9e3-8caa41d89b94\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"2ca7f01aa25914a2be44cd7912e7dd5d80e1720b8858e0c1da90e049d930224d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4df4a02c-eea6-4db8-989f-36f10d66a224\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"68f394f4291179c0aadde84501df7f2ae2a8d46b2a05de1e6c5049847c26ad85\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4636, \"end_char_idx\": 8065, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "doc_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "ref_doc_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "chroma:document": "This guide uses pre-computed Wikipedia article embeddings available in the OpenAI Cookbook `examples` directory so that you do not have to compute embeddings with your own OpenAI credits. \n\nImport the pre-computed embeddings zip file:\n\n\n```python\nimport wget\n\nembeddings_url = \"https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip\"\n\n# The file is ~700 MB. Importing it will take several minutes.\nwget.download(embeddings_url)\n```\n\n\n\n\n    'vector_database_wikipedia_articles_embedded.zip'\n\n\n\nExtract the downloaded zip file:\n\n\n```python\nimport zipfile\nimport os\nimport re\nimport tempfile\n\ncurrent_directory = os.getcwd()\nzip_file_path = os.path.join(current_directory, \"vector_database_wikipedia_articles_embedded.zip\")\noutput_directory = os.path.join(current_directory, \"../../data\")\n\nwith zipfile.ZipFile(zip_file_path, \"r\") as zip_ref:\n    zip_ref.extractall(output_directory)\n\n\n# Check to see if the csv file was extracted\nfile_name = \"vector_database_wikipedia_articles_embedded.csv\"\ndata_directory = os.path.join(current_directory, \"../../data\")\nfile_path = os.path.join(data_directory, file_name)\n\n\nif os.path.exists(file_path):\n    print(f\"The csv file {file_name} exists in the data directory.\")\nelse:\n    print(f\"The csv file {file_name} does not exist in the data directory.\")\n```\n\n    The file vector_database_wikipedia_articles_embedded.csv exists in the data directory.\n\n\n## Create a table and add indexes for your vector embeddings\n\nThe vector table created in your database is called **articles**. Each object has **title** and **content** vectors. \n\nAn index is defined on both the **title** and **content** vector columns.\n\n\n```python\ncreate_table_sql = '''\nCREATE TABLE IF NOT EXISTS public.articles (\n    id INTEGER NOT NULL,\n    url TEXT,\n    title TEXT,\n    content TEXT,\n    title_vector vector(1536),\n    content_vector vector(1536),\n    vector_id INTEGER\n);\n\nALTER TABLE public.articles ADD PRIMARY KEY (id);\n'''\n\n# SQL statement for creating indexes\ncreate_indexes_sql = '''\nCREATE INDEX ON public.articles USING ivfflat (content_vector) WITH (lists = 1000);\n\nCREATE INDEX ON public.articles USING ivfflat (title_vector) WITH (lists = 1000);\n'''\n\n# Execute the SQL statements\ncursor.execute(create_table_sql)\ncursor.execute(create_indexes_sql)\n\n# Commit the changes\nconnection.commit()\n```\n\n## Load the data\n\nLoad the pre-computed vector data into your `articles` table from the `.csv` file. There are 25000 records, so expect the operation to take several minutes.\n\n\n```python\nimport io\n\n# Path to your local CSV file\ncsv_file_path = '../../data/vector_database_wikipedia_articles_embedded.csv'\n\n# Define a generator function to process the csv file\ndef process_file(file_path):\n    with open(file_path, 'r', encoding='utf-8') as file:\n        for line in file:\n            yield line\n\n# Create a StringIO object to store the modified lines\nmodified_lines = io.StringIO(''.join(list(process_file(csv_file_path))))\n\n# Create the COPY command for copy_expert\ncopy_command = '''\nCOPY public.articles (id, url, title, content, title_vector, content_vector, vector_id)\nFROM STDIN WITH (FORMAT CSV, HEADER true, DELIMITER ',');\n'''\n\n# Execute the COPY command using copy_expert\ncursor.copy_expert(copy_command, modified_lines)\n\n# Commit the changes\nconnection.commit()\n```\n\nCheck the number of records to ensure the data has been been loaded. There should be 25000 records."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 4df4a02c-eea6-4db8-989f-36f10d66a224
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb", "title": "Vector similarity search using Neon Postgres", "tokens": 2947, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"4df4a02c-eea6-4db8-989f-36f10d66a224\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e8d39b19-a02d-45ef-8d7f-980a3bd2dee3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3be94c682f3a332b91a21c63fd7e215ba9994445c22203fccf81f907a42dce64\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a71b2dfb-ff1d-43e7-a274-4f8850c289ad\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"30ecde2b292e0d985fc46b74791fd7494e0734a0ba015db2fcd454248cd2e56e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"97478478-fd46-4d38-a203-ec272bb73877\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8e9be04a9b45ed2ad19b08b614cb789a4979286c28fe9e98d54ced49a84e3c85\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6314, \"end_char_idx\": 8914, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "doc_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "ref_doc_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "chroma:document": "```python\ncreate_table_sql = '''\nCREATE TABLE IF NOT EXISTS public.articles (\n    id INTEGER NOT NULL,\n    url TEXT,\n    title TEXT,\n    content TEXT,\n    title_vector vector(1536),\n    content_vector vector(1536),\n    vector_id INTEGER\n);\n\nALTER TABLE public.articles ADD PRIMARY KEY (id);\n'''\n\n# SQL statement for creating indexes\ncreate_indexes_sql = '''\nCREATE INDEX ON public.articles USING ivfflat (content_vector) WITH (lists = 1000);\n\nCREATE INDEX ON public.articles USING ivfflat (title_vector) WITH (lists = 1000);\n'''\n\n# Execute the SQL statements\ncursor.execute(create_table_sql)\ncursor.execute(create_indexes_sql)\n\n# Commit the changes\nconnection.commit()\n```\n\n## Load the data\n\nLoad the pre-computed vector data into your `articles` table from the `.csv` file. There are 25000 records, so expect the operation to take several minutes.\n\n\n```python\nimport io\n\n# Path to your local CSV file\ncsv_file_path = '../../data/vector_database_wikipedia_articles_embedded.csv'\n\n# Define a generator function to process the csv file\ndef process_file(file_path):\n    with open(file_path, 'r', encoding='utf-8') as file:\n        for line in file:\n            yield line\n\n# Create a StringIO object to store the modified lines\nmodified_lines = io.StringIO(''.join(list(process_file(csv_file_path))))\n\n# Create the COPY command for copy_expert\ncopy_command = '''\nCOPY public.articles (id, url, title, content, title_vector, content_vector, vector_id)\nFROM STDIN WITH (FORMAT CSV, HEADER true, DELIMITER ',');\n'''\n\n# Execute the COPY command using copy_expert\ncursor.copy_expert(copy_command, modified_lines)\n\n# Commit the changes\nconnection.commit()\n```\n\nCheck the number of records to ensure the data has been been loaded. There should be 25000 records.\n\n\n```python\n# Check the size of the data\ncount_sql = \"\"\"select count(*) from public.articles;\"\"\"\ncursor.execute(count_sql)\nresult = cursor.fetchone()\nprint(f\"Count:{result[0]}\")\n```\n\n    Count:25000\n\n\n## Search your data\n\nAfter the data is stored in your Neon database, you can query the data for nearest neighbors. \n\nStart by defining the `query_neon` function, which is executed when you run the vector similarity search. The function creates an embedding based on the user's query, prepares the SQL query, and runs the SQL query with the embedding. The pre-computed embeddings that you loaded into your database were created with `text-embedding-3-small` OpenAI model, so you must use the same model to create an embedding for the similarity search.\n\nA `vector_name` parameter is provided that allows you to search based on \"title\" or \"content\"."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 97478478-fd46-4d38-a203-ec272bb73877
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb", "title": "Vector similarity search using Neon Postgres", "tokens": 2947, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"97478478-fd46-4d38-a203-ec272bb73877\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e8d39b19-a02d-45ef-8d7f-980a3bd2dee3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3be94c682f3a332b91a21c63fd7e215ba9994445c22203fccf81f907a42dce64\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4df4a02c-eea6-4db8-989f-36f10d66a224\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7ce91047831847e6a1c0b763c64e890bb84a11b6de8718e482318effced8535c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6c1c6de7-2a0e-4bdc-b019-d32a17ed5026\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"39662add4fd6dd4ac0f012934d2367131f42a71ae125074d41f1561915995d47\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7165, \"end_char_idx\": 9777, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "doc_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "ref_doc_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "chroma:document": "```python\nimport io\n\n# Path to your local CSV file\ncsv_file_path = '../../data/vector_database_wikipedia_articles_embedded.csv'\n\n# Define a generator function to process the csv file\ndef process_file(file_path):\n    with open(file_path, 'r', encoding='utf-8') as file:\n        for line in file:\n            yield line\n\n# Create a StringIO object to store the modified lines\nmodified_lines = io.StringIO(''.join(list(process_file(csv_file_path))))\n\n# Create the COPY command for copy_expert\ncopy_command = '''\nCOPY public.articles (id, url, title, content, title_vector, content_vector, vector_id)\nFROM STDIN WITH (FORMAT CSV, HEADER true, DELIMITER ',');\n'''\n\n# Execute the COPY command using copy_expert\ncursor.copy_expert(copy_command, modified_lines)\n\n# Commit the changes\nconnection.commit()\n```\n\nCheck the number of records to ensure the data has been been loaded. There should be 25000 records.\n\n\n```python\n# Check the size of the data\ncount_sql = \"\"\"select count(*) from public.articles;\"\"\"\ncursor.execute(count_sql)\nresult = cursor.fetchone()\nprint(f\"Count:{result[0]}\")\n```\n\n    Count:25000\n\n\n## Search your data\n\nAfter the data is stored in your Neon database, you can query the data for nearest neighbors. \n\nStart by defining the `query_neon` function, which is executed when you run the vector similarity search. The function creates an embedding based on the user's query, prepares the SQL query, and runs the SQL query with the embedding. The pre-computed embeddings that you loaded into your database were created with `text-embedding-3-small` OpenAI model, so you must use the same model to create an embedding for the similarity search.\n\nA `vector_name` parameter is provided that allows you to search based on \"title\" or \"content\".\n\n\n```python\ndef query_neon(query, collection_name, vector_name=\"title_vector\", top_k=20):\n\n    # Create an embedding vector from the user query\n    embedded_query = openai.Embedding.create(\n        input=query,\n        model=\"text-embedding-3-small\",\n    )[\"data\"][0][\"embedding\"]\n\n    # Convert the embedded_query to PostgreSQL compatible format\n    embedded_query_pg = \"[\" + \",\".join(map(str, embedded_query)) + \"]\"\n\n    # Create the SQL query\n    query_sql = f\"\"\"\n    SELECT id, url, title, l2_distance({vector_name},'{embedded_query_pg}'::VECTOR(1536)) AS similarity\n    FROM {collection_name}\n    ORDER BY {vector_name} <-> '{embedded_query_pg}'::VECTOR(1536)\n    LIMIT {top_k};\n    \"\"\"\n    # Execute the query\n    cursor.execute(query_sql)\n    results = cursor.fetchall()\n\n    return results\n```\n\nRun a similarity search based on `title_vector` embeddings:"}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 6c1c6de7-2a0e-4bdc-b019-d32a17ed5026
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb", "title": "Vector similarity search using Neon Postgres", "tokens": 2947, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"6c1c6de7-2a0e-4bdc-b019-d32a17ed5026\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e8d39b19-a02d-45ef-8d7f-980a3bd2dee3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3be94c682f3a332b91a21c63fd7e215ba9994445c22203fccf81f907a42dce64\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"97478478-fd46-4d38-a203-ec272bb73877\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"dd993023170041653128e1317a01757af1bde7a7360f3dcfeb81e4e1bd21feb9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0265f2a3-eb30-4e34-8343-acca072811e7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"09563670940806d3127e841b07d877ac4df00d9595880483fa895ac33e09c604\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8068, \"end_char_idx\": 10776, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "doc_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "ref_doc_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "chroma:document": "```python\n# Check the size of the data\ncount_sql = \"\"\"select count(*) from public.articles;\"\"\"\ncursor.execute(count_sql)\nresult = cursor.fetchone()\nprint(f\"Count:{result[0]}\")\n```\n\n    Count:25000\n\n\n## Search your data\n\nAfter the data is stored in your Neon database, you can query the data for nearest neighbors. \n\nStart by defining the `query_neon` function, which is executed when you run the vector similarity search. The function creates an embedding based on the user's query, prepares the SQL query, and runs the SQL query with the embedding. The pre-computed embeddings that you loaded into your database were created with `text-embedding-3-small` OpenAI model, so you must use the same model to create an embedding for the similarity search.\n\nA `vector_name` parameter is provided that allows you to search based on \"title\" or \"content\".\n\n\n```python\ndef query_neon(query, collection_name, vector_name=\"title_vector\", top_k=20):\n\n    # Create an embedding vector from the user query\n    embedded_query = openai.Embedding.create(\n        input=query,\n        model=\"text-embedding-3-small\",\n    )[\"data\"][0][\"embedding\"]\n\n    # Convert the embedded_query to PostgreSQL compatible format\n    embedded_query_pg = \"[\" + \",\".join(map(str, embedded_query)) + \"]\"\n\n    # Create the SQL query\n    query_sql = f\"\"\"\n    SELECT id, url, title, l2_distance({vector_name},'{embedded_query_pg}'::VECTOR(1536)) AS similarity\n    FROM {collection_name}\n    ORDER BY {vector_name} <-> '{embedded_query_pg}'::VECTOR(1536)\n    LIMIT {top_k};\n    \"\"\"\n    # Execute the query\n    cursor.execute(query_sql)\n    results = cursor.fetchall()\n\n    return results\n```\n\nRun a similarity search based on `title_vector` embeddings:\n\n\n```python\n# Query based on `title_vector` embeddings\nimport openai\n\nquery_results = query_neon(\"Greek mythology\", \"Articles\")\nfor i, result in enumerate(query_results):\n    print(f\"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})\")\n```\n\n    1. Greek mythology (Score: 0.998)\n    2. Roman mythology (Score: 0.7)\n    3. Greek underworld (Score: 0.637)\n    4. Mythology (Score: 0.635)\n    5. Classical mythology (Score: 0.629)\n    6. Japanese mythology (Score: 0.615)\n    7. Norse mythology (Score: 0.569)\n    8. Greek language (Score: 0.566)\n    9. Zeus (Score: 0.534)\n    10. List of mythologies (Score: 0.531)\n    11. Jupiter (mythology) (Score: 0.53)\n    12. Greek (Score: 0.53)\n    13. Gaia (mythology) (Score: 0.526)\n    14. Titan (mythology) (Score: 0.522)\n    15. Mercury (mythology) (Score: 0.521)\n    16. Ancient Greece (Score: 0.52)\n    17. Greek alphabet (Score: 0.52)\n    18. Venus (mythology) (Score: 0.515)\n    19. Pluto (mythology) (Score: 0.515)\n    20. Athena (Score: 0.514)"}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 0265f2a3-eb30-4e34-8343-acca072811e7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb", "title": "Vector similarity search using Neon Postgres", "tokens": 2947, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0265f2a3-eb30-4e34-8343-acca072811e7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e8d39b19-a02d-45ef-8d7f-980a3bd2dee3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3be94c682f3a332b91a21c63fd7e215ba9994445c22203fccf81f907a42dce64\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6c1c6de7-2a0e-4bdc-b019-d32a17ed5026\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"d6c9aa61b2300b92b01f629b9fac67e2a69930870a4cf7f23eff0c89f0b2fb36\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d59b3a6d-3397-4bc2-bfcf-0187da87c82a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3ebfe531ac4285d22117a4e8d1249a83ec790bdbe515938312a8f6a9c9d15ac5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9780, \"end_char_idx\": 10840, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "doc_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "ref_doc_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "chroma:document": "```python\n# Query based on `title_vector` embeddings\nimport openai\n\nquery_results = query_neon(\"Greek mythology\", \"Articles\")\nfor i, result in enumerate(query_results):\n    print(f\"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})\")\n```\n\n    1. Greek mythology (Score: 0.998)\n    2. Roman mythology (Score: 0.7)\n    3. Greek underworld (Score: 0.637)\n    4. Mythology (Score: 0.635)\n    5. Classical mythology (Score: 0.629)\n    6. Japanese mythology (Score: 0.615)\n    7. Norse mythology (Score: 0.569)\n    8. Greek language (Score: 0.566)\n    9. Zeus (Score: 0.534)\n    10. List of mythologies (Score: 0.531)\n    11. Jupiter (mythology) (Score: 0.53)\n    12. Greek (Score: 0.53)\n    13. Gaia (mythology) (Score: 0.526)\n    14. Titan (mythology) (Score: 0.522)\n    15. Mercury (mythology) (Score: 0.521)\n    16. Ancient Greece (Score: 0.52)\n    17. Greek alphabet (Score: 0.52)\n    18. Venus (mythology) (Score: 0.515)\n    19. Pluto (mythology) (Score: 0.515)\n    20. Athena (Score: 0.514)\n\n\nRun a similarity search based on `content_vector` embeddings:"}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: d59b3a6d-3397-4bc2-bfcf-0187da87c82a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb", "title": "Vector similarity search using Neon Postgres", "tokens": 2947, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"d59b3a6d-3397-4bc2-bfcf-0187da87c82a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e8d39b19-a02d-45ef-8d7f-980a3bd2dee3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3be94c682f3a332b91a21c63fd7e215ba9994445c22203fccf81f907a42dce64\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0265f2a3-eb30-4e34-8343-acca072811e7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"fc882413a7f0609b9737ef81ef6799efe5bb97abbda23aa903adc037dbf08350\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9780, \"end_char_idx\": 11791, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "doc_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "ref_doc_id": "e8d39b19-a02d-45ef-8d7f-980a3bd2dee3", "chroma:document": "```python\n# Query based on `title_vector` embeddings\nimport openai\n\nquery_results = query_neon(\"Greek mythology\", \"Articles\")\nfor i, result in enumerate(query_results):\n    print(f\"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})\")\n```\n\n    1. Greek mythology (Score: 0.998)\n    2. Roman mythology (Score: 0.7)\n    3. Greek underworld (Score: 0.637)\n    4. Mythology (Score: 0.635)\n    5. Classical mythology (Score: 0.629)\n    6. Japanese mythology (Score: 0.615)\n    7. Norse mythology (Score: 0.569)\n    8. Greek language (Score: 0.566)\n    9. Zeus (Score: 0.534)\n    10. List of mythologies (Score: 0.531)\n    11. Jupiter (mythology) (Score: 0.53)\n    12. Greek (Score: 0.53)\n    13. Gaia (mythology) (Score: 0.526)\n    14. Titan (mythology) (Score: 0.522)\n    15. Mercury (mythology) (Score: 0.521)\n    16. Ancient Greece (Score: 0.52)\n    17. Greek alphabet (Score: 0.52)\n    18. Venus (mythology) (Score: 0.515)\n    19. Pluto (mythology) (Score: 0.515)\n    20. Athena (Score: 0.514)\n\n\nRun a similarity search based on `content_vector` embeddings:\n\n\n```python\n# Query based on `content_vector` embeddings\nquery_results = query_neon(\"Famous battles in Greek history\", \"Articles\", \"content_vector\")\nfor i, result in enumerate(query_results):\n    print(f\"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})\")\n```\n\n    1. 222 BC (Score: 0.489)\n    2. Trojan War (Score: 0.458)\n    3. Peloponnesian War (Score: 0.456)\n    4. History of the Peloponnesian War (Score: 0.449)\n    5. 430 BC (Score: 0.441)\n    6. 168 BC (Score: 0.436)\n    7. Ancient Greece (Score: 0.429)\n    8. Classical Athens (Score: 0.428)\n    9. 499 BC (Score: 0.427)\n    10. Leonidas I (Score: 0.426)\n    11. Battle (Score: 0.421)\n    12. Greek War of Independence (Score: 0.421)\n    13. Menelaus (Score: 0.419)\n    14. Thebes, Greece (Score: 0.417)\n    15. Patroclus (Score: 0.417)\n    16. 427 BC (Score: 0.416)\n    17. 429 BC (Score: 0.413)\n    18. August 2 (Score: 0.412)\n    19. Ionia (Score: 0.411)\n    20. 323 (Score: 0.409)"}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: df9ef189-ea00-4723-9db8-542ef6eeb720
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/README.ipynb", "title": "What is Neon?", "tokens": 667, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"df9ef189-ea00-4723-9db8-542ef6eeb720\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/README.ipynb\", \"title\": \"What is Neon?\", \"tokens\": 667, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2b4aeebf-e73e-4e00-a532-aa12262bd561\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/README.ipynb\", \"title\": \"What is Neon?\", \"tokens\": 667, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a08561bdb0259adc50c15e4d4b8468098d850acee49dce8aee6ee3ee94c1be44\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3046, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2b4aeebf-e73e-4e00-a532-aa12262bd561", "doc_id": "2b4aeebf-e73e-4e00-a532-aa12262bd561", "ref_doc_id": "2b4aeebf-e73e-4e00-a532-aa12262bd561", "chroma:document": "# What is Neon?\n\n[Neon](https://neon.tech/) is Serverless Postgres built for the cloud. Neon separates compute and storage to offer modern developer features such as autoscaling, database branching, scale-to-zero, and more.\n\n## Vector search\n\nNeon supports vector search using the [pgvector](https://neon.tech/docs/extensions/pgvector) open-source PostgreSQL extension, which enables Postgres as a vector database for storing and querying embeddings.\n\n## OpenAI cookbook notebook\n\nCheck out the notebook in this repo for working with Neon Serverless Postgres as your vector database.\n\n### Semantic search using Neon Postgres with pgvector and OpenAI\n\nIn this notebook you will learn how to:\n\n1. Use embeddings created by OpenAI API\n2. Store embeddings in a Neon Serverless Postgres database\n3. Convert a raw text query to an embedding with OpenAI API\n4. Use Neon with the `pgvector` extension to perform vector similarity search\n\n## Scaling Support\n\nNeon enables you to scale your AI applications with the following features:\n\n- [Autoscaling](https://neon.tech/docs/introduction/read-replicas): If your AI application experiences heavy load during certain hours of the day or at different times, Neon can automatically scale compute resources without manual intervention. During periods of inactivity, Neon is able to scale to zero.\n- [Instant read replicas](https://neon.tech/docs/introduction/read-replicas): Neon supports instant read replicas, which are independent read-only compute instances designed to perform read operations on the same data as your read-write computes. With read replicas, you can offload reads from your read-write compute instance to a dedicated read-only compute instance for your AI application.\n- [The Neon serverless driver](https://neon.tech/docs/serverless/serverless-driver): Neon supports a low-latency serverless PostgreSQL driver for JavaScript and TypeScript applications that allows you to query data from serverless and edge environments, making it possible to achieve sub-10ms queries.\n\n## More Examples\n\n- [Build an AI-powered semantic search application](https://github.com/neondatabase/yc-idea-matcher) - Submit a startup idea and get a list of similar ideas that YCombinator has invested in before\n- [Build an AI-powered chatbot](https://github.com/neondatabase/ask-neon) - A Postgres Q&A chatbot that uses Postgres as a vector database\n- [Vercel Postgres pgvector Starter](https://vercel.com/templates/next.js/postgres-pgvector) - Vector similarity search with Vercel Postgres (powered by Neon)\n\n## Additional Resources\n\n- [Building AI applications with Neon](https://neon.tech/ai)\n- [Neon AI & embeddings documentation](https://neon.tech/docs/ai/ai-intro)\n- [Building an AI-powered Chatbot using Vercel, OpenAI, and Postgres](neon.tech/blog/building-an-ai-powered-chatbot-using-vercel-openai-and-postgres)\n- [Web-based AI SQL Playground and connecting to Postgres from the browser](https://neon.tech/blog/postgres-ai-playground)\n- [pgvector GitHub repository](https://github.com/pgvector/pgvector)"}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 698542f6-bd76-49a3-b74d-d8a77daab789
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Qdrant and OpenAI", "tokens": 3804, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"698542f6-bd76-49a3-b74d-d8a77daab789\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b0e921a0-f160-4514-ade1-0ca27ff055d3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6ffba8f46e266148633227faf1ec78c9b968b615a7c3cfab385f4740105c85ff\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"efb81734-07e7-4886-83d1-1948abb8345c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6e4f80d4b416bbc3d69a9c1cca689e805b5c73b5f62bb6fe4b88bab3fe449992\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2968, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "doc_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "ref_doc_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "chroma:document": "# Question Answering with Langchain, Qdrant and OpenAI\n\nThis notebook presents how to implement a Question Answering system with Langchain, Qdrant as a knowledge based and OpenAI embeddings. If you are not familiar with Qdrant, it's better to check out the [Getting_started_with_Qdrant_and_OpenAI.ipynb](Getting_started_with_Qdrant_and_OpenAI.ipynb) notebook.\n\nThis notebook presents an end-to-end process of:\n1. Calculating the embeddings with OpenAI API.\n2. Storing the embeddings in a local instance of Qdrant to build a knowledge base.\n3. Converting raw text query to an embedding with OpenAI API.\n4. Using Qdrant to perform the nearest neighbour search in the created collection to find some context.\n5. Asking LLM to find the answer in a given context.\n\nAll the steps will be simplified to calling some corresponding Langchain methods.\n\n## Prerequisites\n\nFor the purposes of this exercise we need to prepare a couple of things:\n\n1. Qdrant server instance. In our case a local Docker container.\n2. The [qdrant-client](https://github.com/qdrant/qdrant_client) library to interact with the vector database.\n3. [Langchain](https://github.com/hwchase17/langchain) as a framework.\n3. An [OpenAI API key](https://beta.openai.com/account/api-keys).\n\n### Start Qdrant server\n\nWe're going to use a local Qdrant instance running in a Docker container. The easiest way to launch it is to use the attached [docker-compose.yaml] file and run the following command:\n\n\n```python\n! docker-compose up -d\n```\n\n    Starting qdrant_qdrant_1 ... \n    \u001b[1Bting qdrant_qdrant_1 ... \u001b[32mdone\u001b[0m\n\nWe might validate if the server was launched successfully by running a simple curl command:\n\n\n```python\n! curl http://localhost:6333\n```\n\n    {\"title\":\"qdrant - vector search engine\",\"version\":\"1.0.1\"}\n\n### Install requirements\n\nThis notebook obviously requires the `openai`, `langchain` and `qdrant-client` packages.\n\n\n\n```python\n! pip install openai qdrant-client \"langchain==0.0.100\" wget\n```\n\n### Prepare your OpenAI API key\n\nThe OpenAI API key is used for vectorization of the documents and queries.\n\nIf you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n\nOnce you get your key, please add it to your environment variables as `OPENAI_API_KEY` by running following command:\n\n\n```python\n! export OPENAI_API_KEY=\"your API key\"\n```\n\n\n```python\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\nimport os\n\n# Note. alternatively you can set a temporary env variable like this:\n# os.environ[\"OPENAI_API_KEY\"] = \"sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\"\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    print(\"OPENAI_API_KEY is ready\")\nelse:\n    print(\"OPENAI_API_KEY environment variable not found\")\n```\n\n    OPENAI_API_KEY is ready"}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: efb81734-07e7-4886-83d1-1948abb8345c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Qdrant and OpenAI", "tokens": 3804, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"efb81734-07e7-4886-83d1-1948abb8345c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b0e921a0-f160-4514-ade1-0ca27ff055d3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6ffba8f46e266148633227faf1ec78c9b968b615a7c3cfab385f4740105c85ff\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"698542f6-bd76-49a3-b74d-d8a77daab789\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"20f1f40d4b0bbf33296a59bfef04729739ad6d843d59e86e32766ed63f8136ea\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"03b1465d-380f-4404-9141-11fddeb04028\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"30e6c5d295957291869f0f904fe996b870949dc58fb92525dc7f224752626f42\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1459, \"end_char_idx\": 4474, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "doc_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "ref_doc_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "chroma:document": "```python\n! docker-compose up -d\n```\n\n    Starting qdrant_qdrant_1 ... \n    \u001b[1Bting qdrant_qdrant_1 ... \u001b[32mdone\u001b[0m\n\nWe might validate if the server was launched successfully by running a simple curl command:\n\n\n```python\n! curl http://localhost:6333\n```\n\n    {\"title\":\"qdrant - vector search engine\",\"version\":\"1.0.1\"}\n\n### Install requirements\n\nThis notebook obviously requires the `openai`, `langchain` and `qdrant-client` packages.\n\n\n\n```python\n! pip install openai qdrant-client \"langchain==0.0.100\" wget\n```\n\n### Prepare your OpenAI API key\n\nThe OpenAI API key is used for vectorization of the documents and queries.\n\nIf you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n\nOnce you get your key, please add it to your environment variables as `OPENAI_API_KEY` by running following command:\n\n\n```python\n! export OPENAI_API_KEY=\"your API key\"\n```\n\n\n```python\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\nimport os\n\n# Note. alternatively you can set a temporary env variable like this:\n# os.environ[\"OPENAI_API_KEY\"] = \"sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\"\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    print(\"OPENAI_API_KEY is ready\")\nelse:\n    print(\"OPENAI_API_KEY environment variable not found\")\n```\n\n    OPENAI_API_KEY is ready\n\n\n## Load data\n\nIn this section we are going to load the data containing some natural questions and answers to them. All the data will be used to create a Langchain application with Qdrant being the knowledge base.\n\n\n```python\nimport wget\n\n# All the examples come from https://ai.google.com/research/NaturalQuestions\n# This is a sample of the training set that we download and extract for some\n# further processing.\nwget.download(\"https://storage.googleapis.com/dataset-natural-questions/questions.json\")\nwget.download(\"https://storage.googleapis.com/dataset-natural-questions/answers.json\")\n```\n\n    100% [..............................................................................] 95372 / 95372\n\n\n\n\n    'answers.json'\n\n\n\n\n```python\nimport json\n\nwith open(\"questions.json\", \"r\") as fp:\n    questions = json.load(fp)\n\nwith open(\"answers.json\", \"r\") as fp:\n    answers = json.load(fp)\n```\n\n\n```python\nprint(questions[0])\n```\n\n    when is the last episode of season 8 of the walking dead\n\n\n\n```python\nprint(answers[0])\n```\n\n    No . overall No. in season Title Directed by Written by Original air date U.S. viewers ( millions ) 100 `` Mercy '' Greg Nicotero Scott M. Gimple October 22 , 2017 ( 2017 - 10 - 22 ) 11.44 Rick , Maggie , and Ezekiel rally their communities together to take down Negan . Gregory attempts to have the Hilltop residents side with Negan , but they all firmly stand behind Maggie . The group attacks the Sanctuary , taking down its fences and flooding the compound with walkers ."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 03b1465d-380f-4404-9141-11fddeb04028
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Qdrant and OpenAI", "tokens": 3804, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"03b1465d-380f-4404-9141-11fddeb04028\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b0e921a0-f160-4514-ade1-0ca27ff055d3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6ffba8f46e266148633227faf1ec78c9b968b615a7c3cfab385f4740105c85ff\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"efb81734-07e7-4886-83d1-1948abb8345c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"b15361f221b5a19192a6dcef3ec8890b1724213e38300656cf563517ce9c503f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"aaac6685-5dec-4d9f-88a0-8a6b0c0ab143\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b35332f55e097d4f1aa8a07acc11ca5866970b94d13a31b95b2a973ffbe3cc73\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2971, \"end_char_idx\": 6111, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "doc_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "ref_doc_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "chroma:document": "## Load data\n\nIn this section we are going to load the data containing some natural questions and answers to them. All the data will be used to create a Langchain application with Qdrant being the knowledge base.\n\n\n```python\nimport wget\n\n# All the examples come from https://ai.google.com/research/NaturalQuestions\n# This is a sample of the training set that we download and extract for some\n# further processing.\nwget.download(\"https://storage.googleapis.com/dataset-natural-questions/questions.json\")\nwget.download(\"https://storage.googleapis.com/dataset-natural-questions/answers.json\")\n```\n\n    100% [..............................................................................] 95372 / 95372\n\n\n\n\n    'answers.json'\n\n\n\n\n```python\nimport json\n\nwith open(\"questions.json\", \"r\") as fp:\n    questions = json.load(fp)\n\nwith open(\"answers.json\", \"r\") as fp:\n    answers = json.load(fp)\n```\n\n\n```python\nprint(questions[0])\n```\n\n    when is the last episode of season 8 of the walking dead\n\n\n\n```python\nprint(answers[0])\n```\n\n    No . overall No. in season Title Directed by Written by Original air date U.S. viewers ( millions ) 100 `` Mercy '' Greg Nicotero Scott M. Gimple October 22 , 2017 ( 2017 - 10 - 22 ) 11.44 Rick , Maggie , and Ezekiel rally their communities together to take down Negan . Gregory attempts to have the Hilltop residents side with Negan , but they all firmly stand behind Maggie . The group attacks the Sanctuary , taking down its fences and flooding the compound with walkers . With the Sanctuary defaced , everyone leaves except Gabriel , who reluctantly stays to save Gregory , but is left behind when Gregory abandons him . Surrounded by walkers , Gabriel hides in a trailer , where he is trapped inside with Negan . 101 `` The Damned '' Rosemary Rodriguez Matthew Negrete & Channing Powell October 29 , 2017 ( 2017 - 10 - 29 ) 8.92 Rick 's forces split into separate parties to attack several of the Saviors ' outposts , during which many members of the group are killed ; Eric is critically injured and rushed away by Aaron . Jesus stops Tara and Morgan from executing a group of surrendered Saviors . While clearing an outpost with Daryl , Rick is confronted and held at gunpoint by Morales , a survivor he met in the initial Atlanta camp , who is now with the Saviors . 102 `` Monsters '' Greg Nicotero Matthew Negrete & Channing Powell November 5 , 2017 ( 2017 - 11 - 05 ) 8.52 Daryl finds Morales threatening Rick and kills him ; the duo then pursue a group of Saviors who are transporting weapons to another outpost . Gregory returns to Hilltop , and after a heated argument , Maggie ultimately allows him back in the community . Eric dies from his injuries , leaving Aaron distraught . Despite Tara and Morgan 's objections , Jesus leads the group of surrendered Saviors to Hilltop . Ezekiel 's group attacks another Savior compound , during which several Kingdommers are shot while protecting Ezekiel . 103 `` Some Guy '' Dan Liu David Leslie Johnson November 12 , 2017 ( 2017 - 11 - 12 ) 8.69 Ezekiel 's group is overwhelmed by the Saviors , who kill all of them except for Ezekiel himself and Jerry ."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: aaac6685-5dec-4d9f-88a0-8a6b0c0ab143
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Qdrant and OpenAI", "tokens": 3804, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"aaac6685-5dec-4d9f-88a0-8a6b0c0ab143\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b0e921a0-f160-4514-ade1-0ca27ff055d3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6ffba8f46e266148633227faf1ec78c9b968b615a7c3cfab385f4740105c85ff\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"03b1465d-380f-4404-9141-11fddeb04028\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7f9a79f427935cfc7b47fef72763843db0454fe0e16eff471cf7cd4fea8db90b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"82fa842d-b756-4cb6-bfc2-f1c7ec789970\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2fd05f792f59a9591da45a93998efb4f4b6fa61d55f24622e819493ad35fb3e8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4475, \"end_char_idx\": 7638, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "doc_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "ref_doc_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "chroma:document": "With the Sanctuary defaced , everyone leaves except Gabriel , who reluctantly stays to save Gregory , but is left behind when Gregory abandons him . Surrounded by walkers , Gabriel hides in a trailer , where he is trapped inside with Negan . 101 `` The Damned '' Rosemary Rodriguez Matthew Negrete & Channing Powell October 29 , 2017 ( 2017 - 10 - 29 ) 8.92 Rick 's forces split into separate parties to attack several of the Saviors ' outposts , during which many members of the group are killed ; Eric is critically injured and rushed away by Aaron . Jesus stops Tara and Morgan from executing a group of surrendered Saviors . While clearing an outpost with Daryl , Rick is confronted and held at gunpoint by Morales , a survivor he met in the initial Atlanta camp , who is now with the Saviors . 102 `` Monsters '' Greg Nicotero Matthew Negrete & Channing Powell November 5 , 2017 ( 2017 - 11 - 05 ) 8.52 Daryl finds Morales threatening Rick and kills him ; the duo then pursue a group of Saviors who are transporting weapons to another outpost . Gregory returns to Hilltop , and after a heated argument , Maggie ultimately allows him back in the community . Eric dies from his injuries , leaving Aaron distraught . Despite Tara and Morgan 's objections , Jesus leads the group of surrendered Saviors to Hilltop . Ezekiel 's group attacks another Savior compound , during which several Kingdommers are shot while protecting Ezekiel . 103 `` Some Guy '' Dan Liu David Leslie Johnson November 12 , 2017 ( 2017 - 11 - 12 ) 8.69 Ezekiel 's group is overwhelmed by the Saviors , who kill all of them except for Ezekiel himself and Jerry . Carol clears the inside of the compound , killing all but two Saviors , who almost escape but are eventually caught by Rick and Daryl . En route to the Kingdom , Ezekiel , Jerry , and Carol are surrounded by walkers , but Shiva sacrifices herself to save them . The trio returns to the Kingdom , where Ezekiel 's confidence in himself as a leader has diminished . 104 5 `` The Big Scary U '' Michael E. Satrazemis Story by : Scott M. Gimple & David Leslie Johnson & Angela Kang Teleplay by : David Leslie Johnson & Angela Kang November 19 , 2017 ( 2017 - 11 - 19 ) 7.85 After confessing their sins to each other , Gabriel and Negan manage to escape from the trailer . Simon and the other lieutenants grow suspicious of each other , knowing that Rick 's forces must have inside information . The workers in the Sanctuary become increasingly frustrated with their living conditions , and a riot nearly ensues , until Negan returns and restores order . Gabriel is locked in a cell , where Eugene discovers him sick and suffering . Meanwhile , Rick and Daryl argue over how to take out the Saviors , leading Daryl to abandon Rick . 105 6 `` The King , the Widow , and Rick '' John Polson Angela Kang & Corey Reed November 26 , 2017 ( 2017 - 11 - 26 ) 8.28 Rick visits Jadis in hopes of convincing her to turn against Negan ; Jadis refuses , and locks Rick in a shipping container . Carl encounters Siddiq in the woods and recruits him to Alexandria . Daryl and Tara plot to deviate from Rick 's plans by destroying the Sanctuary ."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 82fa842d-b756-4cb6-bfc2-f1c7ec789970
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Qdrant and OpenAI", "tokens": 3804, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"82fa842d-b756-4cb6-bfc2-f1c7ec789970\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b0e921a0-f160-4514-ade1-0ca27ff055d3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6ffba8f46e266148633227faf1ec78c9b968b615a7c3cfab385f4740105c85ff\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"aaac6685-5dec-4d9f-88a0-8a6b0c0ab143\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"418c83666e85d5b7534c7a92a499d65eb989dfe3ea8d9e240590807d6e8f8cb1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ad34b393-d5ca-44f3-8187-9f2b8df4a533\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d47d6bb939b32c2a91b07eefa9126c0fb0f63470ba2b8e0d1fd4b5e2c7dca1cc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6112, \"end_char_idx\": 9294, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "doc_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "ref_doc_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "chroma:document": "Carol clears the inside of the compound , killing all but two Saviors , who almost escape but are eventually caught by Rick and Daryl . En route to the Kingdom , Ezekiel , Jerry , and Carol are surrounded by walkers , but Shiva sacrifices herself to save them . The trio returns to the Kingdom , where Ezekiel 's confidence in himself as a leader has diminished . 104 5 `` The Big Scary U '' Michael E. Satrazemis Story by : Scott M. Gimple & David Leslie Johnson & Angela Kang Teleplay by : David Leslie Johnson & Angela Kang November 19 , 2017 ( 2017 - 11 - 19 ) 7.85 After confessing their sins to each other , Gabriel and Negan manage to escape from the trailer . Simon and the other lieutenants grow suspicious of each other , knowing that Rick 's forces must have inside information . The workers in the Sanctuary become increasingly frustrated with their living conditions , and a riot nearly ensues , until Negan returns and restores order . Gabriel is locked in a cell , where Eugene discovers him sick and suffering . Meanwhile , Rick and Daryl argue over how to take out the Saviors , leading Daryl to abandon Rick . 105 6 `` The King , the Widow , and Rick '' John Polson Angela Kang & Corey Reed November 26 , 2017 ( 2017 - 11 - 26 ) 8.28 Rick visits Jadis in hopes of convincing her to turn against Negan ; Jadis refuses , and locks Rick in a shipping container . Carl encounters Siddiq in the woods and recruits him to Alexandria . Daryl and Tara plot to deviate from Rick 's plans by destroying the Sanctuary . Ezekiel isolates himself at the Kingdom , where Carol tries to encourage him to be the leader his people need . Maggie has the group of captured Saviors placed in a holding area and forces Gregory to join them as punishment for betraying Hilltop . 106 7 `` Time for After '' Larry Teng Matthew Negrete & Corey Reed December 3 , 2017 ( 2017 - 12 - 03 ) 7.47 After learning of Dwight 's association with Rick 's group , Eugene affirms his loyalty to Negan and outlines a plan to get rid of the walkers surrounding the Sanctuary . With help from Morgan and Tara , Daryl drives a truck through the Sanctuary 's walls , flooding its interior with walkers , killing many Saviors . Rick finally convinces Jadis and the Scavengers to align with him , and they plan to force the Saviors to surrender . However , when they arrive at the Sanctuary , Rick is horrified to see the breached walls and no sign of the walker herd . 107 8 `` How It 's Gotta Be '' Michael E. Satrazemis David Leslie Johnson & Angela Kang December 10 , 2017 ( 2017 - 12 - 10 ) 7.89 Eugene 's plan allows the Saviors to escape , and separately , the Saviors waylay the Alexandria , Hilltop , and Kingdom forces . The Scavengers abandon Rick , after which he returns to Alexandria . Ezekiel ensures that the Kingdom residents are able to escape before locking himself in the community with the Saviors . Eugene aids Gabriel and Doctor Carson in escaping the Sanctuary in order to ease his conscience . Negan attacks Alexandria , but Carl devises a plan to allow the Alexandria residents to escape into the sewers . Carl reveals he was bitten by a walker while escorting Siddiq to Alexandria ."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: ad34b393-d5ca-44f3-8187-9f2b8df4a533
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Qdrant and OpenAI", "tokens": 3804, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"ad34b393-d5ca-44f3-8187-9f2b8df4a533\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b0e921a0-f160-4514-ade1-0ca27ff055d3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6ffba8f46e266148633227faf1ec78c9b968b615a7c3cfab385f4740105c85ff\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"82fa842d-b756-4cb6-bfc2-f1c7ec789970\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"91e6877ecca7b8b40915a06e82cbb78583b3cba851e9db44b5d681e3f6c6b9ff\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3d3702b4-362e-4db9-92b1-c4ce42f0c12e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4b16789cc35b73c72ee4ef2fefa7b4ee55bb163511a815198e8d154f152909bf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7639, \"end_char_idx\": 10642, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "doc_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "ref_doc_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "chroma:document": "Ezekiel isolates himself at the Kingdom , where Carol tries to encourage him to be the leader his people need . Maggie has the group of captured Saviors placed in a holding area and forces Gregory to join them as punishment for betraying Hilltop . 106 7 `` Time for After '' Larry Teng Matthew Negrete & Corey Reed December 3 , 2017 ( 2017 - 12 - 03 ) 7.47 After learning of Dwight 's association with Rick 's group , Eugene affirms his loyalty to Negan and outlines a plan to get rid of the walkers surrounding the Sanctuary . With help from Morgan and Tara , Daryl drives a truck through the Sanctuary 's walls , flooding its interior with walkers , killing many Saviors . Rick finally convinces Jadis and the Scavengers to align with him , and they plan to force the Saviors to surrender . However , when they arrive at the Sanctuary , Rick is horrified to see the breached walls and no sign of the walker herd . 107 8 `` How It 's Gotta Be '' Michael E. Satrazemis David Leslie Johnson & Angela Kang December 10 , 2017 ( 2017 - 12 - 10 ) 7.89 Eugene 's plan allows the Saviors to escape , and separately , the Saviors waylay the Alexandria , Hilltop , and Kingdom forces . The Scavengers abandon Rick , after which he returns to Alexandria . Ezekiel ensures that the Kingdom residents are able to escape before locking himself in the community with the Saviors . Eugene aids Gabriel and Doctor Carson in escaping the Sanctuary in order to ease his conscience . Negan attacks Alexandria , but Carl devises a plan to allow the Alexandria residents to escape into the sewers . Carl reveals he was bitten by a walker while escorting Siddiq to Alexandria . 108 9 `` Honor '' Greg Nicotero Matthew Negrete & Channing Powell February 25 , 2018 ( 2018 - 02 - 25 ) 8.28 After the Saviors leave Alexandria , the survivors make for the Hilltop while Rick and Michonne stay behind to say their final goodbyes to a dying Carl , who pleads with Rick to build a better future alongside the Saviors before killing himself . In the Kingdom , Morgan and Carol launch a rescue mission for Ezekiel . Although they are successful and retake the Kingdom , the Saviors ' lieutenant Gavin is killed by Benjamin 's vengeful brother Henry . 109 10 `` The Lost and the Plunderers '' TBA TBA March 4 , 2018 ( 2018 - 03 - 04 ) TBD 110 11 `` Dead or Alive Or '' TBA TBA March 11 , 2018 ( 2018 - 03 - 11 ) TBD 111 12 `` The Key '' TBA TBA March 18 , 2018 ( 2018 - 03 - 18 ) TBD\n\n\n## Chain definition\n\nLangchain is already integrated with Qdrant and performs all the indexing for given list of documents. In our case we are going to store the set of answers we have.\n\n\n```python\nfrom langchain.vectorstores import Qdrant\nfrom langchain.embeddings import OpenAIEmbeddings\nfrom langchain import VectorDBQA, OpenAI\n\nembeddings = OpenAIEmbeddings()\ndoc_store = Qdrant.from_texts(\n    answers, embeddings, host=\"localhost\" \n)\n```\n\nAt this stage all the possible answers are already stored in Qdrant, so we can define the whole QA chain."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 3d3702b4-362e-4db9-92b1-c4ce42f0c12e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Qdrant and OpenAI", "tokens": 3804, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"3d3702b4-362e-4db9-92b1-c4ce42f0c12e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b0e921a0-f160-4514-ade1-0ca27ff055d3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6ffba8f46e266148633227faf1ec78c9b968b615a7c3cfab385f4740105c85ff\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ad34b393-d5ca-44f3-8187-9f2b8df4a533\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8382126d532011cea12c7c0414568c78b79aa0b76e94117680b08f8c5f709e7e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e6a1a0c5-85fc-4b0d-bf2d-ab8b77c6b957\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ced1a2c2ed984140d13c0a99963e6a245bd052ba7c456422800649e7529cd17e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9217, \"end_char_idx\": 11400, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "doc_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "ref_doc_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "chroma:document": "Carl reveals he was bitten by a walker while escorting Siddiq to Alexandria . 108 9 `` Honor '' Greg Nicotero Matthew Negrete & Channing Powell February 25 , 2018 ( 2018 - 02 - 25 ) 8.28 After the Saviors leave Alexandria , the survivors make for the Hilltop while Rick and Michonne stay behind to say their final goodbyes to a dying Carl , who pleads with Rick to build a better future alongside the Saviors before killing himself . In the Kingdom , Morgan and Carol launch a rescue mission for Ezekiel . Although they are successful and retake the Kingdom , the Saviors ' lieutenant Gavin is killed by Benjamin 's vengeful brother Henry . 109 10 `` The Lost and the Plunderers '' TBA TBA March 4 , 2018 ( 2018 - 03 - 04 ) TBD 110 11 `` Dead or Alive Or '' TBA TBA March 11 , 2018 ( 2018 - 03 - 11 ) TBD 111 12 `` The Key '' TBA TBA March 18 , 2018 ( 2018 - 03 - 18 ) TBD\n\n\n## Chain definition\n\nLangchain is already integrated with Qdrant and performs all the indexing for given list of documents. In our case we are going to store the set of answers we have.\n\n\n```python\nfrom langchain.vectorstores import Qdrant\nfrom langchain.embeddings import OpenAIEmbeddings\nfrom langchain import VectorDBQA, OpenAI\n\nembeddings = OpenAIEmbeddings()\ndoc_store = Qdrant.from_texts(\n    answers, embeddings, host=\"localhost\" \n)\n```\n\nAt this stage all the possible answers are already stored in Qdrant, so we can define the whole QA chain.\n\n\n```python\nllm = OpenAI()\nqa = VectorDBQA.from_chain_type(\n    llm=llm, \n    chain_type=\"stuff\", \n    vectorstore=doc_store,\n    return_source_documents=False,\n)\n```\n\n## Search data\n\nOnce the data is put into Qdrant we can start asking some questions. A question will be automatically vectorized by OpenAI model, and the created vector will be used to find some possibly matching answers in Qdrant. Once retrieved, the most similar answers will be incorporated into the prompt sent to OpenAI Large Language Model. The communication between all the services is shown on a graph:\n\n![](https://qdrant.tech/articles_data/langchain-integration/flow-diagram.png)\n\n\n\n```python\nimport random\n\nrandom.seed(52)\nselected_questions = random.choices(questions, k=5)\n```"}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: e6a1a0c5-85fc-4b0d-bf2d-ab8b77c6b957
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Qdrant and OpenAI", "tokens": 3804, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e6a1a0c5-85fc-4b0d-bf2d-ab8b77c6b957\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b0e921a0-f160-4514-ade1-0ca27ff055d3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6ffba8f46e266148633227faf1ec78c9b968b615a7c3cfab385f4740105c85ff\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3d3702b4-362e-4db9-92b1-c4ce42f0c12e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"782a8abe0933a3a913ceee02209172a5c655409efec187db6281c731ad9f6d46\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1a5a931f-a8b5-48af-ace7-8c3dc3a955a7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"90b1c9b433ebdd985f3bd3cf588d8179ba401b34dcf06dd9a1a22a6dc44b4c64\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10092, \"end_char_idx\": 13313, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "doc_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "ref_doc_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "chroma:document": "## Chain definition\n\nLangchain is already integrated with Qdrant and performs all the indexing for given list of documents. In our case we are going to store the set of answers we have.\n\n\n```python\nfrom langchain.vectorstores import Qdrant\nfrom langchain.embeddings import OpenAIEmbeddings\nfrom langchain import VectorDBQA, OpenAI\n\nembeddings = OpenAIEmbeddings()\ndoc_store = Qdrant.from_texts(\n    answers, embeddings, host=\"localhost\" \n)\n```\n\nAt this stage all the possible answers are already stored in Qdrant, so we can define the whole QA chain.\n\n\n```python\nllm = OpenAI()\nqa = VectorDBQA.from_chain_type(\n    llm=llm, \n    chain_type=\"stuff\", \n    vectorstore=doc_store,\n    return_source_documents=False,\n)\n```\n\n## Search data\n\nOnce the data is put into Qdrant we can start asking some questions. A question will be automatically vectorized by OpenAI model, and the created vector will be used to find some possibly matching answers in Qdrant. Once retrieved, the most similar answers will be incorporated into the prompt sent to OpenAI Large Language Model. The communication between all the services is shown on a graph:\n\n![](https://qdrant.tech/articles_data/langchain-integration/flow-diagram.png)\n\n\n\n```python\nimport random\n\nrandom.seed(52)\nselected_questions = random.choices(questions, k=5)\n```\n\n\n```python\nfor question in selected_questions:\n    print(\">\", question)\n    print(qa.run(question), end=\"\\n\\n\")\n```\n\n    > where do frankenstein and the monster first meet\n     Victor and the Creature first meet in the mountains.\n    \n    > who are the actors in fast and furious\n     The actors in the Fast and Furious films are Vin Diesel, Paul Walker, Michelle Rodriguez, Jordana Brewster, Tyrese Gibson, Ludacris, Lucas Black, Sung Kang, Gal Gadot, Dwayne Johnson, Matt Schulze, Chad Lindberg, Johnny Strong, Eva Mendes, Devon Aoki, Nathalie Kelley, Bow Wow, Tego Calder\u00f3n, Don Omar, Elsa Pataky, Kurt Russell, Nathalie Emmanuel, Scott Eastwood, Noel Gugliemi, Ja Rule, Thom Barry, Ted Levine, Minka Kelly, James Remar, Amaury Nolasco, Michael Ealy, MC Jin, Brian Goodman, Lynda Boyd, Jason Tobin, Neela, Liza Lapira, Alimi Ballard, Yorgo Constantine, Geoff Meed, Jeimy Osorio, Max William Crane, Charlie & Miller Kimsey, Eden Estrella, Romeo Santos, John Brotherton, Helen Mirren, Celestino Cornielle, Janmarco Santiago, Carlos De La Hoz, James Ayoub, Rick Yune, Cole Hauser, Brian Tee, John Ortiz, Luke Evans, Jason Statham, Charlize Theron, Reggie Lee, Mo Gallini, Roberto Sanchez, Leonardo\n    \n    > properties of red black tree in data structure\n     Red black trees are a type of binary tree with a special set of properties. Each node is either red or black, the root is black, and if a node is red, then both its children are black. Every path from a given node to any of its descendant NIL nodes contains the same number of black nodes. The number of black nodes from the root to a node is the node's black depth, and the uniform number of black nodes in all paths from root to the leaves is called the black-height of the red-black tree.\n    \n    > who designed the national coat of arms of south africa\n     Iaan Bekker\n    \n    > caravaggio's death of the virgin pamela askew\n     I don't know."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: 1a5a931f-a8b5-48af-ace7-8c3dc3a955a7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Qdrant and OpenAI", "tokens": 3804, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"1a5a931f-a8b5-48af-ace7-8c3dc3a955a7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b0e921a0-f160-4514-ade1-0ca27ff055d3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6ffba8f46e266148633227faf1ec78c9b968b615a7c3cfab385f4740105c85ff\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e6a1a0c5-85fc-4b0d-bf2d-ab8b77c6b957\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"690ab512f3ba0bcfe9148b1bb028dbe6c25756cb0a75e524716d5659addd03eb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13321, \"end_char_idx\": 15908, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "doc_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "ref_doc_id": "b0e921a0-f160-4514-ade1-0ca27ff055d3", "chroma:document": "### Custom prompt templates\n\nThe `stuff` chain type in Langchain uses a specific prompt with question and context documents incorporated. This is what the default prompt looks like:\n\n```text\nUse the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.\n{context}\nQuestion: {question}\nHelpful Answer:\n```\n\nWe can, however, provide our prompt template and change the behaviour of the OpenAI LLM, while still using the `stuff` chain type. It is important to keep `{context}` and `{question}` as placeholders.\n\n#### Experimenting with custom prompts\n\nWe can try using a different prompt template, so the model:\n1. Responds with a single-sentence answer if it knows it.\n2. Suggests a random song title if it doesn't know the answer to our question.\n\n\n```python\nfrom langchain.prompts import PromptTemplate\n```\n\n\n```python\ncustom_prompt = \"\"\"\nUse the following pieces of context to answer the question at the end. Please provide\na short single-sentence summary answer only. If you don't know the answer or if it's \nnot present in given context, don't try to make up an answer, but suggest me a random \nunrelated song title I could listen to. \nContext: {context}\nQuestion: {question}\nHelpful Answer:\n\"\"\"\n```\n\n\n```python\ncustom_prompt_template = PromptTemplate(\n    template=custom_prompt, input_variables=[\"context\", \"question\"]\n)\n```\n\n\n```python\ncustom_qa = VectorDBQA.from_chain_type(\n    llm=llm, \n    chain_type=\"stuff\", \n    vectorstore=doc_store,\n    return_source_documents=False,\n    chain_type_kwargs={\"prompt\": custom_prompt_template},\n)\n```\n\n\n```python\nrandom.seed(41)\nfor question in random.choices(questions, k=5):\n    print(\">\", question)\n    print(custom_qa.run(question), end=\"\\n\\n\")\n```\n\n    > what was uncle jesse's original last name on full house\n    Uncle Jesse's original last name on Full House was Cochran.\n    \n    > when did the volcano erupt in indonesia 2018\n    No volcanic eruption is mentioned in the given context. Suggested Song: \"Ring of Fire\" by Johnny Cash.\n    \n    > what does a dualist way of thinking mean\n    Dualist way of thinking means that the mind and body are separate entities, with the mind being a non-physical substance.\n    \n    > the first civil service commission in india was set up on the basis of recommendation of\n    The first Civil Service Commission in India was not set up on the basis of a recommendation.\n    \n    > how old do you have to be to get a tattoo in utah\n    In Utah, you must be at least 18 years old to get a tattoo."}


created_at: 2024-07-28 18:59:18
topic: persistent://default/default/4bfc3c11-fb71-4cc9-8c12-01ed076a0e01
id: cdae6e4c-62ff-4553-a080-f17895f59a8e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb", "title": "Using Qdrant for Embeddings Search", "tokens": 4309, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"cdae6e4c-62ff-4553-a080-f17895f59a8e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4309, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"0a5e2d7a-39ac-470a-86ab-4a93d4efd483\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4309, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"60898b70a600f0a7e8231cd43e833d8fb18d7a9b2db28edcadf9249732cad1c8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"791739f6-ee4c-43d1-8674-117a97e03dd0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"406b371b81037794911c525539b6cbb09fdd41e3aa5e617d72e210e2012f59bc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3579, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "0a5e2d7a-39ac-470a-86ab-4a93d4efd483", "doc_id": "0a5e2d7a-39ac-470a-86ab-4a93d4efd483", "ref_doc_id": "0a5e2d7a-39ac-470a-86ab-4a93d4efd483", "chroma:document": "# Using Qdrant for Embeddings Search\n\nThis notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.\n\n### What is a Vector Database\n\nA vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.\n\n### Why use a Vector Database\n\nVector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.\n\n\n### Demo Flow\nThe demo flow is:\n- **Setup**: Import packages and set any required variables\n- **Load data**: Load a dataset and embed it using OpenAI embeddings\n- **Qdrant**\n    - *Setup*: Here we'll set up the Python client for Qdrant. For more details go [here](https://github.com/qdrant/qdrant_client)\n    - *Index Data*: We'll create a collection with vectors for __titles__ and __content__\n    - *Search Data*: We'll run a few searches to confirm it works\n\nOnce you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.\n\n## Setup\n\nImport the required libraries and set the embedding model that we'd like to use.\n\n\n```python\n# We'll need to install Qdrant client\n!pip install qdrant-client\n```\n\n\n```python\nimport openai\nimport pandas as pd\nfrom ast import literal_eval\nimport qdrant_client # Qdrant's client library for Python\n\n# This can be changed to the embedding model of your choice. Make sure its the same model that is used for generating embeddings\nEMBEDDING_MODEL = \"text-embedding-ada-002\"\n\n# Ignore unclosed SSL socket warnings - optional in case you get these errors\nimport warnings\n\nwarnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning) \n```\n\n## Load data\n\nIn this section we'll load embedded data that we've prepared previous to this session.\n\n\n```python\nimport requests\n\nembeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'\n\n# The file is ~700 MB so this will take some time\nresponse = requests.get(embeddings_url, verify=True)  # Set verify=False to bypass SSL verification\nwith open('vector_database_wikipedia_articles_embedded.zip', 'wb') as file:\n    file.write(response.content)\n```\n\n\n```python\nimport zipfile\nwith zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n    zip_ref.extractall(\"../data\")\n```"}


