# embeddings_queue

created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 79dda517-c141-4a48-bd3d-219e03c342a6
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"79dda517-c141-4a48-bd3d-219e03c342a6\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb\", \"title\": \"Semantic Search with Pinecone and OpenAI\", \"tokens\": 34426, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ee5c68d1-dff6-4673-87a2-e097be084ab5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb\", \"title\": \"Semantic Search with Pinecone and OpenAI\", \"tokens\": 34426, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"a1034ea29a411ebb75c0f53708206ad7a93cc556d4389577ff0ec654637ecfd2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dd960b31-e355-4e05-8961-64f41b7759c5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb\", \"title\": \"Semantic Search with Pinecone and OpenAI\", \"tokens\": 34426, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"379b9c2da6550884aae9e871022de2586303d25a32df33e85dad96535aebd75d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4b42b653-2bf6-4c5c-b240-3f1dfe509ced\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ab1c655e499d9a518b7e01ab19cb6c8a003e7a4c0b2accab96faadf53a027622\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 71768, \"end_char_idx\": 73816, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ee5c68d1-dff6-4673-87a2-e097be084ab5", "doc_id": "ee5c68d1-dff6-4673-87a2-e097be084ab5", "ref_doc_id": "ee5c68d1-dff6-4673-87a2-e097be084ab5", "chroma:document": "-0.007614186033606529, -0.00643457705155015, -0.02566220425069332, 0.012298375368118286, 0.017123356461524963, 0.01261801179498434, -0.001627670251764357, -0.003949787467718124, -0.00968040432780981, -0.03981751203536987, 0.020197950303554535, -0.01837145909667015, -0.0007795884157530963, 0.014581489376723766, 0.04188753664493561, 0.006232901941984892, -0.017001591622829437, 0.019178159534931183, -0.006887394469231367], index=1, object='embedding')], model='text-embedding-3-small', object='list', usage=Usage(prompt_tokens=13, total_tokens=13))\n\n\n\n\n```python\nprint(f\"vector 0: {len(res.data[0].embedding)}\\nvector 1: {len(res.data[1].embedding)}\")\n```\n\n    vector 0: 1536\n    vector 1: 1536\n\n\n\n```python\n# we can extract embeddings to a list\nembeds = [record.embedding for record in res.data]\nlen(embeds)\n```\n\n\n\n\n    2\n\n\n\nNext, we initialize our index to store vector embeddings with Pinecone.\n\n\n```python\nlen(embeds[0])\n```\n\n\n\n\n    1536\n\n\n\nInitialize connection to Pinecone, you can get a free API key in the [Pinecone dashboard](https://app.pinecone.io).\n\n\n```python\nfrom pinecone import Pinecone\n\npc = Pinecone(api_key=\"...\")\n```\n\n\n\n\n```python\nimport time\nfrom pinecone import ServerlessSpec\n\nspec = ServerlessSpec(cloud=\"aws\", region=\"us-west-2\")\n\nindex_name = 'semantic-search-openai'\n\n# check if index already exists (if shouldn't if this is your first run)\nif index_name not in pc.list_indexes().names():\n    # if does not exist, create index\n    pc.create_index(\n        index_name,\n        dimension=len(embeds[0]),  # dimensionality of text-embed-3-small\n        metric='dotproduct',\n        spec=spec\n    )\n    # wait for index to be initialized\n    while not pc.describe_index(index_name).status['ready']:\n        time.sleep(1)\n\n# connect to index\nindex = pc.Index(index_name)\ntime.sleep(1)\n# view index stats\nindex.describe_index_stats()\n```\n\n\n\n\n    {'dimension': 1536,\n     'index_fullness': 0.0,\n     'namespaces': {},\n     'total_vector_count': 0}\n\n\n\n## Populating the Index\n\nNow we will take 1K questions from the TREC dataset"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 4b42b653-2bf6-4c5c-b240-3f1dfe509ced
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"4b42b653-2bf6-4c5c-b240-3f1dfe509ced\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb\", \"title\": \"Semantic Search with Pinecone and OpenAI\", \"tokens\": 34426, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ee5c68d1-dff6-4673-87a2-e097be084ab5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb\", \"title\": \"Semantic Search with Pinecone and OpenAI\", \"tokens\": 34426, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"a1034ea29a411ebb75c0f53708206ad7a93cc556d4389577ff0ec654637ecfd2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"79dda517-c141-4a48-bd3d-219e03c342a6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb\", \"title\": \"Semantic Search with Pinecone and OpenAI\", \"tokens\": 34426, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"30b3b74b5c293c57cf4a9c104d1c686e602928c10422f53982e9e0066a1c27df\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6fd2f4b0-2e51-4ffb-b13a-6ad078ca4bab\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9e6befbadf57f71353b5f661a6939effdefc4571c7e19deb46014bdfef7bbec4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 72467, \"end_char_idx\": 75208, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ee5c68d1-dff6-4673-87a2-e097be084ab5", "doc_id": "ee5c68d1-dff6-4673-87a2-e097be084ab5", "ref_doc_id": "ee5c68d1-dff6-4673-87a2-e097be084ab5", "chroma:document": "```python\n# we can extract embeddings to a list\nembeds = [record.embedding for record in res.data]\nlen(embeds)\n```\n\n\n\n\n    2\n\n\n\nNext, we initialize our index to store vector embeddings with Pinecone.\n\n\n```python\nlen(embeds[0])\n```\n\n\n\n\n    1536\n\n\n\nInitialize connection to Pinecone, you can get a free API key in the [Pinecone dashboard](https://app.pinecone.io).\n\n\n```python\nfrom pinecone import Pinecone\n\npc = Pinecone(api_key=\"...\")\n```\n\n\n\n\n```python\nimport time\nfrom pinecone import ServerlessSpec\n\nspec = ServerlessSpec(cloud=\"aws\", region=\"us-west-2\")\n\nindex_name = 'semantic-search-openai'\n\n# check if index already exists (if shouldn't if this is your first run)\nif index_name not in pc.list_indexes().names():\n    # if does not exist, create index\n    pc.create_index(\n        index_name,\n        dimension=len(embeds[0]),  # dimensionality of text-embed-3-small\n        metric='dotproduct',\n        spec=spec\n    )\n    # wait for index to be initialized\n    while not pc.describe_index(index_name).status['ready']:\n        time.sleep(1)\n\n# connect to index\nindex = pc.Index(index_name)\ntime.sleep(1)\n# view index stats\nindex.describe_index_stats()\n```\n\n\n\n\n    {'dimension': 1536,\n     'index_fullness': 0.0,\n     'namespaces': {},\n     'total_vector_count': 0}\n\n\n\n## Populating the Index\n\nNow we will take 1K questions from the TREC dataset\n\n\n```python\nfrom datasets import load_dataset\n\n# load the first 1K rows of the TREC dataset\ntrec = load_dataset('trec', split='train[:1000]')\ntrec\n```\n\n    /usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:88: UserWarning: \n    The secret `HF_TOKEN` does not exist in your Colab secrets.\n    To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n    You will be able to reuse this secret in all of your notebooks.\n    Please note that authentication is recommended but still optional to access public models or datasets.\n      warnings.warn(\n\n\n\n    Downloading data:   0%|          | 0.00/213k [00:00<?, ?B/s]\n\n\n\n    Downloading data:   0%|          | 0.00/17.1k [00:00<?, ?B/s]\n\n\n\n    Generating train split:   0%|          | 0/5452 [00:00<?, ? examples/s]\n\n\n\n    Generating test split:   0%|          | 0/500 [00:00<?, ? examples/s]\n\n\n\n\n\n    Dataset({\n        features: ['text', 'coarse_label', 'fine_label'],\n        num_rows: 1000\n    })\n\n\n\n\n```python\ntrec[0]\n```\n\n\n\n\n    {'text': 'How did serfdom develop in and then leave Russia ?',\n     'coarse_label': 2,\n     'fine_label': 26}\n\n\n\nThen we create a vector embedding for each phrase using OpenAI, and `upsert` the ID, vector embedding, and original text for each phrase to Pinecone."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 6fd2f4b0-2e51-4ffb-b13a-6ad078ca4bab
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"6fd2f4b0-2e51-4ffb-b13a-6ad078ca4bab\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb\", \"title\": \"Semantic Search with Pinecone and OpenAI\", \"tokens\": 34426, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ee5c68d1-dff6-4673-87a2-e097be084ab5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb\", \"title\": \"Semantic Search with Pinecone and OpenAI\", \"tokens\": 34426, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"a1034ea29a411ebb75c0f53708206ad7a93cc556d4389577ff0ec654637ecfd2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4b42b653-2bf6-4c5c-b240-3f1dfe509ced\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb\", \"title\": \"Semantic Search with Pinecone and OpenAI\", \"tokens\": 34426, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"ea47a73d2304b9d754eacf20a4ff01392203f857f8e9022fb9324328df5f381e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"aa94707f-e406-441f-91c2-873875be9b7d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"69d5087af3cc2eb5e74b6318ec6bca788889aa5cf20d869c5c43d845eb58cd1a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 73819, \"end_char_idx\": 76542, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ee5c68d1-dff6-4673-87a2-e097be084ab5", "doc_id": "ee5c68d1-dff6-4673-87a2-e097be084ab5", "ref_doc_id": "ee5c68d1-dff6-4673-87a2-e097be084ab5", "chroma:document": "```python\nfrom datasets import load_dataset\n\n# load the first 1K rows of the TREC dataset\ntrec = load_dataset('trec', split='train[:1000]')\ntrec\n```\n\n    /usr/local/lib/python3.10/dist-packages/huggingface_hub/utils/_token.py:88: UserWarning: \n    The secret `HF_TOKEN` does not exist in your Colab secrets.\n    To authenticate with the Hugging Face Hub, create a token in your settings tab (https://huggingface.co/settings/tokens), set it as secret in your Google Colab and restart your session.\n    You will be able to reuse this secret in all of your notebooks.\n    Please note that authentication is recommended but still optional to access public models or datasets.\n      warnings.warn(\n\n\n\n    Downloading data:   0%|          | 0.00/213k [00:00<?, ?B/s]\n\n\n\n    Downloading data:   0%|          | 0.00/17.1k [00:00<?, ?B/s]\n\n\n\n    Generating train split:   0%|          | 0/5452 [00:00<?, ? examples/s]\n\n\n\n    Generating test split:   0%|          | 0/500 [00:00<?, ? examples/s]\n\n\n\n\n\n    Dataset({\n        features: ['text', 'coarse_label', 'fine_label'],\n        num_rows: 1000\n    })\n\n\n\n\n```python\ntrec[0]\n```\n\n\n\n\n    {'text': 'How did serfdom develop in and then leave Russia ?',\n     'coarse_label': 2,\n     'fine_label': 26}\n\n\n\nThen we create a vector embedding for each phrase using OpenAI, and `upsert` the ID, vector embedding, and original text for each phrase to Pinecone.\n\n\n```python\nfrom tqdm.auto import tqdm\n\ncount = 0  # we'll use the count to create unique IDs\nbatch_size = 32  # process everything in batches of 32\nfor i in tqdm(range(0, len(trec['text']), batch_size)):\n    # set end position of batch\n    i_end = min(i+batch_size, len(trec['text']))\n    # get batch of lines and IDs\n    lines_batch = trec['text'][i: i+batch_size]\n    ids_batch = [str(n) for n in range(i, i_end)]\n    # create embeddings\n    res = client.embeddings.create(input=lines_batch, model=MODEL)\n    embeds = [record.embedding for record in res.data]\n    # prep metadata and upsert batch\n    meta = [{'text': line} for line in lines_batch]\n    to_upsert = zip(ids_batch, embeds, meta)\n    # upsert to Pinecone\n    index.upsert(vectors=list(to_upsert))\n```\n\n\n      0%|          | 0/32 [00:00<?, ?it/s]\n\n\n---\n\n# Querying\n\nWith our data indexed, we're now ready to move onto performing searches. This follows a similar process to indexing. We start with a text `query`, that we would like to use to find similar sentences. As before we encode this with OpenAI's text similarity Babbage model to create a *query vector* `xq`. We then use `xq` to query the Pinecone index.\n\n\n```python\nquery = \"What caused the 1929 Great Depression?\"\n\nxq = client.embeddings.create(input=query, model=MODEL).data[0].embedding\n```\n\nNow query..."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: aa94707f-e406-441f-91c2-873875be9b7d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"aa94707f-e406-441f-91c2-873875be9b7d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb\", \"title\": \"Semantic Search with Pinecone and OpenAI\", \"tokens\": 34426, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ee5c68d1-dff6-4673-87a2-e097be084ab5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb\", \"title\": \"Semantic Search with Pinecone and OpenAI\", \"tokens\": 34426, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"a1034ea29a411ebb75c0f53708206ad7a93cc556d4389577ff0ec654637ecfd2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6fd2f4b0-2e51-4ffb-b13a-6ad078ca4bab\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb\", \"title\": \"Semantic Search with Pinecone and OpenAI\", \"tokens\": 34426, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"278ac09c6f5f743fe10b6957794c6dc488b1f96463c664c0109947c996f4390f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"855aafe8-9cc8-4bf9-8aec-6af3cd5a9f09\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a4fae49221d2e416814407e8ac90d84d6e9c874cd0d949c7ce6b489fe80f8e76\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 75059, \"end_char_idx\": 77942, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ee5c68d1-dff6-4673-87a2-e097be084ab5", "doc_id": "ee5c68d1-dff6-4673-87a2-e097be084ab5", "ref_doc_id": "ee5c68d1-dff6-4673-87a2-e097be084ab5", "chroma:document": "Then we create a vector embedding for each phrase using OpenAI, and `upsert` the ID, vector embedding, and original text for each phrase to Pinecone.\n\n\n```python\nfrom tqdm.auto import tqdm\n\ncount = 0  # we'll use the count to create unique IDs\nbatch_size = 32  # process everything in batches of 32\nfor i in tqdm(range(0, len(trec['text']), batch_size)):\n    # set end position of batch\n    i_end = min(i+batch_size, len(trec['text']))\n    # get batch of lines and IDs\n    lines_batch = trec['text'][i: i+batch_size]\n    ids_batch = [str(n) for n in range(i, i_end)]\n    # create embeddings\n    res = client.embeddings.create(input=lines_batch, model=MODEL)\n    embeds = [record.embedding for record in res.data]\n    # prep metadata and upsert batch\n    meta = [{'text': line} for line in lines_batch]\n    to_upsert = zip(ids_batch, embeds, meta)\n    # upsert to Pinecone\n    index.upsert(vectors=list(to_upsert))\n```\n\n\n      0%|          | 0/32 [00:00<?, ?it/s]\n\n\n---\n\n# Querying\n\nWith our data indexed, we're now ready to move onto performing searches. This follows a similar process to indexing. We start with a text `query`, that we would like to use to find similar sentences. As before we encode this with OpenAI's text similarity Babbage model to create a *query vector* `xq`. We then use `xq` to query the Pinecone index.\n\n\n```python\nquery = \"What caused the 1929 Great Depression?\"\n\nxq = client.embeddings.create(input=query, model=MODEL).data[0].embedding\n```\n\nNow query...\n\n\n```python\nres = index.query(vector=[xq], top_k=5, include_metadata=True)\nres\n```\n\n\n\n\n    {'matches': [{'id': '932',\n                  'metadata': {'text': 'Why did the world enter a global '\n                                       'depression in 1929 ?'},\n                  'score': 0.751888752,\n                  'values': []},\n                 {'id': '787',\n                  'metadata': {'text': \"When was `` the Great Depression '' ?\"},\n                  'score': 0.597448647,\n                  'values': []},\n                 {'id': '400',\n                  'metadata': {'text': 'What crop failure caused the Irish Famine '\n                                       '?'},\n                  'score': 0.367482603,\n                  'values': []},\n                 {'id': '835',\n                  'metadata': {'text': 'What were popular songs and types of songs '\n                                       'in the 1920s ?'},\n                  'score': 0.324545294,\n                  'values': []},\n                 {'id': '262',\n                  'metadata': {'text': 'When did World War I start ?'},\n                  'score': 0.320995867,\n                  'values': []}],\n     'namespace': '',\n     'usage': {'read_units': 6}}\n\n\n\nThe response from Pinecone includes our original text in the `metadata` field, let's print out the `top_k` most similar questions and their respective similarity scores."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 855aafe8-9cc8-4bf9-8aec-6af3cd5a9f09
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"855aafe8-9cc8-4bf9-8aec-6af3cd5a9f09\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb\", \"title\": \"Semantic Search with Pinecone and OpenAI\", \"tokens\": 34426, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ee5c68d1-dff6-4673-87a2-e097be084ab5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb\", \"title\": \"Semantic Search with Pinecone and OpenAI\", \"tokens\": 34426, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"a1034ea29a411ebb75c0f53708206ad7a93cc556d4389577ff0ec654637ecfd2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"aa94707f-e406-441f-91c2-873875be9b7d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb\", \"title\": \"Semantic Search with Pinecone and OpenAI\", \"tokens\": 34426, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"3339e0f69ada4d845ef6bfec822835c7b5f0dac4652052fd49c786cf2402672b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f209db53-2200-461b-80e4-1329fef75b08\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5013e1c8c1c5943f09b53b1b236b4fa872689bf4b7fc03a3fc3469757c99bb42\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 76391, \"end_char_idx\": 79175, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ee5c68d1-dff6-4673-87a2-e097be084ab5", "doc_id": "ee5c68d1-dff6-4673-87a2-e097be084ab5", "ref_doc_id": "ee5c68d1-dff6-4673-87a2-e097be084ab5", "chroma:document": "```python\nquery = \"What caused the 1929 Great Depression?\"\n\nxq = client.embeddings.create(input=query, model=MODEL).data[0].embedding\n```\n\nNow query...\n\n\n```python\nres = index.query(vector=[xq], top_k=5, include_metadata=True)\nres\n```\n\n\n\n\n    {'matches': [{'id': '932',\n                  'metadata': {'text': 'Why did the world enter a global '\n                                       'depression in 1929 ?'},\n                  'score': 0.751888752,\n                  'values': []},\n                 {'id': '787',\n                  'metadata': {'text': \"When was `` the Great Depression '' ?\"},\n                  'score': 0.597448647,\n                  'values': []},\n                 {'id': '400',\n                  'metadata': {'text': 'What crop failure caused the Irish Famine '\n                                       '?'},\n                  'score': 0.367482603,\n                  'values': []},\n                 {'id': '835',\n                  'metadata': {'text': 'What were popular songs and types of songs '\n                                       'in the 1920s ?'},\n                  'score': 0.324545294,\n                  'values': []},\n                 {'id': '262',\n                  'metadata': {'text': 'When did World War I start ?'},\n                  'score': 0.320995867,\n                  'values': []}],\n     'namespace': '',\n     'usage': {'read_units': 6}}\n\n\n\nThe response from Pinecone includes our original text in the `metadata` field, let's print out the `top_k` most similar questions and their respective similarity scores.\n\n\n```python\nfor match in res['matches']:\n    print(f\"{match['score']:.2f}: {match['metadata']['text']}\")\n```\n\n    0.75: Why did the world enter a global depression in 1929 ?\n    0.60: When was `` the Great Depression '' ?\n    0.37: What crop failure caused the Irish Famine ?\n    0.32: What were popular songs and types of songs in the 1920s ?\n    0.32: When did World War I start ?\n\n\nLooks good, let's make it harder and replace *\"depression\"* with the incorrect term *\"recession\"*.\n\n\n```python\nquery = \"What was the cause of the major recession in the early 20th century?\"\n\n# create the query embedding\nxq = client.embeddings.create(input=query, model=MODEL).data[0].embedding\n\n# query, returning the top 5 most similar results\nres = index.query(vector=[xq], top_k=5, include_metadata=True)\n\nfor match in res['matches']:\n    print(f\"{match['score']:.2f}: {match['metadata']['text']}\")\n```\n\n    0.63: Why did the world enter a global depression in 1929 ?\n    0.55: When was `` the Great Depression '' ?\n    0.34: What were popular songs and types of songs in the 1920s ?\n    0.33: What crop failure caused the Irish Famine ?\n    0.29: What is considered the costliest disaster the insurance industry has ever faced ?\n\n\nAnd again..."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: f209db53-2200-461b-80e4-1329fef75b08
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"f209db53-2200-461b-80e4-1329fef75b08\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb\", \"title\": \"Semantic Search with Pinecone and OpenAI\", \"tokens\": 34426, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ee5c68d1-dff6-4673-87a2-e097be084ab5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb\", \"title\": \"Semantic Search with Pinecone and OpenAI\", \"tokens\": 34426, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"a1034ea29a411ebb75c0f53708206ad7a93cc556d4389577ff0ec654637ecfd2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"855aafe8-9cc8-4bf9-8aec-6af3cd5a9f09\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb\", \"title\": \"Semantic Search with Pinecone and OpenAI\", \"tokens\": 34426, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e5765fd4a76ea84c6ed54c881e3f3657c5f4c90d934bf06b8173bc866e7bb24c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 77773, \"end_char_idx\": 80169, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ee5c68d1-dff6-4673-87a2-e097be084ab5", "doc_id": "ee5c68d1-dff6-4673-87a2-e097be084ab5", "ref_doc_id": "ee5c68d1-dff6-4673-87a2-e097be084ab5", "chroma:document": "The response from Pinecone includes our original text in the `metadata` field, let's print out the `top_k` most similar questions and their respective similarity scores.\n\n\n```python\nfor match in res['matches']:\n    print(f\"{match['score']:.2f}: {match['metadata']['text']}\")\n```\n\n    0.75: Why did the world enter a global depression in 1929 ?\n    0.60: When was `` the Great Depression '' ?\n    0.37: What crop failure caused the Irish Famine ?\n    0.32: What were popular songs and types of songs in the 1920s ?\n    0.32: When did World War I start ?\n\n\nLooks good, let's make it harder and replace *\"depression\"* with the incorrect term *\"recession\"*.\n\n\n```python\nquery = \"What was the cause of the major recession in the early 20th century?\"\n\n# create the query embedding\nxq = client.embeddings.create(input=query, model=MODEL).data[0].embedding\n\n# query, returning the top 5 most similar results\nres = index.query(vector=[xq], top_k=5, include_metadata=True)\n\nfor match in res['matches']:\n    print(f\"{match['score']:.2f}: {match['metadata']['text']}\")\n```\n\n    0.63: Why did the world enter a global depression in 1929 ?\n    0.55: When was `` the Great Depression '' ?\n    0.34: What were popular songs and types of songs in the 1920s ?\n    0.33: What crop failure caused the Irish Famine ?\n    0.29: What is considered the costliest disaster the insurance industry has ever faced ?\n\n\nAnd again...\n\n\n```python\nquery = \"Why was there a long-term economic downturn in the early 20th century?\"\n\n# create the query embedding\nxq = client.embeddings.create(input=query, model=MODEL).data[0].embedding\n\n# query, returning the top 5 most similar results\nres = index.query(vector=[xq], top_k=5, include_metadata=True)\n\nfor match in res['matches']:\n    print(f\"{match['score']:.2f}: {match['metadata']['text']}\")\n```\n\n    0.62: Why did the world enter a global depression in 1929 ?\n    0.54: When was `` the Great Depression '' ?\n    0.34: What were popular songs and types of songs in the 1920s ?\n    0.33: What crop failure caused the Irish Famine ?\n    0.32: What do economists do ?\n\n\nLooks great, our semantic search pipeline is clearly able to identify the meaning between each of our queries and return the most semantically similar questions from the already indexed questions.\n\nOnce we're finished with the index we delete it to save resources.\n\n\n```python\npc.delete_index(index_name)\n```\n\n---"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 520e44c3-18a7-4e43-879f-fa4320920e8d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/README.ipynb", "title": "Pinecone Vector Database", "tokens": 512, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"520e44c3-18a7-4e43-879f-fa4320920e8d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/README.ipynb\", \"title\": \"Pinecone Vector Database\", \"tokens\": 512, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"05b639b1-40e5-4000-b2c1-7c7c3e7f3e5d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/README.ipynb\", \"title\": \"Pinecone Vector Database\", \"tokens\": 512, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"cbeea1136175c986e003b8ae7f67d3e2de290c94c9b69ee9542983ca1ad10e24\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2245, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "05b639b1-40e5-4000-b2c1-7c7c3e7f3e5d", "doc_id": "05b639b1-40e5-4000-b2c1-7c7c3e7f3e5d", "ref_doc_id": "05b639b1-40e5-4000-b2c1-7c7c3e7f3e5d", "chroma:document": "# Pinecone Vector Database\n\n[Vector search](https://www.pinecone.io/learn/vector-search-basics/) is an innovative technology that enables developers and engineers to efficiently store, search, and recommend information by representing complex data as mathematical vectors. By comparing the similarities between these vectors, you can quickly retrieve relevant information in a seamless and intuitive manner.\n\n[Pinecone](https://pinecone.io/) is a [vector database](https://www.pinecone.io/learn/vector-database/) designed with developers and engineers in mind. As a managed service, it alleviates the burden of maintenance and engineering, allowing you to focus on extracting valuable insights from your data. The free tier supports up to 5 million vectors, making it an accessible and cost-effective way to experiment with vector search capabilities. With Pinecone, you'll experience impressive speed, accuracy, and scalability, as well as access to advanced features like single-stage metadata filtering and the cutting-edge sparse-dense index.\n\n## Examples\n\nThis folder contains examples of using Pinecone and OpenAI together. More will be added over time so check back for updates!\n\n| Name | Description | Google Colab |\n| --- | --- | --- |\n| [GPT-4 Retrieval Augmentation](./GPT4_Retrieval_Augmentation.ipynb) | How to supercharge GPT-4 with retrieval augmentation | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openai/openai-cookbook/blob/master/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb) |\n| [Generative Question-Answering](./Gen_QA.ipynb) | A simple walkthrough demonstrating the use of Generative Question-Answering | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openai/openai-cookbook/blob/master/examples/vector_databases/pinecone/Gen_QA.ipynb) |\n| [Semantic Search](./Semantic_Search.ipynb) | A guide to building a simple semantic search process | [![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openai/openai-cookbook/blob/master/examples/vector_databases/pinecone/Semantic_Search.ipynb) |"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: d4963270-1b49-4e43-a3b9-b76f237591c5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb", "title": "Using Pinecone for Embeddings Search", "tokens": 4156, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"d4963270-1b49-4e43-a3b9-b76f237591c5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ba6c4445-95d6-49ac-9867-b741ab0ec83e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"2ac8f36d192668244fd51cf18c2733bc66d5a3a822d68cf42df37b788b20c210\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c6081771-6318-4f77-af9e-8c9a3ff25906\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"373a4a79d637360e114455f80825418f3ef64c0ccdbe8d98290205b3d8f7581b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3350, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "doc_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "ref_doc_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "chroma:document": "# Using Pinecone for Embeddings Search\n\nThis notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.\n\n### What is a Vector Database\n\nA vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.\n\n### Why use a Vector Database\n\nVector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.\n\n\n### Demo Flow\nThe demo flow is:\n- **Setup**: Import packages and set any required variables\n- **Load data**: Load a dataset and embed it using OpenAI embeddings\n- **Pinecone**\n    - *Setup*: Here we'll set up the Python client for Pinecone. For more details go [here](https://docs.pinecone.io/docs/quickstart)\n    - *Index Data*: We'll create an index with namespaces for __titles__ and __content__\n    - *Search Data*: We'll test out both namespaces with search queries to confirm it works\n\nOnce you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.\n\n## Setup\n\nImport the required libraries and set the embedding model that we'd like to use.\n\n\n```python\n# We'll need to install the Pinecone client\n!pip install pinecone-client\n\n#Install wget to pull zip file\n!pip install wget\n```\n\n    Requirement already satisfied: pinecone-client in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (2.2.2)\n    Requirement already satisfied: requests>=2.19.0 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (2.31.0)\n    Requirement already satisfied: pyyaml>=5.4 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (6.0)\n    Requirement already satisfied: loguru>=0.5.0 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (0.7.0)\n    Requirement already satisfied: typing-extensions>=3.7.4 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (4.5."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: c6081771-6318-4f77-af9e-8c9a3ff25906
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb", "title": "Using Pinecone for Embeddings Search", "tokens": 4156, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c6081771-6318-4f77-af9e-8c9a3ff25906\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ba6c4445-95d6-49ac-9867-b741ab0ec83e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"2ac8f36d192668244fd51cf18c2733bc66d5a3a822d68cf42df37b788b20c210\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d4963270-1b49-4e43-a3b9-b76f237591c5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"d5347f41af7862c7ee23ce5c99e353d5d576b409597ff07456a89957f8b7a425\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2cc458db-7d12-44a2-91af-bb63c78759b6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"30fc4eddb067f979f28f1d146c041365475c760fa5524650a25e2e696d6e274a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2320, \"end_char_idx\": 4952, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "doc_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "ref_doc_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "chroma:document": "```python\n# We'll need to install the Pinecone client\n!pip install pinecone-client\n\n#Install wget to pull zip file\n!pip install wget\n```\n\n    Requirement already satisfied: pinecone-client in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (2.2.2)\n    Requirement already satisfied: requests>=2.19.0 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (2.31.0)\n    Requirement already satisfied: pyyaml>=5.4 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (6.0)\n    Requirement already satisfied: loguru>=0.5.0 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (0.7.0)\n    Requirement already satisfied: typing-extensions>=3.7.4 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (4.5.0)\n    Requirement already satisfied: dnspython>=2.0.0 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (2.3.0)\n    Requirement already satisfied: python-dateutil>=2.5.3 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (2.8.2)\n    Requirement already satisfied: urllib3>=1.21.1 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (1.26.16)\n    Requirement already satisfied: tqdm>=4.64.1 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (4.65.0)\n    Requirement already satisfied: numpy>=1.22.0 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (1.25.0)\n    Requirement already satisfied: six>=1.5 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from python-dateutil>=2.5.3->pinecone-client) (1.16.0)\n    Requirement already satisfied: charset-normalizer<4,>=2 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from requests>=2.19.0->pinecone-client) (3.1.0)\n    Requirement already satisfied: idna<4,>=2.5 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from requests>=2.19.0->pinecone-client) (3.4)\n    Requirement already satisfied: certifi>=2017.4.17 in /Users/colin."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 2cc458db-7d12-44a2-91af-bb63c78759b6
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb", "title": "Using Pinecone for Embeddings Search", "tokens": 4156, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"2cc458db-7d12-44a2-91af-bb63c78759b6\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ba6c4445-95d6-49ac-9867-b741ab0ec83e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"2ac8f36d192668244fd51cf18c2733bc66d5a3a822d68cf42df37b788b20c210\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c6081771-6318-4f77-af9e-8c9a3ff25906\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3cbd93896676d5d780bcdf37183f2ae720a858a8e28d026260dbbffa4e0ffd64\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"53337d01-59cc-490c-965c-a2d11abb6ae3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fe7bcc734458c108b3d69987284c9221a52fa010996faf22ef0f978c06951ae2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3611, \"end_char_idx\": 6301, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "doc_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "ref_doc_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "chroma:document": "jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (2.8.2)\n    Requirement already satisfied: urllib3>=1.21.1 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (1.26.16)\n    Requirement already satisfied: tqdm>=4.64.1 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (4.65.0)\n    Requirement already satisfied: numpy>=1.22.0 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from pinecone-client) (1.25.0)\n    Requirement already satisfied: six>=1.5 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from python-dateutil>=2.5.3->pinecone-client) (1.16.0)\n    Requirement already satisfied: charset-normalizer<4,>=2 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from requests>=2.19.0->pinecone-client) (3.1.0)\n    Requirement already satisfied: idna<4,>=2.5 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from requests>=2.19.0->pinecone-client) (3.4)\n    Requirement already satisfied: certifi>=2017.4.17 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from requests>=2.19.0->pinecone-client) (2023.5.7)\n    Requirement already satisfied: wget in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (3.2)\n\n\n\n```python\nimport openai\n\nfrom typing import List, Iterator\nimport pandas as pd\nimport numpy as np\nimport os\nimport wget\nfrom ast import literal_eval\n\n# Pinecone's client library for Python\nimport pinecone\n\n# I've set this to our new embeddings model, this can be changed to the embedding model of your choice\nEMBEDDING_MODEL = \"text-embedding-3-small\"\n\n# Ignore unclosed SSL socket warnings - optional in case you get these errors\nimport warnings\n\nwarnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning) \n```\n\n    /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages/pinecone/index.py:4: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n      from tqdm.autonotebook import tqdm\n\n\n## Load data\n\nIn this section we'll load embedded data that we've prepared [in this article](../../Embedding_Wikipedia_articles_for_search.ipynb)."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 53337d01-59cc-490c-965c-a2d11abb6ae3
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb", "title": "Using Pinecone for Embeddings Search", "tokens": 4156, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"53337d01-59cc-490c-965c-a2d11abb6ae3\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ba6c4445-95d6-49ac-9867-b741ab0ec83e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"2ac8f36d192668244fd51cf18c2733bc66d5a3a822d68cf42df37b788b20c210\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2cc458db-7d12-44a2-91af-bb63c78759b6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"dd45f53a7f844a9495099fe21ffdc480a62525d1b575b9831e1cf584d2f1fef9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c240d86d-641f-47d3-b5b1-5a0ba150db5f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"69b9510b39e14cd0a20c0febcd50d90c9ed37992b3fddd1c8242ac62a4813379\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4818, \"end_char_idx\": 6793, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "doc_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "ref_doc_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "chroma:document": "10/site-packages (from requests>=2.19.0->pinecone-client) (3.4)\n    Requirement already satisfied: certifi>=2017.4.17 in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (from requests>=2.19.0->pinecone-client) (2023.5.7)\n    Requirement already satisfied: wget in /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages (3.2)\n\n\n\n```python\nimport openai\n\nfrom typing import List, Iterator\nimport pandas as pd\nimport numpy as np\nimport os\nimport wget\nfrom ast import literal_eval\n\n# Pinecone's client library for Python\nimport pinecone\n\n# I've set this to our new embeddings model, this can be changed to the embedding model of your choice\nEMBEDDING_MODEL = \"text-embedding-3-small\"\n\n# Ignore unclosed SSL socket warnings - optional in case you get these errors\nimport warnings\n\nwarnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning) \n```\n\n    /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages/pinecone/index.py:4: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n      from tqdm.autonotebook import tqdm\n\n\n## Load data\n\nIn this section we'll load embedded data that we've prepared [in this article](../../Embedding_Wikipedia_articles_for_search.ipynb).\n\n\n```python\nembeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'\n\n# The file is ~700 MB so this will take some time\nwget.download(embeddings_url)\n```\n\n\n```python\nimport zipfile\nwith zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n    zip_ref.extractall(\"../data\")\n```\n\n\n```python\narticle_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')\n```\n\n\n```python\narticle_df.head()\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: c240d86d-641f-47d3-b5b1-5a0ba150db5f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb", "title": "Using Pinecone for Embeddings Search", "tokens": 4156, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c240d86d-641f-47d3-b5b1-5a0ba150db5f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ba6c4445-95d6-49ac-9867-b741ab0ec83e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"2ac8f36d192668244fd51cf18c2733bc66d5a3a822d68cf42df37b788b20c210\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"53337d01-59cc-490c-965c-a2d11abb6ae3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"1bb59fa2b8a4888ceea85a96ea7a8f0d5cc7e7b4d6a728594e6c7dde6ee6cf8a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d61afd84-b088-4d38-82d2-ae45a67b2b29\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"55d75d734c6c465942a4bbe99c9e2550ca1dde90d5af183d906f3fc23ae149c3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5213, \"end_char_idx\": 7809, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "doc_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "ref_doc_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "chroma:document": "10/site-packages (3.2)\n\n\n\n```python\nimport openai\n\nfrom typing import List, Iterator\nimport pandas as pd\nimport numpy as np\nimport os\nimport wget\nfrom ast import literal_eval\n\n# Pinecone's client library for Python\nimport pinecone\n\n# I've set this to our new embeddings model, this can be changed to the embedding model of your choice\nEMBEDDING_MODEL = \"text-embedding-3-small\"\n\n# Ignore unclosed SSL socket warnings - optional in case you get these errors\nimport warnings\n\nwarnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning) \n```\n\n    /Users/colin.jarvis/Documents/dev/cookbook/openai-cookbook/vector_db/lib/python3.10/site-packages/pinecone/index.py:4: TqdmExperimentalWarning: Using `tqdm.autonotebook.tqdm` in notebook mode. Use `tqdm.tqdm` instead to force console mode (e.g. in jupyter console)\n      from tqdm.autonotebook import tqdm\n\n\n## Load data\n\nIn this section we'll load embedded data that we've prepared [in this article](../../Embedding_Wikipedia_articles_for_search.ipynb).\n\n\n```python\nembeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'\n\n# The file is ~700 MB so this will take some time\nwget.download(embeddings_url)\n```\n\n\n```python\nimport zipfile\nwith zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n    zip_ref.extractall(\"../data\")\n```\n\n\n```python\narticle_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')\n```\n\n\n```python\narticle_df.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>url</th>\n      <th>title</th>\n      <th>text</th>\n      <th>title_vector</th>\n      <th>content_vector</th>\n      <th>vector_id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>https://simple.wikipedia.org/wiki/April</td>\n      <td>April</td>\n      <td>April is the fourth month of the year in the J...</td>\n      <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n      <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>https://simple.wikipedia.org/wiki/August</td>\n      <td>August</td>\n      <td>August (Aug.)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: d61afd84-b088-4d38-82d2-ae45a67b2b29
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb", "title": "Using Pinecone for Embeddings Search", "tokens": 4156, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"d61afd84-b088-4d38-82d2-ae45a67b2b29\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ba6c4445-95d6-49ac-9867-b741ab0ec83e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"2ac8f36d192668244fd51cf18c2733bc66d5a3a822d68cf42df37b788b20c210\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c240d86d-641f-47d3-b5b1-5a0ba150db5f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6dfb04795d1c725f8d109e534e96f94377afed63c671eabe0f2735c2e3b707b8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"78f1379f-48ba-4b97-8954-abd467a4fbb9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0eb9f1fbcad7c87e0ae1cc80d55f22c888212d66f0901dbcd88725f848899c3c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6662, \"end_char_idx\": 8840, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "doc_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "ref_doc_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "chroma:document": "```python\narticle_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')\n```\n\n\n```python\narticle_df.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>url</th>\n      <th>title</th>\n      <th>text</th>\n      <th>title_vector</th>\n      <th>content_vector</th>\n      <th>vector_id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>https://simple.wikipedia.org/wiki/April</td>\n      <td>April</td>\n      <td>April is the fourth month of the year in the J...</td>\n      <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n      <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>https://simple.wikipedia.org/wiki/August</td>\n      <td>August</td>\n      <td>August (Aug.) is the eighth month of the year ...</td>\n      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>6</td>\n      <td>https://simple.wikipedia.org/wiki/Art</td>\n      <td>Art</td>\n      <td>Art is a creative activity that expresses imag...</td>\n      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>8</td>\n      <td>https://simple.wikipedia.org/wiki/A</td>\n      <td>A</td>\n      <td>A or a is the first letter of the English alph...</td>\n      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n      <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>9</td>\n      <td>https://simple.wikipedia.org/wiki/Air</td>\n      <td>Air</td>\n      <td>Air refers to the Earth's atmosphere."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 78f1379f-48ba-4b97-8954-abd467a4fbb9
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb", "title": "Using Pinecone for Embeddings Search", "tokens": 4156, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"78f1379f-48ba-4b97-8954-abd467a4fbb9\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ba6c4445-95d6-49ac-9867-b741ab0ec83e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"2ac8f36d192668244fd51cf18c2733bc66d5a3a822d68cf42df37b788b20c210\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d61afd84-b088-4d38-82d2-ae45a67b2b29\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"74fef6fc1b371128b1fb94f46dcf1bac36bec462800fd464396c9d567df91a32\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"927a3975-3461-4a8a-95e3-331ff6596bce\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c804575cf2a9555283227fedab524b65b0f65965e42b43c7f8e78d26607e6b7c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7810, \"end_char_idx\": 10008, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "doc_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "ref_doc_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "chroma:document": "is the eighth month of the year ...</td>\n      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>6</td>\n      <td>https://simple.wikipedia.org/wiki/Art</td>\n      <td>Art</td>\n      <td>Art is a creative activity that expresses imag...</td>\n      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>8</td>\n      <td>https://simple.wikipedia.org/wiki/A</td>\n      <td>A</td>\n      <td>A or a is the first letter of the English alph...</td>\n      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n      <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>9</td>\n      <td>https://simple.wikipedia.org/wiki/Air</td>\n      <td>Air</td>\n      <td>Air refers to the Earth's atmosphere. Air is a...</td>\n      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n      <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n\n```python\n# Read vectors from strings back into a list\narticle_df['title_vector'] = article_df.title_vector.apply(literal_eval)\narticle_df['content_vector'] = article_df.content_vector.apply(literal_eval)\n\n# Set vector_id to be a string\narticle_df['vector_id'] = article_df['vector_id'].apply(str)\n```\n\n\n```python\narticle_df.info(show_counts=True)\n```\n\n    <class 'pandas.core.frame.DataFrame'>\n    RangeIndex: 25000 entries, 0 to 24999\n    Data columns (total 7 columns):\n     #   Column          Non-Null Count  Dtype \n    ---  ------          --------------  ----- \n     0   id              25000 non-null  int64 \n     1   url             25000 non-null  object\n     2   title           25000 non-null  object\n     3   text            25000 non-null  object\n     4   title_vector    25000 non-null  object\n     5   content_vector  25000 non-null  object\n     6   vector_id       25000 non-null  object\n    dtypes: int64(1), object(6)\n    memory usage: 1.3+ MB"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 927a3975-3461-4a8a-95e3-331ff6596bce
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb", "title": "Using Pinecone for Embeddings Search", "tokens": 4156, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"927a3975-3461-4a8a-95e3-331ff6596bce\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ba6c4445-95d6-49ac-9867-b741ab0ec83e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"2ac8f36d192668244fd51cf18c2733bc66d5a3a822d68cf42df37b788b20c210\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"78f1379f-48ba-4b97-8954-abd467a4fbb9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ae95784672a96f287e43885b343c43f1110d28a6d267c792edb25d01e4fb794b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"587434f0-29f4-4187-83ec-adc30982dad7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a378cffb227c89cc8a6212ec668b8fddf15ba9ed117c4970f678783822c355fa\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8841, \"end_char_idx\": 11318, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "doc_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "ref_doc_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "chroma:document": "Air is a...</td>\n      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n      <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n\n```python\n# Read vectors from strings back into a list\narticle_df['title_vector'] = article_df.title_vector.apply(literal_eval)\narticle_df['content_vector'] = article_df.content_vector.apply(literal_eval)\n\n# Set vector_id to be a string\narticle_df['vector_id'] = article_df['vector_id'].apply(str)\n```\n\n\n```python\narticle_df.info(show_counts=True)\n```\n\n    <class 'pandas.core.frame.DataFrame'>\n    RangeIndex: 25000 entries, 0 to 24999\n    Data columns (total 7 columns):\n     #   Column          Non-Null Count  Dtype \n    ---  ------          --------------  ----- \n     0   id              25000 non-null  int64 \n     1   url             25000 non-null  object\n     2   title           25000 non-null  object\n     3   text            25000 non-null  object\n     4   title_vector    25000 non-null  object\n     5   content_vector  25000 non-null  object\n     6   vector_id       25000 non-null  object\n    dtypes: int64(1), object(6)\n    memory usage: 1.3+ MB\n\n\n## Pinecone\n\nThe next option we'll look at is **Pinecone**, a managed vector database which offers a cloud-native option.\n\nBefore you proceed with this step you'll need to navigate to [Pinecone](pinecone.io), sign up and then save your API key as an environment variable titled ```PINECONE_API_KEY```.\n\nFor section we will:\n- Create an index with multiple namespaces for article titles and content\n- Store our data in the index with separate searchable \"namespaces\" for article **titles** and **content**\n- Fire some similarity search queries to verify our setup is working\n\n\n```python\napi_key = os.getenv(\"PINECONE_API_KEY\")\npinecone.init(api_key=api_key)\n```\n\n### Create Index\n\nFirst we will need to create an index, which we'll call `wikipedia-articles`. Once we have an index, we can create multiple namespaces, which can make a single index searchable for various use cases. For more details, consult [Pinecone documentation](https://docs.pinecone.io/docs/namespaces#:~:text=Pinecone%20allows%20you%20to%20partition,different%20subsets%20of%20your%20index.).\n\nIf you want to batch insert to your index in parallel to increase insertion speed then there is a great guide in the Pinecone documentation on [batch inserts in parallel](https://docs.pinecone.io/docs/insert-data#sending-upserts-in-parallel)."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 587434f0-29f4-4187-83ec-adc30982dad7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb", "title": "Using Pinecone for Embeddings Search", "tokens": 4156, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"587434f0-29f4-4187-83ec-adc30982dad7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ba6c4445-95d6-49ac-9867-b741ab0ec83e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"2ac8f36d192668244fd51cf18c2733bc66d5a3a822d68cf42df37b788b20c210\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"927a3975-3461-4a8a-95e3-331ff6596bce\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"4532d0b86464f56c6ef1cf23ff0bacaede1703e1fe5c12a5962ed3b44a486768\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8db54537-57fc-4525-96db-883a81596772\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"88b2c4483d34537c9f78147ba874830172ec1bf61f3a669bd11d37ecd3e18f9f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10011, \"end_char_idx\": 13322, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "doc_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "ref_doc_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "chroma:document": "## Pinecone\n\nThe next option we'll look at is **Pinecone**, a managed vector database which offers a cloud-native option.\n\nBefore you proceed with this step you'll need to navigate to [Pinecone](pinecone.io), sign up and then save your API key as an environment variable titled ```PINECONE_API_KEY```.\n\nFor section we will:\n- Create an index with multiple namespaces for article titles and content\n- Store our data in the index with separate searchable \"namespaces\" for article **titles** and **content**\n- Fire some similarity search queries to verify our setup is working\n\n\n```python\napi_key = os.getenv(\"PINECONE_API_KEY\")\npinecone.init(api_key=api_key)\n```\n\n### Create Index\n\nFirst we will need to create an index, which we'll call `wikipedia-articles`. Once we have an index, we can create multiple namespaces, which can make a single index searchable for various use cases. For more details, consult [Pinecone documentation](https://docs.pinecone.io/docs/namespaces#:~:text=Pinecone%20allows%20you%20to%20partition,different%20subsets%20of%20your%20index.).\n\nIf you want to batch insert to your index in parallel to increase insertion speed then there is a great guide in the Pinecone documentation on [batch inserts in parallel](https://docs.pinecone.io/docs/insert-data#sending-upserts-in-parallel).\n\n\n```python\n# Models a simple batch generator that make chunks out of an input DataFrame\nclass BatchGenerator:\n    \n    \n    def __init__(self, batch_size: int = 10) -> None:\n        self.batch_size = batch_size\n    \n    # Makes chunks out of an input DataFrame\n    def to_batches(self, df: pd.DataFrame) -> Iterator[pd.DataFrame]:\n        splits = self.splits_num(df.shape[0])\n        if splits <= 1:\n            yield df\n        else:\n            for chunk in np.array_split(df, splits):\n                yield chunk\n\n    # Determines how many chunks DataFrame contains\n    def splits_num(self, elements: int) -> int:\n        return round(elements / self.batch_size)\n    \n    __call__ = to_batches\n\ndf_batcher = BatchGenerator(300)\n```\n\n\n```python\n# Pick a name for the new index\nindex_name = 'wikipedia-articles'\n\n# Check whether the index with the same name already exists - if so, delete it\nif index_name in pinecone.list_indexes():\n    pinecone.delete_index(index_name)\n    \n# Creates new index\npinecone.create_index(name=index_name, dimension=len(article_df['content_vector'][0]))\nindex = pinecone.Index(index_name=index_name)\n\n# Confirm our index was created\npinecone.list_indexes()\n```\n\n\n\n\n    ['podcasts', 'wikipedia-articles']\n\n\n\n\n```python\n# Upsert content vectors in content namespace - this can take a few minutes\nprint(\"Uploading vectors to content namespace..\")\nfor batch_df in df_batcher(article_df):\n    index.upsert(vectors=zip(batch_df.vector_id, batch_df.content_vector), namespace='content')\n```\n\n    Uploading vectors to content namespace..\n\n\n\n```python\n# Upsert title vectors in title namespace - this can also take a few minutes\nprint(\"Uploading vectors to title namespace..\")\nfor batch_df in df_batcher(article_df):\n    index.upsert(vectors=zip(batch_df.vector_id, batch_df.title_vector), namespace='title')\n```\n\n    Uploading vectors to title namespace..\n\n\n\n```python\n# Check index size for each namespace to confirm all of our docs have loaded\nindex.describe_index_stats()\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 8db54537-57fc-4525-96db-883a81596772
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb", "title": "Using Pinecone for Embeddings Search", "tokens": 4156, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"8db54537-57fc-4525-96db-883a81596772\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ba6c4445-95d6-49ac-9867-b741ab0ec83e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"2ac8f36d192668244fd51cf18c2733bc66d5a3a822d68cf42df37b788b20c210\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"587434f0-29f4-4187-83ec-adc30982dad7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6c7818bc112730ce8cb22deeee827ced00944475b80076447d88e63bad987ca7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5535880e-b6ad-493c-bafd-f6a2bf806e24\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1a7834183876003050708c032af3cb4596f1001b144c3e1e345778eebd14c495\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12058, \"end_char_idx\": 15251, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "doc_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "ref_doc_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "chroma:document": "```python\n# Pick a name for the new index\nindex_name = 'wikipedia-articles'\n\n# Check whether the index with the same name already exists - if so, delete it\nif index_name in pinecone.list_indexes():\n    pinecone.delete_index(index_name)\n    \n# Creates new index\npinecone.create_index(name=index_name, dimension=len(article_df['content_vector'][0]))\nindex = pinecone.Index(index_name=index_name)\n\n# Confirm our index was created\npinecone.list_indexes()\n```\n\n\n\n\n    ['podcasts', 'wikipedia-articles']\n\n\n\n\n```python\n# Upsert content vectors in content namespace - this can take a few minutes\nprint(\"Uploading vectors to content namespace..\")\nfor batch_df in df_batcher(article_df):\n    index.upsert(vectors=zip(batch_df.vector_id, batch_df.content_vector), namespace='content')\n```\n\n    Uploading vectors to content namespace..\n\n\n\n```python\n# Upsert title vectors in title namespace - this can also take a few minutes\nprint(\"Uploading vectors to title namespace..\")\nfor batch_df in df_batcher(article_df):\n    index.upsert(vectors=zip(batch_df.vector_id, batch_df.title_vector), namespace='title')\n```\n\n    Uploading vectors to title namespace..\n\n\n\n```python\n# Check index size for each namespace to confirm all of our docs have loaded\nindex.describe_index_stats()\n```\n\n\n\n\n    {'dimension': 1536,\n     'index_fullness': 0.1,\n     'namespaces': {'content': {'vector_count': 25000},\n                    'title': {'vector_count': 25000}},\n     'total_vector_count': 50000}\n\n\n\n### Search data\n\nNow we'll enter some dummy searches and check we get decent results back\n\n\n```python\n# First we'll create dictionaries mapping vector IDs to their outputs so we can retrieve the text for our search results\ntitles_mapped = dict(zip(article_df.vector_id,article_df.title))\ncontent_mapped = dict(zip(article_df.vector_id,article_df.text))\n```\n\n\n```python\ndef query_article(query, namespace, top_k=5):\n    '''Queries an article using its title in the specified\n     namespace and prints results.'''\n\n    # Create vector embeddings based on the title column\n    embedded_query = openai.Embedding.create(\n                                            input=query,\n                                            model=EMBEDDING_MODEL,\n                                            )[\"data\"][0]['embedding']\n\n    # Query namespace passed as parameter using title vector\n    query_result = index.query(embedded_query, \n                                      namespace=namespace, \n                                      top_k=top_k)\n\n    # Print query results \n    print(f'\\nMost similar results to {query} in \"{namespace}\" namespace:\\n')\n    if not query_result.matches:\n        print('no query result')\n    \n    matches = query_result.matches\n    ids = [res.id for res in matches]\n    scores = [res.score for res in matches]\n    df = pd.DataFrame({'id':ids, \n                       'score':scores,\n                       'title': [titles_mapped[_id] for _id in ids],\n                       'content': [content_mapped[_id] for _id in ids],\n                       })\n    \n    counter = 0\n    for k,v in df.iterrows():\n        counter += 1\n        print(f'{v.title} (score = {v.score})')\n    \n    print('\\n')\n\n    return df\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 5535880e-b6ad-493c-bafd-f6a2bf806e24
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb", "title": "Using Pinecone for Embeddings Search", "tokens": 4156, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"5535880e-b6ad-493c-bafd-f6a2bf806e24\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ba6c4445-95d6-49ac-9867-b741ab0ec83e\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"2ac8f36d192668244fd51cf18c2733bc66d5a3a822d68cf42df37b788b20c210\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8db54537-57fc-4525-96db-883a81596772\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Using_Pinecone_for_embeddings_search.ipynb\", \"title\": \"Using Pinecone for Embeddings Search\", \"tokens\": 4156, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a62844db0897f2a45d95cdb155a53d7a69dbced3ea8ac06820da4fdc58a64ada\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13527, \"end_char_idx\": 16096, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "doc_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "ref_doc_id": "ba6c4445-95d6-49ac-9867-b741ab0ec83e", "chroma:document": "### Search data\n\nNow we'll enter some dummy searches and check we get decent results back\n\n\n```python\n# First we'll create dictionaries mapping vector IDs to their outputs so we can retrieve the text for our search results\ntitles_mapped = dict(zip(article_df.vector_id,article_df.title))\ncontent_mapped = dict(zip(article_df.vector_id,article_df.text))\n```\n\n\n```python\ndef query_article(query, namespace, top_k=5):\n    '''Queries an article using its title in the specified\n     namespace and prints results.'''\n\n    # Create vector embeddings based on the title column\n    embedded_query = openai.Embedding.create(\n                                            input=query,\n                                            model=EMBEDDING_MODEL,\n                                            )[\"data\"][0]['embedding']\n\n    # Query namespace passed as parameter using title vector\n    query_result = index.query(embedded_query, \n                                      namespace=namespace, \n                                      top_k=top_k)\n\n    # Print query results \n    print(f'\\nMost similar results to {query} in \"{namespace}\" namespace:\\n')\n    if not query_result.matches:\n        print('no query result')\n    \n    matches = query_result.matches\n    ids = [res.id for res in matches]\n    scores = [res.score for res in matches]\n    df = pd.DataFrame({'id':ids, \n                       'score':scores,\n                       'title': [titles_mapped[_id] for _id in ids],\n                       'content': [content_mapped[_id] for _id in ids],\n                       })\n    \n    counter = 0\n    for k,v in df.iterrows():\n        counter += 1\n        print(f'{v.title} (score = {v.score})')\n    \n    print('\\n')\n\n    return df\n```\n\n\n```python\nquery_output = query_article('modern art in Europe','title')\n```\n\n    \n    Most similar results to modern art in Europe in \"title\" namespace:\n    \n    Museum of Modern Art (score = 0.875177085)\n    Western Europe (score = 0.867441177)\n    Renaissance art (score = 0.864156306)\n    Pop art (score = 0.860346854)\n    Northern Europe (score = 0.854658186)\n    \n    \n\n\n\n```python\ncontent_query_output = query_article(\"Famous battles in Scottish history\",'content')\n```\n\n    \n    Most similar results to Famous battles in Scottish history in \"content\" namespace:\n    \n    Battle of Bannockburn (score = 0.869336188)\n    Wars of Scottish Independence (score = 0.861470938)\n    1651 (score = 0.852588475)\n    First War of Scottish Independence (score = 0.84962213)\n    Robert I of Scotland (score = 0.846214116)\n    \n    \n\n\n\n```python\n\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: ea923c49-159c-4987-93f5-c2c6e8ad2ec0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"ea923c49-159c-4987-93f5-c2c6e8ad2ec0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"589ac817-3347-4f65-952d-f2507a7fb865\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1e981be9-d106-4f59-918e-2c41b504ffab\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a96adf98e64c7662f0aa6d45bc7d47fc79c4a1e0dda582e161091a00a7da52f4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 678, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "589ac817-3347-4f65-952d-f2507a7fb865", "doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "ref_doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "chroma:document": "# Retrieval Augmented Generative Question Answering with Pinecone\n\n#### Fixing LLMs that Hallucinate\n\nIn this notebook we will learn how to query relevant contexts to our queries from Pinecone, and pass these to a generative OpenAI model to generate an answer backed by real data sources.\n\nA common problem with using GPT-3 to factually answer questions is that GPT-3 can sometimes make things up. The GPT models have a broad range of general knowledge, but this does not necessarily apply to more specific information. For that we use the Pinecone vector database as our _\"external knowledge base\"_ \u2014 like *long-term memory* for GPT-3.\n\nRequired installs for this notebook are:"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 1e981be9-d106-4f59-918e-2c41b504ffab
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"1e981be9-d106-4f59-918e-2c41b504ffab\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"589ac817-3347-4f65-952d-f2507a7fb865\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ea923c49-159c-4987-93f5-c2c6e8ad2ec0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ac4a4b9a2731eb039aad178797c8152a3ce3772128bc906ff2b04a969ff0c97e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"03c17e62-9803-4b17-90c5-a74f2dd151fe\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b99cd54c1066cfb822bc0d0efb088706a706083892fb5ff41aeab3a5f2cf2024\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2121, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "589ac817-3347-4f65-952d-f2507a7fb865", "doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "ref_doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "chroma:document": "# Retrieval Augmented Generative Question Answering with Pinecone\n\n#### Fixing LLMs that Hallucinate\n\nIn this notebook we will learn how to query relevant contexts to our queries from Pinecone, and pass these to a generative OpenAI model to generate an answer backed by real data sources.\n\nA common problem with using GPT-3 to factually answer questions is that GPT-3 can sometimes make things up. The GPT models have a broad range of general knowledge, but this does not necessarily apply to more specific information. For that we use the Pinecone vector database as our _\"external knowledge base\"_ \u2014 like *long-term memory* for GPT-3.\n\nRequired installs for this notebook are:\n\n\n```python\n!pip install -qU openai pinecone-client datasets\n```\n\n    \u001b[?25l     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m0.0/55.3 KB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\n\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m55.3/55.3 KB\u001b[0m \u001b[31m1.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25h  Installing build dependencies ... \u001b[?25l\u001b[?25hdone\n      Getting requirements to build wheel ... \u001b[?25l\u001b[?25hdone\n      Preparing metadata (pyproject.toml) ... \u001b[?25l\u001b[?25hdone\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m170.6/170.6 KB\u001b[0m \u001b[31m13.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m452.9/452.9 KB\u001b[0m \u001b[31m30.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m58.3/58.3 KB\u001b[0m \u001b[31m6.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m213.0/213.0 KB\u001b[0m \u001b[31m17.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m132.0/132.0 KB\u001b[0m \u001b[31m13.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m182.4/182.4 KB\u001b[0m \u001b[31m18.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m140.6/140.6 KB\u001b[0m \u001b[31m6.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25h  Building wheel for openai (pyproject.toml) ... \u001b[?25l\u001b[?25hdone"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 03c17e62-9803-4b17-90c5-a74f2dd151fe
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"03c17e62-9803-4b17-90c5-a74f2dd151fe\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"589ac817-3347-4f65-952d-f2507a7fb865\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1e981be9-d106-4f59-918e-2c41b504ffab\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"d9d74dc1dfe3ad9bde8547fbb905330c3c64dd21ffc32ad8fc569edd3a4a5c71\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2a5b1a20-8e07-43c9-9b39-97781195a2ed\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"25a0ca9c39dc8dabc3823a17b9f9d0c9ca74ba52939574d67224193c86844d72\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2125, \"end_char_idx\": 4213, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "589ac817-3347-4f65-952d-f2507a7fb865", "doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "ref_doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "chroma:document": "```python\nimport openai\n\n# get API key from top-right dropdown on OpenAI website\nopenai.api_key = \"OPENAI_API_KEY\"\n```\n\nFor many questions *state-of-the-art (SOTA)* LLMs are more than capable of answering correctly.\n\n\n```python\nquery = \"who was the 12th person on the moon and when did they land?\"\n\n# now query `gpt-3.5-turbo-instruct` WITHOUT context\nres = openai.Completion.create(\n    engine='gpt-3.5-turbo-instruct',\n    prompt=query,\n    temperature=0,\n    max_tokens=400,\n    top_p=1,\n    frequency_penalty=0,\n    presence_penalty=0,\n    stop=None\n)\n\nres['choices'][0]['text'].strip()\n```\n\n\n\n\n    'The 12th person on the moon was Harrison Schmitt, and he landed on December 11, 1972.'\n\n\n\nHowever, that isn't always the case. First let's first rewrite the above into a simple function so we're not rewriting this every time.\n\n\n```python\ndef complete(prompt):\n    res = openai.Completion.create(\n        engine='gpt-3.5-turbo-instruct',\n        prompt=prompt,\n        temperature=0,\n        max_tokens=400,\n        top_p=1,\n        frequency_penalty=0,\n        presence_penalty=0,\n        stop=None\n    )\n    return res['choices'][0]['text'].strip()\n```\n\nNow let's ask a more specific question about training a type of transformer model called a *sentence transformer*. The ideal answer we'd be looking for is _\"Multiple Negatives Ranking (MNR) loss\"_.\n\nDon't worry if this is a new term to you, it isn't required to understand what we're doing or demoing here.\n\n\n```python\nquery = (\n    \"Which training method should I use for sentence transformers when \" +\n    \"I only have pairs of related sentences?\"\n)\n\ncomplete(query)\n```\n\n\n\n\n    'If you only have pairs of related sentences, then the best training method to use for sentence transformers is the supervised learning approach. This approach involves providing the model with labeled data, such as pairs of related sentences, and then training the model to learn the relationships between the sentences. This approach is often used for tasks such as natural language inference, semantic similarity, and paraphrase identification.'"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 2a5b1a20-8e07-43c9-9b39-97781195a2ed
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"2a5b1a20-8e07-43c9-9b39-97781195a2ed\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"589ac817-3347-4f65-952d-f2507a7fb865\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"03c17e62-9803-4b17-90c5-a74f2dd151fe\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"39e1cd0670821ed0c99eb639ce7f29e79c587edb32090ffc0177d3270a0ae130\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c7b5284f-d0b8-4fae-be6a-6324f21cd913\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"907b27554daebaefaff6a79e5767ece3c24d8422edd08ec684f233a5ca46405a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2728, \"end_char_idx\": 6379, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "589ac817-3347-4f65-952d-f2507a7fb865", "doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "ref_doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "chroma:document": "'The 12th person on the moon was Harrison Schmitt, and he landed on December 11, 1972.'\n\n\n\nHowever, that isn't always the case. First let's first rewrite the above into a simple function so we're not rewriting this every time.\n\n\n```python\ndef complete(prompt):\n    res = openai.Completion.create(\n        engine='gpt-3.5-turbo-instruct',\n        prompt=prompt,\n        temperature=0,\n        max_tokens=400,\n        top_p=1,\n        frequency_penalty=0,\n        presence_penalty=0,\n        stop=None\n    )\n    return res['choices'][0]['text'].strip()\n```\n\nNow let's ask a more specific question about training a type of transformer model called a *sentence transformer*. The ideal answer we'd be looking for is _\"Multiple Negatives Ranking (MNR) loss\"_.\n\nDon't worry if this is a new term to you, it isn't required to understand what we're doing or demoing here.\n\n\n```python\nquery = (\n    \"Which training method should I use for sentence transformers when \" +\n    \"I only have pairs of related sentences?\"\n)\n\ncomplete(query)\n```\n\n\n\n\n    'If you only have pairs of related sentences, then the best training method to use for sentence transformers is the supervised learning approach. This approach involves providing the model with labeled data, such as pairs of related sentences, and then training the model to learn the relationships between the sentences. This approach is often used for tasks such as natural language inference, semantic similarity, and paraphrase identification.'\n\n\n\nOne of the common answers we get to this is:\n\n```\nThe best training method to use for fine-tuning a pre-trained model with sentence transformers is the Masked Language Model (MLM) training. MLM training involves randomly masking some of the words in a sentence and then training the model to predict the masked words. This helps the model to learn the context of the sentence and better understand the relationships between words.\n```\n\nThis answer seems pretty convincing right? Yet, it's wrong. MLM is typically used in the pretraining step of a transformer model but *\"cannot\"* be used to fine-tune a sentence-transformer, and has nothing to do with having _\"pairs of related sentences\"_.\n\nAn alternative answer we receive (and the one we returned above) is about `supervised learning approach` being the most suitable. This is completely true, but it's not specific and doesn't answer the question.\n\nWe have two options for enabling our LLM in understanding and correctly answering this question:\n\n1. We fine-tune the LLM on text data covering the topic mentioned, likely on articles and papers talking about sentence transformers, semantic search training methods, etc.\n\n2. We use **R**etrieval **A**ugmented **G**eneration (RAG), a technique that implements an information retrieval component to the generation process. Allowing us to retrieve relevant information and feed this information into the generation model as a *secondary* source of information.\n\nWe will demonstrate option **2**.\n\n---\n\n## Building a Knowledge Base\n\nWith option **2** the retrieval of relevant information requires an external _\"Knowledge Base\"_, a place where we can store and use to efficiently retrieve information. We can think of this as the external _long-term memory_ of our LLM.\n\nWe will need to retrieve information that is semantically related to our queries, to do this we need to use _\"dense vector embeddings\"_. These can be thought of as numerical representations of the *meaning* behind our sentences.\n\nTo create these dense vectors we use the `text-embedding-3-small` model.\n\nWe have already authenticated our OpenAI connection, to create an embedding we just do:"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: c7b5284f-d0b8-4fae-be6a-6324f21cd913
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c7b5284f-d0b8-4fae-be6a-6324f21cd913\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"589ac817-3347-4f65-952d-f2507a7fb865\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2a5b1a20-8e07-43c9-9b39-97781195a2ed\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"5adbec0b6dcb6b222ec8534b8610006d44f0ddd4d459d05db47601207b2d511a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0cfa7ecc-3cb7-4743-a0dd-5f2295c06cda\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e167a90b4f37dc6b5700d6bbe1d3c7b88949fb78ab7a22a211106eb6216fd2ea\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6382, \"end_char_idx\": 8695, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "589ac817-3347-4f65-952d-f2507a7fb865", "doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "ref_doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "chroma:document": "```python\nembed_model = \"text-embedding-ada-002\"\n\nres = openai.Embedding.create(\n    input=[\n        \"Sample document text goes here\",\n        \"there will be several phrases in each batch\"\n    ], engine=embed_model\n)\n```\n\nIn the response `res` we will find a JSON-like object containing our new embeddings within the `'data'` field.\n\n\n```python\nres.keys()\n```\n\n\n\n\n    dict_keys(['object', 'data', 'model', 'usage'])\n\n\n\nInside `'data'` we will find two records, one for each of the two sentences we just embedded. Each vector embedding contains `1536` dimensions (the output dimensionality of the `text-embedding-3-small` model.\n\n\n```python\nlen(res['data'])\n```\n\n\n\n\n    2\n\n\n\n\n```python\nlen(res['data'][0]['embedding']), len(res['data'][1]['embedding'])\n```\n\n\n\n\n    (1536, 1536)\n\n\n\nWe will apply this same embedding logic to a dataset containing information relevant to our query (and many other queries on the topics of ML and AI).\n\n### Data Preparation\n\nThe dataset we will be using is the `jamescalam/youtube-transcriptions` from Hugging Face _Datasets_. It contains transcribed audio from several ML and tech YouTube channels. We download it with:\n\n\n```python\nfrom datasets import load_dataset\n\ndata = load_dataset('jamescalam/youtube-transcriptions', split='train')\ndata\n```\n\n    Using custom data configuration jamescalam--youtube-transcriptions-6a482f3df0aedcdb\n    Reusing dataset json (/Users/jamesbriggs/.cache/huggingface/datasets/jamescalam___json/jamescalam--youtube-transcriptions-6a482f3df0aedcdb/0.0.0/ac0ca5f5289a6cf108e706efcf040422dbbfa8e658dee6a819f20d76bb84d26b)\n\n\n\n\n\n    Dataset({\n        features: ['title', 'published', 'url', 'video_id', 'channel_id', 'id', 'text', 'start', 'end'],\n        num_rows: 208619\n    })\n\n\n\n\n```python\ndata[0]\n```\n\n\n\n\n    {'title': 'Training and Testing an Italian BERT - Transformers From Scratch #4',\n     'published': '2021-07-06 13:00:03 UTC',\n     'url': 'https://youtu.be/35Pdoyi6ZoQ',\n     'video_id': '35Pdoyi6ZoQ',\n     'channel_id': 'UCv83tO5cePwHMt1952IVVHw',\n     'id': '35Pdoyi6ZoQ-t0.0',\n     'text': 'Hi, welcome to the video.',\n     'start': 0.0,\n     'end': 9.36}\n\n\n\nThe dataset contains many small snippets of text data. We will need to merge many snippets from each video to create more substantial chunks of text that contain more information."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 0cfa7ecc-3cb7-4743-a0dd-5f2295c06cda
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0cfa7ecc-3cb7-4743-a0dd-5f2295c06cda\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"589ac817-3347-4f65-952d-f2507a7fb865\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c7b5284f-d0b8-4fae-be6a-6324f21cd913\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"1f7c57fb3c73cea1614d1ae58b6027f2e76f6ef5566e48bdc91ada9856d71e38\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"387d64f5-3359-4545-b181-5fbbbc5357ce\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6d3b3955d70abac0aab0497de7e99c523cdf5605a3c7affcd437a7f9aa807a7f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7534, \"end_char_idx\": 9561, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "589ac817-3347-4f65-952d-f2507a7fb865", "doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "ref_doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "chroma:document": "```python\nfrom datasets import load_dataset\n\ndata = load_dataset('jamescalam/youtube-transcriptions', split='train')\ndata\n```\n\n    Using custom data configuration jamescalam--youtube-transcriptions-6a482f3df0aedcdb\n    Reusing dataset json (/Users/jamesbriggs/.cache/huggingface/datasets/jamescalam___json/jamescalam--youtube-transcriptions-6a482f3df0aedcdb/0.0.0/ac0ca5f5289a6cf108e706efcf040422dbbfa8e658dee6a819f20d76bb84d26b)\n\n\n\n\n\n    Dataset({\n        features: ['title', 'published', 'url', 'video_id', 'channel_id', 'id', 'text', 'start', 'end'],\n        num_rows: 208619\n    })\n\n\n\n\n```python\ndata[0]\n```\n\n\n\n\n    {'title': 'Training and Testing an Italian BERT - Transformers From Scratch #4',\n     'published': '2021-07-06 13:00:03 UTC',\n     'url': 'https://youtu.be/35Pdoyi6ZoQ',\n     'video_id': '35Pdoyi6ZoQ',\n     'channel_id': 'UCv83tO5cePwHMt1952IVVHw',\n     'id': '35Pdoyi6ZoQ-t0.0',\n     'text': 'Hi, welcome to the video.',\n     'start': 0.0,\n     'end': 9.36}\n\n\n\nThe dataset contains many small snippets of text data. We will need to merge many snippets from each video to create more substantial chunks of text that contain more information.\n\n\n```python\nfrom tqdm.auto import tqdm\n\nnew_data = []\n\nwindow = 20  # number of sentences to combine\nstride = 4  # number of sentences to 'stride' over, used to create overlap\n\nfor i in tqdm(range(0, len(data), stride)):\n    i_end = min(len(data)-1, i+window)\n    if data[i]['title'] != data[i_end]['title']:\n        # in this case we skip this entry as we have start/end of two videos\n        continue\n    text = ' '.join(data[i:i_end]['text'])\n    # create the new merged dataset\n    new_data.append({\n        'start': data[i]['start'],\n        'end': data[i_end]['end'],\n        'title': data[i]['title'],\n        'text': text,\n        'id': data[i]['id'],\n        'url': data[i]['url'],\n        'published': data[i]['published'],\n        'channel_id': data[i]['channel_id']\n    })\n```\n\n\n      0%|          | 0/52155 [00:00<?, ?it/s]\n\n\n\n```python\nnew_data[0]\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 387d64f5-3359-4545-b181-5fbbbc5357ce
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"387d64f5-3359-4545-b181-5fbbbc5357ce\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"589ac817-3347-4f65-952d-f2507a7fb865\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0cfa7ecc-3cb7-4743-a0dd-5f2295c06cda\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"51976288180c82096053c76804923ff69c191d4e3c73678c18725e6a1c18f702\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f7bbb1db-ccba-4a1e-9819-8d8043646d14\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9bec52cb2282aedac099544f087180820d7919e9e5397f42b8b30afc6309fa02\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8516, \"end_char_idx\": 10941, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "589ac817-3347-4f65-952d-f2507a7fb865", "doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "ref_doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "chroma:document": "The dataset contains many small snippets of text data. We will need to merge many snippets from each video to create more substantial chunks of text that contain more information.\n\n\n```python\nfrom tqdm.auto import tqdm\n\nnew_data = []\n\nwindow = 20  # number of sentences to combine\nstride = 4  # number of sentences to 'stride' over, used to create overlap\n\nfor i in tqdm(range(0, len(data), stride)):\n    i_end = min(len(data)-1, i+window)\n    if data[i]['title'] != data[i_end]['title']:\n        # in this case we skip this entry as we have start/end of two videos\n        continue\n    text = ' '.join(data[i:i_end]['text'])\n    # create the new merged dataset\n    new_data.append({\n        'start': data[i]['start'],\n        'end': data[i_end]['end'],\n        'title': data[i]['title'],\n        'text': text,\n        'id': data[i]['id'],\n        'url': data[i]['url'],\n        'published': data[i]['published'],\n        'channel_id': data[i]['channel_id']\n    })\n```\n\n\n      0%|          | 0/52155 [00:00<?, ?it/s]\n\n\n\n```python\nnew_data[0]\n```\n\n\n\n\n    {'start': 0.0,\n     'end': 74.12,\n     'title': 'Training and Testing an Italian BERT - Transformers From Scratch #4',\n     'text': \"Hi, welcome to the video. So this is the fourth video in a Transformers from Scratch mini series. So if you haven't been following along, we've essentially covered what you can see on the screen. So we got some data. We built a tokenizer with it. And then we've set up our input pipeline ready to begin actually training our model, which is what we're going to cover in this video. So let's move over to the code. And we see here that we have essentially everything we've done so far. So we've built our input data, our input pipeline. And we're now at a point where we have a data loader, PyTorch data loader, ready. And we can begin training a model with it. So there are a few things to be aware of. So I mean, first, let's just have a quick look at the structure of our data.\",\n     'id': '35Pdoyi6ZoQ-t0.0',\n     'url': 'https://youtu.be/35Pdoyi6ZoQ',\n     'published': '2021-07-06 13:00:03 UTC',\n     'channel_id': 'UCv83tO5cePwHMt1952IVVHw'}\n\n\n\nNow we need a place to store these embeddings and enable a efficient _vector search_ through them all. To do that we use **`Pinecone`**, we can get a [free API key](https://app.pinecone.io) and enter it below where we will initialize our connection to `Pinecone` and create a new index."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: f7bbb1db-ccba-4a1e-9819-8d8043646d14
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"f7bbb1db-ccba-4a1e-9819-8d8043646d14\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"589ac817-3347-4f65-952d-f2507a7fb865\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"387d64f5-3359-4545-b181-5fbbbc5357ce\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a19d9e1eab22fdee52fc43a2602787245bf60df53404e2218d623b410c3fcd77\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"73087520-ee64-4d03-a8ba-3b50ec73fa5b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b092832e524d952078343ed675cf6e388505c21a45d340783625e4b7d25166c3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9536, \"end_char_idx\": 11944, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "589ac817-3347-4f65-952d-f2507a7fb865", "doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "ref_doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "chroma:document": "```python\nnew_data[0]\n```\n\n\n\n\n    {'start': 0.0,\n     'end': 74.12,\n     'title': 'Training and Testing an Italian BERT - Transformers From Scratch #4',\n     'text': \"Hi, welcome to the video. So this is the fourth video in a Transformers from Scratch mini series. So if you haven't been following along, we've essentially covered what you can see on the screen. So we got some data. We built a tokenizer with it. And then we've set up our input pipeline ready to begin actually training our model, which is what we're going to cover in this video. So let's move over to the code. And we see here that we have essentially everything we've done so far. So we've built our input data, our input pipeline. And we're now at a point where we have a data loader, PyTorch data loader, ready. And we can begin training a model with it. So there are a few things to be aware of. So I mean, first, let's just have a quick look at the structure of our data.\",\n     'id': '35Pdoyi6ZoQ-t0.0',\n     'url': 'https://youtu.be/35Pdoyi6ZoQ',\n     'published': '2021-07-06 13:00:03 UTC',\n     'channel_id': 'UCv83tO5cePwHMt1952IVVHw'}\n\n\n\nNow we need a place to store these embeddings and enable a efficient _vector search_ through them all. To do that we use **`Pinecone`**, we can get a [free API key](https://app.pinecone.io) and enter it below where we will initialize our connection to `Pinecone` and create a new index.\n\n\n```python\nimport pinecone\n\nindex_name = 'openai-youtube-transcriptions'\n\n# initialize connection to pinecone (get API key at app.pinecone.io)\npinecone.init(\n    api_key=\"PINECONE_API_KEY\",\n    environment=\"us-east1-gcp\"  # may be different, check at app.pinecone.io\n)\n\n# check if index already exists (it shouldn't if this is first time)\nif index_name not in pinecone.list_indexes():\n    # if does not exist, create index\n    pinecone.create_index(\n        index_name,\n        dimension=len(res['data'][0]['embedding']),\n        metric='cosine',\n        metadata_config={'indexed': ['channel_id', 'published']}\n    )\n# connect to index\nindex = pinecone.Index(index_name)\n# view index stats\nindex.describe_index_stats()\n```\n\n\n\n\n    {'dimension': 1536,\n     'index_fullness': 0.0,\n     'namespaces': {},\n     'total_vector_count': 0}\n\n\n\nWe can see the index is currently empty with a `total_vector_count` of `0`. We can begin populating it with OpenAI `text-embedding-3-small` built embeddings like so:"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 73087520-ee64-4d03-a8ba-3b50ec73fa5b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"73087520-ee64-4d03-a8ba-3b50ec73fa5b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"589ac817-3347-4f65-952d-f2507a7fb865\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f7bbb1db-ccba-4a1e-9819-8d8043646d14\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a343269eadb003865388ae311c3463546a1600c73e40fc6db0ce0aee7d702f5b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"25a3f409-6a29-40ef-9d61-f9180e3a72f5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b00234140bd9677416e527d3d90d73e5ada8bf7f13731e0ae3fca0dba8201c7e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10655, \"end_char_idx\": 13467, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "589ac817-3347-4f65-952d-f2507a7fb865", "doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "ref_doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "chroma:document": "Now we need a place to store these embeddings and enable a efficient _vector search_ through them all. To do that we use **`Pinecone`**, we can get a [free API key](https://app.pinecone.io) and enter it below where we will initialize our connection to `Pinecone` and create a new index.\n\n\n```python\nimport pinecone\n\nindex_name = 'openai-youtube-transcriptions'\n\n# initialize connection to pinecone (get API key at app.pinecone.io)\npinecone.init(\n    api_key=\"PINECONE_API_KEY\",\n    environment=\"us-east1-gcp\"  # may be different, check at app.pinecone.io\n)\n\n# check if index already exists (it shouldn't if this is first time)\nif index_name not in pinecone.list_indexes():\n    # if does not exist, create index\n    pinecone.create_index(\n        index_name,\n        dimension=len(res['data'][0]['embedding']),\n        metric='cosine',\n        metadata_config={'indexed': ['channel_id', 'published']}\n    )\n# connect to index\nindex = pinecone.Index(index_name)\n# view index stats\nindex.describe_index_stats()\n```\n\n\n\n\n    {'dimension': 1536,\n     'index_fullness': 0.0,\n     'namespaces': {},\n     'total_vector_count': 0}\n\n\n\nWe can see the index is currently empty with a `total_vector_count` of `0`. We can begin populating it with OpenAI `text-embedding-3-small` built embeddings like so:\n\n\n```python\nfrom tqdm.auto import tqdm\nfrom time import sleep\n\nbatch_size = 100  # how many embeddings we create and insert at once\n\nfor i in tqdm(range(0, len(new_data), batch_size)):\n    # find end of batch\n    i_end = min(len(new_data), i+batch_size)\n    meta_batch = new_data[i:i_end]\n    # get ids\n    ids_batch = [x['id'] for x in meta_batch]\n    # get texts to encode\n    texts = [x['text'] for x in meta_batch]\n    # create embeddings (try-except added to avoid RateLimitError)\n    done = False\n    while not done:\n        try:\n            res = openai.Embedding.create(input=texts, engine=embed_model)\n            done = True\n        except:\n            sleep(5)\n    embeds = [record['embedding'] for record in res['data']]\n    # cleanup metadata\n    meta_batch = [{\n        'start': x['start'],\n        'end': x['end'],\n        'title': x['title'],\n        'text': x['text'],\n        'url': x['url'],\n        'published': x['published'],\n        'channel_id': x['channel_id']\n    } for x in meta_batch]\n    to_upsert = list(zip(ids_batch, embeds, meta_batch))\n    # upsert to Pinecone\n    index.upsert(vectors=to_upsert)\n```\n\n\n      0%|          | 0/487 [00:00<?, ?it/s]\n\n\nNow we search, for this we need to create a _query vector_ `xq`:\n\n\n```python\nres = openai.Embedding.create(\n    input=[query],\n    engine=embed_model\n)\n\n# retrieve from Pinecone\nxq = res['data'][0]['embedding']\n\n# get relevant contexts (including the questions)\nres = index.query(xq, top_k=2, include_metadata=True)\n```\n\n\n```python\nres\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 25a3f409-6a29-40ef-9d61-f9180e3a72f5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"25a3f409-6a29-40ef-9d61-f9180e3a72f5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"589ac817-3347-4f65-952d-f2507a7fb865\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"73087520-ee64-4d03-a8ba-3b50ec73fa5b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"681ac22d8d27c5abcd17e3e720a5df6ecd53d22d2a8c8ffa9c68d811843f0a00\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ef4c2b06-650e-473f-97b7-c57863de1004\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a83fa20bedec161a7539e949bbb37bd023777514fe06fa0640ba7fd6db2684ad\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13088, \"end_char_idx\": 16901, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "589ac817-3347-4f65-952d-f2507a7fb865", "doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "ref_doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "chroma:document": "0%|          | 0/487 [00:00<?, ?it/s]\n\n\nNow we search, for this we need to create a _query vector_ `xq`:\n\n\n```python\nres = openai.Embedding.create(\n    input=[query],\n    engine=embed_model\n)\n\n# retrieve from Pinecone\nxq = res['data'][0]['embedding']\n\n# get relevant contexts (including the questions)\nres = index.query(xq, top_k=2, include_metadata=True)\n```\n\n\n```python\nres\n```\n\n\n\n\n    {'matches': [{'id': 'pNvujJ1XyeQ-t418.88',\n                  'metadata': {'channel_id': 'UCv83tO5cePwHMt1952IVVHw',\n                               'end': 568.4,\n                               'published': datetime.date(2021, 11, 24),\n                               'start': 418.88,\n                               'text': 'pairs of related sentences you can go '\n                                       'ahead and actually try training or '\n                                       'fine-tuning using NLI with multiple '\n                                       \"negative ranking loss. If you don't have \"\n                                       'that fine. Another option is that you have '\n                                       'a semantic textual similarity data set or '\n                                       'STS and what this is is you have so you '\n                                       'have sentence A here, sentence B here and '\n                                       'then you have a score from from 0 to 1 '\n                                       'that tells you the similarity between '\n                                       'those two scores and you would train this '\n                                       'using something like cosine similarity '\n                                       \"loss. Now if that's not an option and your \"\n                                       'focus or use case is on building a '\n                                       'sentence transformer for another language '\n                                       'where there is no current sentence '\n                                       'transformer you can use multilingual '\n                                       'parallel data. So what I mean by that is '\n                                       'so parallel data just means translation '\n                                       'pairs so if you have for example a English '\n                                       'sentence and then you have another '\n                                       'language here so it can it can be anything '\n                                       \"I'm just going to put XX and that XX is \"\n                                       'your target language you can fine-tune a '\n                                       'model using something called multilingual '\n                                       'knowledge distillation and what that does '\n                                       'is takes a monolingual model for example '\n                                       'in English and using those translation '\n                                       'pairs it distills the knowledge the '\n                                       'semantic similarity knowledge from that '\n                                       'monolingual English model into a '\n                                       'multilingual model which can handle both '\n                                       'English and your target language. So '\n                                       \"they're three options quite popular very \"\n                                       'common that you can go for and as a '\n                                       'supervised methods the chances are that '\n                                       'probably going to outperform anything you '\n                                       'do with unsupervised training at least for '\n                                       'now."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: ef4c2b06-650e-473f-97b7-c57863de1004
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"ef4c2b06-650e-473f-97b7-c57863de1004\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"589ac817-3347-4f65-952d-f2507a7fb865\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"25a3f409-6a29-40ef-9d61-f9180e3a72f5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"230abe3fca201fc905420f118ea2e392e807a7736a2c56cc177d961a5674efc5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cf516927-dcc2-4f63-b6c3-a25f3744ec06\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2cb54677131be85f029cad20fd372c02ae14f928baba44f5c7cc0be67293e37a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14056, \"end_char_idx\": 19068, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "589ac817-3347-4f65-952d-f2507a7fb865", "doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "ref_doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "chroma:document": "If you don't have \"\n                                       'that fine. Another option is that you have '\n                                       'a semantic textual similarity data set or '\n                                       'STS and what this is is you have so you '\n                                       'have sentence A here, sentence B here and '\n                                       'then you have a score from from 0 to 1 '\n                                       'that tells you the similarity between '\n                                       'those two scores and you would train this '\n                                       'using something like cosine similarity '\n                                       \"loss. Now if that's not an option and your \"\n                                       'focus or use case is on building a '\n                                       'sentence transformer for another language '\n                                       'where there is no current sentence '\n                                       'transformer you can use multilingual '\n                                       'parallel data. So what I mean by that is '\n                                       'so parallel data just means translation '\n                                       'pairs so if you have for example a English '\n                                       'sentence and then you have another '\n                                       'language here so it can it can be anything '\n                                       \"I'm just going to put XX and that XX is \"\n                                       'your target language you can fine-tune a '\n                                       'model using something called multilingual '\n                                       'knowledge distillation and what that does '\n                                       'is takes a monolingual model for example '\n                                       'in English and using those translation '\n                                       'pairs it distills the knowledge the '\n                                       'semantic similarity knowledge from that '\n                                       'monolingual English model into a '\n                                       'multilingual model which can handle both '\n                                       'English and your target language. So '\n                                       \"they're three options quite popular very \"\n                                       'common that you can go for and as a '\n                                       'supervised methods the chances are that '\n                                       'probably going to outperform anything you '\n                                       'do with unsupervised training at least for '\n                                       'now. So if none of those sound like '\n                                       'something',\n                               'title': 'Today Unsupervised Sentence Transformers, '\n                                        'Tomorrow Skynet (how TSDAE works)',\n                               'url': 'https://youtu.be/pNvujJ1XyeQ'},\n                  'score': 0.865277052,\n                  'sparseValues': {},\n                  'values': []},\n                 {'id': 'WS1uVMGhlWQ-t737.28',\n                  'metadata': {'channel_id': 'UCv83tO5cePwHMt1952IVVHw',\n                               'end': 900.72,\n                               'published': datetime.date(2021, 10, 20),\n                               'start': 737.28,\n                               'text': \"were actually more accurate. So we can't \"\n                                       \"really do that. We can't use this what is \"\n                                       'called a mean pooling approach. Or we '\n                                       \"can't use it in its current form. Now the \"\n                                       'solution to this problem was introduced by '\n                                       'two people in 2019 Nils Reimers and Irenia '\n                                       'Gurevich. They introduced what is the '\n                                       'first sentence transformer or sentence '\n                                       'BERT. And it was found that sentence BERT '\n                                       'or S BERT outformed all of the previous '\n                                       'Save the Art models on pretty much all '\n                                       'benchmarks. Not all of them but most of '\n                                       'them. And it did it in a very quick time. '\n                                       'So if we compare it to BERT, if we wanted '\n                                       'to find the most similar sentence pair '\n                                       'from 10,000 sentences in that 2019 paper '\n                                       'they found that with BERT that took 65 '\n                                       'hours."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: cf516927-dcc2-4f63-b6c3-a25f3744ec06
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"cf516927-dcc2-4f63-b6c3-a25f3744ec06\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"589ac817-3347-4f65-952d-f2507a7fb865\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ef4c2b06-650e-473f-97b7-c57863de1004\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"4a9cc008a02ea251492a18d124e9234cd352c05ced1f7dcfbe7adbb299ef178b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"63b8ea87-b494-4288-bc80-1af62ec7d4ab\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8d2a0f253254d69b238a99c1386c1b64d40f3ac8ee394d0943fcf5203141fd40\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16902, \"end_char_idx\": 21472, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "589ac817-3347-4f65-952d-f2507a7fb865", "doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "ref_doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "chroma:document": "So if none of those sound like '\n                                       'something',\n                               'title': 'Today Unsupervised Sentence Transformers, '\n                                        'Tomorrow Skynet (how TSDAE works)',\n                               'url': 'https://youtu.be/pNvujJ1XyeQ'},\n                  'score': 0.865277052,\n                  'sparseValues': {},\n                  'values': []},\n                 {'id': 'WS1uVMGhlWQ-t737.28',\n                  'metadata': {'channel_id': 'UCv83tO5cePwHMt1952IVVHw',\n                               'end': 900.72,\n                               'published': datetime.date(2021, 10, 20),\n                               'start': 737.28,\n                               'text': \"were actually more accurate. So we can't \"\n                                       \"really do that. We can't use this what is \"\n                                       'called a mean pooling approach. Or we '\n                                       \"can't use it in its current form. Now the \"\n                                       'solution to this problem was introduced by '\n                                       'two people in 2019 Nils Reimers and Irenia '\n                                       'Gurevich. They introduced what is the '\n                                       'first sentence transformer or sentence '\n                                       'BERT. And it was found that sentence BERT '\n                                       'or S BERT outformed all of the previous '\n                                       'Save the Art models on pretty much all '\n                                       'benchmarks. Not all of them but most of '\n                                       'them. And it did it in a very quick time. '\n                                       'So if we compare it to BERT, if we wanted '\n                                       'to find the most similar sentence pair '\n                                       'from 10,000 sentences in that 2019 paper '\n                                       'they found that with BERT that took 65 '\n                                       'hours. With S BERT embeddings they could '\n                                       'create all the embeddings in just around '\n                                       'five seconds. And then they could compare '\n                                       'all those with cosine similarity in 0.01 '\n                                       \"seconds. So it's a lot faster. We go from \"\n                                       '65 hours to just over five seconds which '\n                                       'is I think pretty incredible. Now I think '\n                                       \"that's pretty much all the context we need \"\n                                       'behind sentence transformers. And what we '\n                                       'do now is dive into a little bit of how '\n                                       'they actually work. Now we said before we '\n                                       'have the core transform models and what S '\n                                       'BERT does is fine tunes on sentence pairs '\n                                       'using what is called a Siamese '\n                                       'architecture or Siamese network. What we '\n                                       'mean by a Siamese network is that we have '\n                                       'what we can see, what can view as two BERT '\n                                       'models that are identical and the weights '\n                                       'between those two models are tied. Now in '\n                                       'reality when implementing this we just use '\n                                       'a single BERT model. And what we do is we '\n                                       'process one sentence, a sentence A through '\n                                       'the model and then we process another '\n                                       'sentence, sentence B through the model. '\n                                       \"And that's the sentence pair. So with our \"\n                                       'cross-linked we were processing the '\n                                       'sentence pair together. We were putting '\n                                       'them both together, processing them all at '\n                                       'once. This time we process them '\n                                       'separately."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 63b8ea87-b494-4288-bc80-1af62ec7d4ab
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"63b8ea87-b494-4288-bc80-1af62ec7d4ab\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"589ac817-3347-4f65-952d-f2507a7fb865\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cf516927-dcc2-4f63-b6c3-a25f3744ec06\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"4acd459784d0569cd6a53761abf54e0ccc798bebca918e2c271913ad1c253491\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b5fc7ab3-b241-471d-b87a-d159846f12de\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9642748df13be8897161feb5b6caaba1ed9846066ec02f5fef13e5b9cec0a390\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19069, \"end_char_idx\": 23138, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "589ac817-3347-4f65-952d-f2507a7fb865", "doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "ref_doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "chroma:document": "With S BERT embeddings they could '\n                                       'create all the embeddings in just around '\n                                       'five seconds. And then they could compare '\n                                       'all those with cosine similarity in 0.01 '\n                                       \"seconds. So it's a lot faster. We go from \"\n                                       '65 hours to just over five seconds which '\n                                       'is I think pretty incredible. Now I think '\n                                       \"that's pretty much all the context we need \"\n                                       'behind sentence transformers. And what we '\n                                       'do now is dive into a little bit of how '\n                                       'they actually work. Now we said before we '\n                                       'have the core transform models and what S '\n                                       'BERT does is fine tunes on sentence pairs '\n                                       'using what is called a Siamese '\n                                       'architecture or Siamese network. What we '\n                                       'mean by a Siamese network is that we have '\n                                       'what we can see, what can view as two BERT '\n                                       'models that are identical and the weights '\n                                       'between those two models are tied. Now in '\n                                       'reality when implementing this we just use '\n                                       'a single BERT model. And what we do is we '\n                                       'process one sentence, a sentence A through '\n                                       'the model and then we process another '\n                                       'sentence, sentence B through the model. '\n                                       \"And that's the sentence pair. So with our \"\n                                       'cross-linked we were processing the '\n                                       'sentence pair together. We were putting '\n                                       'them both together, processing them all at '\n                                       'once. This time we process them '\n                                       'separately. And during training what '\n                                       'happens is the weights',\n                               'title': 'Intro to Sentence Embeddings with '\n                                        'Transformers',\n                               'url': 'https://youtu.be/WS1uVMGhlWQ'},\n                  'score': 0.85855335,\n                  'sparseValues': {},\n                  'values': []}],\n     'namespace': ''}\n\n\n\n\n```python\nlimit = 3750\n\ndef retrieve(query):\n    res = openai.Embedding.create(\n        input=[query],\n        engine=embed_model\n    )\n\n    # retrieve from Pinecone\n    xq = res['data'][0]['embedding']\n\n    # get relevant contexts\n    res = index.query(xq, top_k=3, include_metadata=True)\n    contexts = [\n        x['metadata']['text'] for x in res['matches']\n    ]\n\n    # build our prompt with the retrieved contexts included\n    prompt_start = (\n        \"Answer the question based on the context below.\\n\\n\"+\n        \"Context:\\n\"\n    )\n    prompt_end = (\n        f\"\\n\\nQuestion: {query}\\nAnswer:\"\n    )\n    # append contexts until hitting limit\n    for i in range(1, len(contexts)):\n        if len(\"\\n\\n---\\n\\n\".join(contexts[:i])) >= limit:\n            prompt = (\n                prompt_start +\n                \"\\n\\n---\\n\\n\".join(contexts[:i-1]) +\n                prompt_end\n            )\n            break\n        elif i == len(contexts)-1:\n            prompt = (\n                prompt_start +\n                \"\\n\\n---\\n\\n\".join(contexts) +\n                prompt_end\n            )\n    return prompt\n```\n\n\n```python\n# first we retrieve relevant items from Pinecone\nquery_with_contexts = retrieve(query)\nquery_with_contexts\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: b5fc7ab3-b241-471d-b87a-d159846f12de
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"b5fc7ab3-b241-471d-b87a-d159846f12de\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"589ac817-3347-4f65-952d-f2507a7fb865\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"63b8ea87-b494-4288-bc80-1af62ec7d4ab\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"0e053cd1e57d1c9e74a03f92a55861b98280f55e0e011ee7d3f342b527e7af8c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dd91c44c-acfb-4970-86f2-9a2743fccc8b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1ada838368526ceb327e8610d5483bd454bd64380bbaf5f776ab1f796622882a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21393, \"end_char_idx\": 24922, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "589ac817-3347-4f65-952d-f2507a7fb865", "doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "ref_doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "chroma:document": "This time we process them '\n                                       'separately. And during training what '\n                                       'happens is the weights',\n                               'title': 'Intro to Sentence Embeddings with '\n                                        'Transformers',\n                               'url': 'https://youtu.be/WS1uVMGhlWQ'},\n                  'score': 0.85855335,\n                  'sparseValues': {},\n                  'values': []}],\n     'namespace': ''}\n\n\n\n\n```python\nlimit = 3750\n\ndef retrieve(query):\n    res = openai.Embedding.create(\n        input=[query],\n        engine=embed_model\n    )\n\n    # retrieve from Pinecone\n    xq = res['data'][0]['embedding']\n\n    # get relevant contexts\n    res = index.query(xq, top_k=3, include_metadata=True)\n    contexts = [\n        x['metadata']['text'] for x in res['matches']\n    ]\n\n    # build our prompt with the retrieved contexts included\n    prompt_start = (\n        \"Answer the question based on the context below.\\n\\n\"+\n        \"Context:\\n\"\n    )\n    prompt_end = (\n        f\"\\n\\nQuestion: {query}\\nAnswer:\"\n    )\n    # append contexts until hitting limit\n    for i in range(1, len(contexts)):\n        if len(\"\\n\\n---\\n\\n\".join(contexts[:i])) >= limit:\n            prompt = (\n                prompt_start +\n                \"\\n\\n---\\n\\n\".join(contexts[:i-1]) +\n                prompt_end\n            )\n            break\n        elif i == len(contexts)-1:\n            prompt = (\n                prompt_start +\n                \"\\n\\n---\\n\\n\".join(contexts) +\n                prompt_end\n            )\n    return prompt\n```\n\n\n```python\n# first we retrieve relevant items from Pinecone\nquery_with_contexts = retrieve(query)\nquery_with_contexts\n```\n\n\n\n\n    \"Answer the question based on the context below.\\n\\nContext:\\npairs of related sentences you can go ahead and actually try training or fine-tuning using NLI with multiple negative ranking loss. If you don't have that fine. Another option is that you have a semantic textual similarity data set or STS and what this is is you have so you have sentence A here, sentence B here and then you have a score from from 0 to 1 that tells you the similarity between those two scores and you would train this using something like cosine similarity loss. Now if that's not an option and your focus or use case is on building a sentence transformer for another language where there is no current sentence transformer you can use multilingual parallel data. So what I mean by that is so parallel data just means translation pairs so if you have for example a English sentence and then you have another language here so it can it can be anything I'm just going to put XX and that XX is your target language you can fine-tune a model using something called multilingual knowledge distillation and what that does is takes a monolingual model for example in English and using those translation pairs it distills the knowledge the semantic similarity knowledge from that monolingual English model into a multilingual model which can handle both English and your target language. So they're three options quite popular very common that you can go for and as a supervised methods the chances are that probably going to outperform anything you do with unsupervised training at least for now. So if none of those sound like something\\n\\n---\\n\\nwere actually more accurate. So we can't really do that. We can't use this what is called a mean pooling approach. Or we can't use it in its current form."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: dd91c44c-acfb-4970-86f2-9a2743fccc8b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"dd91c44c-acfb-4970-86f2-9a2743fccc8b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"589ac817-3347-4f65-952d-f2507a7fb865\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b5fc7ab3-b241-471d-b87a-d159846f12de\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7c736400978f041b600d7822c32c022845a228b0bf893f560e6fee9e3e69d5d7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"134874c4-0821-4bce-b276-b224c7cb5263\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"df5db25bd1aee09d8e65c26743f2013f3c77dc990cf80ad4b5b60c37220b2734\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 23018, \"end_char_idx\": 26475, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "589ac817-3347-4f65-952d-f2507a7fb865", "doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "ref_doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "chroma:document": "```python\n# first we retrieve relevant items from Pinecone\nquery_with_contexts = retrieve(query)\nquery_with_contexts\n```\n\n\n\n\n    \"Answer the question based on the context below.\\n\\nContext:\\npairs of related sentences you can go ahead and actually try training or fine-tuning using NLI with multiple negative ranking loss. If you don't have that fine. Another option is that you have a semantic textual similarity data set or STS and what this is is you have so you have sentence A here, sentence B here and then you have a score from from 0 to 1 that tells you the similarity between those two scores and you would train this using something like cosine similarity loss. Now if that's not an option and your focus or use case is on building a sentence transformer for another language where there is no current sentence transformer you can use multilingual parallel data. So what I mean by that is so parallel data just means translation pairs so if you have for example a English sentence and then you have another language here so it can it can be anything I'm just going to put XX and that XX is your target language you can fine-tune a model using something called multilingual knowledge distillation and what that does is takes a monolingual model for example in English and using those translation pairs it distills the knowledge the semantic similarity knowledge from that monolingual English model into a multilingual model which can handle both English and your target language. So they're three options quite popular very common that you can go for and as a supervised methods the chances are that probably going to outperform anything you do with unsupervised training at least for now. So if none of those sound like something\\n\\n---\\n\\nwere actually more accurate. So we can't really do that. We can't use this what is called a mean pooling approach. Or we can't use it in its current form. Now the solution to this problem was introduced by two people in 2019 Nils Reimers and Irenia Gurevich. They introduced what is the first sentence transformer or sentence BERT. And it was found that sentence BERT or S BERT outformed all of the previous Save the Art models on pretty much all benchmarks. Not all of them but most of them. And it did it in a very quick time. So if we compare it to BERT, if we wanted to find the most similar sentence pair from 10,000 sentences in that 2019 paper they found that with BERT that took 65 hours. With S BERT embeddings they could create all the embeddings in just around five seconds. And then they could compare all those with cosine similarity in 0.01 seconds. So it's a lot faster. We go from 65 hours to just over five seconds which is I think pretty incredible. Now I think that's pretty much all the context we need behind sentence transformers. And what we do now is dive into a little bit of how they actually work. Now we said before we have the core transform models and what S BERT does is fine tunes on sentence pairs using what is called a Siamese architecture or Siamese network. What we mean by a Siamese network is that we have what we can see, what can view as two BERT models that are identical and the weights between those two models are tied. Now in reality when implementing this we just use a single BERT model. And what we do is we process one sentence, a sentence A through the model and then we process another sentence, sentence B through the model. And that's the sentence pair."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 134874c4-0821-4bce-b276-b224c7cb5263
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"134874c4-0821-4bce-b276-b224c7cb5263\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"589ac817-3347-4f65-952d-f2507a7fb865\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dd91c44c-acfb-4970-86f2-9a2743fccc8b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"e02a5a65d348c8771c4b7055d79f29dcf2917709014ae47efa8522110b08969c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"207ac258-1706-4a67-9de6-96bd1453004f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"189755547952daeaed473ce5344dc206add394dadb0d8f96d42ede0a639b6051\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 24825, \"end_char_idx\": 28068, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "589ac817-3347-4f65-952d-f2507a7fb865", "doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "ref_doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "chroma:document": "We can't use this what is called a mean pooling approach. Or we can't use it in its current form. Now the solution to this problem was introduced by two people in 2019 Nils Reimers and Irenia Gurevich. They introduced what is the first sentence transformer or sentence BERT. And it was found that sentence BERT or S BERT outformed all of the previous Save the Art models on pretty much all benchmarks. Not all of them but most of them. And it did it in a very quick time. So if we compare it to BERT, if we wanted to find the most similar sentence pair from 10,000 sentences in that 2019 paper they found that with BERT that took 65 hours. With S BERT embeddings they could create all the embeddings in just around five seconds. And then they could compare all those with cosine similarity in 0.01 seconds. So it's a lot faster. We go from 65 hours to just over five seconds which is I think pretty incredible. Now I think that's pretty much all the context we need behind sentence transformers. And what we do now is dive into a little bit of how they actually work. Now we said before we have the core transform models and what S BERT does is fine tunes on sentence pairs using what is called a Siamese architecture or Siamese network. What we mean by a Siamese network is that we have what we can see, what can view as two BERT models that are identical and the weights between those two models are tied. Now in reality when implementing this we just use a single BERT model. And what we do is we process one sentence, a sentence A through the model and then we process another sentence, sentence B through the model. And that's the sentence pair. So with our cross-linked we were processing the sentence pair together. We were putting them both together, processing them all at once. This time we process them separately. And during training what happens is the weights\\n\\n---\\n\\nTransformer-based Sequential Denoising Autoencoder. So what we'll do is jump straight into it and take a look at where we might want to use this training approach and and how we can actually implement it. So the first question we need to ask is do we really need to resort to unsupervised training? Now what we're going to do here is just have a look at a few of the most popular training approaches and what sort of data we need for that. So the first one we're looking at here is Natural Language Inference or NLI and NLI requires that we have pairs of sentences that are labeled as either contradictory, neutral which means they're not necessarily related or as entailing or as inferring each other. So you have pairs that entail each other so they are both very similar pairs that are neutral and also pairs that are contradictory. And this is the traditional NLI data. Now using another version of fine-tuning with NLI called a multiple negatives ranking loss you can get by with only entailment pairs so pairs that are related to each other or positive pairs and it can also use contradictory pairs to improve the performance of training as well but you don't need it. So if you have positive pairs of related sentences you can go ahead and actually try training or fine-tuning using NLI with multiple negative ranking loss. If you don't have that fine."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 207ac258-1706-4a67-9de6-96bd1453004f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb", "title": "Retrieval Augmented Generative Question Answering with Pinecone", "tokens": 6559, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"207ac258-1706-4a67-9de6-96bd1453004f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"589ac817-3347-4f65-952d-f2507a7fb865\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"07667fc3f6e7fea2522ca28f9b3e0cab0d2b258e6889dbc2ccb70f9042002e0f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"134874c4-0821-4bce-b276-b224c7cb5263\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Gen_QA.ipynb\", \"title\": \"Retrieval Augmented Generative Question Answering with Pinecone\", \"tokens\": 6559, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ec6b25456bccafc04e1b73847976164c559f7a2252fd7195c2ee806f4e59cd7f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 26233, \"end_char_idx\": 28675, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "589ac817-3347-4f65-952d-f2507a7fb865", "doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "ref_doc_id": "589ac817-3347-4f65-952d-f2507a7fb865", "chroma:document": "Now in reality when implementing this we just use a single BERT model. And what we do is we process one sentence, a sentence A through the model and then we process another sentence, sentence B through the model. And that's the sentence pair. So with our cross-linked we were processing the sentence pair together. We were putting them both together, processing them all at once. This time we process them separately. And during training what happens is the weights\\n\\n---\\n\\nTransformer-based Sequential Denoising Autoencoder. So what we'll do is jump straight into it and take a look at where we might want to use this training approach and and how we can actually implement it. So the first question we need to ask is do we really need to resort to unsupervised training? Now what we're going to do here is just have a look at a few of the most popular training approaches and what sort of data we need for that. So the first one we're looking at here is Natural Language Inference or NLI and NLI requires that we have pairs of sentences that are labeled as either contradictory, neutral which means they're not necessarily related or as entailing or as inferring each other. So you have pairs that entail each other so they are both very similar pairs that are neutral and also pairs that are contradictory. And this is the traditional NLI data. Now using another version of fine-tuning with NLI called a multiple negatives ranking loss you can get by with only entailment pairs so pairs that are related to each other or positive pairs and it can also use contradictory pairs to improve the performance of training as well but you don't need it. So if you have positive pairs of related sentences you can go ahead and actually try training or fine-tuning using NLI with multiple negative ranking loss. If you don't have that fine. Another option is that you have a semantic textual similarity data set or STS and what this is is you have so you have sentence A here, sentence B\\n\\nQuestion: Which training method should I use for sentence transformers when I only have pairs of related sentences?\\nAnswer:\"\n\n\n\n\n```python\n# then we complete the context-infused query\ncomplete(query_with_contexts)\n```\n\n\n\n\n    'You should use Natural Language Inference (NLI) with multiple negative ranking loss.'\n\n\n\nAnd we get a pretty great answer straight away, specifying to use _multiple-rankings loss_ (also called _multiple negatives ranking loss_)."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: e3a4fb4c-2f00-4f72-bf0d-6aab2c2379ac
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e3a4fb4c-2f00-4f72-bf0d-6aab2c2379ac\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fb6f226e-e286-4750-9744-c62096e3194a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"33fa5220faea5bfd8fe4f54ee19a884847c32fa2af84ff1fe447e99a7369ce46\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1673, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "# Retrieval Augmentation for GPT-4 using Pinecone\n\n#### Fixing LLMs that Hallucinate\n\nIn this notebook we will learn how to query relevant contexts to our queries from Pinecone, and pass these to a GPT-4 model to generate an answer backed by real data sources.\n\nGPT-4 is a big step up from previous OpenAI completion models. It also exclusively uses the `ChatCompletion` endpoint, so we must use it in a slightly different way to usual. However, the power of the model makes the change worthwhile, particularly when augmented with an external knowledge base like the Pinecone vector database.\n\nRequired installs for this notebook are:\n\n\n```python\n!pip install -qU bs4 tiktoken openai langchain pinecone-client[grpc]\n```\n\n    \u001b[?25l     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m0.0/1.7 MB\u001b[0m \u001b[31m?\u001b[0m eta \u001b[36m-:--:--\u001b[0m\n\u001b[2K     \u001b[91m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m\u001b[91m\u2578\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m71.4 MB/s\u001b[0m eta \u001b[36m0:00:01\u001b[0m\n\u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.7/1.7 MB\u001b[0m \u001b[31m41.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m70.1/70.1 KB\u001b[0m \u001b[31m6.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m396.0/396.0 KB\u001b[0m \u001b[31m28.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m177.2/177.2 KB\u001b[0m \u001b[31m12.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m62.8/62.8 KB\u001b[0m \u001b[31m4.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.0/1.0 MB\u001b[0m \u001b[31m4."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: fb6f226e-e286-4750-9744-c62096e3194a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"fb6f226e-e286-4750-9744-c62096e3194a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e3a4fb4c-2f00-4f72-bf0d-6aab2c2379ac\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"8729dacca690457bb82592b5adfe93af37c350dd8cc32e6fb9062ffc0111d68e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f04e803e-8c56-4a49-8691-f608b089358d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6b1f62835839d80a6698b640686e58de5455b9917b873b62e21dec198ebc2fdb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1032, \"end_char_idx\": 2293, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "7/1.7 MB\u001b[0m \u001b[31m41.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m70.1/70.1 KB\u001b[0m \u001b[31m6.5 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m396.0/396.0 KB\u001b[0m \u001b[31m28.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m177.2/177.2 KB\u001b[0m \u001b[31m12.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m62.8/62.8 KB\u001b[0m \u001b[31m4.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.0/1.0 MB\u001b[0m \u001b[31m4.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m58.3/58.3 KB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m43.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m77.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m158.8/158.8 KB\u001b[0m \u001b[31m19.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m199.2/199.2 KB\u001b[0m \u001b[31m26."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: f04e803e-8c56-4a49-8691-f608b089358d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"f04e803e-8c56-4a49-8691-f608b089358d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fb6f226e-e286-4750-9744-c62096e3194a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"85171f2f2dbf462eb9972e6582a16c9eb0dc2eb6ecb795c9b1a07b44b2e215bd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"adb39d11-9263-4dbf-ac82-812c655d48d7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7651f0edd1aa663f4a2cf2b3472ca1dc01bf96aacf6f3c0de5d7b76a4619100f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1653, \"end_char_idx\": 2917, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "0/1.0 MB\u001b[0m \u001b[31m4.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m58.3/58.3 KB\u001b[0m \u001b[31m8.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.1/1.1 MB\u001b[0m \u001b[31m43.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m1.3/1.3 MB\u001b[0m \u001b[31m77.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m158.8/158.8 KB\u001b[0m \u001b[31m19.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m199.2/199.2 KB\u001b[0m \u001b[31m26.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m264.6/264.6 KB\u001b[0m \u001b[31m35.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m114.2/114.2 KB\u001b[0m \u001b[31m15.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m49.1/49.1 KB\u001b[0m \u001b[31m7.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m218.0/218.0 KB\u001b[0m \u001b[31m27.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m218.0/218.0 KB\u001b[0m \u001b[31m28."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: adb39d11-9263-4dbf-ac82-812c655d48d7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"adb39d11-9263-4dbf-ac82-812c655d48d7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f04e803e-8c56-4a49-8691-f608b089358d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4814286645455af5e261bbab2e281d5845c031285a090d9695b91aa415f017ac\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b1c7946d-4b5c-466e-bcd0-853ffa0d51a8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4af58b1b57bedcabe1f1f1c8e2d73085660a1fa12e6a30f57541b4da00f9a45a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2270, \"end_char_idx\": 3650, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "2/199.2 KB\u001b[0m \u001b[31m26.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m264.6/264.6 KB\u001b[0m \u001b[31m35.1 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m114.2/114.2 KB\u001b[0m \u001b[31m15.6 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m49.1/49.1 KB\u001b[0m \u001b[31m7.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m218.0/218.0 KB\u001b[0m \u001b[31m27.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m218.0/218.0 KB\u001b[0m \u001b[31m28.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m211.7/211.7 KB\u001b[0m \u001b[31m12.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25h\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n    google-cloud-translate 3.8.4 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\n    google-cloud-language 2.6.1 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: b1c7946d-4b5c-466e-bcd0-853ffa0d51a8
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"b1c7946d-4b5c-466e-bcd0-853ffa0d51a8\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"adb39d11-9263-4dbf-ac82-812c655d48d7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"939fda61730eecb9c5c28e66f1d8d19044eb384cd432449ad0771beeff89f973\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c062f02b-55ba-4233-8d3d-39497da32780\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5981b35275acee87e1de108f9acff08c84e42b89fb461f0d3e4658537cbc9714\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2792, \"end_char_idx\": 4232, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m218.0/218.0 KB\u001b[0m \u001b[31m28.7 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m211.7/211.7 KB\u001b[0m \u001b[31m12.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25h\u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n    google-cloud-translate 3.8.4 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\n    google-cloud-language 2.6.1 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\n    google-cloud-firestore 2.7.3 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\n    google-cloud-datastore 2.11.1 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\n    google-cloud-bigquery 3.4.2 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: c062f02b-55ba-4233-8d3d-39497da32780
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c062f02b-55ba-4233-8d3d-39497da32780\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b1c7946d-4b5c-466e-bcd0-853ffa0d51a8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"36bb417b286c1fda73e46db556fc31aa09245f69d078e42495b472ec6f8041f7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5b06fdfb-e8a7-4cdf-8043-2935ef43df4d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3b4df178cd3ed185eb91620d1f33ad7ed08521e4dd9de110d07a265fb46a2112\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3462, \"end_char_idx\": 5119, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "google-cloud-language 2.6.1 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\n    google-cloud-firestore 2.7.3 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\n    google-cloud-datastore 2.11.1 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\n    google-cloud-bigquery 3.4.2 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\n    google-cloud-bigquery-storage 2.19.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\n    google-api-core 2.11.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\u001b[0m\u001b[31m\n    \u001b[0m\n\n## Preparing the Data\n\nIn this example, we will download the LangChain docs from [langchain.readthedocs.io/](https://langchain.readthedocs.io/latest/en/). We get all `.html` files located on the site like so:\n\n\n```python\n!wget -r -A.html -P rtdocs https://python.langchain.com/en/latest/\n```\n\n\n\n\n    <Response [200]>\n\n\n\nThis downloads all HTML into the `rtdocs` directory. Now we can use LangChain itself to process these docs. We do this using the `ReadTheDocsLoader` like so:"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 5b06fdfb-e8a7-4cdf-8043-2935ef43df4d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"5b06fdfb-e8a7-4cdf-8043-2935ef43df4d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c062f02b-55ba-4233-8d3d-39497da32780\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"dbd627b97251beb2478a1594283d7109726abce79fd911b433779b9d2f62676b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"072b7034-8041-4114-ad4f-bf248145fe74\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"047fa90d76d86dbb4dc2d7b37b486c2d03bad7b2e6e873c723d50f14d9922768\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4237, \"end_char_idx\": 7142, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "google-cloud-bigquery-storage 2.19.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\n    google-api-core 2.11.0 requires protobuf!=3.20.0,!=3.20.1,!=4.21.0,!=4.21.1,!=4.21.2,!=4.21.3,!=4.21.4,!=4.21.5,<5.0.0dev,>=3.19.5, but you have protobuf 3.19.3 which is incompatible.\u001b[0m\u001b[31m\n    \u001b[0m\n\n## Preparing the Data\n\nIn this example, we will download the LangChain docs from [langchain.readthedocs.io/](https://langchain.readthedocs.io/latest/en/). We get all `.html` files located on the site like so:\n\n\n```python\n!wget -r -A.html -P rtdocs https://python.langchain.com/en/latest/\n```\n\n\n\n\n    <Response [200]>\n\n\n\nThis downloads all HTML into the `rtdocs` directory. Now we can use LangChain itself to process these docs. We do this using the `ReadTheDocsLoader` like so:\n\n\n```python\nfrom langchain.document_loaders import ReadTheDocsLoader\n\nloader = ReadTheDocsLoader('rtdocs')\ndocs = loader.load()\nlen(docs)\n```\n\n    .rst .pdf Welcome to LangChain Contents Getting Started Modules Use Cases Reference Docs LangChain Ecosystem Additional Resources Welcome to LangChain# Large language models (LLMs) are emerging as a transformative technology, enabling developers to build applications that they previously could not. But using these LLMs in isolation is often not enough to create a truly powerful app - the real power comes when you are able to combine them with other sources of computation or knowledge. This library is aimed at assisting in the development of those types of applications. Common examples of these types of applications include: \u2753 Question Answering over specific documents Documentation End-to-end Example: Question Answering over Notion Database \ud83d\udcac Chatbots Documentation End-to-end Example: Chat-LangChain \ud83e\udd16 Agents Documentation End-to-end Example: GPT+WolframAlpha Getting Started# Checkout the below guide for a walkthrough of how to get started using LangChain to create an Language Model application. Getting Started Documentation Modules# There are several main modules that LangChain provides support for. For each module we provide some examples to get started, how-to guides, reference docs, and conceptual guides. These modules are, in increasing order of complexity: Prompts: This includes prompt management, prompt optimization, and prompt serialization. LLMs: This includes a generic interface for all LLMs, and common utilities for working with LLMs. Document Loaders: This includes a standard interface for loading documents, as well as specific integrations to all types of text data sources. Utils: Language models are often more powerful when interacting with other sources of knowledge or computation. This can include Python REPLs, embeddings, search engines, and more. LangChain provides a large collection of common utils to use in your application."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 072b7034-8041-4114-ad4f-bf248145fe74
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"072b7034-8041-4114-ad4f-bf248145fe74\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5b06fdfb-e8a7-4cdf-8043-2935ef43df4d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"7a32b5b2991ca8e0f94078a7e56f53cc416c72fc6672f4b0c2a5f0c9791dc914\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0add6fd9-45a3-4a77-adfb-8462e77316f5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b5fa00a64712c7c6789a8ff7498ba7e6e7c3be5299febcf5d0f7a7303955d9e9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5567, \"end_char_idx\": 9357, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "But using these LLMs in isolation is often not enough to create a truly powerful app - the real power comes when you are able to combine them with other sources of computation or knowledge. This library is aimed at assisting in the development of those types of applications. Common examples of these types of applications include: \u2753 Question Answering over specific documents Documentation End-to-end Example: Question Answering over Notion Database \ud83d\udcac Chatbots Documentation End-to-end Example: Chat-LangChain \ud83e\udd16 Agents Documentation End-to-end Example: GPT+WolframAlpha Getting Started# Checkout the below guide for a walkthrough of how to get started using LangChain to create an Language Model application. Getting Started Documentation Modules# There are several main modules that LangChain provides support for. For each module we provide some examples to get started, how-to guides, reference docs, and conceptual guides. These modules are, in increasing order of complexity: Prompts: This includes prompt management, prompt optimization, and prompt serialization. LLMs: This includes a generic interface for all LLMs, and common utilities for working with LLMs. Document Loaders: This includes a standard interface for loading documents, as well as specific integrations to all types of text data sources. Utils: Language models are often more powerful when interacting with other sources of knowledge or computation. This can include Python REPLs, embeddings, search engines, and more. LangChain provides a large collection of common utils to use in your application. Chains: Chains go beyond just a single LLM call, and are sequences of calls (whether to an LLM or a different utility). LangChain provides a standard interface for chains, lots of integrations with other tools, and end-to-end chains for common applications. Indexes: Language models are often more powerful when combined with your own text data - this module covers best practices for doing exactly that. Agents: Agents involve an LLM making decisions about which Actions to take, taking that Action, seeing an Observation, and repeating that until done. LangChain provides a standard interface for agents, a selection of agents to choose from, and examples of end to end agents. Memory: Memory is the concept of persisting state between calls of a chain/agent. LangChain provides a standard interface for memory, a collection of memory implementations, and examples of chains/agents that use memory. Chat: Chat models are a variation on Language Models that expose a different API - rather than working with raw text, they work with messages. LangChain provides a standard interface for working with them and doing all the same things as above. Use Cases# The above modules can be used in a variety of ways. LangChain also provides guidance and assistance in this. Below are some of the common use cases LangChain supports. Agents: Agents are systems that use a language model to interact with other tools. These can be used to do more grounded question/answering, interact with APIs, or even take actions. Chatbots: Since language models are good at producing text, that makes them ideal for creating chatbots. Data Augmented Generation: Data Augmented Generation involves specific types of chains that first interact with an external datasource to fetch data to use in the generation step. Examples of this include summarization of long pieces of text and question/answering over specific data sources. Question Answering: Answering questions over specific documents, only utilizing the information in those documents to construct an answer. A type of Data Augmented Generation. Summarization: Summarizing longer documents into shorter, more condensed chunks of information. A type of Data Augmented Generation."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 0add6fd9-45a3-4a77-adfb-8462e77316f5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0add6fd9-45a3-4a77-adfb-8462e77316f5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"072b7034-8041-4114-ad4f-bf248145fe74\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"f6dcb4f6476138af315f53b8c1065777db5e592581960433def9a51924a7d01b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5253e28f-869d-47a6-b740-1c26118e6598\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ccd77e967d5990671fee9e6299da6e48b22574b57fe08035a4d356ef69c0da86\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7401, \"end_char_idx\": 11127, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "Indexes: Language models are often more powerful when combined with your own text data - this module covers best practices for doing exactly that. Agents: Agents involve an LLM making decisions about which Actions to take, taking that Action, seeing an Observation, and repeating that until done. LangChain provides a standard interface for agents, a selection of agents to choose from, and examples of end to end agents. Memory: Memory is the concept of persisting state between calls of a chain/agent. LangChain provides a standard interface for memory, a collection of memory implementations, and examples of chains/agents that use memory. Chat: Chat models are a variation on Language Models that expose a different API - rather than working with raw text, they work with messages. LangChain provides a standard interface for working with them and doing all the same things as above. Use Cases# The above modules can be used in a variety of ways. LangChain also provides guidance and assistance in this. Below are some of the common use cases LangChain supports. Agents: Agents are systems that use a language model to interact with other tools. These can be used to do more grounded question/answering, interact with APIs, or even take actions. Chatbots: Since language models are good at producing text, that makes them ideal for creating chatbots. Data Augmented Generation: Data Augmented Generation involves specific types of chains that first interact with an external datasource to fetch data to use in the generation step. Examples of this include summarization of long pieces of text and question/answering over specific data sources. Question Answering: Answering questions over specific documents, only utilizing the information in those documents to construct an answer. A type of Data Augmented Generation. Summarization: Summarizing longer documents into shorter, more condensed chunks of information. A type of Data Augmented Generation. Evaluation: Generative models are notoriously hard to evaluate with traditional metrics. One new way of evaluating them is using language models themselves to do the evaluation. LangChain provides some prompts/chains for assisting in this. Generate similar examples: Generating similar examples to a given input. This is a common use case for many applications, and LangChain provides some prompts/chains for assisting in this. Compare models: Experimenting with different prompts, models, and chains is a big part of developing the best possible application. The ModelLaboratory makes it easy to do so. Reference Docs# All of LangChain\u2019s reference documentation, in one place. Full documentation on all methods, classes, installation methods, and integration setups for LangChain. Reference Documentation LangChain Ecosystem# Guides for how other companies/products can be used with LangChain LangChain Ecosystem Additional Resources# Additional collection of resources we think may be useful as you develop your application! LangChainHub: The LangChainHub is a place to share and explore other prompts, chains, and agents. Glossary: A glossary of all related terms, papers, methods, etc. Whether implemented in LangChain or not! Gallery: A collection of our favorite projects that use LangChain. Useful for finding inspiration or seeing how things were done in other applications. Deployments: A collection of instructions, code snippets, and template repositories for deploying LangChain apps. Discord: Join us on our Discord to discuss all things LangChain! Tracing: A guide on using tracing in LangChain to visualize the execution of chains and agents. Production Support: As you move your LangChains into production, we\u2019d love to offer more comprehensive support."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 5253e28f-869d-47a6-b740-1c26118e6598
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"5253e28f-869d-47a6-b740-1c26118e6598\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0add6fd9-45a3-4a77-adfb-8462e77316f5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"3ed80ffea10eccbb63d07abf18c31df265fd1905a4c96924d5bafea00646cac2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1aa74647-9ddc-4784-bf37-5d936cc4a310\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"546e6f96602b15542a2a11e010892fbbfe2864bff31dc4751e5575c46ce46016\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9188, \"end_char_idx\": 12529, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "A type of Data Augmented Generation. Summarization: Summarizing longer documents into shorter, more condensed chunks of information. A type of Data Augmented Generation. Evaluation: Generative models are notoriously hard to evaluate with traditional metrics. One new way of evaluating them is using language models themselves to do the evaluation. LangChain provides some prompts/chains for assisting in this. Generate similar examples: Generating similar examples to a given input. This is a common use case for many applications, and LangChain provides some prompts/chains for assisting in this. Compare models: Experimenting with different prompts, models, and chains is a big part of developing the best possible application. The ModelLaboratory makes it easy to do so. Reference Docs# All of LangChain\u2019s reference documentation, in one place. Full documentation on all methods, classes, installation methods, and integration setups for LangChain. Reference Documentation LangChain Ecosystem# Guides for how other companies/products can be used with LangChain LangChain Ecosystem Additional Resources# Additional collection of resources we think may be useful as you develop your application! LangChainHub: The LangChainHub is a place to share and explore other prompts, chains, and agents. Glossary: A glossary of all related terms, papers, methods, etc. Whether implemented in LangChain or not! Gallery: A collection of our favorite projects that use LangChain. Useful for finding inspiration or seeing how things were done in other applications. Deployments: A collection of instructions, code snippets, and template repositories for deploying LangChain apps. Discord: Join us on our Discord to discuss all things LangChain! Tracing: A guide on using tracing in LangChain to visualize the execution of chains and agents. Production Support: As you move your LangChains into production, we\u2019d love to offer more comprehensive support. Please fill out this form and we\u2019ll set up a dedicated support Slack channel. next Quickstart Guide Contents Getting Started Modules Use Cases Reference Docs LangChain Ecosystem Additional Resources By Harrison Chase \u00a9 Copyright 2022, Harrison Chase. Last updated on Mar 15, 2023.\n\n\nThis leaves us with hundreds of processed doc pages. Let's take a look at the format each one contains:\n\n\n```python\ndocs[0]\n```\n\nWe access the plaintext page content like so:\n\n\n```python\nprint(docs[0].page_content)\n```\n\n\n```python\nprint(docs[5].page_content)\n```\n\nWe can also find the source of each document:\n\n\n```python\ndocs[5].metadata['source'].replace('rtdocs/', 'https://')\n```\n\nWe can use these to create our `data` list:\n\n\n```python\ndata = []\n\nfor doc in docs:\n    data.append({\n        'url': doc.metadata['source'].replace('rtdocs/', 'https://'),\n        'text': doc.page_content\n    })\n```\n\n\n```python\ndata[3]\n```\n\n\n\n\n    {'url': 'https://langchain.readthedocs.io/en/latest/modules/memory/types/entity_summary_memory.html',\n     'text': '.ipynb .pdf Entity Memory Contents Using in a chain Inspecting the memory store Entity Memory# This notebook shows how to work with a memory module that remembers things about specific entities. It extracts information on entities (using LLMs) and builds up its knowledge about that entity over time (also using LLMs). Let\u2019s first walk through using this functionality."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 1aa74647-9ddc-4784-bf37-5d936cc4a310
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"1aa74647-9ddc-4784-bf37-5d936cc4a310\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5253e28f-869d-47a6-b740-1c26118e6598\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e65fb1ac832c8f6252e556c9c1995382cec518a7b911f30e790d5695ee8271db\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f53d38bc-5a84-4aae-bd3b-e758735dd275\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"eb0b4c77e92b7376399c30deb3690ca2c3a7c9f91e36ad5e8afbf75e86e174e8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10855, \"end_char_idx\": 13916, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "Discord: Join us on our Discord to discuss all things LangChain! Tracing: A guide on using tracing in LangChain to visualize the execution of chains and agents. Production Support: As you move your LangChains into production, we\u2019d love to offer more comprehensive support. Please fill out this form and we\u2019ll set up a dedicated support Slack channel. next Quickstart Guide Contents Getting Started Modules Use Cases Reference Docs LangChain Ecosystem Additional Resources By Harrison Chase \u00a9 Copyright 2022, Harrison Chase. Last updated on Mar 15, 2023.\n\n\nThis leaves us with hundreds of processed doc pages. Let's take a look at the format each one contains:\n\n\n```python\ndocs[0]\n```\n\nWe access the plaintext page content like so:\n\n\n```python\nprint(docs[0].page_content)\n```\n\n\n```python\nprint(docs[5].page_content)\n```\n\nWe can also find the source of each document:\n\n\n```python\ndocs[5].metadata['source'].replace('rtdocs/', 'https://')\n```\n\nWe can use these to create our `data` list:\n\n\n```python\ndata = []\n\nfor doc in docs:\n    data.append({\n        'url': doc.metadata['source'].replace('rtdocs/', 'https://'),\n        'text': doc.page_content\n    })\n```\n\n\n```python\ndata[3]\n```\n\n\n\n\n    {'url': 'https://langchain.readthedocs.io/en/latest/modules/memory/types/entity_summary_memory.html',\n     'text': '.ipynb .pdf Entity Memory Contents Using in a chain Inspecting the memory store Entity Memory# This notebook shows how to work with a memory module that remembers things about specific entities. It extracts information on entities (using LLMs) and builds up its knowledge about that entity over time (also using LLMs). Let\u2019s first walk through using this functionality. from langchain.llms import OpenAI from langchain.memory import ConversationEntityMemory llm = OpenAI(temperature=0) memory = ConversationEntityMemory(llm=llm) _input = {\"input\": \"Deven & Sam are working on a hackathon project\"} memory.load_memory_variables(_input) memory.save_context( _input, {\"ouput\": \" That sounds like a great project! What kind of project are they working on?\"} ) memory.load_memory_variables({\"input\": \\'who is Sam\\'}) {\\'history\\': \\'Human: Deven & Sam are working on a hackathon project\\\\nAI: That sounds like a great project! What kind of project are they working on?\\', \\'entities\\': {\\'Sam\\': \\'Sam is working on a hackathon project with Deven.\\'}} memory = ConversationEntityMemory(llm=llm, return_messages=True) _input = {\"input\": \"Deven & Sam are working on a hackathon project\"} memory.load_memory_variables(_input) memory.save_context( _input, {\"ouput\": \" That sounds like a great project! What kind of project are they working on?\"} ) memory.load_memory_variables({\"input\": \\'who is Sam\\'}) {\\'history\\': [HumanMessage(content=\\'Deven & Sam are working on a hackathon project\\', additional_kwargs={}), AIMessage(content=\\' That sounds like a great project! What kind of project are they working on?\\', additional_kwargs={})], \\'entities\\': {\\'Sam\\': \\'Sam is working on a hackathon project with Deven.\\'}} Using in a chain# Let\u2019s now use it in a chain!"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: f53d38bc-5a84-4aae-bd3b-e758735dd275
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"f53d38bc-5a84-4aae-bd3b-e758735dd275\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1aa74647-9ddc-4784-bf37-5d936cc4a310\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"cecd15a4d5c470a67807b3bb66c2ee5038ec8402bfbdd50fad66031e265c5a8c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"474ff370-8c09-4b59-a9d4-cb6f25ea06c6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"93a94fad140377b4fb75dcedd534cdc40b04b0030773f7bad3fb8d462f608936\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12355, \"end_char_idx\": 15767, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "It extracts information on entities (using LLMs) and builds up its knowledge about that entity over time (also using LLMs). Let\u2019s first walk through using this functionality. from langchain.llms import OpenAI from langchain.memory import ConversationEntityMemory llm = OpenAI(temperature=0) memory = ConversationEntityMemory(llm=llm) _input = {\"input\": \"Deven & Sam are working on a hackathon project\"} memory.load_memory_variables(_input) memory.save_context( _input, {\"ouput\": \" That sounds like a great project! What kind of project are they working on?\"} ) memory.load_memory_variables({\"input\": \\'who is Sam\\'}) {\\'history\\': \\'Human: Deven & Sam are working on a hackathon project\\\\nAI: That sounds like a great project! What kind of project are they working on?\\', \\'entities\\': {\\'Sam\\': \\'Sam is working on a hackathon project with Deven.\\'}} memory = ConversationEntityMemory(llm=llm, return_messages=True) _input = {\"input\": \"Deven & Sam are working on a hackathon project\"} memory.load_memory_variables(_input) memory.save_context( _input, {\"ouput\": \" That sounds like a great project! What kind of project are they working on?\"} ) memory.load_memory_variables({\"input\": \\'who is Sam\\'}) {\\'history\\': [HumanMessage(content=\\'Deven & Sam are working on a hackathon project\\', additional_kwargs={}), AIMessage(content=\\' That sounds like a great project! What kind of project are they working on?\\', additional_kwargs={})], \\'entities\\': {\\'Sam\\': \\'Sam is working on a hackathon project with Deven.\\'}} Using in a chain# Let\u2019s now use it in a chain! from langchain.chains import ConversationChain from langchain.memory import ConversationEntityMemory from langchain.memory.prompt import ENTITY_MEMORY_CONVERSATION_TEMPLATE from pydantic import BaseModel from typing import List, Dict, Any conversation = ConversationChain( llm=llm, verbose=True, prompt=ENTITY_MEMORY_CONVERSATION_TEMPLATE, memory=ConversationEntityMemory(llm=llm) ) conversation.predict(input=\"Deven & Sam are working on a hackathon project\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI. You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 474ff370-8c09-4b59-a9d4-cb6f25ea06c6
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"474ff370-8c09-4b59-a9d4-cb6f25ea06c6\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f53d38bc-5a84-4aae-bd3b-e758735dd275\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"89af7ccefa09c4a4e39820edb1492229d5ecd9fa1f934fe5b69d483ec3538e21\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ca062848-ecae-4edc-a7b8-c6a022d9092c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"03547a629b37fc8603bba0e32de65c7732bf561fa4cbcc427b5f453c13c8d94a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13917, \"end_char_idx\": 17645, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "from langchain.chains import ConversationChain from langchain.memory import ConversationEntityMemory from langchain.memory.prompt import ENTITY_MEMORY_CONVERSATION_TEMPLATE from pydantic import BaseModel from typing import List, Dict, Any conversation = ConversationChain( llm=llm, verbose=True, prompt=ENTITY_MEMORY_CONVERSATION_TEMPLATE, memory=ConversationEntityMemory(llm=llm) ) conversation.predict(input=\"Deven & Sam are working on a hackathon project\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI. You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist. Context: {\\'Deven\\': \\'\\', \\'Sam\\': \\'\\'} Current conversation: Last line: Human: Deven & Sam are working on a hackathon project You: > Finished chain. \\' That sounds like a great project! What kind of project are they working on?\\' conversation.memory.store {\\'Deven\\': \\'Deven is working on a hackathon project with Sam.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven.\\'} conversation.predict(input=\"They are trying to add more complex memory structures to Langchain\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI. You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: ca062848-ecae-4edc-a7b8-c6a022d9092c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"ca062848-ecae-4edc-a7b8-c6a022d9092c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"474ff370-8c09-4b59-a9d4-cb6f25ea06c6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"d7488772900bd8b20547485ab55317b653c7a2b0442409710b0099a03223bee3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"646e41f7-c150-44a0-a4cd-88d9cb08ac6d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"73b65920df4df45507208d9e663600963566a119a9a3dbf58ef5150a103d72f6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15768, \"end_char_idx\": 19392, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "Context: {\\'Deven\\': \\'\\', \\'Sam\\': \\'\\'} Current conversation: Last line: Human: Deven & Sam are working on a hackathon project You: > Finished chain. \\' That sounds like a great project! What kind of project are they working on?\\' conversation.memory.store {\\'Deven\\': \\'Deven is working on a hackathon project with Sam.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven.\\'} conversation.predict(input=\"They are trying to add more complex memory structures to Langchain\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI. You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist. Context: {\\'Deven\\': \\'Deven is working on a hackathon project with Sam.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven.\\', \\'Langchain\\': \\'\\'} Current conversation: Human: Deven & Sam are working on a hackathon project AI: That sounds like a great project! What kind of project are they working on? Last line: Human: They are trying to add more complex memory structures to Langchain You: > Finished chain. \\' That sounds like an interesting project! What kind of memory structures are they trying to add?\\' conversation.predict(input=\"They are adding in a key-value store for entities mentioned so far in the conversation.\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI. You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 646e41f7-c150-44a0-a4cd-88d9cb08ac6d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"646e41f7-c150-44a0-a4cd-88d9cb08ac6d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ca062848-ecae-4edc-a7b8-c6a022d9092c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"f092d6c94646f26b16bb6abec0412376672cda66365701d81ea86aaabbe20465\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9292521e-6565-42d8-a437-3f2dbb15bd81\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4bbc045004cbbbe3e448cd9602c4263ad9580a4dfc81b60eaff21735afcfd15c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17505, \"end_char_idx\": 20991, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist. Context: {\\'Deven\\': \\'Deven is working on a hackathon project with Sam.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven.\\', \\'Langchain\\': \\'\\'} Current conversation: Human: Deven & Sam are working on a hackathon project AI: That sounds like a great project! What kind of project are they working on? Last line: Human: They are trying to add more complex memory structures to Langchain You: > Finished chain. \\' That sounds like an interesting project! What kind of memory structures are they trying to add?\\' conversation.predict(input=\"They are adding in a key-value store for entities mentioned so far in the conversation.\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI. You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist. Context: {\\'Deven\\': \\'Deven is working on a hackathon project with Sam, attempting to add more complex memory structures to Langchain.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven, trying to add more complex memory structures to Langchain.\\', \\'Langchain\\': \\'Langchain is a project that is trying to add more complex memory structures.\\', \\'Key-Value Store\\': \\'\\'} Current conversation: Human: Deven & Sam are working on a hackathon project AI: That sounds like a great project! What kind of project are they working on? Human: They are trying to add more complex memory structures to Langchain AI: That sounds like an interesting project! What kind of memory structures are they trying to add? Last line: Human: They are adding in a key-value store for entities mentioned so far in the conversation. You: > Finished chain. \\' That sounds like a great idea! How will the key-value store work?\\' conversation.predict(input=\"What do you know about Deven & Sam?\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI. You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 9292521e-6565-42d8-a437-3f2dbb15bd81
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"9292521e-6565-42d8-a437-3f2dbb15bd81\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"646e41f7-c150-44a0-a4cd-88d9cb08ac6d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"92e4d306c08f3221a6c1cc4bd99714d9f0ece5faec60093629d381991ecbbd86\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"75840d66-caad-4a83-b1e2-799c9d8d780f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a91c5fe92b5b92b2d19d4dcaa0b5218db6190b093fbcf6073ff67b3e735a8dc1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19201, \"end_char_idx\": 22803, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist. Context: {\\'Deven\\': \\'Deven is working on a hackathon project with Sam, attempting to add more complex memory structures to Langchain.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven, trying to add more complex memory structures to Langchain.\\', \\'Langchain\\': \\'Langchain is a project that is trying to add more complex memory structures.\\', \\'Key-Value Store\\': \\'\\'} Current conversation: Human: Deven & Sam are working on a hackathon project AI: That sounds like a great project! What kind of project are they working on? Human: They are trying to add more complex memory structures to Langchain AI: That sounds like an interesting project! What kind of memory structures are they trying to add? Last line: Human: They are adding in a key-value store for entities mentioned so far in the conversation. You: > Finished chain. \\' That sounds like a great idea! How will the key-value store work?\\' conversation.predict(input=\"What do you know about Deven & Sam?\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI. You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist. Context: {\\'Deven\\': \\'Deven is working on a hackathon project with Sam, attempting to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven, trying to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.\\'} Current conversation: Human: Deven & Sam are working on a hackathon project AI: That sounds like a great project! What kind of project are they working on? Human: They are trying to add more complex memory structures to Langchain AI: That sounds like an interesting project! What kind of memory structures are they trying to add?"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 75840d66-caad-4a83-b1e2-799c9d8d780f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"75840d66-caad-4a83-b1e2-799c9d8d780f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9292521e-6565-42d8-a437-3f2dbb15bd81\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"8f290fca4ce9be228e62291118fbf5fb7b7b0ba509dcd3668925c0596088c9f9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"db7fab6e-5015-4c96-b197-6ec1e98b015b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"595458a7c2454b396a75fee8e4d50db9cd33a2c6355d1fbf712cedcb45e0945f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 20816, \"end_char_idx\": 24473, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist. Context: {\\'Deven\\': \\'Deven is working on a hackathon project with Sam, attempting to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven, trying to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.\\'} Current conversation: Human: Deven & Sam are working on a hackathon project AI: That sounds like a great project! What kind of project are they working on? Human: They are trying to add more complex memory structures to Langchain AI: That sounds like an interesting project! What kind of memory structures are they trying to add? Human: They are adding in a key-value store for entities mentioned so far in the conversation. AI: That sounds like a great idea! How will the key-value store work? Last line: Human: What do you know about Deven & Sam? You: > Finished chain. \\' Deven and Sam are working on a hackathon project together, attempting to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.\\' Inspecting the memory store# We can also inspect the memory store directly. In the following examaples, we look at it directly, and then go through some examples of adding information and watch how it changes. from pprint import pprint pprint(conversation.memory.store) {\\'Deven\\': \\'Deven is working on a hackathon project with Sam, attempting to add \\' \\'more complex memory structures to Langchain, including a key-value \\' \\'store for entities mentioned so far in the conversation.\\', \\'Key-Value Store\\': \\'A key-value store that stores entities mentioned in the \\' \\'conversation.\\', \\'Langchain\\': \\'Langchain is a project that is trying to add more complex \\' \\'memory structures, including a key-value store for entities \\' \\'mentioned so far in the conversation.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven, attempting to add \\' \\'more complex memory structures to Langchain, including a key-value \\' \\'store for entities mentioned so far in the conversation.\\'} conversation.predict(input=\"Sam is the founder of a company called Daimon.\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: db7fab6e-5015-4c96-b197-6ec1e98b015b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"db7fab6e-5015-4c96-b197-6ec1e98b015b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"75840d66-caad-4a83-b1e2-799c9d8d780f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"8979d6204ba339247e408d696754c052595525a29946c157526b9e37df2ef53e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"06a88cff-2102-43fd-b0e4-1c075a9898d8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7974fd0b4d1cad29b62feef85eeace12c90f951f9e5629f0395848040e803ba8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 22749, \"end_char_idx\": 26337, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "What kind of memory structures are they trying to add? Human: They are adding in a key-value store for entities mentioned so far in the conversation. AI: That sounds like a great idea! How will the key-value store work? Last line: Human: What do you know about Deven & Sam? You: > Finished chain. \\' Deven and Sam are working on a hackathon project together, attempting to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.\\' Inspecting the memory store# We can also inspect the memory store directly. In the following examaples, we look at it directly, and then go through some examples of adding information and watch how it changes. from pprint import pprint pprint(conversation.memory.store) {\\'Deven\\': \\'Deven is working on a hackathon project with Sam, attempting to add \\' \\'more complex memory structures to Langchain, including a key-value \\' \\'store for entities mentioned so far in the conversation.\\', \\'Key-Value Store\\': \\'A key-value store that stores entities mentioned in the \\' \\'conversation.\\', \\'Langchain\\': \\'Langchain is a project that is trying to add more complex \\' \\'memory structures, including a key-value store for entities \\' \\'mentioned so far in the conversation.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven, attempting to add \\' \\'more complex memory structures to Langchain, including a key-value \\' \\'store for entities mentioned so far in the conversation.\\'} conversation.predict(input=\"Sam is the founder of a company called Daimon.\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI. You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist. Context: {\\'Daimon\\': \\'\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.\\'} Current conversation: Human: They are trying to add more complex memory structures to Langchain AI: That sounds like an interesting project! What kind of memory structures are they trying to add? Human: They are adding in a key-value store for entities mentioned so far in the conversation. AI: That sounds like a great idea! How will the key-value store work? Human: What do you know about Deven & Sam?"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 06a88cff-2102-43fd-b0e4-1c075a9898d8
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"06a88cff-2102-43fd-b0e4-1c075a9898d8\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"db7fab6e-5015-4c96-b197-6ec1e98b015b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"f5a7f50dcec839e1fabb745a7b3fe822531125167d3b937c084fe959ef740a3e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0e6aedd0-fa29-4fae-a3b6-3935b818c251\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"396ad241a19796d845203f794bbdaf4b24872224f53d2d1e0a4537c7ddf19f6b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 24474, \"end_char_idx\": 27912, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist. Context: {\\'Daimon\\': \\'\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation.\\'} Current conversation: Human: They are trying to add more complex memory structures to Langchain AI: That sounds like an interesting project! What kind of memory structures are they trying to add? Human: They are adding in a key-value store for entities mentioned so far in the conversation. AI: That sounds like a great idea! How will the key-value store work? Human: What do you know about Deven & Sam? AI: Deven and Sam are working on a hackathon project to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to be very motivated and passionate about their project, and are working hard to make it a success. Last line: Human: Sam is the founder of a company called Daimon. You: > Finished chain. \"\\\\nThat\\'s impressive! It sounds like Sam is a very successful entrepreneur. What kind of company is Daimon?\" from pprint import pprint pprint(conversation.memory.store) {\\'Daimon\\': \\'Daimon is a company founded by Sam.\\', \\'Deven\\': \\'Deven is working on a hackathon project with Sam to add more \\' \\'complex memory structures to Langchain, including a key-value store \\' \\'for entities mentioned so far in the conversation.\\', \\'Key-Value Store\\': \\'Key-Value Store: A data structure that stores values \\' \\'associated with a unique key, allowing for efficient \\' \\'retrieval of values. Deven and Sam are adding a key-value \\' \\'store for entities mentioned so far in the conversation.\\', \\'Langchain\\': \\'Langchain is a project that seeks to add more complex memory \\' \\'structures, including a key-value store for entities mentioned \\' \\'so far in the conversation.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven to add more complex \\' \\'memory structures to Langchain, including a key-value store for \\' \\'entities mentioned so far in the conversation. He is also the founder \\' \\'of a company called Daimon.\\'} conversation.predict(input=\"What do you know about Sam?\")"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 0e6aedd0-fa29-4fae-a3b6-3935b818c251
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0e6aedd0-fa29-4fae-a3b6-3935b818c251\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"06a88cff-2102-43fd-b0e4-1c075a9898d8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"50d53b8a2ad61fe7e3765829484602531a2c5f1b45a94c7d62cebe680e8fb81e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7482badd-56fd-4818-b415-1efbadda726c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e0cb814d232a103a4a519ed24d8cd7bd263f368a9edac5a3e5315bfd97cfeb62\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 26225, \"end_char_idx\": 29728, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "AI: That sounds like a great idea! How will the key-value store work? Human: What do you know about Deven & Sam? AI: Deven and Sam are working on a hackathon project to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to be very motivated and passionate about their project, and are working hard to make it a success. Last line: Human: Sam is the founder of a company called Daimon. You: > Finished chain. \"\\\\nThat\\'s impressive! It sounds like Sam is a very successful entrepreneur. What kind of company is Daimon?\" from pprint import pprint pprint(conversation.memory.store) {\\'Daimon\\': \\'Daimon is a company founded by Sam.\\', \\'Deven\\': \\'Deven is working on a hackathon project with Sam to add more \\' \\'complex memory structures to Langchain, including a key-value store \\' \\'for entities mentioned so far in the conversation.\\', \\'Key-Value Store\\': \\'Key-Value Store: A data structure that stores values \\' \\'associated with a unique key, allowing for efficient \\' \\'retrieval of values. Deven and Sam are adding a key-value \\' \\'store for entities mentioned so far in the conversation.\\', \\'Langchain\\': \\'Langchain is a project that seeks to add more complex memory \\' \\'structures, including a key-value store for entities mentioned \\' \\'so far in the conversation.\\', \\'Sam\\': \\'Sam is working on a hackathon project with Deven to add more complex \\' \\'memory structures to Langchain, including a key-value store for \\' \\'entities mentioned so far in the conversation. He is also the founder \\' \\'of a company called Daimon.\\'} conversation.predict(input=\"What do you know about Sam?\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI. You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist. Context: {\\'Sam\\': \\'Sam is working on a hackathon project with Deven to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. He is also the founder of a company called Daimon.\\', \\'Daimon\\': \\'Daimon is a company founded by Sam.\\'} Current conversation: Human: They are adding in a key-value store for entities mentioned so far in the conversation."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 7482badd-56fd-4818-b415-1efbadda726c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"7482badd-56fd-4818-b415-1efbadda726c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0e6aedd0-fa29-4fae-a3b6-3935b818c251\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"9eac9a40892cedf57479eb2e7e26c73ec28161ba7c80c2146a51df89dd0ed474\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5b110d72-fe0d-4ee6-a0a4-cd861ca685be\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6c9bc8316944935518f43d74919a37c910648fea6da92b13b4432636fbab4853\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 27796, \"end_char_idx\": 31126, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "He is also the founder \\' \\'of a company called Daimon.\\'} conversation.predict(input=\"What do you know about Sam?\") > Entering new ConversationChain chain... Prompt after formatting: You are an assistant to a human, powered by a large language model trained by OpenAI. You are designed to be able to assist with a wide range of tasks, from answering simple questions to providing in-depth explanations and discussions on a wide range of topics. As a language model, you are able to generate human-like text based on the input you receive, allowing you to engage in natural-sounding conversations and provide responses that are coherent and relevant to the topic at hand. You are constantly learning and improving, and your capabilities are constantly evolving. You are able to process and understand large amounts of text, and can use this knowledge to provide accurate and informative responses to a wide range of questions. You have access to some personalized information provided by the human in the Context section below. Additionally, you are able to generate your own text based on the input you receive, allowing you to engage in discussions and provide explanations and descriptions on a wide range of topics. Overall, you are a powerful tool that can help with a wide range of tasks and provide valuable insights and information on a wide range of topics. Whether the human needs help with a specific question or just wants to have a conversation about a particular topic, you are here to assist. Context: {\\'Sam\\': \\'Sam is working on a hackathon project with Deven to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. He is also the founder of a company called Daimon.\\', \\'Daimon\\': \\'Daimon is a company founded by Sam.\\'} Current conversation: Human: They are adding in a key-value store for entities mentioned so far in the conversation. AI: That sounds like a great idea! How will the key-value store work? Human: What do you know about Deven & Sam? AI: Deven and Sam are working on a hackathon project to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to be very motivated and passionate about their project, and are working hard to make it a success. Human: Sam is the founder of a company called Daimon. AI: That\\'s impressive! It sounds like Sam is a very successful entrepreneur. What kind of company is Daimon? Last line: Human: What do you know about Sam? You: > Finished chain. \\' Sam is the founder of a company called Daimon. He is also working on a hackathon project with Deven to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. He seems to be very motivated and passionate about his project, and is working hard to make it a success.\\' previous ConversationBufferWindowMemory next Conversation Knowledge Graph Memory Contents Using in a chain Inspecting the memory store By Harrison Chase \u00a9 Copyright 2022, Harrison Chase. Last updated on Mar 15, 2023.'}\n\n\n\nIt's pretty ugly but it's good enough for now. Let's see how we can process all of these. We will chunk everything into ~400 token chunks, we can do this easily with `langchain` and `tiktoken`:"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 5b110d72-fe0d-4ee6-a0a4-cd861ca685be
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"5b110d72-fe0d-4ee6-a0a4-cd861ca685be\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7482badd-56fd-4818-b415-1efbadda726c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"d69b3db5002183e2d307d6b76ce67f821657f3ee5c3d2e38250aed215ffd3890\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"196fb696-ec59-474c-aecc-6ed17777fc08\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c81be2465c67526e3139ce57ef60dc00f760905647105a886c02041ad22fb64e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 29505, \"end_char_idx\": 32649, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "He is also the founder of a company called Daimon.\\', \\'Daimon\\': \\'Daimon is a company founded by Sam.\\'} Current conversation: Human: They are adding in a key-value store for entities mentioned so far in the conversation. AI: That sounds like a great idea! How will the key-value store work? Human: What do you know about Deven & Sam? AI: Deven and Sam are working on a hackathon project to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. They seem to be very motivated and passionate about their project, and are working hard to make it a success. Human: Sam is the founder of a company called Daimon. AI: That\\'s impressive! It sounds like Sam is a very successful entrepreneur. What kind of company is Daimon? Last line: Human: What do you know about Sam? You: > Finished chain. \\' Sam is the founder of a company called Daimon. He is also working on a hackathon project with Deven to add more complex memory structures to Langchain, including a key-value store for entities mentioned so far in the conversation. He seems to be very motivated and passionate about his project, and is working hard to make it a success.\\' previous ConversationBufferWindowMemory next Conversation Knowledge Graph Memory Contents Using in a chain Inspecting the memory store By Harrison Chase \u00a9 Copyright 2022, Harrison Chase. Last updated on Mar 15, 2023.'}\n\n\n\nIt's pretty ugly but it's good enough for now. Let's see how we can process all of these. We will chunk everything into ~400 token chunks, we can do this easily with `langchain` and `tiktoken`:\n\n\n```python\nimport tiktoken\n\ntokenizer = tiktoken.get_encoding('p50k_base')\n\n# create the length function\ndef tiktoken_len(text):\n    tokens = tokenizer.encode(\n        text,\n        disallowed_special=()\n    )\n    return len(tokens)\n```\n\n\n```python\nfrom langchain.text_splitter import RecursiveCharacterTextSplitter\n\ntext_splitter = RecursiveCharacterTextSplitter(\n    chunk_size=400,\n    chunk_overlap=20,\n    length_function=tiktoken_len,\n    separators=[\"\\n\\n\", \"\\n\", \" \", \"\"]\n)\n```\n\nProcess the `data` into more chunks using this approach.\n\n\n```python\nfrom uuid import uuid4\nfrom tqdm.auto import tqdm\n\nchunks = []\n\nfor idx, record in enumerate(tqdm(data)):\n    texts = text_splitter.split_text(record['text'])\n    chunks.extend([{\n        'id': str(uuid4()),\n        'text': texts[i],\n        'chunk': i,\n        'url': record['url']\n    } for i in range(len(texts))])\n```\n\n\n      0%|          | 0/231 [00:00<?, ?it/s]\n\n\nOur chunks are ready so now we move onto embedding and indexing everything.\n\n## Initialize Embedding Model\n\nWe use `text-embedding-3-small` as the embedding model. We can embed text like so:\n\n\n```python\nimport openai\n\n# initialize openai API key\nopenai.api_key = \"sk-...\"\n\nembed_model = \"text-embedding-3-small\"\n\nres = openai.Embedding.create(\n    input=[\n        \"Sample document text goes here\",\n        \"there will be several phrases in each batch\"\n    ], engine=embed_model\n)\n```\n\nIn the response `res` we will find a JSON-like object containing our new embeddings within the `'data'` field."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 196fb696-ec59-474c-aecc-6ed17777fc08
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"196fb696-ec59-474c-aecc-6ed17777fc08\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5b110d72-fe0d-4ee6-a0a4-cd861ca685be\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"912e4aec2676f9c4ee4e04f93c38ad84bf5352a689dba8143a484c331f547bb5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dfbecbbd-096f-474a-80c4-3b4ed9735da6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"372401475dc1c570393ceb3af2e55d3bbffa5542f795935a80703302d6bcf014\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 31129, \"end_char_idx\": 33560, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "```python\nimport tiktoken\n\ntokenizer = tiktoken.get_encoding('p50k_base')\n\n# create the length function\ndef tiktoken_len(text):\n    tokens = tokenizer.encode(\n        text,\n        disallowed_special=()\n    )\n    return len(tokens)\n```\n\n\n```python\nfrom langchain.text_splitter import RecursiveCharacterTextSplitter\n\ntext_splitter = RecursiveCharacterTextSplitter(\n    chunk_size=400,\n    chunk_overlap=20,\n    length_function=tiktoken_len,\n    separators=[\"\\n\\n\", \"\\n\", \" \", \"\"]\n)\n```\n\nProcess the `data` into more chunks using this approach.\n\n\n```python\nfrom uuid import uuid4\nfrom tqdm.auto import tqdm\n\nchunks = []\n\nfor idx, record in enumerate(tqdm(data)):\n    texts = text_splitter.split_text(record['text'])\n    chunks.extend([{\n        'id': str(uuid4()),\n        'text': texts[i],\n        'chunk': i,\n        'url': record['url']\n    } for i in range(len(texts))])\n```\n\n\n      0%|          | 0/231 [00:00<?, ?it/s]\n\n\nOur chunks are ready so now we move onto embedding and indexing everything.\n\n## Initialize Embedding Model\n\nWe use `text-embedding-3-small` as the embedding model. We can embed text like so:\n\n\n```python\nimport openai\n\n# initialize openai API key\nopenai.api_key = \"sk-...\"\n\nembed_model = \"text-embedding-3-small\"\n\nres = openai.Embedding.create(\n    input=[\n        \"Sample document text goes here\",\n        \"there will be several phrases in each batch\"\n    ], engine=embed_model\n)\n```\n\nIn the response `res` we will find a JSON-like object containing our new embeddings within the `'data'` field.\n\n\n```python\nres.keys()\n```\n\n\n\n\n    dict_keys(['object', 'data', 'model', 'usage'])\n\n\n\nInside `'data'` we will find two records, one for each of the two sentences we just embedded. Each vector embedding contains `1536` dimensions (the output dimensionality of the `text-embedding-3-small` model.\n\n\n```python\nlen(res['data'])\n```\n\n\n\n\n    2\n\n\n\n\n```python\nlen(res['data'][0]['embedding']), len(res['data'][1]['embedding'])\n```\n\n\n\n\n    (1536, 1536)\n\n\n\nWe will apply this same embedding logic to the langchain docs dataset we've just scraped. But before doing so we must create a place to store the embeddings.\n\n## Initializing the Index\n\nNow we need a place to store these embeddings and enable a efficient vector search through them all. To do that we use Pinecone, we can get a [free API key](https://app.pinecone.io/) and enter it below where we will initialize our connection to Pinecone and create a new index."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: dfbecbbd-096f-474a-80c4-3b4ed9735da6
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"dfbecbbd-096f-474a-80c4-3b4ed9735da6\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"196fb696-ec59-474c-aecc-6ed17777fc08\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1db5d4b7fc7a983bf814875b1df803e7469a63d8db40dedb512d603658d3d523\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"72faa1c3-d36f-4a83-b242-d7847443d0dd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ab6c9a29118bd61a3da4af121a3aa0002fb567936f08c4464280544caa8d3275\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 32014, \"end_char_idx\": 34484, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "0%|          | 0/231 [00:00<?, ?it/s]\n\n\nOur chunks are ready so now we move onto embedding and indexing everything.\n\n## Initialize Embedding Model\n\nWe use `text-embedding-3-small` as the embedding model. We can embed text like so:\n\n\n```python\nimport openai\n\n# initialize openai API key\nopenai.api_key = \"sk-...\"\n\nembed_model = \"text-embedding-3-small\"\n\nres = openai.Embedding.create(\n    input=[\n        \"Sample document text goes here\",\n        \"there will be several phrases in each batch\"\n    ], engine=embed_model\n)\n```\n\nIn the response `res` we will find a JSON-like object containing our new embeddings within the `'data'` field.\n\n\n```python\nres.keys()\n```\n\n\n\n\n    dict_keys(['object', 'data', 'model', 'usage'])\n\n\n\nInside `'data'` we will find two records, one for each of the two sentences we just embedded. Each vector embedding contains `1536` dimensions (the output dimensionality of the `text-embedding-3-small` model.\n\n\n```python\nlen(res['data'])\n```\n\n\n\n\n    2\n\n\n\n\n```python\nlen(res['data'][0]['embedding']), len(res['data'][1]['embedding'])\n```\n\n\n\n\n    (1536, 1536)\n\n\n\nWe will apply this same embedding logic to the langchain docs dataset we've just scraped. But before doing so we must create a place to store the embeddings.\n\n## Initializing the Index\n\nNow we need a place to store these embeddings and enable a efficient vector search through them all. To do that we use Pinecone, we can get a [free API key](https://app.pinecone.io/) and enter it below where we will initialize our connection to Pinecone and create a new index.\n\n\n```python\nimport pinecone\n\nindex_name = 'gpt-4-langchain-docs'\n\n# initialize connection to pinecone\npinecone.init(\n    api_key=\"PINECONE_API_KEY\",  # app.pinecone.io (console)\n    environment=\"PINECONE_ENVIRONMENT\"  # next to API key in console\n)\n\n# check if index already exists (it shouldn't if this is first time)\nif index_name not in pinecone.list_indexes():\n    # if does not exist, create index\n    pinecone.create_index(\n        index_name,\n        dimension=len(res['data'][0]['embedding']),\n        metric='dotproduct'\n    )\n# connect to index\nindex = pinecone.GRPCIndex(index_name)\n# view index stats\nindex.describe_index_stats()\n```\n\n\n\n\n    {'dimension': 1536,\n     'index_fullness': 0.0,\n     'namespaces': {},\n     'total_vector_count': 0}\n\n\n\nWe can see the index is currently empty with a `total_vector_count` of `0`. We can begin populating it with OpenAI `text-embedding-3-small` built embeddings like so:"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 72faa1c3-d36f-4a83-b242-d7847443d0dd
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"72faa1c3-d36f-4a83-b242-d7847443d0dd\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dfbecbbd-096f-474a-80c4-3b4ed9735da6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"246d28d2b6b03bf6f7d5e495b6f8b8fae23b3eee30c68adb88844d77a8e8aeea\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"31fcf034-1311-4cdf-ab68-1dd6d440094a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"26dd93ef9645fc2e23b364e6181d28409cd9c492f68aff09f0a39a8b4dc37903\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 32947, \"end_char_idx\": 35681, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "```python\nlen(res['data'])\n```\n\n\n\n\n    2\n\n\n\n\n```python\nlen(res['data'][0]['embedding']), len(res['data'][1]['embedding'])\n```\n\n\n\n\n    (1536, 1536)\n\n\n\nWe will apply this same embedding logic to the langchain docs dataset we've just scraped. But before doing so we must create a place to store the embeddings.\n\n## Initializing the Index\n\nNow we need a place to store these embeddings and enable a efficient vector search through them all. To do that we use Pinecone, we can get a [free API key](https://app.pinecone.io/) and enter it below where we will initialize our connection to Pinecone and create a new index.\n\n\n```python\nimport pinecone\n\nindex_name = 'gpt-4-langchain-docs'\n\n# initialize connection to pinecone\npinecone.init(\n    api_key=\"PINECONE_API_KEY\",  # app.pinecone.io (console)\n    environment=\"PINECONE_ENVIRONMENT\"  # next to API key in console\n)\n\n# check if index already exists (it shouldn't if this is first time)\nif index_name not in pinecone.list_indexes():\n    # if does not exist, create index\n    pinecone.create_index(\n        index_name,\n        dimension=len(res['data'][0]['embedding']),\n        metric='dotproduct'\n    )\n# connect to index\nindex = pinecone.GRPCIndex(index_name)\n# view index stats\nindex.describe_index_stats()\n```\n\n\n\n\n    {'dimension': 1536,\n     'index_fullness': 0.0,\n     'namespaces': {},\n     'total_vector_count': 0}\n\n\n\nWe can see the index is currently empty with a `total_vector_count` of `0`. We can begin populating it with OpenAI `text-embedding-3-small` built embeddings like so:\n\n\n```python\nfrom tqdm.auto import tqdm\nimport datetime\nfrom time import sleep\n\nbatch_size = 100  # how many embeddings we create and insert at once\n\nfor i in tqdm(range(0, len(chunks), batch_size)):\n    # find end of batch\n    i_end = min(len(chunks), i+batch_size)\n    meta_batch = chunks[i:i_end]\n    # get ids\n    ids_batch = [x['id'] for x in meta_batch]\n    # get texts to encode\n    texts = [x['text'] for x in meta_batch]\n    # create embeddings (try-except added to avoid RateLimitError)\n    try:\n        res = openai.Embedding.create(input=texts, engine=embed_model)\n    except:\n        done = False\n        while not done:\n            sleep(5)\n            try:\n                res = openai.Embedding.create(input=texts, engine=embed_model)\n                done = True\n            except:\n                pass\n    embeds = [record['embedding'] for record in res['data']]\n    # cleanup metadata\n    meta_batch = [{\n        'text': x['text'],\n        'chunk': x['chunk'],\n        'url': x['url']\n    } for x in meta_batch]\n    to_upsert = list(zip(ids_batch, embeds, meta_batch))\n    # upsert to Pinecone\n    index.upsert(vectors=to_upsert)\n```\n\n\n      0%|          | 0/12 [00:00<?, ?it/s]"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 31fcf034-1311-4cdf-ab68-1dd6d440094a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"31fcf034-1311-4cdf-ab68-1dd6d440094a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"72faa1c3-d36f-4a83-b242-d7847443d0dd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"d5a718bbe9db768dc878beb3f2bb668d02d198ab09b5a0b796703f5e18a19ee1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9b31dd7f-bacd-43cf-adc5-7c908368b6ff\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"57703511b8d18ccbc3f90990500aa373c7ede900ce8d0cd64ef167f94f1e03f3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 34215, \"end_char_idx\": 37551, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "{'dimension': 1536,\n     'index_fullness': 0.0,\n     'namespaces': {},\n     'total_vector_count': 0}\n\n\n\nWe can see the index is currently empty with a `total_vector_count` of `0`. We can begin populating it with OpenAI `text-embedding-3-small` built embeddings like so:\n\n\n```python\nfrom tqdm.auto import tqdm\nimport datetime\nfrom time import sleep\n\nbatch_size = 100  # how many embeddings we create and insert at once\n\nfor i in tqdm(range(0, len(chunks), batch_size)):\n    # find end of batch\n    i_end = min(len(chunks), i+batch_size)\n    meta_batch = chunks[i:i_end]\n    # get ids\n    ids_batch = [x['id'] for x in meta_batch]\n    # get texts to encode\n    texts = [x['text'] for x in meta_batch]\n    # create embeddings (try-except added to avoid RateLimitError)\n    try:\n        res = openai.Embedding.create(input=texts, engine=embed_model)\n    except:\n        done = False\n        while not done:\n            sleep(5)\n            try:\n                res = openai.Embedding.create(input=texts, engine=embed_model)\n                done = True\n            except:\n                pass\n    embeds = [record['embedding'] for record in res['data']]\n    # cleanup metadata\n    meta_batch = [{\n        'text': x['text'],\n        'chunk': x['chunk'],\n        'url': x['url']\n    } for x in meta_batch]\n    to_upsert = list(zip(ids_batch, embeds, meta_batch))\n    # upsert to Pinecone\n    index.upsert(vectors=to_upsert)\n```\n\n\n      0%|          | 0/12 [00:00<?, ?it/s]\n\n\nNow we've added all of our langchain docs to the index. With that we can move on to retrieval and then answer generation using GPT-4.\n\n## Retrieval\n\nTo search through our documents we first need to create a query vector `xq`. Using `xq` we will retrieve the most relevant chunks from the LangChain docs, like so:\n\n\n```python\nquery = \"how do I use the LLMChain in LangChain?\"\n\nres = openai.Embedding.create(\n    input=[query],\n    engine=embed_model\n)\n\n# retrieve from Pinecone\nxq = res['data'][0]['embedding']\n\n# get relevant contexts (including the questions)\nres = index.query(xq, top_k=5, include_metadata=True)\n```\n\n\n```python\nres\n```\n\n\n\n\n    {'matches': [{'id': '1fec660b-9937-4f7e-9692-280c8cc7ce0d',\n                  'metadata': {'chunk': 0.0,\n                               'text': '.rst .pdf Chains Chains# Using an LLM in '\n                                       'isolation is fine for some simple '\n                                       'applications, but many more complex ones '\n                                       'require chaining LLMs - either with each '\n                                       'other or with other experts. LangChain '\n                                       'provides a standard interface for Chains, '\n                                       'as well as some common implementations of '\n                                       'chains for ease of use. The following '\n                                       'sections of documentation are provided: '\n                                       'Getting Started: A getting started guide '\n                                       'for chains, to get you up and running '\n                                       'quickly. Key Concepts: A conceptual guide '\n                                       'going over the various concepts related to '\n                                       'chains."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 9b31dd7f-bacd-43cf-adc5-7c908368b6ff
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"9b31dd7f-bacd-43cf-adc5-7c908368b6ff\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"31fcf034-1311-4cdf-ab68-1dd6d440094a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"4411630be21542803dc174853a157e685cd3d0584a1eb6168df53eee8104e7f8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"71b7eea7-3435-4e98-b1c1-00d3101b29f3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7d6e4fb075b39bb9773c9cec5dd2b9d9f95da53633593b12529a1e35fe4a4038\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 35645, \"end_char_idx\": 39783, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "0%|          | 0/12 [00:00<?, ?it/s]\n\n\nNow we've added all of our langchain docs to the index. With that we can move on to retrieval and then answer generation using GPT-4.\n\n## Retrieval\n\nTo search through our documents we first need to create a query vector `xq`. Using `xq` we will retrieve the most relevant chunks from the LangChain docs, like so:\n\n\n```python\nquery = \"how do I use the LLMChain in LangChain?\"\n\nres = openai.Embedding.create(\n    input=[query],\n    engine=embed_model\n)\n\n# retrieve from Pinecone\nxq = res['data'][0]['embedding']\n\n# get relevant contexts (including the questions)\nres = index.query(xq, top_k=5, include_metadata=True)\n```\n\n\n```python\nres\n```\n\n\n\n\n    {'matches': [{'id': '1fec660b-9937-4f7e-9692-280c8cc7ce0d',\n                  'metadata': {'chunk': 0.0,\n                               'text': '.rst .pdf Chains Chains# Using an LLM in '\n                                       'isolation is fine for some simple '\n                                       'applications, but many more complex ones '\n                                       'require chaining LLMs - either with each '\n                                       'other or with other experts. LangChain '\n                                       'provides a standard interface for Chains, '\n                                       'as well as some common implementations of '\n                                       'chains for ease of use. The following '\n                                       'sections of documentation are provided: '\n                                       'Getting Started: A getting started guide '\n                                       'for chains, to get you up and running '\n                                       'quickly. Key Concepts: A conceptual guide '\n                                       'going over the various concepts related to '\n                                       'chains. How-To Guides: A collection of '\n                                       'how-to guides. These highlight how to use '\n                                       'various types of chains. Reference: API '\n                                       'reference documentation for all Chain '\n                                       'classes. previous Vector DB Text '\n                                       'Generation next Getting Started By '\n                                       'Harrison Chase \u00a9 Copyright 2022, Harrison '\n                                       'Chase. Last updated on Mar 15, 2023.',\n                               'url': 'https://langchain.readthedocs.io/en/latest/modules/chains.html'},\n                  'score': 0.8848499,\n                  'sparse_values': {'indices': [], 'values': []},\n                  'values': []},\n                 {'id': 'fe48438d-228a-4e0e-b41e-5cb5c6ba1482',\n                  'metadata': {'chunk': 0.0,\n                               'text': '.rst .pdf LLMs LLMs# Large Language Models '\n                                       '(LLMs) are a core component of LangChain. '\n                                       'LangChain is not a provider of LLMs, but '\n                                       'rather provides a standard interface '\n                                       'through which you can interact with a '\n                                       'variety of LLMs. The following sections of '\n                                       'documentation are provided: Getting '\n                                       'Started: An overview of all the '\n                                       'functionality the LangChain LLM class '\n                                       'provides. Key Concepts: A conceptual guide '\n                                       'going over the various concepts related to '\n                                       'LLMs. How-To Guides: A collection of '\n                                       'how-to guides. These highlight how to '\n                                       'accomplish various objectives with our LLM '\n                                       'class, as well as how to integrate with '\n                                       'various LLM providers."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 71b7eea7-3435-4e98-b1c1-00d3101b29f3
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"71b7eea7-3435-4e98-b1c1-00d3101b29f3\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9b31dd7f-bacd-43cf-adc5-7c908368b6ff\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"689d56ac6bf36358c07b0501457016dc6b82c0e3960699ced8c5b1066014443a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"be9e49cf-938f-440b-82e8-f63b5758e567\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ad8e6690aa47ff2de0fbf024473e14a0b6c4ff7fc219184db95bb1fd67aa6d38\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 37384, \"end_char_idx\": 41997, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "Key Concepts: A conceptual guide '\n                                       'going over the various concepts related to '\n                                       'chains. How-To Guides: A collection of '\n                                       'how-to guides. These highlight how to use '\n                                       'various types of chains. Reference: API '\n                                       'reference documentation for all Chain '\n                                       'classes. previous Vector DB Text '\n                                       'Generation next Getting Started By '\n                                       'Harrison Chase \u00a9 Copyright 2022, Harrison '\n                                       'Chase. Last updated on Mar 15, 2023.',\n                               'url': 'https://langchain.readthedocs.io/en/latest/modules/chains.html'},\n                  'score': 0.8848499,\n                  'sparse_values': {'indices': [], 'values': []},\n                  'values': []},\n                 {'id': 'fe48438d-228a-4e0e-b41e-5cb5c6ba1482',\n                  'metadata': {'chunk': 0.0,\n                               'text': '.rst .pdf LLMs LLMs# Large Language Models '\n                                       '(LLMs) are a core component of LangChain. '\n                                       'LangChain is not a provider of LLMs, but '\n                                       'rather provides a standard interface '\n                                       'through which you can interact with a '\n                                       'variety of LLMs. The following sections of '\n                                       'documentation are provided: Getting '\n                                       'Started: An overview of all the '\n                                       'functionality the LangChain LLM class '\n                                       'provides. Key Concepts: A conceptual guide '\n                                       'going over the various concepts related to '\n                                       'LLMs. How-To Guides: A collection of '\n                                       'how-to guides. These highlight how to '\n                                       'accomplish various objectives with our LLM '\n                                       'class, as well as how to integrate with '\n                                       'various LLM providers. Reference: API '\n                                       'reference documentation for all LLM '\n                                       'classes. previous Example Selector next '\n                                       'Getting Started By Harrison Chase \u00a9 '\n                                       'Copyright 2022, Harrison Chase. Last '\n                                       'updated on Mar 15, 2023.',\n                               'url': 'https://langchain.readthedocs.io/en/latest/modules/llms.html'},\n                  'score': 0.8595519,\n                  'sparse_values': {'indices': [], 'values': []},\n                  'values': []},\n                 {'id': '60df5bff-5f79-46ee-9456-534d42f6a94e',\n                  'metadata': {'chunk': 0.0,\n                               'text': '.ipynb .pdf Getting Started Contents Why '\n                                       'do we need chains? Query an LLM with the '\n                                       'LLMChain Combine chains with the '\n                                       'SequentialChain Create a custom chain with '\n                                       'the Chain class Getting Started# In this '\n                                       'tutorial, we will learn about creating '\n                                       'simple chains in LangChain. We will learn '\n                                       'how to create a chain, add components to '\n                                       'it, and run it. In this tutorial, we will '\n                                       'cover: Using a simple LLM chain Creating '\n                                       'sequential chains Creating a custom chain '\n                                       'Why do we need chains?# Chains allow us to '\n                                       'combine multiple components together to '\n                                       'create a single, coherent application. For '\n                                       'example, we can create a chain that takes '\n                                       'user input, formats it with a '\n                                       'PromptTemplate, and then passes the '\n                                       'formatted response to an LLM."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: be9e49cf-938f-440b-82e8-f63b5758e567
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"be9e49cf-938f-440b-82e8-f63b5758e567\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"71b7eea7-3435-4e98-b1c1-00d3101b29f3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"231fea718d92d007ef9c1f922b3d69feba58ca80e62fcdd951ffbc61711e4672\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d1fc4c05-59ec-4b39-adc9-8fafc600eb37\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"88c539957dadbad9293c32fd7cc518ffb82640856baa3c300aff401ca978432a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 39529, \"end_char_idx\": 43942, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "These highlight how to '\n                                       'accomplish various objectives with our LLM '\n                                       'class, as well as how to integrate with '\n                                       'various LLM providers. Reference: API '\n                                       'reference documentation for all LLM '\n                                       'classes. previous Example Selector next '\n                                       'Getting Started By Harrison Chase \u00a9 '\n                                       'Copyright 2022, Harrison Chase. Last '\n                                       'updated on Mar 15, 2023.',\n                               'url': 'https://langchain.readthedocs.io/en/latest/modules/llms.html'},\n                  'score': 0.8595519,\n                  'sparse_values': {'indices': [], 'values': []},\n                  'values': []},\n                 {'id': '60df5bff-5f79-46ee-9456-534d42f6a94e',\n                  'metadata': {'chunk': 0.0,\n                               'text': '.ipynb .pdf Getting Started Contents Why '\n                                       'do we need chains? Query an LLM with the '\n                                       'LLMChain Combine chains with the '\n                                       'SequentialChain Create a custom chain with '\n                                       'the Chain class Getting Started# In this '\n                                       'tutorial, we will learn about creating '\n                                       'simple chains in LangChain. We will learn '\n                                       'how to create a chain, add components to '\n                                       'it, and run it. In this tutorial, we will '\n                                       'cover: Using a simple LLM chain Creating '\n                                       'sequential chains Creating a custom chain '\n                                       'Why do we need chains?# Chains allow us to '\n                                       'combine multiple components together to '\n                                       'create a single, coherent application. For '\n                                       'example, we can create a chain that takes '\n                                       'user input, formats it with a '\n                                       'PromptTemplate, and then passes the '\n                                       'formatted response to an LLM. We can build '\n                                       'more complex chains by combining multiple '\n                                       'chains together, or by combining chains '\n                                       'with other components. Query an LLM with '\n                                       'the LLMChain# The LLMChain is a simple '\n                                       'chain that takes in a prompt template, '\n                                       'formats it with the user input and returns '\n                                       'the response from an LLM. To use the '\n                                       'LLMChain, first create a prompt template. '\n                                       'from langchain.prompts import '\n                                       'PromptTemplate from langchain.llms import '\n                                       'OpenAI llm = OpenAI(temperature=0.9) '\n                                       'prompt = PromptTemplate( '\n                                       'input_variables=[\"product\"], '\n                                       'template=\"What is a good',\n                               'url': 'https://langchain.readthedocs.io/en/latest/modules/chains/getting_started.html'},\n                  'score': 0.8462403,\n                  'sparse_values': {'indices': [], 'values': []},\n                  'values': []},\n                 {'id': '2f11beb1-3935-447e-b565-b20383dc4544',\n                  'metadata': {'chunk': 1.0,\n                               'text': 'chain first uses a LLM to construct the '\n                                       'url to hit, then makes that request with '\n                                       'the Requests wrapper, and finally runs '\n                                       'that result through the language model '\n                                       'again in order to product a natural '\n                                       'language response."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: d1fc4c05-59ec-4b39-adc9-8fafc600eb37
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"d1fc4c05-59ec-4b39-adc9-8fafc600eb37\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"be9e49cf-938f-440b-82e8-f63b5758e567\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"769f79a351cf8d5c2f96033f5c1657be6e0aac8fd7b1698e88b15bb3ed8652ec\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a001684c-e083-46a7-a5a8-5942c7670c8b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"25791e540d6d8b7bd3641afb0ba41d2b68f776c61826ae81d962861fe7414ce3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 41688, \"end_char_idx\": 45748, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "For '\n                                       'example, we can create a chain that takes '\n                                       'user input, formats it with a '\n                                       'PromptTemplate, and then passes the '\n                                       'formatted response to an LLM. We can build '\n                                       'more complex chains by combining multiple '\n                                       'chains together, or by combining chains '\n                                       'with other components. Query an LLM with '\n                                       'the LLMChain# The LLMChain is a simple '\n                                       'chain that takes in a prompt template, '\n                                       'formats it with the user input and returns '\n                                       'the response from an LLM. To use the '\n                                       'LLMChain, first create a prompt template. '\n                                       'from langchain.prompts import '\n                                       'PromptTemplate from langchain.llms import '\n                                       'OpenAI llm = OpenAI(temperature=0.9) '\n                                       'prompt = PromptTemplate( '\n                                       'input_variables=[\"product\"], '\n                                       'template=\"What is a good',\n                               'url': 'https://langchain.readthedocs.io/en/latest/modules/chains/getting_started.html'},\n                  'score': 0.8462403,\n                  'sparse_values': {'indices': [], 'values': []},\n                  'values': []},\n                 {'id': '2f11beb1-3935-447e-b565-b20383dc4544',\n                  'metadata': {'chunk': 1.0,\n                               'text': 'chain first uses a LLM to construct the '\n                                       'url to hit, then makes that request with '\n                                       'the Requests wrapper, and finally runs '\n                                       'that result through the language model '\n                                       'again in order to product a natural '\n                                       'language response. Example Notebook '\n                                       'LLMBash Chain Links Used: BashProcess, '\n                                       'LLMChain Notes: This chain takes user '\n                                       'input (a question), uses an LLM chain to '\n                                       'convert it to a bash command to run in the '\n                                       'terminal, and then returns that as the '\n                                       'result. Example Notebook LLMChecker Chain '\n                                       'Links Used: LLMChain Notes: This chain '\n                                       'takes user input (a question), uses an LLM '\n                                       'chain to answer that question, and then '\n                                       'uses other LLMChains to self-check that '\n                                       'answer. Example Notebook LLMRequests Chain '\n                                       'Links Used: Requests, LLMChain Notes: This '\n                                       'chain takes a URL and other inputs, uses '\n                                       'Requests to get the data at that URL, and '\n                                       'then passes that along with the other '\n                                       'inputs into an LLMChain to generate a '\n                                       'response. The example included shows how '\n                                       'to ask a question to Google - it firsts '\n                                       'constructs a Google url, then fetches the '\n                                       'data there, then passes that data + the '\n                                       'original question into an LLMChain to get '\n                                       'an answer."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: a001684c-e083-46a7-a5a8-5942c7670c8b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a001684c-e083-46a7-a5a8-5942c7670c8b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d1fc4c05-59ec-4b39-adc9-8fafc600eb37\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"b9d0f41e0621954efa9783cd581e35b427f3635e3b1c5d0ffafdaa6d4349e03e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7fbcbb7d-4c1f-4f8c-8a40-e316b096c2c1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c683b285916e74a0c203e83f06cf97e34e052e5679ba964719c322794f0ad425\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 43943, \"end_char_idx\": 48773, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "Example Notebook '\n                                       'LLMBash Chain Links Used: BashProcess, '\n                                       'LLMChain Notes: This chain takes user '\n                                       'input (a question), uses an LLM chain to '\n                                       'convert it to a bash command to run in the '\n                                       'terminal, and then returns that as the '\n                                       'result. Example Notebook LLMChecker Chain '\n                                       'Links Used: LLMChain Notes: This chain '\n                                       'takes user input (a question), uses an LLM '\n                                       'chain to answer that question, and then '\n                                       'uses other LLMChains to self-check that '\n                                       'answer. Example Notebook LLMRequests Chain '\n                                       'Links Used: Requests, LLMChain Notes: This '\n                                       'chain takes a URL and other inputs, uses '\n                                       'Requests to get the data at that URL, and '\n                                       'then passes that along with the other '\n                                       'inputs into an LLMChain to generate a '\n                                       'response. The example included shows how '\n                                       'to ask a question to Google - it firsts '\n                                       'constructs a Google url, then fetches the '\n                                       'data there, then passes that data + the '\n                                       'original question into an LLMChain to get '\n                                       'an answer. Example Notebook Moderation '\n                                       'Chain Links Used: LLMChain, '\n                                       'ModerationChain Notes: This chain shows '\n                                       'how to use OpenAI\u2019s content',\n                               'url': 'https://langchain.readthedocs.io/en/latest/modules/chains/utility_how_to.html'},\n                  'score': 0.8451743,\n                  'sparse_values': {'indices': [], 'values': []},\n                  'values': []},\n                 {'id': 'f3ed41eb-063c-407f-bdaa-706a8c6a2091',\n                  'metadata': {'chunk': 1.0,\n                               'text': 'Prompts: This includes prompt management, '\n                                       'prompt optimization, and prompt '\n                                       'serialization. LLMs: This includes a '\n                                       'generic interface for all LLMs, and common '\n                                       'utilities for working with LLMs. Document '\n                                       'Loaders: This includes a standard '\n                                       'interface for loading documents, as well '\n                                       'as specific integrations to all types of '\n                                       'text data sources. Utils: Language models '\n                                       'are often more powerful when interacting '\n                                       'with other sources of knowledge or '\n                                       'computation. This can include Python '\n                                       'REPLs, embeddings, search engines, and '\n                                       'more. LangChain provides a large '\n                                       'collection of common utils to use in your '\n                                       'application. Chains: Chains go beyond just '\n                                       'a single LLM call, and are sequences of '\n                                       'calls (whether to an LLM or a different '\n                                       'utility). LangChain provides a standard '\n                                       'interface for chains, lots of integrations '\n                                       'with other tools, and end-to-end chains '\n                                       'for common applications. Indexes: Language '\n                                       'models are often more powerful when '\n                                       'combined with your own text data - this '\n                                       'module covers best practices for doing '\n                                       'exactly that. Agents: Agents involve an '\n                                       'LLM making decisions about which Actions '\n                                       'to take, taking that Action, seeing an '\n                                       'Observation, and repeating that until '\n                                       'done."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 7fbcbb7d-4c1f-4f8c-8a40-e316b096c2c1
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"7fbcbb7d-4c1f-4f8c-8a40-e316b096c2c1\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a001684c-e083-46a7-a5a8-5942c7670c8b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"c040997da5d97b3193ef2bed1b71b983c3090a4afb5b92d0504a30726aff7a95\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2d164f35-a735-4fea-821d-d7954588389d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4f04a6d632c60fbd474f6a50912e97c0d9f6b30b5ff68a975801aeb1d3ef1b60\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 46580, \"end_char_idx\": 50800, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "LLMs: This includes a '\n                                       'generic interface for all LLMs, and common '\n                                       'utilities for working with LLMs. Document '\n                                       'Loaders: This includes a standard '\n                                       'interface for loading documents, as well '\n                                       'as specific integrations to all types of '\n                                       'text data sources. Utils: Language models '\n                                       'are often more powerful when interacting '\n                                       'with other sources of knowledge or '\n                                       'computation. This can include Python '\n                                       'REPLs, embeddings, search engines, and '\n                                       'more. LangChain provides a large '\n                                       'collection of common utils to use in your '\n                                       'application. Chains: Chains go beyond just '\n                                       'a single LLM call, and are sequences of '\n                                       'calls (whether to an LLM or a different '\n                                       'utility). LangChain provides a standard '\n                                       'interface for chains, lots of integrations '\n                                       'with other tools, and end-to-end chains '\n                                       'for common applications. Indexes: Language '\n                                       'models are often more powerful when '\n                                       'combined with your own text data - this '\n                                       'module covers best practices for doing '\n                                       'exactly that. Agents: Agents involve an '\n                                       'LLM making decisions about which Actions '\n                                       'to take, taking that Action, seeing an '\n                                       'Observation, and repeating that until '\n                                       'done. LangChain provides a standard '\n                                       'interface for agents, a selection of '\n                                       'agents to choose from, and examples of end '\n                                       'to end agents. Memory: Memory is the',\n                               'url': 'https://langchain.readthedocs.io/en/latest/'},\n                  'score': 0.84271824,\n                  'sparse_values': {'indices': [], 'values': []},\n                  'values': []}],\n     'namespace': ''}\n\n\n\nWith retrieval complete, we move on to feeding these into GPT-4 to produce answers.\n\n## Retrieval Augmented Generation\n\nGPT-4 is currently accessed via the `ChatCompletions` endpoint of OpenAI. To add the information we retrieved into the model, we need to pass it into our user prompts *alongside* our original query. We can do that like so:\n\n\n```python\n# get list of retrieved text\ncontexts = [item['metadata']['text'] for item in res['matches']]\n\naugmented_query = \"\\n\\n---\\n\\n\".join(contexts)+\"\\n\\n-----\\n\\n\"+query\n```\n\n\n```python\nprint(augmented_query)\n```\n\n    .rst .pdf Chains Chains# Using an LLM in isolation is fine for some simple applications, but many more complex ones require chaining LLMs - either with each other or with other experts. LangChain provides a standard interface for Chains, as well as some common implementations of chains for ease of use. The following sections of documentation are provided: Getting Started: A getting started guide for chains, to get you up and running quickly. Key Concepts: A conceptual guide going over the various concepts related to chains. How-To Guides: A collection of how-to guides. These highlight how to use various types of chains. Reference: API reference documentation for all Chain classes. previous Vector DB Text Generation next Getting Started By Harrison Chase \u00a9 Copyright 2022, Harrison Chase. Last updated on Mar 15, 2023.\n    \n    ---\n    \n    .rst .pdf LLMs LLMs# Large Language Models (LLMs) are a core component of LangChain."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 2d164f35-a735-4fea-821d-d7954588389d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"2d164f35-a735-4fea-821d-d7954588389d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7fbcbb7d-4c1f-4f8c-8a40-e316b096c2c1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"7002cae8f529b7be7a6990aac707767969b0c808e0fa6e8f53057e69ca29011d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4a45ef53-aefd-4743-897a-f9a419dd67e7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"db7b162fa7f1ee7e7b3e69cec0df5b97072d857ce4fae315a496a3d6f6c3818d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 49299, \"end_char_idx\": 52589, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "With retrieval complete, we move on to feeding these into GPT-4 to produce answers.\n\n## Retrieval Augmented Generation\n\nGPT-4 is currently accessed via the `ChatCompletions` endpoint of OpenAI. To add the information we retrieved into the model, we need to pass it into our user prompts *alongside* our original query. We can do that like so:\n\n\n```python\n# get list of retrieved text\ncontexts = [item['metadata']['text'] for item in res['matches']]\n\naugmented_query = \"\\n\\n---\\n\\n\".join(contexts)+\"\\n\\n-----\\n\\n\"+query\n```\n\n\n```python\nprint(augmented_query)\n```\n\n    .rst .pdf Chains Chains# Using an LLM in isolation is fine for some simple applications, but many more complex ones require chaining LLMs - either with each other or with other experts. LangChain provides a standard interface for Chains, as well as some common implementations of chains for ease of use. The following sections of documentation are provided: Getting Started: A getting started guide for chains, to get you up and running quickly. Key Concepts: A conceptual guide going over the various concepts related to chains. How-To Guides: A collection of how-to guides. These highlight how to use various types of chains. Reference: API reference documentation for all Chain classes. previous Vector DB Text Generation next Getting Started By Harrison Chase \u00a9 Copyright 2022, Harrison Chase. Last updated on Mar 15, 2023.\n    \n    ---\n    \n    .rst .pdf LLMs LLMs# Large Language Models (LLMs) are a core component of LangChain. LangChain is not a provider of LLMs, but rather provides a standard interface through which you can interact with a variety of LLMs. The following sections of documentation are provided: Getting Started: An overview of all the functionality the LangChain LLM class provides. Key Concepts: A conceptual guide going over the various concepts related to LLMs. How-To Guides: A collection of how-to guides. These highlight how to accomplish various objectives with our LLM class, as well as how to integrate with various LLM providers. Reference: API reference documentation for all LLM classes. previous Example Selector next Getting Started By Harrison Chase \u00a9 Copyright 2022, Harrison Chase. Last updated on Mar 15, 2023.\n    \n    ---\n    \n    .ipynb .pdf Getting Started Contents Why do we need chains? Query an LLM with the LLMChain Combine chains with the SequentialChain Create a custom chain with the Chain class Getting Started# In this tutorial, we will learn about creating simple chains in LangChain. We will learn how to create a chain, add components to it, and run it. In this tutorial, we will cover: Using a simple LLM chain Creating sequential chains Creating a custom chain Why do we need chains?# Chains allow us to combine multiple components together to create a single, coherent application. For example, we can create a chain that takes user input, formats it with a PromptTemplate, and then passes the formatted response to an LLM. We can build more complex chains by combining multiple chains together, or by combining chains with other components. Query an LLM with the LLMChain# The LLMChain is a simple chain that takes in a prompt template, formats it with the user input and returns the response from an LLM. To use the LLMChain, first create a prompt template."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 4a45ef53-aefd-4743-897a-f9a419dd67e7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"4a45ef53-aefd-4743-897a-f9a419dd67e7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2d164f35-a735-4fea-821d-d7954588389d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"70654a9ae6abf0b18d85d924139a528c87f9af3a68206c8e0150633fb8aad416\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"05c31335-5189-419f-9ecf-89042ef1c9cc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c9f6170af0eaee77d8c905637b70dc9626460b3a075d22e1f706e6297785b007\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 50801, \"end_char_idx\": 54228, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "LangChain is not a provider of LLMs, but rather provides a standard interface through which you can interact with a variety of LLMs. The following sections of documentation are provided: Getting Started: An overview of all the functionality the LangChain LLM class provides. Key Concepts: A conceptual guide going over the various concepts related to LLMs. How-To Guides: A collection of how-to guides. These highlight how to accomplish various objectives with our LLM class, as well as how to integrate with various LLM providers. Reference: API reference documentation for all LLM classes. previous Example Selector next Getting Started By Harrison Chase \u00a9 Copyright 2022, Harrison Chase. Last updated on Mar 15, 2023.\n    \n    ---\n    \n    .ipynb .pdf Getting Started Contents Why do we need chains? Query an LLM with the LLMChain Combine chains with the SequentialChain Create a custom chain with the Chain class Getting Started# In this tutorial, we will learn about creating simple chains in LangChain. We will learn how to create a chain, add components to it, and run it. In this tutorial, we will cover: Using a simple LLM chain Creating sequential chains Creating a custom chain Why do we need chains?# Chains allow us to combine multiple components together to create a single, coherent application. For example, we can create a chain that takes user input, formats it with a PromptTemplate, and then passes the formatted response to an LLM. We can build more complex chains by combining multiple chains together, or by combining chains with other components. Query an LLM with the LLMChain# The LLMChain is a simple chain that takes in a prompt template, formats it with the user input and returns the response from an LLM. To use the LLMChain, first create a prompt template. from langchain.prompts import PromptTemplate from langchain.llms import OpenAI llm = OpenAI(temperature=0.9) prompt = PromptTemplate( input_variables=[\"product\"], template=\"What is a good\n    \n    ---\n    \n    chain first uses a LLM to construct the url to hit, then makes that request with the Requests wrapper, and finally runs that result through the language model again in order to product a natural language response. Example Notebook LLMBash Chain Links Used: BashProcess, LLMChain Notes: This chain takes user input (a question), uses an LLM chain to convert it to a bash command to run in the terminal, and then returns that as the result. Example Notebook LLMChecker Chain Links Used: LLMChain Notes: This chain takes user input (a question), uses an LLM chain to answer that question, and then uses other LLMChains to self-check that answer. Example Notebook LLMRequests Chain Links Used: Requests, LLMChain Notes: This chain takes a URL and other inputs, uses Requests to get the data at that URL, and then passes that along with the other inputs into an LLMChain to generate a response. The example included shows how to ask a question to Google - it firsts constructs a Google url, then fetches the data there, then passes that data + the original question into an LLMChain to get an answer. Example Notebook Moderation Chain Links Used: LLMChain, ModerationChain Notes: This chain shows how to use OpenAI\u2019s content\n    \n    ---\n    \n    Prompts: This includes prompt management, prompt optimization, and prompt serialization. LLMs: This includes a generic interface for all LLMs, and common utilities for working with LLMs."}


