# embeddings_queue

created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 8eba3e05-0831-4436-a2c1-3a7a6b057c6a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"8eba3e05-0831-4436-a2c1-3a7a6b057c6a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"66597835-989b-42d1-907a-969fec381a64\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"5bf7250ac75635842d8521cb0812470a1faef05758e6c5f8c6ce97d961b67946\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"81dbfa72-5dc3-426c-9ddb-ed40d5fb8bba\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3a8293023104a5c65c389a14cc7d2c3875dfc149c27663b687af99e7c89d2e3e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4417, \"end_char_idx\": 6142, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "0.4-py3-none-any.whl.metadata (6.6 kB)\n    Collecting certifi>=2017.4.17 (from requests>=2.20->openai)\n      Using cached certifi-2023.5.7-py3-none-any.whl (156 kB)\n    Collecting attrs>=17.3.0 (from aiohttp->openai)\n      Using cached attrs-23.1.0-py3-none-any.whl (61 kB)\n    Collecting multidict<7.0,>=4.5 (from aiohttp->openai)\n      Using cached multidict-6.0.4-cp310-cp310-macosx_11_0_arm64.whl (29 kB)\n    Collecting async-timeout<5.0,>=4.0.0a3 (from aiohttp->openai)\n      Using cached async_timeout-4.0.2-py3-none-any.whl (5.8 kB)\n    Collecting yarl<2.0,>=1.0 (from aiohttp->openai)\n      Using cached yarl-1.9.2-cp310-cp310-macosx_11_0_arm64.whl (62 kB)\n    Collecting frozenlist>=1.1.1 (from aiohttp->openai)\n      Obtaining dependency information for frozenlist>=1.1.1 from https://files.pythonhosted.org/packages/67/6a/55a49da0fa373ac9aa49ccd5b6393ecc183e2a0904d9449ea3ee1163e0b1/frozenlist-1.4.0-cp310-cp310-macosx_11_0_arm64.whl.metadata\n      Downloading frozenlist-1.4.0-cp310-cp310-macosx_11_0_arm64.whl.metadata (5.2 kB)\n    Collecting aiosignal>=1.1.2 (from aiohttp->openai)\n      Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)\n    Using cached openai-0.27.8-py3-none-any.whl (73 kB)\n    Using cached requests-2.31.0-py3-none-any.whl (62 kB)\n    Downloading aiohttp-3.8.5-cp310-cp310-macosx_11_0_arm64.whl (343 kB)\n    \u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m343.9/343.9 kB\u001b[0m \u001b[31m11.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25hUsing cached charset_normalizer-3.2.0-cp310-cp310-macosx_11_0_arm64.whl (124 kB)\n    Downloading frozenlist-1.4.0-cp310-cp310-macosx_11_0_arm64.whl (46 kB)\n    \u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m46.0/46.0 kB\u001b[0m \u001b[31m4."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 81dbfa72-5dc3-426c-9ddb-ed40d5fb8bba
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"81dbfa72-5dc3-426c-9ddb-ed40d5fb8bba\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8eba3e05-0831-4436-a2c1-3a7a6b057c6a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"3183224ab57868e1f7b74f168f8714b045de535cfbeaec8894902ecaff40cc78\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7bac77ff-1d24-48c9-bcea-32a1f482a161\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cbbb86efe862822f9bafe2078da11a3fb6af7839abee83bb576b461a70874b54\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5324, \"end_char_idx\": 7067, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "4.0-cp310-cp310-macosx_11_0_arm64.whl.metadata\n      Downloading frozenlist-1.4.0-cp310-cp310-macosx_11_0_arm64.whl.metadata (5.2 kB)\n    Collecting aiosignal>=1.1.2 (from aiohttp->openai)\n      Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)\n    Using cached openai-0.27.8-py3-none-any.whl (73 kB)\n    Using cached requests-2.31.0-py3-none-any.whl (62 kB)\n    Downloading aiohttp-3.8.5-cp310-cp310-macosx_11_0_arm64.whl (343 kB)\n    \u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m343.9/343.9 kB\u001b[0m \u001b[31m11.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25hUsing cached charset_normalizer-3.2.0-cp310-cp310-macosx_11_0_arm64.whl (124 kB)\n    Downloading frozenlist-1.4.0-cp310-cp310-macosx_11_0_arm64.whl (46 kB)\n    \u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m46.0/46.0 kB\u001b[0m \u001b[31m4.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25hDownloading urllib3-2.0.4-py3-none-any.whl (123 kB)\n    \u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m123.9/123.9 kB\u001b[0m \u001b[31m20.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25hInstalling collected packages: urllib3, tqdm, multidict, idna, frozenlist, charset-normalizer, certifi, attrs, async-timeout, yarl, requests, aiosignal, aiohttp, openai\n    Successfully installed aiohttp-3.8.5 aiosignal-1.3.1 async-timeout-4.0.2 attrs-23.1.0 certifi-2023.5.7 charset-normalizer-3.2.0 frozenlist-1.4.0 idna-3.4 multidict-6.0.4 openai-0.27.8 requests-2.31.0 tqdm-4.65.0 urllib3-2.0.4 yarl-1.9.2\n    Note: you may need to restart the kernel to use updated packages.\n    Collecting chromadb\n      Obtaining dependency information for chromadb from https://files.pythonhosted.org/packages/47/b7/41d975f02818c965cdb8a119cab5a38cfb08e0c1abb18efebe9a373ea97b/chromadb-0.4.2-py3-none-any."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 7bac77ff-1d24-48c9-bcea-32a1f482a161
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"7bac77ff-1d24-48c9-bcea-32a1f482a161\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"81dbfa72-5dc3-426c-9ddb-ed40d5fb8bba\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1bb8f0b915b789dc47ed17f5aca6414674ff23d44854e36f8d8c4b1c92678faf\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8e58a4e6-06c7-4d64-9d9d-702823f0bdab\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f0bff1d46574a874bc313737bc10eea73393130c9a01b4ed313b415ebcd929ea\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6121, \"end_char_idx\": 8015, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "0/46.0 kB\u001b[0m \u001b[31m4.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25hDownloading urllib3-2.0.4-py3-none-any.whl (123 kB)\n    \u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m123.9/123.9 kB\u001b[0m \u001b[31m20.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25hInstalling collected packages: urllib3, tqdm, multidict, idna, frozenlist, charset-normalizer, certifi, attrs, async-timeout, yarl, requests, aiosignal, aiohttp, openai\n    Successfully installed aiohttp-3.8.5 aiosignal-1.3.1 async-timeout-4.0.2 attrs-23.1.0 certifi-2023.5.7 charset-normalizer-3.2.0 frozenlist-1.4.0 idna-3.4 multidict-6.0.4 openai-0.27.8 requests-2.31.0 tqdm-4.65.0 urllib3-2.0.4 yarl-1.9.2\n    Note: you may need to restart the kernel to use updated packages.\n    Collecting chromadb\n      Obtaining dependency information for chromadb from https://files.pythonhosted.org/packages/47/b7/41d975f02818c965cdb8a119cab5a38cfb08e0c1abb18efebe9a373ea97b/chromadb-0.4.2-py3-none-any.whl.metadata\n      Downloading chromadb-0.4.2-py3-none-any.whl.metadata (6.9 kB)\n    Collecting pandas>=1.3 (from chromadb)\n      Obtaining dependency information for pandas>=1.3 from https://files.pythonhosted.org/packages/4a/f6/f620ca62365d83e663a255a41b08d2fc2eaf304e0b8b21bb6d62a7390fe3/pandas-2.0.3-cp310-cp310-macosx_11_0_arm64.whl.metadata\n      Using cached pandas-2.0.3-cp310-cp310-macosx_11_0_arm64.whl.metadata (18 kB)\n    Requirement already satisfied: requests>=2.28 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from chromadb) (2.31.0)\n    Collecting pydantic<2.0,>=1.9 (from chromadb)\n      Obtaining dependency information for pydantic<2.0,>=1.9 from https://files.pythonhosted.org/packages/79/3e/6b4d0fb2174beceac9a991ba8e67158b45c35faca9ea4545ae32d47096cd/pydantic-1.10.11-cp310-cp310-macosx_11_0_arm64.whl.metadata\n      Using cached pydantic-1.10.11-cp310-cp310-macosx_11_0_arm64."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 8e58a4e6-06c7-4d64-9d9d-702823f0bdab
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"8e58a4e6-06c7-4d64-9d9d-702823f0bdab\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7bac77ff-1d24-48c9-bcea-32a1f482a161\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"57367f8bcb998b4bdec07caa2bb404c03a5fe0a707eee56eb12240cfa59a8a4b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f17390e6-ccb0-43b3-aabd-5bd5f800515c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a747c43aaf3aa919f95211780385a6dba693592494771bcde4005c8b729b5b67\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7050, \"end_char_idx\": 8929, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "4.2-py3-none-any.whl.metadata\n      Downloading chromadb-0.4.2-py3-none-any.whl.metadata (6.9 kB)\n    Collecting pandas>=1.3 (from chromadb)\n      Obtaining dependency information for pandas>=1.3 from https://files.pythonhosted.org/packages/4a/f6/f620ca62365d83e663a255a41b08d2fc2eaf304e0b8b21bb6d62a7390fe3/pandas-2.0.3-cp310-cp310-macosx_11_0_arm64.whl.metadata\n      Using cached pandas-2.0.3-cp310-cp310-macosx_11_0_arm64.whl.metadata (18 kB)\n    Requirement already satisfied: requests>=2.28 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from chromadb) (2.31.0)\n    Collecting pydantic<2.0,>=1.9 (from chromadb)\n      Obtaining dependency information for pydantic<2.0,>=1.9 from https://files.pythonhosted.org/packages/79/3e/6b4d0fb2174beceac9a991ba8e67158b45c35faca9ea4545ae32d47096cd/pydantic-1.10.11-cp310-cp310-macosx_11_0_arm64.whl.metadata\n      Using cached pydantic-1.10.11-cp310-cp310-macosx_11_0_arm64.whl.metadata (148 kB)\n    Collecting chroma-hnswlib==0.7.1 (from chromadb)\n      Obtaining dependency information for chroma-hnswlib==0.7.1 from https://files.pythonhosted.org/packages/a5/d5/54947127f5cb2a1fcef40877fb3e6044495eec0a158ba0956babe4ab2a77/chroma_hnswlib-0.7.1-cp310-cp310-macosx_13_0_arm64.whl.metadata\n      Using cached chroma_hnswlib-0.7.1-cp310-cp310-macosx_13_0_arm64.whl.metadata (252 bytes)\n    Collecting fastapi<0.100.0,>=0.95.2 (from chromadb)\n      Obtaining dependency information for fastapi<0.100.0,>=0.95.2 from https://files.pythonhosted.org/packages/73/eb/03b691afa0b5ffa1e93ed34f97ec1e7855c758efbdcfb16c209af0b0506b/fastapi-0.99.1-py3-none-any.whl.metadata\n      Using cached fastapi-0.99.1-py3-none-any.whl.metadata (23 kB)\n    Collecting uvicorn[standard]>=0.18.3 (from chromadb)\n      Obtaining dependency information for uvicorn[standard]>=0.18.3 from https://files.pythonhosted."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: f17390e6-ccb0-43b3-aabd-5bd5f800515c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"f17390e6-ccb0-43b3-aabd-5bd5f800515c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8e58a4e6-06c7-4d64-9d9d-702823f0bdab\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"21722b67ffa496f9d8e8a8876a692e0cc74e02df264be26c0797895ab765d41e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9a73e628-5103-458d-a5e6-bd9f586aeae4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"63a74415aa0316047bba32682a9b6ff1252e608c967c6a44c612e413f620213f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7903, \"end_char_idx\": 9906, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "11-cp310-cp310-macosx_11_0_arm64.whl.metadata\n      Using cached pydantic-1.10.11-cp310-cp310-macosx_11_0_arm64.whl.metadata (148 kB)\n    Collecting chroma-hnswlib==0.7.1 (from chromadb)\n      Obtaining dependency information for chroma-hnswlib==0.7.1 from https://files.pythonhosted.org/packages/a5/d5/54947127f5cb2a1fcef40877fb3e6044495eec0a158ba0956babe4ab2a77/chroma_hnswlib-0.7.1-cp310-cp310-macosx_13_0_arm64.whl.metadata\n      Using cached chroma_hnswlib-0.7.1-cp310-cp310-macosx_13_0_arm64.whl.metadata (252 bytes)\n    Collecting fastapi<0.100.0,>=0.95.2 (from chromadb)\n      Obtaining dependency information for fastapi<0.100.0,>=0.95.2 from https://files.pythonhosted.org/packages/73/eb/03b691afa0b5ffa1e93ed34f97ec1e7855c758efbdcfb16c209af0b0506b/fastapi-0.99.1-py3-none-any.whl.metadata\n      Using cached fastapi-0.99.1-py3-none-any.whl.metadata (23 kB)\n    Collecting uvicorn[standard]>=0.18.3 (from chromadb)\n      Obtaining dependency information for uvicorn[standard]>=0.18.3 from https://files.pythonhosted.org/packages/5d/07/b9eac057f7efa56900640a233c1ed63db83568322c6bcbabe98f741d5289/uvicorn-0.23.1-py3-none-any.whl.metadata\n      Using cached uvicorn-0.23.1-py3-none-any.whl.metadata (6.2 kB)\n    Collecting numpy>=1.21.6 (from chromadb)\n      Obtaining dependency information for numpy>=1.21.6 from https://files.pythonhosted.org/packages/1b/cd/9e8313ffd849626c836fffd7881296a74f53a7739bd9ce7a6e22b1fc843b/numpy-1.25.1-cp310-cp310-macosx_11_0_arm64.whl.metadata\n      Using cached numpy-1.25.1-cp310-cp310-macosx_11_0_arm64.whl.metadata (5.6 kB)\n    Collecting posthog>=2.4.0 (from chromadb)\n      Using cached posthog-3.0.1-py2.py3-none-any.whl (37 kB)\n    Requirement already satisfied: typing-extensions>=4.5.0 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from chromadb) (4.7.1)\n    Collecting pulsar-client>=3.1.0 (from chromadb)\n      Obtaining dependency information for pulsar-client>=3.1.0 from https://files.pythonhosted."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 9a73e628-5103-458d-a5e6-bd9f586aeae4
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"9a73e628-5103-458d-a5e6-bd9f586aeae4\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f17390e6-ccb0-43b3-aabd-5bd5f800515c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"828bf73fee962d40b49987729ec15243a4626889a2e15e526ca281f0ae72c8f1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1c2d3419-b938-485c-83ef-57bf79ac529e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"31e5bc06ffe5c69f796c05e65e3e7d768b09ec188e5420d2f9a2d54a84f3f413\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8810, \"end_char_idx\": 10806, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "3 (from chromadb)\n      Obtaining dependency information for uvicorn[standard]>=0.18.3 from https://files.pythonhosted.org/packages/5d/07/b9eac057f7efa56900640a233c1ed63db83568322c6bcbabe98f741d5289/uvicorn-0.23.1-py3-none-any.whl.metadata\n      Using cached uvicorn-0.23.1-py3-none-any.whl.metadata (6.2 kB)\n    Collecting numpy>=1.21.6 (from chromadb)\n      Obtaining dependency information for numpy>=1.21.6 from https://files.pythonhosted.org/packages/1b/cd/9e8313ffd849626c836fffd7881296a74f53a7739bd9ce7a6e22b1fc843b/numpy-1.25.1-cp310-cp310-macosx_11_0_arm64.whl.metadata\n      Using cached numpy-1.25.1-cp310-cp310-macosx_11_0_arm64.whl.metadata (5.6 kB)\n    Collecting posthog>=2.4.0 (from chromadb)\n      Using cached posthog-3.0.1-py2.py3-none-any.whl (37 kB)\n    Requirement already satisfied: typing-extensions>=4.5.0 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from chromadb) (4.7.1)\n    Collecting pulsar-client>=3.1.0 (from chromadb)\n      Obtaining dependency information for pulsar-client>=3.1.0 from https://files.pythonhosted.org/packages/43/85/ab0455008ce3335a1c75a7c500fd8921ab166f34821fa67dc91ae9687a40/pulsar_client-3.2.0-cp310-cp310-macosx_10_15_universal2.whl.metadata\n      Using cached pulsar_client-3.2.0-cp310-cp310-macosx_10_15_universal2.whl.metadata (1.0 kB)\n    Collecting onnxruntime>=1.14.1 (from chromadb)\n      Obtaining dependency information for onnxruntime>=1.14.1 from https://files.pythonhosted.org/packages/cf/06/0c6e355b9ddbebc34d0e21bc5be1e4bd2c124ebd9030525838fa6e65eaa8/onnxruntime-1.15.1-cp310-cp310-macosx_11_0_arm64.whl.metadata\n      Using cached onnxruntime-1.15.1-cp310-cp310-macosx_11_0_arm64.whl.metadata (4.0 kB)\n    Collecting tokenizers>=0.13.2 (from chromadb)\n      Using cached tokenizers-0.13.3-cp310-cp310-macosx_12_0_arm64.whl (3.9 MB)\n    Collecting pypika>=0.48.9 (from chromadb)\n      Using cached PyPika-0.48.9-py2.py3-none-any.whl\n    Requirement already satisfied: tqdm>=4.65."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 1c2d3419-b938-485c-83ef-57bf79ac529e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"1c2d3419-b938-485c-83ef-57bf79ac529e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9a73e628-5103-458d-a5e6-bd9f586aeae4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"72e29c6be555eab86497a73b53edaf55a16a54e1705bf4df788ea1a8098c5cde\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c20f8957-3544-4404-9927-15ddfa054af4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a35813e3cee8b0f51f9ff668bfae389c2851389d3717cdb4e0bcd321fabf5503\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9790, \"end_char_idx\": 11821, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "1.0 (from chromadb)\n      Obtaining dependency information for pulsar-client>=3.1.0 from https://files.pythonhosted.org/packages/43/85/ab0455008ce3335a1c75a7c500fd8921ab166f34821fa67dc91ae9687a40/pulsar_client-3.2.0-cp310-cp310-macosx_10_15_universal2.whl.metadata\n      Using cached pulsar_client-3.2.0-cp310-cp310-macosx_10_15_universal2.whl.metadata (1.0 kB)\n    Collecting onnxruntime>=1.14.1 (from chromadb)\n      Obtaining dependency information for onnxruntime>=1.14.1 from https://files.pythonhosted.org/packages/cf/06/0c6e355b9ddbebc34d0e21bc5be1e4bd2c124ebd9030525838fa6e65eaa8/onnxruntime-1.15.1-cp310-cp310-macosx_11_0_arm64.whl.metadata\n      Using cached onnxruntime-1.15.1-cp310-cp310-macosx_11_0_arm64.whl.metadata (4.0 kB)\n    Collecting tokenizers>=0.13.2 (from chromadb)\n      Using cached tokenizers-0.13.3-cp310-cp310-macosx_12_0_arm64.whl (3.9 MB)\n    Collecting pypika>=0.48.9 (from chromadb)\n      Using cached PyPika-0.48.9-py2.py3-none-any.whl\n    Requirement already satisfied: tqdm>=4.65.0 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from chromadb) (4.65.0)\n    Collecting overrides>=7.3.1 (from chromadb)\n      Using cached overrides-7.3.1-py3-none-any.whl (17 kB)\n    Collecting importlib-resources (from chromadb)\n      Obtaining dependency information for importlib-resources from https://files.pythonhosted.org/packages/29/d1/bed03eca30aa05aaf6e0873de091f9385c48705c4a607c2dfe3edbe543e8/importlib_resources-6.0.0-py3-none-any.whl.metadata\n      Using cached importlib_resources-6.0.0-py3-none-any.whl.metadata (4.2 kB)\n    Collecting starlette<0.28.0,>=0.27.0 (from fastapi<0.100.0,>=0.95.2->chromadb)\n      Obtaining dependency information for starlette<0.28.0,>=0.27.0 from https://files.pythonhosted.org/packages/58/f8/e2cca22387965584a409795913b774235752be4176d276714e15e1a58884/starlette-0.27.0-py3-none-any.whl.metadata\n      Using cached starlette-0.27.0-py3-none-any.whl.metadata (5.8 kB)\n    Collecting coloredlogs (from onnxruntime>=1.14."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: c20f8957-3544-4404-9927-15ddfa054af4
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c20f8957-3544-4404-9927-15ddfa054af4\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1c2d3419-b938-485c-83ef-57bf79ac529e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"031b9c8f34ad1821435ae6571ee8b816be4d5bfdcf82d45c4e8cd5566cc300e6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e8d9ad6a-14da-46f7-ab3b-d7a3e766879d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a9771477b207bbed33c5eec2fcbe187b1b39d6c7b39c7b4079f475af7dbab9cc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10734, \"end_char_idx\": 12870, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "48.9-py2.py3-none-any.whl\n    Requirement already satisfied: tqdm>=4.65.0 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from chromadb) (4.65.0)\n    Collecting overrides>=7.3.1 (from chromadb)\n      Using cached overrides-7.3.1-py3-none-any.whl (17 kB)\n    Collecting importlib-resources (from chromadb)\n      Obtaining dependency information for importlib-resources from https://files.pythonhosted.org/packages/29/d1/bed03eca30aa05aaf6e0873de091f9385c48705c4a607c2dfe3edbe543e8/importlib_resources-6.0.0-py3-none-any.whl.metadata\n      Using cached importlib_resources-6.0.0-py3-none-any.whl.metadata (4.2 kB)\n    Collecting starlette<0.28.0,>=0.27.0 (from fastapi<0.100.0,>=0.95.2->chromadb)\n      Obtaining dependency information for starlette<0.28.0,>=0.27.0 from https://files.pythonhosted.org/packages/58/f8/e2cca22387965584a409795913b774235752be4176d276714e15e1a58884/starlette-0.27.0-py3-none-any.whl.metadata\n      Using cached starlette-0.27.0-py3-none-any.whl.metadata (5.8 kB)\n    Collecting coloredlogs (from onnxruntime>=1.14.1->chromadb)\n      Using cached coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)\n    Collecting flatbuffers (from onnxruntime>=1.14.1->chromadb)\n      Obtaining dependency information for flatbuffers from https://files.pythonhosted.org/packages/6f/12/d5c79ee252793ffe845d58a913197bfa02ae9a0b5c9bc3dc4b58d477b9e7/flatbuffers-23.5.26-py2.py3-none-any.whl.metadata\n      Using cached flatbuffers-23.5.26-py2.py3-none-any.whl.metadata (850 bytes)\n    Requirement already satisfied: packaging in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from onnxruntime>=1.14.1->chromadb) (23.1)\n    Collecting protobuf (from onnxruntime>=1.14.1->chromadb)\n      Obtaining dependency information for protobuf from https://files.pythonhosted.org/packages/cb/d3/a164038605494d49acc4f9cda1c0bc200b96382c53edd561387263bb181d/protobuf-4.23.4-cp37-abi3-macosx_10_9_universal2.whl.metadata\n      Using cached protobuf-4.23.4-cp37-abi3-macosx_10_9_universal2.whl.metadata (540 bytes)\n    Collecting sympy (from onnxruntime>=1.14."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: e8d9ad6a-14da-46f7-ab3b-d7a3e766879d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e8d9ad6a-14da-46f7-ab3b-d7a3e766879d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c20f8957-3544-4404-9927-15ddfa054af4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"b5b542927892ef31a0faebb5247e4e457d78e42331639f76fcdbc0bf66228ed9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"20631516-73b8-45e5-aed3-1cd3b88a7997\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0c870e984ba4ac3159bd8b49355cba154ae47ec5b1d2e520684c6cae098b0c69\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11748, \"end_char_idx\": 13910, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "whl.metadata (5.8 kB)\n    Collecting coloredlogs (from onnxruntime>=1.14.1->chromadb)\n      Using cached coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)\n    Collecting flatbuffers (from onnxruntime>=1.14.1->chromadb)\n      Obtaining dependency information for flatbuffers from https://files.pythonhosted.org/packages/6f/12/d5c79ee252793ffe845d58a913197bfa02ae9a0b5c9bc3dc4b58d477b9e7/flatbuffers-23.5.26-py2.py3-none-any.whl.metadata\n      Using cached flatbuffers-23.5.26-py2.py3-none-any.whl.metadata (850 bytes)\n    Requirement already satisfied: packaging in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from onnxruntime>=1.14.1->chromadb) (23.1)\n    Collecting protobuf (from onnxruntime>=1.14.1->chromadb)\n      Obtaining dependency information for protobuf from https://files.pythonhosted.org/packages/cb/d3/a164038605494d49acc4f9cda1c0bc200b96382c53edd561387263bb181d/protobuf-4.23.4-cp37-abi3-macosx_10_9_universal2.whl.metadata\n      Using cached protobuf-4.23.4-cp37-abi3-macosx_10_9_universal2.whl.metadata (540 bytes)\n    Collecting sympy (from onnxruntime>=1.14.1->chromadb)\n      Using cached sympy-1.12-py3-none-any.whl (5.7 MB)\n    Requirement already satisfied: python-dateutil>=2.8.2 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from pandas>=1.3->chromadb) (2.8.2)\n    Collecting pytz>=2020.1 (from pandas>=1.3->chromadb)\n      Using cached pytz-2023.3-py2.py3-none-any.whl (502 kB)\n    Collecting tzdata>=2022.1 (from pandas>=1.3->chromadb)\n      Using cached tzdata-2023.3-py2.py3-none-any.whl (341 kB)\n    Requirement already satisfied: six>=1.5 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from posthog>=2.4.0->chromadb) (1.16.0)\n    Collecting monotonic>=1.5 (from posthog>=2.4.0->chromadb)\n      Using cached monotonic-1.6-py2.py3-none-any.whl (8.2 kB)\n    Collecting backoff>=1.10.0 (from posthog>=2.4.0->chromadb)\n      Using cached backoff-2.2.1-py3-none-any.whl (15 kB)\n    Requirement already satisfied: certifi in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 20631516-73b8-45e5-aed3-1cd3b88a7997
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"20631516-73b8-45e5-aed3-1cd3b88a7997\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e8d9ad6a-14da-46f7-ab3b-d7a3e766879d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"e96b6263a2c5130a5c9b3a102b6d0dd189f73875534b4b88b2c4cc4ab7898ab7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9a0eaebb-45cc-47fa-b617-594f520ba7fe\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a89dd86bd6173f8cf6ca5d19f2ea4f78e3482602aae1162d619b5ac1b4ea8642\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12800, \"end_char_idx\": 14928, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "whl.metadata (540 bytes)\n    Collecting sympy (from onnxruntime>=1.14.1->chromadb)\n      Using cached sympy-1.12-py3-none-any.whl (5.7 MB)\n    Requirement already satisfied: python-dateutil>=2.8.2 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from pandas>=1.3->chromadb) (2.8.2)\n    Collecting pytz>=2020.1 (from pandas>=1.3->chromadb)\n      Using cached pytz-2023.3-py2.py3-none-any.whl (502 kB)\n    Collecting tzdata>=2022.1 (from pandas>=1.3->chromadb)\n      Using cached tzdata-2023.3-py2.py3-none-any.whl (341 kB)\n    Requirement already satisfied: six>=1.5 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from posthog>=2.4.0->chromadb) (1.16.0)\n    Collecting monotonic>=1.5 (from posthog>=2.4.0->chromadb)\n      Using cached monotonic-1.6-py2.py3-none-any.whl (8.2 kB)\n    Collecting backoff>=1.10.0 (from posthog>=2.4.0->chromadb)\n      Using cached backoff-2.2.1-py3-none-any.whl (15 kB)\n    Requirement already satisfied: certifi in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from pulsar-client>=3.1.0->chromadb) (2023.5.7)\n    Requirement already satisfied: charset-normalizer<4,>=2 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from requests>=2.28->chromadb) (3.2.0)\n    Requirement already satisfied: idna<4,>=2.5 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from requests>=2.28->chromadb) (3.4)\n    Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from requests>=2.28->chromadb) (2.0.4)\n    Collecting click>=7.0 (from uvicorn[standard]>=0.18.3->chromadb)\n      Obtaining dependency information for click>=7.0 from https://files.pythonhosted.org/packages/1a/70/e63223f8116931d365993d4a6b7ef653a4d920b41d03de7c59499962821f/click-8.1.6-py3-none-any.whl.metadata\n      Using cached click-8.1.6-py3-none-any.whl.metadata (3.0 kB)\n    Collecting h11>=0.8 (from uvicorn[standard]>=0.18."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 9a0eaebb-45cc-47fa-b617-594f520ba7fe
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"9a0eaebb-45cc-47fa-b617-594f520ba7fe\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"20631516-73b8-45e5-aed3-1cd3b88a7997\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"67e92f787292ab99157f792b7893d834bbe64a8ec5fb6d0b9fd7e11b3a62de28\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5ba613c4-7b47-426d-9d4c-9137306049ad\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4d4960667d4c09bc9bad056f6c3e5c657eb02277118bfed7992df10cfc65cdec\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13910, \"end_char_idx\": 15912, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "10/site-packages (from pulsar-client>=3.1.0->chromadb) (2023.5.7)\n    Requirement already satisfied: charset-normalizer<4,>=2 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from requests>=2.28->chromadb) (3.2.0)\n    Requirement already satisfied: idna<4,>=2.5 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from requests>=2.28->chromadb) (3.4)\n    Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from requests>=2.28->chromadb) (2.0.4)\n    Collecting click>=7.0 (from uvicorn[standard]>=0.18.3->chromadb)\n      Obtaining dependency information for click>=7.0 from https://files.pythonhosted.org/packages/1a/70/e63223f8116931d365993d4a6b7ef653a4d920b41d03de7c59499962821f/click-8.1.6-py3-none-any.whl.metadata\n      Using cached click-8.1.6-py3-none-any.whl.metadata (3.0 kB)\n    Collecting h11>=0.8 (from uvicorn[standard]>=0.18.3->chromadb)\n      Using cached h11-0.14.0-py3-none-any.whl (58 kB)\n    Collecting httptools>=0.5.0 (from uvicorn[standard]>=0.18.3->chromadb)\n      Obtaining dependency information for httptools>=0.5.0 from https://files.pythonhosted.org/packages/8f/71/d535e9f6967958d21b8fe1baeb7efb6304b86e8fcff44d0bda8690e0aec9/httptools-0.6.0-cp310-cp310-macosx_10_9_universal2.whl.metadata\n      Using cached httptools-0.6.0-cp310-cp310-macosx_10_9_universal2.whl.metadata (3.6 kB)\n    Collecting python-dotenv>=0.13 (from uvicorn[standard]>=0.18.3->chromadb)\n      Using cached python_dotenv-1.0.0-py3-none-any.whl (19 kB)\n    Collecting pyyaml>=5.1 (from uvicorn[standard]>=0.18.3->chromadb)\n      Obtaining dependency information for pyyaml>=5.1 from https://files.pythonhosted.org/packages/5b/07/10033a403b23405a8fc48975444463d3d10a5c2736b7eb2550b07b367429/PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl.metadata\n      Using cached PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl.metadata (2."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 5ba613c4-7b47-426d-9d4c-9137306049ad
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"5ba613c4-7b47-426d-9d4c-9137306049ad\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9a0eaebb-45cc-47fa-b617-594f520ba7fe\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"8c5a59adb1728a915245929e31a77a048e968586ac74d3fc8abca2cf58e2e21a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"197a2123-a39c-4f9b-8eda-191c1bf8ccd8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d3d57c5eba9d2e0f23e4154c915b76b48b8d72298d685642b4ca716a0f19d8d4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14928, \"end_char_idx\": 16767, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "3->chromadb)\n      Using cached h11-0.14.0-py3-none-any.whl (58 kB)\n    Collecting httptools>=0.5.0 (from uvicorn[standard]>=0.18.3->chromadb)\n      Obtaining dependency information for httptools>=0.5.0 from https://files.pythonhosted.org/packages/8f/71/d535e9f6967958d21b8fe1baeb7efb6304b86e8fcff44d0bda8690e0aec9/httptools-0.6.0-cp310-cp310-macosx_10_9_universal2.whl.metadata\n      Using cached httptools-0.6.0-cp310-cp310-macosx_10_9_universal2.whl.metadata (3.6 kB)\n    Collecting python-dotenv>=0.13 (from uvicorn[standard]>=0.18.3->chromadb)\n      Using cached python_dotenv-1.0.0-py3-none-any.whl (19 kB)\n    Collecting pyyaml>=5.1 (from uvicorn[standard]>=0.18.3->chromadb)\n      Obtaining dependency information for pyyaml>=5.1 from https://files.pythonhosted.org/packages/5b/07/10033a403b23405a8fc48975444463d3d10a5c2736b7eb2550b07b367429/PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl.metadata\n      Using cached PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl.metadata (2.1 kB)\n    Collecting uvloop!=0.15.0,!=0.15.1,>=0.14.0 (from uvicorn[standard]>=0.18.3->chromadb)\n      Using cached uvloop-0.17.0-cp310-cp310-macosx_10_9_universal2.whl (2.1 MB)\n    Collecting watchfiles>=0.13 (from uvicorn[standard]>=0.18.3->chromadb)\n      Using cached watchfiles-0.19.0-cp37-abi3-macosx_11_0_arm64.whl (388 kB)\n    Collecting websockets>=10.4 (from uvicorn[standard]>=0.18.3->chromadb)\n      Using cached websockets-11.0.3-cp310-cp310-macosx_11_0_arm64.whl (121 kB)\n    Collecting anyio<5,>=3.4.0 (from starlette<0.28.0,>=0.27.0->fastapi<0.100.0,>=0.95.2->chromadb)\n      Obtaining dependency information for anyio<5,>=3.4.0 from https://files.pythonhosted.org/packages/19/24/44299477fe7dcc9cb58d0a57d5a7588d6af2ff403fdd2d47a246c91a3246/anyio-3.7.1-py3-none-any.whl.metadata\n      Using cached anyio-3.7.1-py3-none-any.whl.metadata (4."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 197a2123-a39c-4f9b-8eda-191c1bf8ccd8
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"197a2123-a39c-4f9b-8eda-191c1bf8ccd8\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5ba613c4-7b47-426d-9d4c-9137306049ad\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"2a61254633a16831d1fd46a8cdd7d21bc1749a661e71d44998c53b35463ae29c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0bbe83ae-a6b1-4a6c-a86f-2de559cd8195\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"532a6220d13418b7ed177e24bc801f08a6a339411958723f50510aeee323d0ef\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15821, \"end_char_idx\": 17678, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "whl.metadata\n      Using cached PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl.metadata (2.1 kB)\n    Collecting uvloop!=0.15.0,!=0.15.1,>=0.14.0 (from uvicorn[standard]>=0.18.3->chromadb)\n      Using cached uvloop-0.17.0-cp310-cp310-macosx_10_9_universal2.whl (2.1 MB)\n    Collecting watchfiles>=0.13 (from uvicorn[standard]>=0.18.3->chromadb)\n      Using cached watchfiles-0.19.0-cp37-abi3-macosx_11_0_arm64.whl (388 kB)\n    Collecting websockets>=10.4 (from uvicorn[standard]>=0.18.3->chromadb)\n      Using cached websockets-11.0.3-cp310-cp310-macosx_11_0_arm64.whl (121 kB)\n    Collecting anyio<5,>=3.4.0 (from starlette<0.28.0,>=0.27.0->fastapi<0.100.0,>=0.95.2->chromadb)\n      Obtaining dependency information for anyio<5,>=3.4.0 from https://files.pythonhosted.org/packages/19/24/44299477fe7dcc9cb58d0a57d5a7588d6af2ff403fdd2d47a246c91a3246/anyio-3.7.1-py3-none-any.whl.metadata\n      Using cached anyio-3.7.1-py3-none-any.whl.metadata (4.7 kB)\n    Collecting humanfriendly>=9.1 (from coloredlogs->onnxruntime>=1.14.1->chromadb)\n      Using cached humanfriendly-10.0-py2.py3-none-any.whl (86 kB)\n    Collecting mpmath>=0.19 (from sympy->onnxruntime>=1.14.1->chromadb)\n      Using cached mpmath-1.3.0-py3-none-any.whl (536 kB)\n    Collecting sniffio>=1.1 (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi<0.100.0,>=0.95.2->chromadb)\n      Using cached sniffio-1.3.0-py3-none-any.whl (10 kB)\n    Collecting exceptiongroup (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi<0.100.0,>=0.95.2->chromadb)\n      Obtaining dependency information for exceptiongroup from https://files.pythonhosted.org/packages/fe/17/f43b7c9ccf399d72038042ee72785c305f6c6fdc6231942f8ab99d995742/exceptiongroup-1.1.2-py3-none-any.whl.metadata\n      Using cached exceptiongroup-1.1.2-py3-none-any.whl.metadata (6.1 kB)\n    Downloading chromadb-0.4.2-py3-none-any."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 0bbe83ae-a6b1-4a6c-a86f-2de559cd8195
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0bbe83ae-a6b1-4a6c-a86f-2de559cd8195\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"197a2123-a39c-4f9b-8eda-191c1bf8ccd8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"43bf7045ae076615ddd241a374727713a0d71da849a9d6e669dad81a3da3432e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e26ab46e-ebf3-4b7d-87a5-0f8fa8a8e492\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fc48370691c2af01a41535cb3dc824f9ffb04d104cbfc3a7c4260dc312e6f4ca\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16677, \"end_char_idx\": 18461, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "7.1-py3-none-any.whl.metadata\n      Using cached anyio-3.7.1-py3-none-any.whl.metadata (4.7 kB)\n    Collecting humanfriendly>=9.1 (from coloredlogs->onnxruntime>=1.14.1->chromadb)\n      Using cached humanfriendly-10.0-py2.py3-none-any.whl (86 kB)\n    Collecting mpmath>=0.19 (from sympy->onnxruntime>=1.14.1->chromadb)\n      Using cached mpmath-1.3.0-py3-none-any.whl (536 kB)\n    Collecting sniffio>=1.1 (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi<0.100.0,>=0.95.2->chromadb)\n      Using cached sniffio-1.3.0-py3-none-any.whl (10 kB)\n    Collecting exceptiongroup (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi<0.100.0,>=0.95.2->chromadb)\n      Obtaining dependency information for exceptiongroup from https://files.pythonhosted.org/packages/fe/17/f43b7c9ccf399d72038042ee72785c305f6c6fdc6231942f8ab99d995742/exceptiongroup-1.1.2-py3-none-any.whl.metadata\n      Using cached exceptiongroup-1.1.2-py3-none-any.whl.metadata (6.1 kB)\n    Downloading chromadb-0.4.2-py3-none-any.whl (399 kB)\n    \u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m399.3/399.3 kB\u001b[0m \u001b[31m12.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25hUsing cached chroma_hnswlib-0.7.1-cp310-cp310-macosx_13_0_arm64.whl (195 kB)\n    Using cached fastapi-0.99.1-py3-none-any.whl (58 kB)\n    Using cached numpy-1.25.1-cp310-cp310-macosx_11_0_arm64.whl (14.0 MB)\n    Using cached onnxruntime-1.15.1-cp310-cp310-macosx_11_0_arm64.whl (6.1 MB)\n    Using cached pandas-2.0.3-cp310-cp310-macosx_11_0_arm64.whl (10.8 MB)\n    Using cached pulsar_client-3.2.0-cp310-cp310-macosx_10_15_universal2.whl (10.8 MB)\n    Using cached pydantic-1.10.11-cp310-cp310-macosx_11_0_arm64.whl (2.5 MB)\n    Using cached importlib_resources-6.0.0-py3-none-any.whl (31 kB)\n    Using cached click-8.1.6-py3-none-any."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: e26ab46e-ebf3-4b7d-87a5-0f8fa8a8e492
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e26ab46e-ebf3-4b7d-87a5-0f8fa8a8e492\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0bbe83ae-a6b1-4a6c-a86f-2de559cd8195\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"a6c631432a4fd44f8849d95a92bac843ee984095a43aa88bf26a65aee7b5a2c9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b6f35b47-e3f9-4759-ba22-15794970a214\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"be62f03cdee8aa6b42d698d98c708d7dcfddc1eb2c2ce61ffa020c5c87f7cea7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17595, \"end_char_idx\": 19469, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "1.2-py3-none-any.whl.metadata (6.1 kB)\n    Downloading chromadb-0.4.2-py3-none-any.whl (399 kB)\n    \u001b[2K   \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m399.3/399.3 kB\u001b[0m \u001b[31m12.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25hUsing cached chroma_hnswlib-0.7.1-cp310-cp310-macosx_13_0_arm64.whl (195 kB)\n    Using cached fastapi-0.99.1-py3-none-any.whl (58 kB)\n    Using cached numpy-1.25.1-cp310-cp310-macosx_11_0_arm64.whl (14.0 MB)\n    Using cached onnxruntime-1.15.1-cp310-cp310-macosx_11_0_arm64.whl (6.1 MB)\n    Using cached pandas-2.0.3-cp310-cp310-macosx_11_0_arm64.whl (10.8 MB)\n    Using cached pulsar_client-3.2.0-cp310-cp310-macosx_10_15_universal2.whl (10.8 MB)\n    Using cached pydantic-1.10.11-cp310-cp310-macosx_11_0_arm64.whl (2.5 MB)\n    Using cached importlib_resources-6.0.0-py3-none-any.whl (31 kB)\n    Using cached click-8.1.6-py3-none-any.whl (97 kB)\n    Using cached httptools-0.6.0-cp310-cp310-macosx_10_9_universal2.whl (237 kB)\n    Using cached PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl (169 kB)\n    Using cached starlette-0.27.0-py3-none-any.whl (66 kB)\n    Using cached flatbuffers-23.5.26-py2.py3-none-any.whl (26 kB)\n    Using cached protobuf-4.23.4-cp37-abi3-macosx_10_9_universal2.whl (400 kB)\n    Using cached uvicorn-0.23.1-py3-none-any.whl (59 kB)\n    Using cached anyio-3.7.1-py3-none-any.whl (80 kB)\n    Using cached exceptiongroup-1.1.2-py3-none-any.whl (14 kB)\n    Installing collected packages: tokenizers, pytz, pypika, mpmath, monotonic, flatbuffers, websockets, uvloop, tzdata, sympy, sniffio, pyyaml, python-dotenv, pydantic, pulsar-client, protobuf, overrides, numpy, importlib-resources, humanfriendly, httptools, h11, exceptiongroup, click, backoff, uvicorn, posthog, pandas, coloredlogs, chroma-hnswlib, anyio, watchfiles, starlette, onnxruntime, fastapi, chromadb\n    Successfully installed anyio-3.7.1 backoff-2.2."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: b6f35b47-e3f9-4759-ba22-15794970a214
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"b6f35b47-e3f9-4759-ba22-15794970a214\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e26ab46e-ebf3-4b7d-87a5-0f8fa8a8e492\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"eefa500abf2c84b6c61c909f2583dd50554d96b4d7ab0f955acf8f8100976f97\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b7c035a2-45e1-4111-a572-58aa56b32d3a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0648dffc3582738fbfdcfe4d638c81845d49076868d83d09ff3cb1ab80c6b817\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18390, \"end_char_idx\": 20088, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "0.0-py3-none-any.whl (31 kB)\n    Using cached click-8.1.6-py3-none-any.whl (97 kB)\n    Using cached httptools-0.6.0-cp310-cp310-macosx_10_9_universal2.whl (237 kB)\n    Using cached PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl (169 kB)\n    Using cached starlette-0.27.0-py3-none-any.whl (66 kB)\n    Using cached flatbuffers-23.5.26-py2.py3-none-any.whl (26 kB)\n    Using cached protobuf-4.23.4-cp37-abi3-macosx_10_9_universal2.whl (400 kB)\n    Using cached uvicorn-0.23.1-py3-none-any.whl (59 kB)\n    Using cached anyio-3.7.1-py3-none-any.whl (80 kB)\n    Using cached exceptiongroup-1.1.2-py3-none-any.whl (14 kB)\n    Installing collected packages: tokenizers, pytz, pypika, mpmath, monotonic, flatbuffers, websockets, uvloop, tzdata, sympy, sniffio, pyyaml, python-dotenv, pydantic, pulsar-client, protobuf, overrides, numpy, importlib-resources, humanfriendly, httptools, h11, exceptiongroup, click, backoff, uvicorn, posthog, pandas, coloredlogs, chroma-hnswlib, anyio, watchfiles, starlette, onnxruntime, fastapi, chromadb\n    Successfully installed anyio-3.7.1 backoff-2.2.1 chroma-hnswlib-0.7.1 chromadb-0.4.2 click-8.1.6 coloredlogs-15.0.1 exceptiongroup-1.1.2 fastapi-0.99.1 flatbuffers-23.5.26 h11-0.14.0 httptools-0.6.0 humanfriendly-10.0 importlib-resources-6.0.0 monotonic-1.6 mpmath-1.3.0 numpy-1.25.1 onnxruntime-1.15.1 overrides-7.3.1 pandas-2.0.3 posthog-3.0.1 protobuf-4.23.4 pulsar-client-3.2.0 pydantic-1.10.11 pypika-0.48.9 python-dotenv-1.0.0 pytz-2023.3 pyyaml-6.0.1 sniffio-1.3.0 starlette-0.27.0 sympy-1.12 tokenizers-0.13.3 tzdata-2023.3 uvicorn-0.23.1 uvloop-0.17.0 watchfiles-0.19.0 websockets-11.0.3\n    Note: you may need to restart the kernel to use updated packages."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: b7c035a2-45e1-4111-a572-58aa56b32d3a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"b7c035a2-45e1-4111-a572-58aa56b32d3a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b6f35b47-e3f9-4759-ba22-15794970a214\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"6e16a96eea041d876d6b2c6736e6c7515f0109628d44896f40ef1ee99470ba2d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"78fef95d-4d0d-48b3-84c3-613a627b65b5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bb97a35ff3877fb498af5413fb94a64d443cfb7e6fa92291c630aa3306f2ccca\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19110, \"end_char_idx\": 20985, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "uvloop, tzdata, sympy, sniffio, pyyaml, python-dotenv, pydantic, pulsar-client, protobuf, overrides, numpy, importlib-resources, humanfriendly, httptools, h11, exceptiongroup, click, backoff, uvicorn, posthog, pandas, coloredlogs, chroma-hnswlib, anyio, watchfiles, starlette, onnxruntime, fastapi, chromadb\n    Successfully installed anyio-3.7.1 backoff-2.2.1 chroma-hnswlib-0.7.1 chromadb-0.4.2 click-8.1.6 coloredlogs-15.0.1 exceptiongroup-1.1.2 fastapi-0.99.1 flatbuffers-23.5.26 h11-0.14.0 httptools-0.6.0 humanfriendly-10.0 importlib-resources-6.0.0 monotonic-1.6 mpmath-1.3.0 numpy-1.25.1 onnxruntime-1.15.1 overrides-7.3.1 pandas-2.0.3 posthog-3.0.1 protobuf-4.23.4 pulsar-client-3.2.0 pydantic-1.10.11 pypika-0.48.9 python-dotenv-1.0.0 pytz-2023.3 pyyaml-6.0.1 sniffio-1.3.0 starlette-0.27.0 sympy-1.12 tokenizers-0.13.3 tzdata-2023.3 uvicorn-0.23.1 uvloop-0.17.0 watchfiles-0.19.0 websockets-11.0.3\n    Note: you may need to restart the kernel to use updated packages.\n    Collecting wget\n      Using cached wget-3.2.zip (10 kB)\n      Preparing metadata (setup.py) ... \u001b[?25ldone\n    \u001b[?25hBuilding wheels for collected packages: wget\n      Building wheel for wget (setup.py) ... \u001b[?25ldone\n    \u001b[?25h  Created wheel for wget: filename=wget-3.2-py3-none-any.whl size=9657 sha256=b2d83c5fcdeab398d0a4e9808a470bbf725fffea4a6130e731c6097b9561005b\n      Stored in directory: /Users/antontroynikov/Library/Caches/pip/wheels/8b/f1/7f/5c94f0a7a505ca1c81cd1d9208ae2064675d97582078e6c769\n    Successfully built wget\n    Installing collected packages: wget\n    Successfully installed wget-3.2\n    Note: you may need to restart the kernel to use updated packages.\n    Requirement already satisfied: numpy in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (1.25.1)\n    Note: you may need to restart the kernel to use updated packages."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 78fef95d-4d0d-48b3-84c3-613a627b65b5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"78fef95d-4d0d-48b3-84c3-613a627b65b5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b7c035a2-45e1-4111-a572-58aa56b32d3a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"a694c829048ae6db02e60c5b4e5721efeef0ec63277083f57de3fb15e770e9f3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f4164929-d6ce-44b5-9cc4-b0e6005c6b2e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cc5552ba2aa38db470f007100d75c8ebf3eaacbb31b332e54b1079d48a6a9459\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19851, \"end_char_idx\": 22189, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "0.0 pytz-2023.3 pyyaml-6.0.1 sniffio-1.3.0 starlette-0.27.0 sympy-1.12 tokenizers-0.13.3 tzdata-2023.3 uvicorn-0.23.1 uvloop-0.17.0 watchfiles-0.19.0 websockets-11.0.3\n    Note: you may need to restart the kernel to use updated packages.\n    Collecting wget\n      Using cached wget-3.2.zip (10 kB)\n      Preparing metadata (setup.py) ... \u001b[?25ldone\n    \u001b[?25hBuilding wheels for collected packages: wget\n      Building wheel for wget (setup.py) ... \u001b[?25ldone\n    \u001b[?25h  Created wheel for wget: filename=wget-3.2-py3-none-any.whl size=9657 sha256=b2d83c5fcdeab398d0a4e9808a470bbf725fffea4a6130e731c6097b9561005b\n      Stored in directory: /Users/antontroynikov/Library/Caches/pip/wheels/8b/f1/7f/5c94f0a7a505ca1c81cd1d9208ae2064675d97582078e6c769\n    Successfully built wget\n    Installing collected packages: wget\n    Successfully installed wget-3.2\n    Note: you may need to restart the kernel to use updated packages.\n    Requirement already satisfied: numpy in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (1.25.1)\n    Note: you may need to restart the kernel to use updated packages.\n\n\n\n```python\nimport openai\nimport pandas as pd\nimport os\nimport wget\nfrom ast import literal_eval\n\n# Chroma's client library for Python\nimport chromadb\n\n# I've set this to our new embeddings model, this can be changed to the embedding model of your choice\nEMBEDDING_MODEL = \"text-embedding-3-small\"\n\n# Ignore unclosed SSL socket warnings - optional in case you get these errors\nimport warnings\n\nwarnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning) \n```\n\n## Load data\n\nIn this section we'll load embedded data that we've prepared previous to this session.\n\n\n```python\nembeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'\n\n# The file is ~700 MB so this will take some time\nwget.download(embeddings_url)\n```\n\n\n\n\n    'vector_database_wikipedia_articles_embedded.zip'\n\n\n\n\n```python\nimport zipfile\nwith zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n    zip_ref.extractall(\"../data\")\n```\n\n\n```python\narticle_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')\n```\n\n\n```python\narticle_df.head()\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: f4164929-d6ce-44b5-9cc4-b0e6005c6b2e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"f4164929-d6ce-44b5-9cc4-b0e6005c6b2e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"78fef95d-4d0d-48b3-84c3-613a627b65b5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"bf42f226b94a5c82b0ea8c4149292011f2eff6474192952a02215daa5b86db73\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b726e3c9-eb22-4b81-a44b-f6d00daa5d4b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"55d75d734c6c465942a4bbe99c9e2550ca1dde90d5af183d906f3fc23ae149c3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 20777, \"end_char_idx\": 23205, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "Requirement already satisfied: numpy in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (1.25.1)\n    Note: you may need to restart the kernel to use updated packages.\n\n\n\n```python\nimport openai\nimport pandas as pd\nimport os\nimport wget\nfrom ast import literal_eval\n\n# Chroma's client library for Python\nimport chromadb\n\n# I've set this to our new embeddings model, this can be changed to the embedding model of your choice\nEMBEDDING_MODEL = \"text-embedding-3-small\"\n\n# Ignore unclosed SSL socket warnings - optional in case you get these errors\nimport warnings\n\nwarnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning) \n```\n\n## Load data\n\nIn this section we'll load embedded data that we've prepared previous to this session.\n\n\n```python\nembeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'\n\n# The file is ~700 MB so this will take some time\nwget.download(embeddings_url)\n```\n\n\n\n\n    'vector_database_wikipedia_articles_embedded.zip'\n\n\n\n\n```python\nimport zipfile\nwith zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n    zip_ref.extractall(\"../data\")\n```\n\n\n```python\narticle_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')\n```\n\n\n```python\narticle_df.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>url</th>\n      <th>title</th>\n      <th>text</th>\n      <th>title_vector</th>\n      <th>content_vector</th>\n      <th>vector_id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>https://simple.wikipedia.org/wiki/April</td>\n      <td>April</td>\n      <td>April is the fourth month of the year in the J...</td>\n      <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n      <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>https://simple.wikipedia.org/wiki/August</td>\n      <td>August</td>\n      <td>August (Aug.)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: b726e3c9-eb22-4b81-a44b-f6d00daa5d4b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"b726e3c9-eb22-4b81-a44b-f6d00daa5d4b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f4164929-d6ce-44b5-9cc4-b0e6005c6b2e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"5ab0f70936dfab0b4238ff107eed102792d23e0750a4ec7fc444661a843d47b0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6420e2cc-8203-4f93-9d24-e0701e68c45b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0eb9f1fbcad7c87e0ae1cc80d55f22c888212d66f0901dbcd88725f848899c3c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 22058, \"end_char_idx\": 24236, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "```python\narticle_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')\n```\n\n\n```python\narticle_df.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>url</th>\n      <th>title</th>\n      <th>text</th>\n      <th>title_vector</th>\n      <th>content_vector</th>\n      <th>vector_id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>https://simple.wikipedia.org/wiki/April</td>\n      <td>April</td>\n      <td>April is the fourth month of the year in the J...</td>\n      <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n      <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>https://simple.wikipedia.org/wiki/August</td>\n      <td>August</td>\n      <td>August (Aug.) is the eighth month of the year ...</td>\n      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>6</td>\n      <td>https://simple.wikipedia.org/wiki/Art</td>\n      <td>Art</td>\n      <td>Art is a creative activity that expresses imag...</td>\n      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>8</td>\n      <td>https://simple.wikipedia.org/wiki/A</td>\n      <td>A</td>\n      <td>A or a is the first letter of the English alph...</td>\n      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n      <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>9</td>\n      <td>https://simple.wikipedia.org/wiki/Air</td>\n      <td>Air</td>\n      <td>Air refers to the Earth's atmosphere."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 6420e2cc-8203-4f93-9d24-e0701e68c45b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"6420e2cc-8203-4f93-9d24-e0701e68c45b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b726e3c9-eb22-4b81-a44b-f6d00daa5d4b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"5e7cf39b2939fc387f82f1b1edcd54492de9e9c10b9794bd1c595ccf56724885\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"aef7dd39-207a-42c9-8625-97dec779cb2f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e31dbdf83f0b175955f34de70464687d224d388059d362c5fb12d62947de6bde\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 23206, \"end_char_idx\": 25404, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "is the eighth month of the year ...</td>\n      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>6</td>\n      <td>https://simple.wikipedia.org/wiki/Art</td>\n      <td>Art</td>\n      <td>Art is a creative activity that expresses imag...</td>\n      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>8</td>\n      <td>https://simple.wikipedia.org/wiki/A</td>\n      <td>A</td>\n      <td>A or a is the first letter of the English alph...</td>\n      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n      <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>9</td>\n      <td>https://simple.wikipedia.org/wiki/Air</td>\n      <td>Air</td>\n      <td>Air refers to the Earth's atmosphere. Air is a...</td>\n      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n      <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n\n```python\n# Read vectors from strings back into a list\narticle_df['title_vector'] = article_df.title_vector.apply(literal_eval)\narticle_df['content_vector'] = article_df.content_vector.apply(literal_eval)\n\n# Set vector_id to be a string\narticle_df['vector_id'] = article_df['vector_id'].apply(str)\n```\n\n\n```python\narticle_df.info(show_counts=True)\n```\n\n    <class 'pandas.core.frame.DataFrame'>\n    RangeIndex: 25000 entries, 0 to 24999\n    Data columns (total 7 columns):\n     #   Column          Non-Null Count  Dtype \n    ---  ------          --------------  ----- \n     0   id              25000 non-null  int64 \n     1   url             25000 non-null  object\n     2   title           25000 non-null  object\n     3   text            25000 non-null  object\n     4   title_vector    25000 non-null  object\n     5   content_vector  25000 non-null  object\n     6   vector_id       25000 non-null  object\n    dtypes: int64(1), object(6)\n    memory usage: 1.3+ MB"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: aef7dd39-207a-42c9-8625-97dec779cb2f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"aef7dd39-207a-42c9-8625-97dec779cb2f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6420e2cc-8203-4f93-9d24-e0701e68c45b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"2ece711683c6310cd604958b8cd5be14dbba53c8c66e3636993a862eb40a39da\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"af718734-5c1f-444a-b8b0-52c0f41af662\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6915acaf8b4f99ccbaf24180f03eb4a16e0105699601e93c153d4568a5fff1dd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 24237, \"end_char_idx\": 27234, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "Air is a...</td>\n      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n      <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n\n```python\n# Read vectors from strings back into a list\narticle_df['title_vector'] = article_df.title_vector.apply(literal_eval)\narticle_df['content_vector'] = article_df.content_vector.apply(literal_eval)\n\n# Set vector_id to be a string\narticle_df['vector_id'] = article_df['vector_id'].apply(str)\n```\n\n\n```python\narticle_df.info(show_counts=True)\n```\n\n    <class 'pandas.core.frame.DataFrame'>\n    RangeIndex: 25000 entries, 0 to 24999\n    Data columns (total 7 columns):\n     #   Column          Non-Null Count  Dtype \n    ---  ------          --------------  ----- \n     0   id              25000 non-null  int64 \n     1   url             25000 non-null  object\n     2   title           25000 non-null  object\n     3   text            25000 non-null  object\n     4   title_vector    25000 non-null  object\n     5   content_vector  25000 non-null  object\n     6   vector_id       25000 non-null  object\n    dtypes: int64(1), object(6)\n    memory usage: 1.3+ MB\n\n\n# Chroma\n\nWe'll index these embedded documents in a vector database and search them. The first option we'll look at is **Chroma**, an easy to use open-source self-hosted in-memory vector database, designed for working with embeddings together with LLMs. \n\nIn this section, we will:\n- Instantiate the Chroma client\n- Create collections for each class of embedding \n- Query each collection \n\n### Instantiate the Chroma client\n\nCreate the Chroma client. By default, Chroma is ephemeral and runs in memory. \nHowever, you can easily set up a persistent configuration which writes to disk.\n\n\n```python\nchroma_client = chromadb.EphemeralClient() # Equivalent to chromadb.Client(), ephemeral.\n# Uncomment for persistent client\n# chroma_client = chromadb.PersistentClient()\n```\n\n### Create collections\n\nChroma collections allow you to store and filter with arbitrary metadata, making it easy to query subsets of the embedded data. \n\nChroma is already integrated with OpenAI's embedding functions. The best way to use them is on construction of a collection, as follows.\nAlternatively, you can 'bring your own embeddings'. More information can be found [here](https://docs.trychroma.com/embeddings)\n\n\n```python\nfrom chromadb.utils.embedding_functions import OpenAIEmbeddingFunction\n\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\n\n# Note. alternatively you can set a temporary env variable like this:\n# os.environ[\"OPENAI_API_KEY\"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    openai.api_key = os.getenv(\"OPENAI_API_KEY\")\n    print (\"OPENAI_API_KEY is ready\")\nelse:\n    print (\"OPENAI_API_KEY environment variable not found\")"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: af718734-5c1f-444a-b8b0-52c0f41af662
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"af718734-5c1f-444a-b8b0-52c0f41af662\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"aef7dd39-207a-42c9-8625-97dec779cb2f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"d856fba364552c3d638a374629a47bba05767b95bc0837ae152492f23757fee6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e514a9be-ec80-4132-835c-95515201e90a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4cab9f149edcde8976c9c43075473e4fd728f89cf4451a534e42faddc7ac53e0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 25407, \"end_char_idx\": 28948, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "# Chroma\n\nWe'll index these embedded documents in a vector database and search them. The first option we'll look at is **Chroma**, an easy to use open-source self-hosted in-memory vector database, designed for working with embeddings together with LLMs. \n\nIn this section, we will:\n- Instantiate the Chroma client\n- Create collections for each class of embedding \n- Query each collection \n\n### Instantiate the Chroma client\n\nCreate the Chroma client. By default, Chroma is ephemeral and runs in memory. \nHowever, you can easily set up a persistent configuration which writes to disk.\n\n\n```python\nchroma_client = chromadb.EphemeralClient() # Equivalent to chromadb.Client(), ephemeral.\n# Uncomment for persistent client\n# chroma_client = chromadb.PersistentClient()\n```\n\n### Create collections\n\nChroma collections allow you to store and filter with arbitrary metadata, making it easy to query subsets of the embedded data. \n\nChroma is already integrated with OpenAI's embedding functions. The best way to use them is on construction of a collection, as follows.\nAlternatively, you can 'bring your own embeddings'. More information can be found [here](https://docs.trychroma.com/embeddings)\n\n\n```python\nfrom chromadb.utils.embedding_functions import OpenAIEmbeddingFunction\n\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\n\n# Note. alternatively you can set a temporary env variable like this:\n# os.environ[\"OPENAI_API_KEY\"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    openai.api_key = os.getenv(\"OPENAI_API_KEY\")\n    print (\"OPENAI_API_KEY is ready\")\nelse:\n    print (\"OPENAI_API_KEY environment variable not found\")\n\n\nembedding_function = OpenAIEmbeddingFunction(api_key=os.environ.get('OPENAI_API_KEY'), model_name=EMBEDDING_MODEL)\n\nwikipedia_content_collection = chroma_client.create_collection(name='wikipedia_content', embedding_function=embedding_function)\nwikipedia_title_collection = chroma_client.create_collection(name='wikipedia_titles', embedding_function=embedding_function)\n```\n\n    OPENAI_API_KEY is ready\n\n\n### Populate the collections\n\nChroma collections allow you to populate, and filter on, whatever metadata you like. Chroma can also store the text alongside the vectors, and return everything in a single `query` call, when this is more convenient. \n\nFor this use-case, we'll just store the embeddings and IDs, and use these to index the original dataframe. \n\n\n```python\n# Add the content vectors\nwikipedia_content_collection.add(\n    ids=article_df.vector_id.tolist(),\n    embeddings=article_df.content_vector.tolist(),\n)\n\n# Add the title vectors\nwikipedia_title_collection.add(\n    ids=article_df.vector_id.tolist(),\n    embeddings=article_df.title_vector.tolist(),\n)\n```\n\n### Search the collections\n\nChroma handles embedding queries for you if an embedding function is set, like in this example.\n\n\n```python\ndef query_collection(collection, query, max_results, dataframe):\n    results = collection.query(query_texts=query, n_results=max_results, include=['distances']) \n    df = pd.DataFrame({\n                'id':results['ids'][0], \n                'score':results['distances'][0],\n                'title': dataframe[dataframe.vector_id.isin(results['ids'][0])]['title'],\n                'content': dataframe[dataframe.vector_id.isin(results['ids'][0])]['text'],\n                })\n    \n    return df\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: e514a9be-ec80-4132-835c-95515201e90a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e514a9be-ec80-4132-835c-95515201e90a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"af718734-5c1f-444a-b8b0-52c0f41af662\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"a5064641359a1e2fbf00e5df2f2ed51559f34313de1413b20581dc9b1f00528b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"186230b9-d662-477c-a848-e7afe7297cff\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3b60c49431a6444e1de36ffa84f1f5ac907c937da763d81595ffcda70ae57123\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 27237, \"end_char_idx\": 29153, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "embedding_function = OpenAIEmbeddingFunction(api_key=os.environ.get('OPENAI_API_KEY'), model_name=EMBEDDING_MODEL)\n\nwikipedia_content_collection = chroma_client.create_collection(name='wikipedia_content', embedding_function=embedding_function)\nwikipedia_title_collection = chroma_client.create_collection(name='wikipedia_titles', embedding_function=embedding_function)\n```\n\n    OPENAI_API_KEY is ready\n\n\n### Populate the collections\n\nChroma collections allow you to populate, and filter on, whatever metadata you like. Chroma can also store the text alongside the vectors, and return everything in a single `query` call, when this is more convenient. \n\nFor this use-case, we'll just store the embeddings and IDs, and use these to index the original dataframe. \n\n\n```python\n# Add the content vectors\nwikipedia_content_collection.add(\n    ids=article_df.vector_id.tolist(),\n    embeddings=article_df.content_vector.tolist(),\n)\n\n# Add the title vectors\nwikipedia_title_collection.add(\n    ids=article_df.vector_id.tolist(),\n    embeddings=article_df.title_vector.tolist(),\n)\n```\n\n### Search the collections\n\nChroma handles embedding queries for you if an embedding function is set, like in this example.\n\n\n```python\ndef query_collection(collection, query, max_results, dataframe):\n    results = collection.query(query_texts=query, n_results=max_results, include=['distances']) \n    df = pd.DataFrame({\n                'id':results['ids'][0], \n                'score':results['distances'][0],\n                'title': dataframe[dataframe.vector_id.isin(results['ids'][0])]['title'],\n                'content': dataframe[dataframe.vector_id.isin(results['ids'][0])]['text'],\n                })\n    \n    return df\n```\n\n\n```python\ntitle_query_result = query_collection(\n    collection=wikipedia_title_collection,\n    query=\"modern art in Europe\",\n    max_results=10,\n    dataframe=article_df\n)\ntitle_query_result.head()\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 186230b9-d662-477c-a848-e7afe7297cff
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"186230b9-d662-477c-a848-e7afe7297cff\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e514a9be-ec80-4132-835c-95515201e90a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"f707ac8cbb10a5cc5e3139209e55fd723b86a39e6894247a51d6b44bef087599\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1fae1473-358c-4a40-8221-d51d93698a8a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8261dc1b40eb4d98829a6e56f7c6e2ebddb0e2ea3c57980015bc72ad04bd8391\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 27641, \"end_char_idx\": 30512, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "### Populate the collections\n\nChroma collections allow you to populate, and filter on, whatever metadata you like. Chroma can also store the text alongside the vectors, and return everything in a single `query` call, when this is more convenient. \n\nFor this use-case, we'll just store the embeddings and IDs, and use these to index the original dataframe. \n\n\n```python\n# Add the content vectors\nwikipedia_content_collection.add(\n    ids=article_df.vector_id.tolist(),\n    embeddings=article_df.content_vector.tolist(),\n)\n\n# Add the title vectors\nwikipedia_title_collection.add(\n    ids=article_df.vector_id.tolist(),\n    embeddings=article_df.title_vector.tolist(),\n)\n```\n\n### Search the collections\n\nChroma handles embedding queries for you if an embedding function is set, like in this example.\n\n\n```python\ndef query_collection(collection, query, max_results, dataframe):\n    results = collection.query(query_texts=query, n_results=max_results, include=['distances']) \n    df = pd.DataFrame({\n                'id':results['ids'][0], \n                'score':results['distances'][0],\n                'title': dataframe[dataframe.vector_id.isin(results['ids'][0])]['title'],\n                'content': dataframe[dataframe.vector_id.isin(results['ids'][0])]['text'],\n                })\n    \n    return df\n```\n\n\n```python\ntitle_query_result = query_collection(\n    collection=wikipedia_title_collection,\n    query=\"modern art in Europe\",\n    max_results=10,\n    dataframe=article_df\n)\ntitle_query_result.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>score</th>\n      <th>title</th>\n      <th>content</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>2</th>\n      <td>23266</td>\n      <td>0.249646</td>\n      <td>Art</td>\n      <td>Art is a creative activity that expresses imag...</td>\n    </tr>\n    <tr>\n      <th>11777</th>\n      <td>15436</td>\n      <td>0.271688</td>\n      <td>Hellenistic art</td>\n      <td>The art of the Hellenistic time (from 400 B.C....</td>\n    </tr>\n    <tr>\n      <th>12178</th>\n      <td>23265</td>\n      <td>0.279306</td>\n      <td>Byzantine art</td>\n      <td>Byzantine art is a form of Christian Greek art...</td>\n    </tr>\n    <tr>\n      <th>13215</th>\n      <td>11777</td>\n      <td>0.294415</td>\n      <td>Art film</td>\n      <td>Art films are a type of movie that is very dif...</td>\n    </tr>\n    <tr>\n      <th>15436</th>\n      <td>22108</td>\n      <td>0.305937</td>\n      <td>Renaissance art</td>\n      <td>Many of the most famous and best-loved works o...</td>\n    </tr>\n  </tbody>\n</table>\n</div>"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 1fae1473-358c-4a40-8221-d51d93698a8a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"1fae1473-358c-4a40-8221-d51d93698a8a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7c54ae26-1c8c-487f-bf17-dd7249ab7b6c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"186230b9-d662-477c-a848-e7afe7297cff\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"15f1d4fcb09d1998d0efb4b49cbb3c4072d5dd45a01649a0304ada84c6a7d110\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 30517, \"end_char_idx\": 32346, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "ref_doc_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "chroma:document": "```python\ncontent_query_result = query_collection(\n    collection=wikipedia_content_collection,\n    query=\"Famous battles in Scottish history\",\n    max_results=10,\n    dataframe=article_df\n)\ncontent_query_result.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>score</th>\n      <th>title</th>\n      <th>content</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>2923</th>\n      <td>13135</td>\n      <td>0.261328</td>\n      <td>1651</td>\n      <td>\\n\\nEvents \\n January 1 \u2013 Charles II crowned K...</td>\n    </tr>\n    <tr>\n      <th>3694</th>\n      <td>13571</td>\n      <td>0.277058</td>\n      <td>Stirling</td>\n      <td>Stirling () is a city in the middle of Scotlan...</td>\n    </tr>\n    <tr>\n      <th>6248</th>\n      <td>2923</td>\n      <td>0.294823</td>\n      <td>841</td>\n      <td>\\n\\nEvents \\n June 25: Battle of Fontenay \u2013 Lo...</td>\n    </tr>\n    <tr>\n      <th>6297</th>\n      <td>13568</td>\n      <td>0.300756</td>\n      <td>1746</td>\n      <td>\\n\\nEvents \\n January 8 \u2013 Bonnie Prince Charli...</td>\n    </tr>\n    <tr>\n      <th>11702</th>\n      <td>11708</td>\n      <td>0.307572</td>\n      <td>William Wallace</td>\n      <td>William Wallace was a Scottish knight who foug...</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\nNow that you've got a basic embeddings search running, you can [hop over to the Chroma docs](https://docs.trychroma.com/usage-guide#using-where-filters) to learn more about how to add filters to your query, update/delete data in your collections, and deploy Chroma."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 0ec70657-4275-4e5e-af3d-2cbec0f22732
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0ec70657-4275-4e5e-af3d-2cbec0f22732\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fbfe1fcf-38c8-4c66-af50-49fe591c21b7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d052b78d-de76-46ee-ae30-cd00595572c2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2a683a846ba84360e06402d30d14238dd58a09a52d66cff465ff09bba1bf294b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3231, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "ref_doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "chroma:document": "# Robust Question Answering with Chroma and OpenAI \n\nThis notebook guides you step-by-step through answering questions about a collection of data, using [Chroma](https://trychroma.com), an open-source embeddings database, along with OpenAI's [text embeddings](https://platform.openai.com/docs/guides/embeddings/use-cases) and [chat completion](https://platform.openai.com/docs/guides/chat) API's. \n\nAdditionally, this notebook demonstrates some of the tradeoffs in making a question answering system more robust. As we shall see, *simple querying doesn't always create the best results*! \n\n## Question Answering with LLMs\n\nLarge language models (LLMs) like OpenAI's ChatGPT can be used to answer questions about data that the model may not have been trained on, or have access to. For example;\n\n- Personal data like e-mails and notes\n- Highly specialized data like archival or legal documents\n- Newly created data like recent news stories\n\nIn order to overcome this limitation, we can use a data store which is amenable to querying in natural language, just like the LLM itself. An embeddings store like Chroma represents documents as [embeddings](https://openai.com/blog/introducing-text-and-code-embeddings), alongside the documents themselves. \n\nBy embedding a text query, Chroma can find relevant documents, which we can then pass to the LLM to answer our question. We'll show detailed examples and variants of this approach. \n\n# Setup and preliminaries\n\nFirst we make sure the python dependencies we need are installed. \n\n\n```python\n%pip install -qU openai chromadb pandas\n```\n\n    Note: you may need to restart the kernel to use updated packages.\n\n\nWe use OpenAI's API's throughout this notebook. You can get an API key from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys)\n\nYou can add your API key as an environment variable by executing the command `export OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx` in a terminal. Note that you will need to reload the notebook if the environment variable wasn't set yet. Alternatively, you can set it in the notebook, see below. \n\n\n```python\nimport os\n\n# Uncomment the following line to set the environment variable in the notebook\n# os.environ[\"OPENAI_API_KEY\"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    print(\"OPENAI_API_KEY is ready\")\n    import openai\n    openai.api_key = os.getenv(\"OPENAI_API_KEY\")\nelse:\n    print(\"OPENAI_API_KEY environment variable not found\")\n```\n\n    OPENAI_API_KEY is ready\n\n\n# Dataset\n\nThroughout this notebook, we use the [SciFact dataset](https://github.com/allenai/scifact). This is a curated dataset of expert annotated scientific claims, with an accompanying text corpus of paper titles and abstracts. Each claim may be supported, contradicted, or not have enough evidence either way, according to the documents in the corpus. \n\nHaving the corpus available as ground-truth allows us to investigate how well the following approaches to LLM question answering perform. \n\n\n```python\n# Load the claim dataset\nimport pandas as pd\n\ndata_path = '../../data'\n\nclaim_df = pd.read_json(f'{data_path}/scifact_claims.jsonl', lines=True)\nclaim_df.head()\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: d052b78d-de76-46ee-ae30-cd00595572c2
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"d052b78d-de76-46ee-ae30-cd00595572c2\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fbfe1fcf-38c8-4c66-af50-49fe591c21b7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0ec70657-4275-4e5e-af3d-2cbec0f22732\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"e5dad5a280180581156d07349f86803495a87fe18d2cc48935707f2548b0f9ca\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f118969f-d730-4335-8618-5897da4e0a3c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"266c03c3164aa015d5aee257ec93c88a5e2583aa158dfe4fe1195110e1268966\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1528, \"end_char_idx\": 4626, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "ref_doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "chroma:document": "```python\n%pip install -qU openai chromadb pandas\n```\n\n    Note: you may need to restart the kernel to use updated packages.\n\n\nWe use OpenAI's API's throughout this notebook. You can get an API key from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys)\n\nYou can add your API key as an environment variable by executing the command `export OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx` in a terminal. Note that you will need to reload the notebook if the environment variable wasn't set yet. Alternatively, you can set it in the notebook, see below. \n\n\n```python\nimport os\n\n# Uncomment the following line to set the environment variable in the notebook\n# os.environ[\"OPENAI_API_KEY\"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    print(\"OPENAI_API_KEY is ready\")\n    import openai\n    openai.api_key = os.getenv(\"OPENAI_API_KEY\")\nelse:\n    print(\"OPENAI_API_KEY environment variable not found\")\n```\n\n    OPENAI_API_KEY is ready\n\n\n# Dataset\n\nThroughout this notebook, we use the [SciFact dataset](https://github.com/allenai/scifact). This is a curated dataset of expert annotated scientific claims, with an accompanying text corpus of paper titles and abstracts. Each claim may be supported, contradicted, or not have enough evidence either way, according to the documents in the corpus. \n\nHaving the corpus available as ground-truth allows us to investigate how well the following approaches to LLM question answering perform. \n\n\n```python\n# Load the claim dataset\nimport pandas as pd\n\ndata_path = '../../data'\n\nclaim_df = pd.read_json(f'{data_path}/scifact_claims.jsonl', lines=True)\nclaim_df.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>claim</th>\n      <th>evidence</th>\n      <th>cited_doc_ids</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>0-dimensional biomaterials show inductive prop...</td>\n      <td>{}</td>\n      <td>[31715818]</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>3</td>\n      <td>1,000 genomes project enables mapping of genet...</td>\n      <td>{'14717500': [{'sentences': [2, 5], 'label': '...</td>\n      <td>[14717500]</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>5</td>\n      <td>1/2000 in UK have abnormal PrP positivity.</td>\n      <td>{'13734012': [{'sentences': [4], 'label': 'SUP...</td>\n      <td>[13734012]</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>13</td>\n      <td>5% of perinatal mortality is due to low birth ...</td>\n      <td>{}</td>\n      <td>[1606628]</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>36</td>\n      <td>A deficiency of vitamin B12 increases blood le...</td>\n      <td>{}</td>\n      <td>[5152028, 11705328]</td>\n    </tr>\n  </tbody>\n</table>\n</div>"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: f118969f-d730-4335-8618-5897da4e0a3c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"f118969f-d730-4335-8618-5897da4e0a3c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fbfe1fcf-38c8-4c66-af50-49fe591c21b7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d052b78d-de76-46ee-ae30-cd00595572c2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"68c24ecd4ade173279c9351091e77c19fb71d14bb1aae504c4aac90a6a212f07\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a21fb14e-af3a-4fcc-9ecc-40fd6cc10fb6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"355fbc1e7a71331fb5247d3623d9b8fb0538274ed8dba5cc363dcba2fd1c0cbf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4630, \"end_char_idx\": 7216, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "ref_doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "chroma:document": "# Just asking the model\n\nGPT-3.5 was trained on a large amount of scientific information. As a baseline, we'd like to understand what the model already knows without any further context. This will allow us to calibrate overall performance. \n\nWe construct an appropriate prompt, with some example facts, then query the model with each claim in the dataset. We ask the model to assess a claim as 'True', 'False', or 'NEE' if there is not enough evidence one way or the other. \n\n\n```python\ndef build_prompt(claim):\n    return [\n        {\"role\": \"system\", \"content\": \"I will ask you to assess a scientific claim. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence.\"},\n        {\"role\": \"user\", \"content\": f\"\"\"        \nExample:\n\nClaim:\n0-dimensional biomaterials show inductive properties.\n\nAssessment:\nFalse\n\nClaim:\n1/2000 in UK have abnormal PrP positivity.\n\nAssessment:\nTrue\n\nClaim:\nAspirin inhibits the production of PGE2.\n\nAssessment:\nFalse\n\nEnd of examples. Assess the following claim:\n\nClaim:\n{claim}\n\nAssessment:\n\"\"\"}\n    ]\n\n\ndef assess_claims(claims):\n    responses = []\n    # Query the OpenAI API\n    for claim in claims:\n        response = openai.ChatCompletion.create(\n            model='gpt-3.5-turbo',\n            messages=build_prompt(claim),\n            max_tokens=3,\n        )\n        # Strip any punctuation or whitespace from the response\n        responses.append(response.choices[0].message.content.strip('., '))\n\n    return responses\n```\n\nWe sample 100 claims from the dataset\n\n\n```python\n# Let's take a look at 100 claims\nsamples = claim_df.sample(50)\n\nclaims = samples['claim'].tolist() \n\n```\n\nWe evaluate the ground-truth according to the dataset. From the dataset description, each claim is either supported or contradicted by the evidence, or else there isn't enough evidence either way. \n\n\n```python\ndef get_groundtruth(evidence):\n    groundtruth = []\n    for e in evidence:\n        # Evidence is empty \n        if len(e) == 0:\n            groundtruth.append('NEE')\n        else:\n            # In this dataset, all evidence for a given claim is consistent, either SUPPORT or CONTRADICT\n            if list(e.values())[0][0]['label'] == 'SUPPORT':\n                groundtruth.append('True')\n            else:\n                groundtruth.append('False')\n    return groundtruth\n```\n\n\n```python\nevidence = samples['evidence'].tolist()\ngroundtruth = get_groundtruth(evidence)\n```\n\nWe also output the confusion matrix, comparing the model's assessments with the ground truth, in an easy to read table."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: a21fb14e-af3a-4fcc-9ecc-40fd6cc10fb6
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a21fb14e-af3a-4fcc-9ecc-40fd6cc10fb6\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fbfe1fcf-38c8-4c66-af50-49fe591c21b7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f118969f-d730-4335-8618-5897da4e0a3c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"1353bc944e06942d6f92ed95dc0e4a2aa4d95ff79e82e6bcf6c69da7b0ff93fb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4f52e5c9-4f28-4552-9cdb-9becebc0e331\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2585e3d072f75550210dd85dba3e086b3331ed428f25dc567392fffc97703a33\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5728, \"end_char_idx\": 8574, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "ref_doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "chroma:document": "def assess_claims(claims):\n    responses = []\n    # Query the OpenAI API\n    for claim in claims:\n        response = openai.ChatCompletion.create(\n            model='gpt-3.5-turbo',\n            messages=build_prompt(claim),\n            max_tokens=3,\n        )\n        # Strip any punctuation or whitespace from the response\n        responses.append(response.choices[0].message.content.strip('., '))\n\n    return responses\n```\n\nWe sample 100 claims from the dataset\n\n\n```python\n# Let's take a look at 100 claims\nsamples = claim_df.sample(50)\n\nclaims = samples['claim'].tolist() \n\n```\n\nWe evaluate the ground-truth according to the dataset. From the dataset description, each claim is either supported or contradicted by the evidence, or else there isn't enough evidence either way. \n\n\n```python\ndef get_groundtruth(evidence):\n    groundtruth = []\n    for e in evidence:\n        # Evidence is empty \n        if len(e) == 0:\n            groundtruth.append('NEE')\n        else:\n            # In this dataset, all evidence for a given claim is consistent, either SUPPORT or CONTRADICT\n            if list(e.values())[0][0]['label'] == 'SUPPORT':\n                groundtruth.append('True')\n            else:\n                groundtruth.append('False')\n    return groundtruth\n```\n\n\n```python\nevidence = samples['evidence'].tolist()\ngroundtruth = get_groundtruth(evidence)\n```\n\nWe also output the confusion matrix, comparing the model's assessments with the ground truth, in an easy to read table. \n\n\n```python\ndef confusion_matrix(inferred, groundtruth):\n    assert len(inferred) == len(groundtruth)\n    confusion = {\n        'True': {'True': 0, 'False': 0, 'NEE': 0},\n        'False': {'True': 0, 'False': 0, 'NEE': 0},\n        'NEE': {'True': 0, 'False': 0, 'NEE': 0},\n    }\n    for i, g in zip(inferred, groundtruth):\n        confusion[i][g] += 1\n\n    # Pretty print the confusion matrix\n    print('\\tGroundtruth')\n    print('\\tTrue\\tFalse\\tNEE')\n    for i in confusion:\n        print(i, end='\\t')\n        for g in confusion[i]:\n            print(confusion[i][g], end='\\t')\n        print()\n\n    return confusion\n```\n\nWe ask the model to directly assess the claims, without additional context. \n\n\n```python\ngpt_inferred = assess_claims(claims)\nconfusion_matrix(gpt_inferred, groundtruth)\n```\n\n    \tGroundtruth\n    \tTrue\tFalse\tNEE\n    True\t15\t5\t14\t\n    False\t0\t2\t1\t\n    NEE\t3\t3\t7\t\n\n\n\n\n\n    {'True': {'True': 15, 'False': 5, 'NEE': 14},\n     'False': {'True': 0, 'False': 2, 'NEE': 1},\n     'NEE': {'True': 3, 'False': 3, 'NEE': 7}}\n\n\n\n## Results\n\nFrom these results we see that the LLM is strongly biased to assess claims as true, even when they are false, and also tends to assess false claims as not having enough evidence. Note that 'not enough evidence' is with respect to the model's assessment of the claim in a vacuum, without additional context."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 4f52e5c9-4f28-4552-9cdb-9becebc0e331
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"4f52e5c9-4f28-4552-9cdb-9becebc0e331\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fbfe1fcf-38c8-4c66-af50-49fe591c21b7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a21fb14e-af3a-4fcc-9ecc-40fd6cc10fb6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"65855cb9c8543815167a0c2902bf9fc5a5f8d6bf925dfee8644a570afe3d979b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d8da2860-024b-46fd-a3ec-f02d9593ff75\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dbcd26f0518ab002681e4ec6edc0956a9caf6d5ea8419af30b57c73648576cf6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7919, \"end_char_idx\": 8951, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "ref_doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "chroma:document": "```python\ngpt_inferred = assess_claims(claims)\nconfusion_matrix(gpt_inferred, groundtruth)\n```\n\n    \tGroundtruth\n    \tTrue\tFalse\tNEE\n    True\t15\t5\t14\t\n    False\t0\t2\t1\t\n    NEE\t3\t3\t7\t\n\n\n\n\n\n    {'True': {'True': 15, 'False': 5, 'NEE': 14},\n     'False': {'True': 0, 'False': 2, 'NEE': 1},\n     'NEE': {'True': 3, 'False': 3, 'NEE': 7}}\n\n\n\n## Results\n\nFrom these results we see that the LLM is strongly biased to assess claims as true, even when they are false, and also tends to assess false claims as not having enough evidence. Note that 'not enough evidence' is with respect to the model's assessment of the claim in a vacuum, without additional context.\n\n\n# Adding context \n\nWe now add the additional context available from the corpus of paper titles and abstracts. This section shows how to load a text corpus into Chroma, using OpenAI text embeddings. \n\nFirst, we load the text corpus. \n\n\n```python\n# Load the corpus into a dataframe\ncorpus_df = pd.read_json(f'{data_path}/scifact_corpus.jsonl', lines=True)\ncorpus_df.head()\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: d8da2860-024b-46fd-a3ec-f02d9593ff75
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"d8da2860-024b-46fd-a3ec-f02d9593ff75\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fbfe1fcf-38c8-4c66-af50-49fe591c21b7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4f52e5c9-4f28-4552-9cdb-9becebc0e331\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"e2705f8b9280fe72f564b5247c116abc6f1866ebcd2e6ea120aec0d1c98e5e3a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ee3df3ca-4837-4112-8d96-59f341717590\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"76197a26079d0781a448b68ca74f4b47cd67db1631ef0f0f6dc0a516ac4e0d45\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7919, \"end_char_idx\": 10477, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "ref_doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "chroma:document": "```python\ngpt_inferred = assess_claims(claims)\nconfusion_matrix(gpt_inferred, groundtruth)\n```\n\n    \tGroundtruth\n    \tTrue\tFalse\tNEE\n    True\t15\t5\t14\t\n    False\t0\t2\t1\t\n    NEE\t3\t3\t7\t\n\n\n\n\n\n    {'True': {'True': 15, 'False': 5, 'NEE': 14},\n     'False': {'True': 0, 'False': 2, 'NEE': 1},\n     'NEE': {'True': 3, 'False': 3, 'NEE': 7}}\n\n\n\n## Results\n\nFrom these results we see that the LLM is strongly biased to assess claims as true, even when they are false, and also tends to assess false claims as not having enough evidence. Note that 'not enough evidence' is with respect to the model's assessment of the claim in a vacuum, without additional context.\n\n\n# Adding context \n\nWe now add the additional context available from the corpus of paper titles and abstracts. This section shows how to load a text corpus into Chroma, using OpenAI text embeddings. \n\nFirst, we load the text corpus. \n\n\n```python\n# Load the corpus into a dataframe\ncorpus_df = pd.read_json(f'{data_path}/scifact_corpus.jsonl', lines=True)\ncorpus_df.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>doc_id</th>\n      <th>title</th>\n      <th>abstract</th>\n      <th>structured</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>4983</td>\n      <td>Microstructural development of human newborn c...</td>\n      <td>[Alterations of the architecture of cerebral w...</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>5836</td>\n      <td>Induction of myelodysplasia by myeloid-derived...</td>\n      <td>[Myelodysplastic syndromes (MDS) are age-depen...</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>7912</td>\n      <td>BC1 RNA, the transcript from a master gene for...</td>\n      <td>[ID elements are short interspersed elements (...</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>18670</td>\n      <td>The DNA Methylome of Human Peripheral Blood Mo...</td>\n      <td>[DNA methylation plays an important role in bi...</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>19238</td>\n      <td>The human myelin basic protein gene is include...</td>\n      <td>[Two human Golli (for gene expressed in the ol...</td>\n      <td>False</td>\n    </tr>\n  </tbody>\n</table>\n</div>"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: ee3df3ca-4837-4112-8d96-59f341717590
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"ee3df3ca-4837-4112-8d96-59f341717590\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fbfe1fcf-38c8-4c66-af50-49fe591c21b7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d8da2860-024b-46fd-a3ec-f02d9593ff75\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"70b42dac93d846ba1ddf24ce895f1bb40d71e03fc8f46791f42dfca237e69716\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"84a65079-cd27-4ed9-88af-b687742dbfdd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4141e1ba657640c16d1e19319fd2889088058dddfcd42bf528197d0f5098e009\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10481, \"end_char_idx\": 13640, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "ref_doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "chroma:document": "## Loading the corpus into Chroma\n\nThe next step is to load the corpus into Chroma. Given an embedding function, Chroma will automatically handle embedding each document, and will store it alongside its text and metadata, making it simple to query.\n\nWe instantiate a (ephemeral) Chroma client, and create a collection for the SciFact title and abstract corpus. \nChroma can also be instantiated in a persisted configuration; learn more at the [Chroma docs](https://docs.trychroma.com/usage-guide?lang=py). \n\n\n```python\nimport chromadb\nfrom chromadb.utils.embedding_functions import OpenAIEmbeddingFunction\n\n# We initialize an embedding function, and provide it to the collection.\nembedding_function = OpenAIEmbeddingFunction(api_key=os.getenv(\"OPENAI_API_KEY\"))\n\nchroma_client = chromadb.Client() # Ephemeral by default\nscifact_corpus_collection = chroma_client.create_collection(name='scifact_corpus', embedding_function=embedding_function)\n```\n\n    Running Chroma using direct local API.\n    Using DuckDB in-memory for database. Data will be transient.\n\n\nNext we load the corpus into Chroma. Because this data loading is memory intensive, we recommend using a batched loading scheme in batches of 50-1000. For this example it should take just over one minute for the entire corpus. It's being embedded in the background, automatically, using the `embedding_function` we specified earlier.\n\n\n```python\nbatch_size = 100\n\nfor i in range(0, len(corpus_df), batch_size):\n    batch_df = corpus_df[i:i+batch_size]\n    scifact_corpus_collection.add(\n        ids=batch_df['doc_id'].apply(lambda x: str(x)).tolist(), # Chroma takes string IDs.\n        documents=(batch_df['title'] + '. ' + batch_df['abstract'].apply(lambda x: ' '.join(x))).to_list(), # We concatenate the title and abstract.\n        metadatas=[{\"structured\": structured} for structured in batch_df['structured'].to_list()] # We also store the metadata, though we don't use it in this example.\n    )\n```\n\n## Retrieving context\n\nNext we retrieve documents from the corpus which may be relevant to each claim in our sample. We want to provide these as context to the LLM for evaluating the claims. We retrieve the 3 most relevant documents for each claim, according to the embedding distance. \n\n\n```python\nclaim_query_result = scifact_corpus_collection.query(query_texts=claims, include=['documents', 'distances'], n_results=3)\n```\n\nWe create a new prompt, this time taking into account the additional context we retrieve from the corpus. \n\n\n```python\ndef build_prompt_with_context(claim, context):\n    return [{'role': 'system', 'content': \"I will ask you to assess whether a particular scientific claim, based on evidence provided. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence.\"}, \n            {'role': 'user', 'content': f\"\"\"\"\nThe evidence is the following:\n\n{' '.join(context)}\n\nAssess the following claim on the basis of the evidence. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence. Do not output any other text. \n\nClaim:\n{claim}\n\nAssessment:\n\"\"\"}]"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 84a65079-cd27-4ed9-88af-b687742dbfdd
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"84a65079-cd27-4ed9-88af-b687742dbfdd\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fbfe1fcf-38c8-4c66-af50-49fe591c21b7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ee3df3ca-4837-4112-8d96-59f341717590\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"4c5a5cca7fac41d07e1d3db95a8568244b9a239bf3b430fd4ebb19a8dfa5a1cc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"96666921-c260-4430-99c3-9e0485855e52\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0628ac2ec49633ccfa9cdc83edaaf9ad092c43126c21199ecdd621dbcda51b1d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12733, \"end_char_idx\": 15739, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "ref_doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "chroma:document": "```python\nclaim_query_result = scifact_corpus_collection.query(query_texts=claims, include=['documents', 'distances'], n_results=3)\n```\n\nWe create a new prompt, this time taking into account the additional context we retrieve from the corpus. \n\n\n```python\ndef build_prompt_with_context(claim, context):\n    return [{'role': 'system', 'content': \"I will ask you to assess whether a particular scientific claim, based on evidence provided. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence.\"}, \n            {'role': 'user', 'content': f\"\"\"\"\nThe evidence is the following:\n\n{' '.join(context)}\n\nAssess the following claim on the basis of the evidence. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence. Do not output any other text. \n\nClaim:\n{claim}\n\nAssessment:\n\"\"\"}]\n\n\ndef assess_claims_with_context(claims, contexts):\n    responses = []\n    # Query the OpenAI API\n    for claim, context in zip(claims, contexts):\n        # If no evidence is provided, return NEE\n        if len(context) == 0:\n            responses.append('NEE')\n            continue\n        response = openai.ChatCompletion.create(\n            model='gpt-3.5-turbo',\n            messages=build_prompt_with_context(claim=claim, context=context),\n            max_tokens=3,\n        )\n        # Strip any punctuation or whitespace from the response\n        responses.append(response.choices[0].message.content.strip('., '))\n\n    return responses\n```\n\nThen ask the model to evaluate the claims with the retrieved context. \n\n\n```python\ngpt_with_context_evaluation = assess_claims_with_context(claims, claim_query_result['documents'])\nconfusion_matrix(gpt_with_context_evaluation, groundtruth)\n```\n\n    \tGroundtruth\n    \tTrue\tFalse\tNEE\n    True\t16\t2\t8\t\n    False\t1\t6\t5\t\n    NEE\t1\t2\t9\t\n\n\n\n\n\n    {'True': {'True': 16, 'False': 2, 'NEE': 8},\n     'False': {'True': 1, 'False': 6, 'NEE': 5},\n     'NEE': {'True': 1, 'False': 2, 'NEE': 9}}\n\n\n\n## Results\n\nWe see that the model is a lot less likely to evaluate a False claim as true (2 instances VS 5 previously), but that claims without enough evidence are still often assessed as True or False.\n\nTaking a look at the retrieved documents, we see that they are sometimes not relevant to the claim - this causes the model to be confused by the extra information, and it may decide that sufficient evidence is present, even when the information is irrelevant. This happens because we always ask for the 3 'most' relevant documents, but these might not be relevant at all beyond a certain point.  \n\n## Filtering context on relevance\n\nAlong with the documents themselves, Chroma returns a distance score. We can try thresholding on distance, so that fewer irrelevant documents make it into the context we provide the model. \n\nIf, after filtering on the threshold, no context documents remain, we bypass the model and simply return that there is not enough evidence."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 96666921-c260-4430-99c3-9e0485855e52
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"96666921-c260-4430-99c3-9e0485855e52\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fbfe1fcf-38c8-4c66-af50-49fe591c21b7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"84a65079-cd27-4ed9-88af-b687742dbfdd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a2034b8a20f8446978940ff03e81095fa0b76e455ed1e0671ad3cbf96ccbc2f1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fa59e943-f7dc-4b8a-9271-1af2e38a2e4f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0debb42270b184f1c2c79545a873540128782d62b2012be14f939979e8a2e7a5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14361, \"end_char_idx\": 16808, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "ref_doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "chroma:document": "```python\ngpt_with_context_evaluation = assess_claims_with_context(claims, claim_query_result['documents'])\nconfusion_matrix(gpt_with_context_evaluation, groundtruth)\n```\n\n    \tGroundtruth\n    \tTrue\tFalse\tNEE\n    True\t16\t2\t8\t\n    False\t1\t6\t5\t\n    NEE\t1\t2\t9\t\n\n\n\n\n\n    {'True': {'True': 16, 'False': 2, 'NEE': 8},\n     'False': {'True': 1, 'False': 6, 'NEE': 5},\n     'NEE': {'True': 1, 'False': 2, 'NEE': 9}}\n\n\n\n## Results\n\nWe see that the model is a lot less likely to evaluate a False claim as true (2 instances VS 5 previously), but that claims without enough evidence are still often assessed as True or False.\n\nTaking a look at the retrieved documents, we see that they are sometimes not relevant to the claim - this causes the model to be confused by the extra information, and it may decide that sufficient evidence is present, even when the information is irrelevant. This happens because we always ask for the 3 'most' relevant documents, but these might not be relevant at all beyond a certain point.  \n\n## Filtering context on relevance\n\nAlong with the documents themselves, Chroma returns a distance score. We can try thresholding on distance, so that fewer irrelevant documents make it into the context we provide the model. \n\nIf, after filtering on the threshold, no context documents remain, we bypass the model and simply return that there is not enough evidence. \n\n\n```python\ndef filter_query_result(query_result, distance_threshold=0.25):\n# For each query result, retain only the documents whose distance is below the threshold\n    for ids, docs, distances in zip(query_result['ids'], query_result['documents'], query_result['distances']):\n        for i in range(len(ids)-1, -1, -1):\n            if distances[i] > distance_threshold:\n                ids.pop(i)\n                docs.pop(i)\n                distances.pop(i)\n    return query_result\n\n```\n\n\n```python\nfiltered_claim_query_result = filter_query_result(claim_query_result)\n```\n\nNow we assess the claims using this cleaner context. \n\n\n```python\ngpt_with_filtered_context_evaluation = assess_claims_with_context(claims, filtered_claim_query_result['documents'])\nconfusion_matrix(gpt_with_filtered_context_evaluation, groundtruth)\n```\n\n    \tGroundtruth\n    \tTrue\tFalse\tNEE\n    True\t10\t2\t1\t\n    False\t0\t2\t1\t\n    NEE\t8\t6\t20\t\n\n\n\n\n\n    {'True': {'True': 10, 'False': 2, 'NEE': 1},\n     'False': {'True': 0, 'False': 2, 'NEE': 1},\n     'NEE': {'True': 8, 'False': 6, 'NEE': 20}}"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: fa59e943-f7dc-4b8a-9271-1af2e38a2e4f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"fa59e943-f7dc-4b8a-9271-1af2e38a2e4f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fbfe1fcf-38c8-4c66-af50-49fe591c21b7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"96666921-c260-4430-99c3-9e0485855e52\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"4cf8d2f39e5c8c2728a6728861bd1206e9ac12b6b3652c3e533cad19a75c30c1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"be9edb9d-8e8a-4bed-85a3-3594eee5bdc6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"725e8f3d5074fb077f7affdec9d8e9d486e4728c381e176262d779f22aaa5afd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15743, \"end_char_idx\": 18984, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "ref_doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "chroma:document": "```python\ndef filter_query_result(query_result, distance_threshold=0.25):\n# For each query result, retain only the documents whose distance is below the threshold\n    for ids, docs, distances in zip(query_result['ids'], query_result['documents'], query_result['distances']):\n        for i in range(len(ids)-1, -1, -1):\n            if distances[i] > distance_threshold:\n                ids.pop(i)\n                docs.pop(i)\n                distances.pop(i)\n    return query_result\n\n```\n\n\n```python\nfiltered_claim_query_result = filter_query_result(claim_query_result)\n```\n\nNow we assess the claims using this cleaner context. \n\n\n```python\ngpt_with_filtered_context_evaluation = assess_claims_with_context(claims, filtered_claim_query_result['documents'])\nconfusion_matrix(gpt_with_filtered_context_evaluation, groundtruth)\n```\n\n    \tGroundtruth\n    \tTrue\tFalse\tNEE\n    True\t10\t2\t1\t\n    False\t0\t2\t1\t\n    NEE\t8\t6\t20\t\n\n\n\n\n\n    {'True': {'True': 10, 'False': 2, 'NEE': 1},\n     'False': {'True': 0, 'False': 2, 'NEE': 1},\n     'NEE': {'True': 8, 'False': 6, 'NEE': 20}}\n\n\n\n## Results\n\nThe model now assesses many fewer claims as True or False when there is not enough evidence present. However, it now biases away from certainty. Most claims are now assessed as having not enough evidence, because a large fraction of them are filtered out by the distance threshold. It's possible to tune the distance threshold to find the optimal operating point, but this can be difficult, and is dataset and embedding model dependent. \n\n# Hypothetical Document Embeddings: Using hallucinations productively\n\nWe want to be able to retrieve relevant documents, without retrieving less relevant ones which might confuse the model. One way to accomplish this is to improve the retrieval query. \n\nUntil now, we have queried the dataset using _claims_ which are single sentence statements, while the corpus contains _abstracts_ describing a scientific paper. Intuitively, while these might be related, there are significant differences in their structure and meaning. These differences are encoded by the embedding model, and so influence the distances between the query and the most relevant results. \n\nWe can overcome this by leveraging the power of LLMs to generate relevant text. While the facts might be hallucinated, the content and structure of the documents the models generate is more similar to the documents in our corpus, than the queries are. This could lead to better queries and hence better results. \n\nThis approach is called [Hypothetical Document Embeddings (HyDE)](https://arxiv.org/abs/2212.10496), and has been shown to be quite good at the retrieval task. It should help us bring more relevant information into the context, without polluting it.  \n\nTL;DR:\n- you get much better matches when you embed whole abstracts rather than single sentences\n- but claims are usually single sentences\n- So HyDE shows that using GPT3 to expand claims into hallucinated abstracts and then searching based on those abstracts works (claims -> abstracts -> results) better than searching directly (claims -> results)\n\nFirst, we use in-context examples to prompt the model to generate documents similar to what's in the corpus, for each claim we want to assess."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: be9edb9d-8e8a-4bed-85a3-3594eee5bdc6
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"be9edb9d-8e8a-4bed-85a3-3594eee5bdc6\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fbfe1fcf-38c8-4c66-af50-49fe591c21b7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fa59e943-f7dc-4b8a-9271-1af2e38a2e4f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"16d3c426bc313b207cfe8f5e8df5dc47b41e085a89aa8ae9c8e5e4e3edc801e9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dd6cb8da-0cfb-4656-9008-ab517c7c8712\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3ce52a35ffd3ed9ac882558ad576d3512cb63f01ed926b3e9164054a92403d5f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18988, \"end_char_idx\": 22163, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "ref_doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "chroma:document": "```python\ndef build_hallucination_prompt(claim):\n    return [{'role': 'system', 'content': \"\"\"I will ask you to write an abstract for a scientific paper which supports or refutes a given claim. It should be written in scientific language, include a title. Output only one abstract, then stop.\n    \n    An Example:\n\n    Claim:\n    A high microerythrocyte count raises vulnerability to severe anemia in homozygous alpha (+)- thalassemia trait subjects.\n\n    Abstract:\n    BACKGROUND The heritable haemoglobinopathy alpha(+)-thalassaemia is caused by the reduced synthesis of alpha-globin chains that form part of normal adult haemoglobin (Hb). Individuals homozygous for alpha(+)-thalassaemia have microcytosis and an increased erythrocyte count. Alpha(+)-thalassaemia homozygosity confers considerable protection against severe malaria, including severe malarial anaemia (SMA) (Hb concentration < 50 g/l), but does not influence parasite count. We tested the hypothesis that the erythrocyte indices associated with alpha(+)-thalassaemia homozygosity provide a haematological benefit during acute malaria.   \n    METHODS AND FINDINGS Data from children living on the north coast of Papua New Guinea who had participated in a case-control study of the protection afforded by alpha(+)-thalassaemia against severe malaria were reanalysed to assess the genotype-specific reduction in erythrocyte count and Hb levels associated with acute malarial disease. We observed a reduction in median erythrocyte count of approximately 1.5 x 10(12)/l in all children with acute falciparum malaria relative to values in community children (p < 0.001). We developed a simple mathematical model of the linear relationship between Hb concentration and erythrocyte count. This model predicted that children homozygous for alpha(+)-thalassaemia lose less Hb than children of normal genotype for a reduction in erythrocyte count of >1.1 x 10(12)/l as a result of the reduced mean cell Hb in homozygous alpha(+)-thalassaemia. In addition, children homozygous for alpha(+)-thalassaemia require a 10% greater reduction in erythrocyte count than children of normal genotype (p = 0.02) for Hb concentration to fall to 50 g/l, the cutoff for SMA. We estimated that the haematological profile in children homozygous for alpha(+)-thalassaemia reduces the risk of SMA during acute malaria compared to children of normal genotype (relative risk 0.52; 95% confidence interval [CI] 0.24-1.12, p = 0.09).   \n    CONCLUSIONS The increased erythrocyte count and microcytosis in children homozygous for alpha(+)-thalassaemia may contribute substantially to their protection against SMA. A lower concentration of Hb per erythrocyte and a larger population of erythrocytes may be a biologically advantageous strategy against the significant reduction in erythrocyte count that occurs during acute infection with the malaria parasite Plasmodium falciparum. This haematological profile may reduce the risk of anaemia by other Plasmodium species, as well as other causes of anaemia. Other host polymorphisms that induce an increased erythrocyte count and microcytosis may confer a similar advantage.\n\n    End of example."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: dd6cb8da-0cfb-4656-9008-ab517c7c8712
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"dd6cb8da-0cfb-4656-9008-ab517c7c8712\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fbfe1fcf-38c8-4c66-af50-49fe591c21b7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"be9edb9d-8e8a-4bed-85a3-3594eee5bdc6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c725155626775468ccff264bfb16d4b7caaee7ebebe6cc2f91f148bf87509b1d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"16beaa96-bd45-4f5c-93a9-471daf9703b9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"89ee72a4c1a935544a9770be131bcf7707e09d9833f115e340c7d1adc505a3f5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 20622, \"end_char_idx\": 23668, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "ref_doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "chroma:document": "We developed a simple mathematical model of the linear relationship between Hb concentration and erythrocyte count. This model predicted that children homozygous for alpha(+)-thalassaemia lose less Hb than children of normal genotype for a reduction in erythrocyte count of >1.1 x 10(12)/l as a result of the reduced mean cell Hb in homozygous alpha(+)-thalassaemia. In addition, children homozygous for alpha(+)-thalassaemia require a 10% greater reduction in erythrocyte count than children of normal genotype (p = 0.02) for Hb concentration to fall to 50 g/l, the cutoff for SMA. We estimated that the haematological profile in children homozygous for alpha(+)-thalassaemia reduces the risk of SMA during acute malaria compared to children of normal genotype (relative risk 0.52; 95% confidence interval [CI] 0.24-1.12, p = 0.09).   \n    CONCLUSIONS The increased erythrocyte count and microcytosis in children homozygous for alpha(+)-thalassaemia may contribute substantially to their protection against SMA. A lower concentration of Hb per erythrocyte and a larger population of erythrocytes may be a biologically advantageous strategy against the significant reduction in erythrocyte count that occurs during acute infection with the malaria parasite Plasmodium falciparum. This haematological profile may reduce the risk of anaemia by other Plasmodium species, as well as other causes of anaemia. Other host polymorphisms that induce an increased erythrocyte count and microcytosis may confer a similar advantage.\n\n    End of example. \n    \n    \"\"\"}, {'role': 'user', 'content': f\"\"\"\"\n    Perform the task for the following claim.\n\n    Claim:\n    {claim}\n\n    Abstract:\n    \"\"\"}]\n\n\ndef hallucinate_evidence(claims):\n    # Query the OpenAI API\n    responses = []\n    # Query the OpenAI API\n    for claim in claims:\n        response = openai.ChatCompletion.create(\n            model='gpt-3.5-turbo',\n            messages=build_hallucination_prompt(claim),\n        )\n        responses.append(response.choices[0].message.content)\n    return responses\n```\n\nWe hallucinate a document for each claim.\n\n*NB: This can take a while, about 30m for 100 claims*. You can reduce the number of claims we want to assess to get results more quickly. \n\n\n```python\nhallucinated_evidence = hallucinate_evidence(claims)\n```\n\nWe use the hallucinated documents as queries into the corpus, and filter the results using the same distance threshold. \n\n\n```python\nhallucinated_query_result = scifact_corpus_collection.query(query_texts=hallucinated_evidence, include=['documents', 'distances'], n_results=3)\nfiltered_hallucinated_query_result = filter_query_result(hallucinated_query_result)\n```\n\nWe then ask the model to assess the claims, using the new context. \n\n\n```python\ngpt_with_hallucinated_context_evaluation = assess_claims_with_context(claims, filtered_hallucinated_query_result['documents'])\nconfusion_matrix(gpt_with_hallucinated_context_evaluation, groundtruth)\n```\n\n    \tGroundtruth\n    \tTrue\tFalse\tNEE\n    True\t15\t2\t5\t\n    False\t1\t5\t4\t\n    NEE\t2\t3\t13"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 16beaa96-bd45-4f5c-93a9-471daf9703b9
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"16beaa96-bd45-4f5c-93a9-471daf9703b9\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fbfe1fcf-38c8-4c66-af50-49fe591c21b7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dd6cb8da-0cfb-4656-9008-ab517c7c8712\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"b2a33432a960093a8a6bfd74a74c1ea6ffb5ad60bb14553270d5bc28d560f2ea\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 22148, \"end_char_idx\": 24502, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "ref_doc_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "chroma:document": "End of example. \n    \n    \"\"\"}, {'role': 'user', 'content': f\"\"\"\"\n    Perform the task for the following claim.\n\n    Claim:\n    {claim}\n\n    Abstract:\n    \"\"\"}]\n\n\ndef hallucinate_evidence(claims):\n    # Query the OpenAI API\n    responses = []\n    # Query the OpenAI API\n    for claim in claims:\n        response = openai.ChatCompletion.create(\n            model='gpt-3.5-turbo',\n            messages=build_hallucination_prompt(claim),\n        )\n        responses.append(response.choices[0].message.content)\n    return responses\n```\n\nWe hallucinate a document for each claim.\n\n*NB: This can take a while, about 30m for 100 claims*. You can reduce the number of claims we want to assess to get results more quickly. \n\n\n```python\nhallucinated_evidence = hallucinate_evidence(claims)\n```\n\nWe use the hallucinated documents as queries into the corpus, and filter the results using the same distance threshold. \n\n\n```python\nhallucinated_query_result = scifact_corpus_collection.query(query_texts=hallucinated_evidence, include=['documents', 'distances'], n_results=3)\nfiltered_hallucinated_query_result = filter_query_result(hallucinated_query_result)\n```\n\nWe then ask the model to assess the claims, using the new context. \n\n\n```python\ngpt_with_hallucinated_context_evaluation = assess_claims_with_context(claims, filtered_hallucinated_query_result['documents'])\nconfusion_matrix(gpt_with_hallucinated_context_evaluation, groundtruth)\n```\n\n    \tGroundtruth\n    \tTrue\tFalse\tNEE\n    True\t15\t2\t5\t\n    False\t1\t5\t4\t\n    NEE\t2\t3\t13\t\n\n\n\n\n\n    {'True': {'True': 15, 'False': 2, 'NEE': 5},\n     'False': {'True': 1, 'False': 5, 'NEE': 4},\n     'NEE': {'True': 2, 'False': 3, 'NEE': 13}}\n\n\n\n## Results\n\nCombining HyDE with a simple distance threshold leads to a significant improvement. The model no longer biases assessing claims as True, nor toward their not being enough evidence. It also correctly assesses when there isn't enough evidence more often.\n\n# Conclusion\n\nEquipping LLMs with a context based on a corpus of documents is a powerful technique for bringing the general reasoning and natural language interactions of LLMs to your own data. However, it's important to know that naive query and retrieval may not produce the best possible results! Ultimately understanding the data will help get the most out of the retrieval based question-answering approach."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 5da032a0-baaa-4a90-ae69-00f333102cda
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"5da032a0-baaa-4a90-ae69-00f333102cda\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"44faa2e7-58cb-4f5b-852e-4f211cd17acd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f95f57ee-b1d7-4445-800f-5521d01819f0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"088acc09ac6d00e2f69d60f7eaec540a731a22f958b8c7d25f609591117eb096\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3626, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "ref_doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "chroma:document": "# Using Redis for Embeddings Search\n\nThis notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.\n\n### What is a Vector Database\n\nA vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.\n\n### Why use a Vector Database\n\nVector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.\n\n\n### Demo Flow\nThe demo flow is:\n- **Setup**: Import packages and set any required variables\n- **Load data**: Load a dataset and embed it using OpenAI embeddings\n- **Redis**\n    - *Setup*: Set up the Redis-Py client. For more details go [here](https://github.com/redis/redis-py)\n    - *Index Data*: Create the search index for vector search and hybrid search (vector + full-text search) on all available fields.\n    - *Search Data*: Run a few example queries with various goals in mind.\n\nOnce you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.\n\n## Setup\n\nImport the required libraries and set the embedding model that we'd like to use.\n\n\n```python\n# We'll need to install the Redis client\n!pip install redis\n\n#Install wget to pull zip file\n!pip install wget\n```\n\n\n```python\nimport openai\n\nfrom typing import List, Iterator\nimport pandas as pd\nimport numpy as np\nimport os\nimport wget\nfrom ast import literal_eval\n\n# Redis client library for Python\nimport redis\n\n# I've set this to our new embeddings model, this can be changed to the embedding model of your choice\nEMBEDDING_MODEL = \"text-embedding-3-small\"\n\n# Ignore unclosed SSL socket warnings - optional in case you get these errors\nimport warnings\n\nwarnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning) \n```\n\n## Load data\n\nIn this section we'll load embedded data that we've prepared previous to this session.\n\n\n```python\nembeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'\n\n# The file is ~700 MB so this will take some time\nwget.download(embeddings_url)\n```\n\n\n```python\nimport zipfile\nwith zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n    zip_ref.extractall(\"../data\")\n```\n\n\n```python\narticle_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')\n```\n\n\n```python\narticle_df.head()\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: f95f57ee-b1d7-4445-800f-5521d01819f0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"f95f57ee-b1d7-4445-800f-5521d01819f0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"44faa2e7-58cb-4f5b-852e-4f211cd17acd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5da032a0-baaa-4a90-ae69-00f333102cda\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6cb0e5791b2675568000a729aced58af9bb81fb4436908ee7d55e1b5282e3bf8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"95ca9e9e-50be-410b-94c2-1a3be3e9b92b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"55d75d734c6c465942a4bbe99c9e2550ca1dde90d5af183d906f3fc23ae149c3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2312, \"end_char_idx\": 4642, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "ref_doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "chroma:document": "```python\n# We'll need to install the Redis client\n!pip install redis\n\n#Install wget to pull zip file\n!pip install wget\n```\n\n\n```python\nimport openai\n\nfrom typing import List, Iterator\nimport pandas as pd\nimport numpy as np\nimport os\nimport wget\nfrom ast import literal_eval\n\n# Redis client library for Python\nimport redis\n\n# I've set this to our new embeddings model, this can be changed to the embedding model of your choice\nEMBEDDING_MODEL = \"text-embedding-3-small\"\n\n# Ignore unclosed SSL socket warnings - optional in case you get these errors\nimport warnings\n\nwarnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning) \n```\n\n## Load data\n\nIn this section we'll load embedded data that we've prepared previous to this session.\n\n\n```python\nembeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'\n\n# The file is ~700 MB so this will take some time\nwget.download(embeddings_url)\n```\n\n\n```python\nimport zipfile\nwith zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n    zip_ref.extractall(\"../data\")\n```\n\n\n```python\narticle_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')\n```\n\n\n```python\narticle_df.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>url</th>\n      <th>title</th>\n      <th>text</th>\n      <th>title_vector</th>\n      <th>content_vector</th>\n      <th>vector_id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>https://simple.wikipedia.org/wiki/April</td>\n      <td>April</td>\n      <td>April is the fourth month of the year in the J...</td>\n      <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n      <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>https://simple.wikipedia.org/wiki/August</td>\n      <td>August</td>\n      <td>August (Aug.)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 95ca9e9e-50be-410b-94c2-1a3be3e9b92b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"95ca9e9e-50be-410b-94c2-1a3be3e9b92b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"44faa2e7-58cb-4f5b-852e-4f211cd17acd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f95f57ee-b1d7-4445-800f-5521d01819f0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a689ea010c1bc5251888130f4573694442c5ee0d666e0b4fb260dcaf8e91fe5e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8a68892d-dd5c-48aa-a03d-041dcbb591e7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"18e7b60db5538910c0b015b09df347c6a085ba812f72bb29fa3eeccb225ebb90\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3495, \"end_char_idx\": 5673, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "ref_doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "chroma:document": "```python\narticle_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')\n```\n\n\n```python\narticle_df.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>url</th>\n      <th>title</th>\n      <th>text</th>\n      <th>title_vector</th>\n      <th>content_vector</th>\n      <th>vector_id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>https://simple.wikipedia.org/wiki/April</td>\n      <td>April</td>\n      <td>April is the fourth month of the year in the J...</td>\n      <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n      <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>https://simple.wikipedia.org/wiki/August</td>\n      <td>August</td>\n      <td>August (Aug.) is the eighth month of the year ...</td>\n      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>6</td>\n      <td>https://simple.wikipedia.org/wiki/Art</td>\n      <td>Art</td>\n      <td>Art is a creative activity that expresses imag...</td>\n      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>8</td>\n      <td>https://simple.wikipedia.org/wiki/A</td>\n      <td>A</td>\n      <td>A or a is the first letter of the English alph...</td>\n      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n      <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>9</td>\n      <td>https://simple.wikipedia.org/wiki/Air</td>\n      <td>Air</td>\n      <td>Air refers to the Earth's atmosphere."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 8a68892d-dd5c-48aa-a03d-041dcbb591e7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"8a68892d-dd5c-48aa-a03d-041dcbb591e7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"44faa2e7-58cb-4f5b-852e-4f211cd17acd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"95ca9e9e-50be-410b-94c2-1a3be3e9b92b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ba98c8b990e273960d99a1a8b9c6fac5aa624908bdcb9d71eb7037bfc7172e54\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c15a8f4c-6528-4401-9590-8362e85e5977\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"647a53a8136e9b26b4b4b4cefb65311c24b5a2c31beb16c74dc5582c5bf676be\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4643, \"end_char_idx\": 6972, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "ref_doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "chroma:document": "is the eighth month of the year ...</td>\n      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>6</td>\n      <td>https://simple.wikipedia.org/wiki/Art</td>\n      <td>Art</td>\n      <td>Art is a creative activity that expresses imag...</td>\n      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>8</td>\n      <td>https://simple.wikipedia.org/wiki/A</td>\n      <td>A</td>\n      <td>A or a is the first letter of the English alph...</td>\n      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n      <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>9</td>\n      <td>https://simple.wikipedia.org/wiki/Air</td>\n      <td>Air</td>\n      <td>Air refers to the Earth's atmosphere. Air is a...</td>\n      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n      <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n\n```python\n# Read vectors from strings back into a list\narticle_df['title_vector'] = article_df.title_vector.apply(literal_eval)\narticle_df['content_vector'] = article_df.content_vector.apply(literal_eval)\n\n# Set vector_id to be a string\narticle_df['vector_id'] = article_df['vector_id'].apply(str)\n```\n\n\n```python\narticle_df.info(show_counts=True)\n```\n\n    <class 'pandas.core.frame.DataFrame'>\n    RangeIndex: 25000 entries, 0 to 24999\n    Data columns (total 7 columns):\n     #   Column          Non-Null Count  Dtype \n    ---  ------          --------------  ----- \n     0   id              25000 non-null  int64 \n     1   url             25000 non-null  object\n     2   title           25000 non-null  object\n     3   text            25000 non-null  object\n     4   title_vector    25000 non-null  object\n     5   content_vector  25000 non-null  object\n     6   vector_id       25000 non-null  object\n    dtypes: int64(1), object(6)\n    memory usage: 1.3+ MB\n\n\n# Redis\n\nThe next vector database covered in this tutorial is **[Redis](https://redis.io)**. You most likely already know Redis."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: c15a8f4c-6528-4401-9590-8362e85e5977
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c15a8f4c-6528-4401-9590-8362e85e5977\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"44faa2e7-58cb-4f5b-852e-4f211cd17acd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8a68892d-dd5c-48aa-a03d-041dcbb591e7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c28052e04f52063da569dfeb941e12b763aed2cd054bf995ed1bacca8fbfbb15\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0730bc98-a0b2-480e-a975-95037cfe6fec\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"06777b422b8a962b9e3d3037f35aa5facb415c0cb6155bf970990a0f6ca82fbf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5674, \"end_char_idx\": 8400, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "ref_doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "chroma:document": "Air is a...</td>\n      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n      <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n\n```python\n# Read vectors from strings back into a list\narticle_df['title_vector'] = article_df.title_vector.apply(literal_eval)\narticle_df['content_vector'] = article_df.content_vector.apply(literal_eval)\n\n# Set vector_id to be a string\narticle_df['vector_id'] = article_df['vector_id'].apply(str)\n```\n\n\n```python\narticle_df.info(show_counts=True)\n```\n\n    <class 'pandas.core.frame.DataFrame'>\n    RangeIndex: 25000 entries, 0 to 24999\n    Data columns (total 7 columns):\n     #   Column          Non-Null Count  Dtype \n    ---  ------          --------------  ----- \n     0   id              25000 non-null  int64 \n     1   url             25000 non-null  object\n     2   title           25000 non-null  object\n     3   text            25000 non-null  object\n     4   title_vector    25000 non-null  object\n     5   content_vector  25000 non-null  object\n     6   vector_id       25000 non-null  object\n    dtypes: int64(1), object(6)\n    memory usage: 1.3+ MB\n\n\n# Redis\n\nThe next vector database covered in this tutorial is **[Redis](https://redis.io)**. You most likely already know Redis. What you might not be aware of is the [RediSearch module](https://github.com/RediSearch/RediSearch). Enterprises have been using Redis with the RediSearch module for years now across all major cloud providers, Redis Cloud, and on premise. Recently, the Redis team added vector storage and search capability to this module in addition to the features RediSearch already had.\n\nGiven the large ecosystem around Redis, there are most likely client libraries in the language you need. You can use any standard Redis client library to run RediSearch commands, but it's easiest to use a library that wraps the RediSearch API. Below are a few examples, but you can find more client libraries [here](https://redis.io/resources/clients/).\n\n| Project | Language | License | Author | Stars |\n|----------|---------|--------|---------|-------|\n| [jedis][jedis-url] | Java | MIT | [Redis][redis-url] | ![Stars][jedis-stars] |\n| [redis-py][redis-py-url] | Python | MIT | [Redis][redis-url] | ![Stars][redis-py-stars] |\n| [node-redis][node-redis-url] | Node.js | MIT | [Redis][redis-url] | ![Stars][node-redis-stars] |\n| [nredisstack][nredisstack-url] | .NET | MIT | [Redis][redis-url] | ![Stars][nredisstack-stars] |\n| [redisearch-go][redisearch-go-url] | Go | BSD | [Redis][redisearch-go-author] | [![redisearch-go-stars]][redisearch-go-url] |\n| [redisearch-api-rs][redisearch-api-rs-url] | Rust | BSD | [Redis][redisearch-api-rs-author] | [!"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 0730bc98-a0b2-480e-a975-95037cfe6fec
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0730bc98-a0b2-480e-a975-95037cfe6fec\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"44faa2e7-58cb-4f5b-852e-4f211cd17acd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c15a8f4c-6528-4401-9590-8362e85e5977\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"1ca1cfe187425171a779b3643ef7c9357ba9a22e6d91c49e34c0351af54af32d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"16df6f21-0d61-45d9-8ddb-3d6b3418eb1f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"845a012f52e37a98b828c0f17f31fdfd1a68f9a6186ab60e3678e5e6f272f198\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6937, \"end_char_idx\": 10175, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "ref_doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "chroma:document": "You most likely already know Redis. What you might not be aware of is the [RediSearch module](https://github.com/RediSearch/RediSearch). Enterprises have been using Redis with the RediSearch module for years now across all major cloud providers, Redis Cloud, and on premise. Recently, the Redis team added vector storage and search capability to this module in addition to the features RediSearch already had.\n\nGiven the large ecosystem around Redis, there are most likely client libraries in the language you need. You can use any standard Redis client library to run RediSearch commands, but it's easiest to use a library that wraps the RediSearch API. Below are a few examples, but you can find more client libraries [here](https://redis.io/resources/clients/).\n\n| Project | Language | License | Author | Stars |\n|----------|---------|--------|---------|-------|\n| [jedis][jedis-url] | Java | MIT | [Redis][redis-url] | ![Stars][jedis-stars] |\n| [redis-py][redis-py-url] | Python | MIT | [Redis][redis-url] | ![Stars][redis-py-stars] |\n| [node-redis][node-redis-url] | Node.js | MIT | [Redis][redis-url] | ![Stars][node-redis-stars] |\n| [nredisstack][nredisstack-url] | .NET | MIT | [Redis][redis-url] | ![Stars][nredisstack-stars] |\n| [redisearch-go][redisearch-go-url] | Go | BSD | [Redis][redisearch-go-author] | [![redisearch-go-stars]][redisearch-go-url] |\n| [redisearch-api-rs][redisearch-api-rs-url] | Rust | BSD | [Redis][redisearch-api-rs-author] | [![redisearch-api-rs-stars]][redisearch-api-rs-url] |\n\n[redis-url]: https://redis.com\n\n[redis-py-url]: https://github.com/redis/redis-py\n[redis-py-stars]: https://img.shields.io/github/stars/redis/redis-py.svg?style=social&amp;label=Star&amp;maxAge=2592000\n[redis-py-package]: https://pypi.python.org/pypi/redis\n\n[jedis-url]: https://github.com/redis/jedis\n[jedis-stars]: https://img.shields.io/github/stars/redis/jedis.svg?style=social&amp;label=Star&amp;maxAge=2592000\n[Jedis-package]: https://search.maven.org/artifact/redis.clients/jedis\n\n[nredisstack-url]: https://github.com/redis/nredisstack\n[nredisstack-stars]: https://img.shields.io/github/stars/redis/nredisstack.svg?style=social&amp;label=Star&amp;maxAge=2592000\n[nredisstack-package]: https://www.nuget.org/packages/nredisstack/\n\n[node-redis-url]: https://github.com/redis/node-redis\n[node-redis-stars]: https://img.shields.io/github/stars/redis/node-redis.svg?style=social&amp;label=Star&amp;maxAge=2592000\n[node-redis-package]: https://www.npmjs.com/package/redis\n\n[redis-om-python-url]: https://github.com/redis/redis-om-python\n[redis-om-python-author]: https://redis.com\n[redis-om-python-stars]: https://img.shields.io/github/stars/redis/redis-om-python.svg?style=social&amp;label=Star&amp;maxAge=2592000\n\n[redisearch-go-url]: https://github.com/RediSearch/redisearch-go\n[redisearch-go-author]: https://redis.com\n[redisearch-go-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-go.svg?style=social&amp;label=Star&amp;maxAge=2592000\n\n[redisearch-api-rs-url]: https://github.com/RediSearch/redisearch-api-rs\n[redisearch-api-rs-author]: https://redis.com\n[redisearch-api-rs-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-api-rs.svg?style=social&amp;label=Star&amp;maxAge=2592000"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 16df6f21-0d61-45d9-8ddb-3d6b3418eb1f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"16df6f21-0d61-45d9-8ddb-3d6b3418eb1f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"44faa2e7-58cb-4f5b-852e-4f211cd17acd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0730bc98-a0b2-480e-a975-95037cfe6fec\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"b39f91b81ebb87ad5a5b4803bbe38240f481907ca6a60356b248d670b3652a4a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"26673c34-bef4-488c-94a2-b82669a7107c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4b9ab54977f54ba8b00795c25cff3079b45023950e8f5130d45a560a1d613543\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10178, \"end_char_idx\": 13003, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "ref_doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "chroma:document": "In the below cells, we will walk you through using Redis as a vector database. Since many of you are likely already used to the Redis API, this should be familiar to most.\n\n## Setup\n\nThere are many ways to deploy Redis with RediSearch. The easiest way to get started is to use Docker, but there are are many potential options for deployment. For other deployment options, see the [redis directory](./redis) in this repo.\n\nFor this tutorial, we will use Redis Stack on Docker.\n\nStart a version of Redis with RediSearch (Redis Stack) by running the following docker command\n\n```bash\n$ cd redis\n$ docker compose up -d\n```\nThis also includes the [RedisInsight](https://redis.com/redis-enterprise/redis-insight/) GUI for managing your Redis database which you can view at [http://localhost:8001](http://localhost:8001) once you start the docker container.\n\nYou're all set up and ready to go! Next, we import and create our client for communicating with the Redis database we just created.\n\n\n```python\nimport redis\nfrom redis.commands.search.indexDefinition import (\n    IndexDefinition,\n    IndexType\n)\nfrom redis.commands.search.query import Query\nfrom redis.commands.search.field import (\n    TextField,\n    VectorField\n)\n\nREDIS_HOST =  \"localhost\"\nREDIS_PORT = 6379\nREDIS_PASSWORD = \"\" # default for passwordless Redis\n\n# Connect to Redis\nredis_client = redis.Redis(\n    host=REDIS_HOST,\n    port=REDIS_PORT,\n    password=REDIS_PASSWORD\n)\nredis_client.ping()\n```\n\n\n\n\n    True\n\n\n\n## Creating a Search Index\n\nThe below cells will show how to specify and create a search index in Redis. We will\n\n1. Set some constants for defining our index like the distance metric and the index name\n2. Define the index schema with RediSearch fields\n3. Create the index\n\n\n\n```python\n# Constants\nVECTOR_DIM = len(article_df['title_vector'][0]) # length of the vectors\nVECTOR_NUMBER = len(article_df)                 # initial number of vectors\nINDEX_NAME = \"embeddings-index\"                 # name of the search index\nPREFIX = \"doc\"                                  # prefix for the document keys\nDISTANCE_METRIC = \"COSINE\"                      # distance metric for the vectors (ex. COSINE, IP, L2)\n```\n\n\n```python\n# Define RediSearch fields for each of the columns in the dataset\ntitle = TextField(name=\"title\")\nurl = TextField(name=\"url\")\ntext = TextField(name=\"text\")\ntitle_embedding = VectorField(\"title_vector\",\n    \"FLAT\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER,\n    }\n)\ntext_embedding = VectorField(\"content_vector\",\n    \"FLAT\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER,\n    }\n)\nfields = [title, url, text, title_embedding, text_embedding]\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 26673c34-bef4-488c-94a2-b82669a7107c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"26673c34-bef4-488c-94a2-b82669a7107c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"44faa2e7-58cb-4f5b-852e-4f211cd17acd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"16df6f21-0d61-45d9-8ddb-3d6b3418eb1f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"d43be6244787ff80f97e206d0363aebc1629a61e741af83d693c19e759c9d6ac\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a2813576-adcd-4560-9642-1bd0cb006199\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"39ab50975f0a16a65a7b2763f6d87b336d84a7e1a0abc07e558b57eddd8b6425\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11647, \"end_char_idx\": 14591, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "ref_doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "chroma:document": "True\n\n\n\n## Creating a Search Index\n\nThe below cells will show how to specify and create a search index in Redis. We will\n\n1. Set some constants for defining our index like the distance metric and the index name\n2. Define the index schema with RediSearch fields\n3. Create the index\n\n\n\n```python\n# Constants\nVECTOR_DIM = len(article_df['title_vector'][0]) # length of the vectors\nVECTOR_NUMBER = len(article_df)                 # initial number of vectors\nINDEX_NAME = \"embeddings-index\"                 # name of the search index\nPREFIX = \"doc\"                                  # prefix for the document keys\nDISTANCE_METRIC = \"COSINE\"                      # distance metric for the vectors (ex. COSINE, IP, L2)\n```\n\n\n```python\n# Define RediSearch fields for each of the columns in the dataset\ntitle = TextField(name=\"title\")\nurl = TextField(name=\"url\")\ntext = TextField(name=\"text\")\ntitle_embedding = VectorField(\"title_vector\",\n    \"FLAT\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER,\n    }\n)\ntext_embedding = VectorField(\"content_vector\",\n    \"FLAT\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER,\n    }\n)\nfields = [title, url, text, title_embedding, text_embedding]\n```\n\n\n```python\n# Check if index exists\ntry:\n    redis_client.ft(INDEX_NAME).info()\n    print(\"Index already exists\")\nexcept:\n    # Create RediSearch Index\n    redis_client.ft(INDEX_NAME).create_index(\n        fields = fields,\n        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n    )\n```\n\n## Load Documents into the Index\n\nNow that we have a search index, we can load documents into it. We will use the same documents we used in the previous examples. In Redis, either the Hash or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The below cells will show how to load documents into the index.\n\n\n```python\ndef index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):\n    records = documents.to_dict(\"records\")\n    for doc in records:\n        key = f\"{prefix}:{str(doc['id'])}\"\n\n        # create byte vectors for title and content\n        title_embedding = np.array(doc[\"title_vector\"], dtype=np.float32).tobytes()\n        content_embedding = np.array(doc[\"content_vector\"], dtype=np.float32).tobytes()\n\n        # replace list of floats with byte vectors\n        doc[\"title_vector\"] = title_embedding\n        doc[\"content_vector\"] = content_embedding\n\n        client.hset(key, mapping = doc)\n```\n\n\n```python\nindex_documents(redis_client, PREFIX, article_df)\nprint(f\"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}\")\n```\n\n    Loaded 25000 documents in Redis search index with name: embeddings-index"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: a2813576-adcd-4560-9642-1bd0cb006199
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a2813576-adcd-4560-9642-1bd0cb006199\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"44faa2e7-58cb-4f5b-852e-4f211cd17acd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"26673c34-bef4-488c-94a2-b82669a7107c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"dc23a3b876b5ee4ec3874bfa359efa6e05ce21be0c374f5d89fde4da54009918\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d3f14ea9-3f39-4556-8d9e-b710e32649c3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"14c41fb6388a9b347e8d726c9e619ceb3216ba48c7a37f4be7844bc2417bedb4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13006, \"end_char_idx\": 15568, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "ref_doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "chroma:document": "```python\n# Check if index exists\ntry:\n    redis_client.ft(INDEX_NAME).info()\n    print(\"Index already exists\")\nexcept:\n    # Create RediSearch Index\n    redis_client.ft(INDEX_NAME).create_index(\n        fields = fields,\n        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n    )\n```\n\n## Load Documents into the Index\n\nNow that we have a search index, we can load documents into it. We will use the same documents we used in the previous examples. In Redis, either the Hash or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The below cells will show how to load documents into the index.\n\n\n```python\ndef index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):\n    records = documents.to_dict(\"records\")\n    for doc in records:\n        key = f\"{prefix}:{str(doc['id'])}\"\n\n        # create byte vectors for title and content\n        title_embedding = np.array(doc[\"title_vector\"], dtype=np.float32).tobytes()\n        content_embedding = np.array(doc[\"content_vector\"], dtype=np.float32).tobytes()\n\n        # replace list of floats with byte vectors\n        doc[\"title_vector\"] = title_embedding\n        doc[\"content_vector\"] = content_embedding\n\n        client.hset(key, mapping = doc)\n```\n\n\n```python\nindex_documents(redis_client, PREFIX, article_df)\nprint(f\"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}\")\n```\n\n    Loaded 25000 documents in Redis search index with name: embeddings-index\n\n\n## Running Search Queries\n\nNow that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database. Each example will demonstrate specific features to keep in mind when developing your search application with Redis.\n\n1. **Return Fields**: You can specify which fields you want to return in the search results. This is useful if you only want to return a subset of the fields in your documents and doesn't require a separate call to retrieve documents. In the below example, we will only return the `title` field in the search results.\n2. **Hybrid Search**: You can combine vector search with any of the other RediSearch fields for hybrid search such as full text search, tag, geo, and numeric. In the below example, we will combine vector search with full text search."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: d3f14ea9-3f39-4556-8d9e-b710e32649c3
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"d3f14ea9-3f39-4556-8d9e-b710e32649c3\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"44faa2e7-58cb-4f5b-852e-4f211cd17acd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a2813576-adcd-4560-9642-1bd0cb006199\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7ed0b7e400e1aecc8f3b033407fd9a2ae028bb8bab93f18befcf9df90ba71122\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f5a70c4d-ebc4-48e0-97c8-aae3dd590af8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5765ce945710b5e3b8c7b151bd19436eaaff407dedd1df6a8bf9fdfd431e2c4a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13720, \"end_char_idx\": 16762, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "ref_doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "chroma:document": "```python\ndef index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):\n    records = documents.to_dict(\"records\")\n    for doc in records:\n        key = f\"{prefix}:{str(doc['id'])}\"\n\n        # create byte vectors for title and content\n        title_embedding = np.array(doc[\"title_vector\"], dtype=np.float32).tobytes()\n        content_embedding = np.array(doc[\"content_vector\"], dtype=np.float32).tobytes()\n\n        # replace list of floats with byte vectors\n        doc[\"title_vector\"] = title_embedding\n        doc[\"content_vector\"] = content_embedding\n\n        client.hset(key, mapping = doc)\n```\n\n\n```python\nindex_documents(redis_client, PREFIX, article_df)\nprint(f\"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}\")\n```\n\n    Loaded 25000 documents in Redis search index with name: embeddings-index\n\n\n## Running Search Queries\n\nNow that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database. Each example will demonstrate specific features to keep in mind when developing your search application with Redis.\n\n1. **Return Fields**: You can specify which fields you want to return in the search results. This is useful if you only want to return a subset of the fields in your documents and doesn't require a separate call to retrieve documents. In the below example, we will only return the `title` field in the search results.\n2. **Hybrid Search**: You can combine vector search with any of the other RediSearch fields for hybrid search such as full text search, tag, geo, and numeric. In the below example, we will combine vector search with full text search.\n\n\n\n```python\ndef search_redis(\n    redis_client: redis.Redis,\n    user_query: str,\n    index_name: str = \"embeddings-index\",\n    vector_field: str = \"title_vector\",\n    return_fields: list = [\"title\", \"url\", \"text\", \"vector_score\"],\n    hybrid_fields = \"*\",\n    k: int = 20,\n) -> List[dict]:\n\n    # Creates embedding vector from user query\n    embedded_query = openai.Embedding.create(input=user_query,\n                                            model=EMBEDDING_MODEL,\n                                            )[\"data\"][0]['embedding']\n\n    # Prepare the Query\n    base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'\n    query = (\n        Query(base_query)\n         .return_fields(*return_fields)\n         .sort_by(\"vector_score\")\n         .paging(0, k)\n         .dialect(2)\n    )\n    params_dict = {\"vector\": np.array(embedded_query).astype(dtype=np.float32).tobytes()}\n\n    # perform vector search\n    results = redis_client.ft(index_name).search(query, params_dict)\n    for i, article in enumerate(results.docs):\n        score = 1 - float(article.vector_score)\n        print(f\"{i}. {article.title} (Score: {round(score ,3) })\")\n    return results.docs\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: f5a70c4d-ebc4-48e0-97c8-aae3dd590af8
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"f5a70c4d-ebc4-48e0-97c8-aae3dd590af8\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"44faa2e7-58cb-4f5b-852e-4f211cd17acd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d3f14ea9-3f39-4556-8d9e-b710e32649c3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"e09f9719e2068d70221001567bc4fa69fb93f8b47e0b190d1879427f99fdb0e6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b5d89480-daae-4ed4-b9df-d701333f0aa6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b87ee3e2b78c1e1e0d083d238aa850d7def6c3e19e6b872ed42bcf58c8d1f8d0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15572, \"end_char_idx\": 18252, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "ref_doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "chroma:document": "```python\ndef search_redis(\n    redis_client: redis.Redis,\n    user_query: str,\n    index_name: str = \"embeddings-index\",\n    vector_field: str = \"title_vector\",\n    return_fields: list = [\"title\", \"url\", \"text\", \"vector_score\"],\n    hybrid_fields = \"*\",\n    k: int = 20,\n) -> List[dict]:\n\n    # Creates embedding vector from user query\n    embedded_query = openai.Embedding.create(input=user_query,\n                                            model=EMBEDDING_MODEL,\n                                            )[\"data\"][0]['embedding']\n\n    # Prepare the Query\n    base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'\n    query = (\n        Query(base_query)\n         .return_fields(*return_fields)\n         .sort_by(\"vector_score\")\n         .paging(0, k)\n         .dialect(2)\n    )\n    params_dict = {\"vector\": np.array(embedded_query).astype(dtype=np.float32).tobytes()}\n\n    # perform vector search\n    results = redis_client.ft(index_name).search(query, params_dict)\n    for i, article in enumerate(results.docs):\n        score = 1 - float(article.vector_score)\n        print(f\"{i}. {article.title} (Score: {round(score ,3) })\")\n    return results.docs\n```\n\n\n```python\n# For using OpenAI to generate query embedding\nopenai.api_key = os.getenv(\"OPENAI_API_KEY\", \"sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\")\nresults = search_redis(redis_client, 'modern art in Europe', k=10)\n```\n\n    0. Museum of Modern Art (Score: 0.875)\n    1. Western Europe (Score: 0.867)\n    2. Renaissance art (Score: 0.864)\n    3. Pop art (Score: 0.86)\n    4. Northern Europe (Score: 0.855)\n    5. Hellenistic art (Score: 0.853)\n    6. Modernist literature (Score: 0.847)\n    7. Art film (Score: 0.843)\n    8. Central Europe (Score: 0.843)\n    9. European (Score: 0.841)\n\n\n\n```python\nresults = search_redis(redis_client, 'Famous battles in Scottish history', vector_field='content_vector', k=10)\n```\n\n    0. Battle of Bannockburn (Score: 0.869)\n    1. Wars of Scottish Independence (Score: 0.861)\n    2. 1651 (Score: 0.853)\n    3. First War of Scottish Independence (Score: 0.85)\n    4. Robert I of Scotland (Score: 0.846)\n    5. 841 (Score: 0.844)\n    6. 1716 (Score: 0.844)\n    7. 1314 (Score: 0.837)\n    8. 1263 (Score: 0.836)\n    9. William Wallace (Score: 0.835)\n\n\n## Hybrid Queries with Redis\n\nThe previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the below example, we will combine vector search with full text search.\n\n\n```python\ndef create_hybrid_field(field_name: str, value: str) -> str:\n    return f'@{field_name}:\"{value}\"'\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: b5d89480-daae-4ed4-b9df-d701333f0aa6
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"b5d89480-daae-4ed4-b9df-d701333f0aa6\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"44faa2e7-58cb-4f5b-852e-4f211cd17acd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f5a70c4d-ebc4-48e0-97c8-aae3dd590af8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"0502f54a747773e52cc858c77c9c8a4fcec263117b8cfac90516a862cf82df60\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"19643cb5-4125-4fd6-a148-ccccc1f62d3a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6cb263833ce2db73fc6ea420ea725838d71a38887aa9f8c5fbb0f7d90ed6e5ce\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17354, \"end_char_idx\": 20462, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "ref_doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "chroma:document": "```python\nresults = search_redis(redis_client, 'Famous battles in Scottish history', vector_field='content_vector', k=10)\n```\n\n    0. Battle of Bannockburn (Score: 0.869)\n    1. Wars of Scottish Independence (Score: 0.861)\n    2. 1651 (Score: 0.853)\n    3. First War of Scottish Independence (Score: 0.85)\n    4. Robert I of Scotland (Score: 0.846)\n    5. 841 (Score: 0.844)\n    6. 1716 (Score: 0.844)\n    7. 1314 (Score: 0.837)\n    8. 1263 (Score: 0.836)\n    9. William Wallace (Score: 0.835)\n\n\n## Hybrid Queries with Redis\n\nThe previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the below example, we will combine vector search with full text search.\n\n\n```python\ndef create_hybrid_field(field_name: str, value: str) -> str:\n    return f'@{field_name}:\"{value}\"'\n```\n\n\n```python\n# search the content vector for articles about famous battles in Scottish history and only include results with Scottish in the title\nresults = search_redis(redis_client,\n                       \"Famous battles in Scottish history\",\n                       vector_field=\"title_vector\",\n                       k=5,\n                       hybrid_fields=create_hybrid_field(\"title\", \"Scottish\")\n                       )\n```\n\n    0. First War of Scottish Independence (Score: 0.892)\n    1. Wars of Scottish Independence (Score: 0.889)\n    2. Second War of Scottish Independence (Score: 0.879)\n    3. List of Scottish monarchs (Score: 0.873)\n    4. Scottish Borders (Score: 0.863)\n\n\n\n```python\n# run a hybrid query for articles about Art in the title vector and only include results with the phrase \"Leonardo da Vinci\" in the text\nresults = search_redis(redis_client,\n                       \"Art\",\n                       vector_field=\"title_vector\",\n                       k=5,\n                       hybrid_fields=create_hybrid_field(\"text\", \"Leonardo da Vinci\")\n                       )\n\n# find specific mention of Leonardo da Vinci in the text that our full-text-search query returned\nmention = [sentence for sentence in results[0].text.split(\"\\n\") if \"Leonardo da Vinci\" in sentence][0]\nmention\n```\n\n    0. Art (Score: 1.0)\n    1. Paint (Score: 0.896)\n    2. Renaissance art (Score: 0.88)\n    3. Painting (Score: 0.874)\n    4. Renaissance (Score: 0.846)\n\n\n\n\n\n    'In Europe, after the Middle Ages, there was a \"Renaissance\" which means \"rebirth\". People rediscovered science and artists were allowed to paint subjects other than religious subjects. People like Michelangelo and Leonardo da Vinci still painted religious pictures, but they also now could paint mythological pictures too. These artists also invented perspective where things in the distance look smaller in the picture. This was new because in the Middle Ages people would paint all the figures close up and just overlapping each other. These artists used nudity regularly in their art.'\n\n\n\nFor more example with Redis as a vector database, see the README and examples within the ``vector_databases/redis`` directory of this repository"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 19643cb5-4125-4fd6-a148-ccccc1f62d3a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"19643cb5-4125-4fd6-a148-ccccc1f62d3a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"44faa2e7-58cb-4f5b-852e-4f211cd17acd\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b5d89480-daae-4ed4-b9df-d701333f0aa6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"1455475e46fc448513bd9d1ee7df7b5daaf42774e6fae9058747d1ec6747a942\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18942, \"end_char_idx\": 20479, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "ref_doc_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "chroma:document": "```python\n# run a hybrid query for articles about Art in the title vector and only include results with the phrase \"Leonardo da Vinci\" in the text\nresults = search_redis(redis_client,\n                       \"Art\",\n                       vector_field=\"title_vector\",\n                       k=5,\n                       hybrid_fields=create_hybrid_field(\"text\", \"Leonardo da Vinci\")\n                       )\n\n# find specific mention of Leonardo da Vinci in the text that our full-text-search query returned\nmention = [sentence for sentence in results[0].text.split(\"\\n\") if \"Leonardo da Vinci\" in sentence][0]\nmention\n```\n\n    0. Art (Score: 1.0)\n    1. Paint (Score: 0.896)\n    2. Renaissance art (Score: 0.88)\n    3. Painting (Score: 0.874)\n    4. Renaissance (Score: 0.846)\n\n\n\n\n\n    'In Europe, after the Middle Ages, there was a \"Renaissance\" which means \"rebirth\". People rediscovered science and artists were allowed to paint subjects other than religious subjects. People like Michelangelo and Leonardo da Vinci still painted religious pictures, but they also now could paint mythological pictures too. These artists also invented perspective where things in the distance look smaller in the picture. This was new because in the Middle Ages people would paint all the figures close up and just overlapping each other. These artists used nudity regularly in their art.'\n\n\n\nFor more example with Redis as a vector database, see the README and examples within the ``vector_databases/redis`` directory of this repository\n\n\n```python\n\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: f3c97012-a116-4e56-9a79-37e3e6e861fb
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"f3c97012-a116-4e56-9a79-37e3e6e861fb\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb\", \"title\": \"Redis\", \"tokens\": 2363, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2aa7946d-a00f-4f81-bdd6-e48e11274ff7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb\", \"title\": \"Redis\", \"tokens\": 2363, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"b08d2ce2199cb0ef05b32554508f11d39da0689c732dce096a9cbf3dd901eca0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"968cd1e8-9d08-41c1-a291-1f6c231698b0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"60252f48545fffa75c59a4eb516fd84fc4c1ac8dcd6a08004bbf9cef8f9bfd57\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2501, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2aa7946d-a00f-4f81-bdd6-e48e11274ff7", "doc_id": "2aa7946d-a00f-4f81-bdd6-e48e11274ff7", "ref_doc_id": "2aa7946d-a00f-4f81-bdd6-e48e11274ff7", "chroma:document": "# Redis\n\n### What is Redis?\n\nMost developers from a web services background are probably familiar with Redis. At it's core, Redis is an open-source key-value store that can be used as a cache, message broker, and database. Developers choice Redis because it is fast, has a large ecosystem of client libraries, and has been deployed by major enterprises for years.\n\nIn addition to the traditional uses of Redis. Redis also provides [Redis Modules](https://redis.io/modules) which are a way to extend Redis with new capabilities, commands and data types. Example modules include [RedisJSON](https://redis.io/docs/stack/json/), [RedisTimeSeries](https://redis.io/docs/stack/timeseries/), [RedisBloom](https://redis.io/docs/stack/bloom/) and [RediSearch](https://redis.io/docs/stack/search/).\n\n\n### Deployment options\n\nThere are a number of ways to deploy Redis. For local development, the quickest method is to use the [Redis Stack docker container](https://hub.docker.com/r/redis/redis-stack) which we will use here. Redis Stack contains a number of Redis modules that can be used together to create a fast, multi-model data store and query engine.\n\nFor production use cases, The easiest way to get started is to use the [Redis Cloud](https://redislabs.com/redis-enterprise-cloud/overview/) service. Redis Cloud is a fully managed Redis service. You can also deploy Redis on your own infrastructure using [Redis Enterprise](https://redislabs.com/redis-enterprise/overview/). Redis Enterprise is a fully managed Redis service that can be deployed in kubernetes, on-premises or in the cloud.\n\nAdditionally, every major cloud provider ([AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-e6y7ork67pjwg?sr=0-2&ref_=beagle&applicationId=AWSMPContessa), [Google Marketplace](https://console.cloud.google.com/marketplace/details/redislabs-public/redis-enterprise?pli=1), or [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/garantiadata.redis_enterprise_1sp_public_preview?tab=Overview)) offers Redis Enterprise in a marketplace offering.\n\n\n### What is RediSearch?\n\nRediSearch is a [Redis module](https://redis.io/modules) that provides querying, secondary indexing, full-text search and vector search for Redis. To use RediSearch, you first declare indexes on your Redis data. You can then use the RediSearch clients to query that data. For more information on the feature set of RediSearch, see the [RediSearch documentation](https://redis.io/docs/stack/search/)."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 968cd1e8-9d08-41c1-a291-1f6c231698b0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"968cd1e8-9d08-41c1-a291-1f6c231698b0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb\", \"title\": \"Redis\", \"tokens\": 2363, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2aa7946d-a00f-4f81-bdd6-e48e11274ff7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb\", \"title\": \"Redis\", \"tokens\": 2363, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"b08d2ce2199cb0ef05b32554508f11d39da0689c732dce096a9cbf3dd901eca0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f3c97012-a116-4e56-9a79-37e3e6e861fb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb\", \"title\": \"Redis\", \"tokens\": 2363, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"80a34cf61b078186617c84d636e9a15d7ce53212072a42d702fa9297589564f7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"33bfa666-f704-4905-b5d8-30be7abd894f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cb7257b683082a9e2a155f60bb6de86a717353be30ded39f860e2f3d5be40b32\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 791, \"end_char_idx\": 3909, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2aa7946d-a00f-4f81-bdd6-e48e11274ff7", "doc_id": "2aa7946d-a00f-4f81-bdd6-e48e11274ff7", "ref_doc_id": "2aa7946d-a00f-4f81-bdd6-e48e11274ff7", "chroma:document": "### Deployment options\n\nThere are a number of ways to deploy Redis. For local development, the quickest method is to use the [Redis Stack docker container](https://hub.docker.com/r/redis/redis-stack) which we will use here. Redis Stack contains a number of Redis modules that can be used together to create a fast, multi-model data store and query engine.\n\nFor production use cases, The easiest way to get started is to use the [Redis Cloud](https://redislabs.com/redis-enterprise-cloud/overview/) service. Redis Cloud is a fully managed Redis service. You can also deploy Redis on your own infrastructure using [Redis Enterprise](https://redislabs.com/redis-enterprise/overview/). Redis Enterprise is a fully managed Redis service that can be deployed in kubernetes, on-premises or in the cloud.\n\nAdditionally, every major cloud provider ([AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-e6y7ork67pjwg?sr=0-2&ref_=beagle&applicationId=AWSMPContessa), [Google Marketplace](https://console.cloud.google.com/marketplace/details/redislabs-public/redis-enterprise?pli=1), or [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/garantiadata.redis_enterprise_1sp_public_preview?tab=Overview)) offers Redis Enterprise in a marketplace offering.\n\n\n### What is RediSearch?\n\nRediSearch is a [Redis module](https://redis.io/modules) that provides querying, secondary indexing, full-text search and vector search for Redis. To use RediSearch, you first declare indexes on your Redis data. You can then use the RediSearch clients to query that data. For more information on the feature set of RediSearch, see the [RediSearch documentation](https://redis.io/docs/stack/search/).\n\n\n### Features\n\nRediSearch uses compressed, inverted indexes for fast indexing with a low memory footprint. RediSearch indexes enhance Redis by providing exact-phrase matching, fuzzy search, and numeric filtering, among many other features. Such as:\n\n* Full-Text indexing of multiple fields in Redis hashes\n* Incremental indexing without performance loss\n* Vector similarity search\n* Document ranking (using [tf-idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf), with optional user-provided weights)\n* Field weighting\n* Complex boolean queries with AND, OR, and NOT operators\n* Prefix matching, fuzzy matching, and exact-phrase queries\n* Support for [double-metaphone phonetic matching](https://redis.io/docs/stack/search/reference/phonetic_matching/)\n* Auto-complete suggestions (with fuzzy prefix suggestions)\n* Stemming-based query expansion in [many languages](https://redis.io/docs/stack/search/reference/stemming/) (using [Snowball](http://snowballstem.org/))\n* Support for Chinese-language tokenization and querying (using [Friso](https://github.com/lionsoul2014/friso))\n* Numeric filters and ranges\n* Geospatial searches using [Redis geospatial indexing](/commands/georadius)\n* A powerful aggregations engine\n* Supports for all utf-8 encoded text\n* Retrieve full documents, selected fields, or only the document IDs\n* Sorting results (for example, by creation date)\n* JSON support through RedisJSON"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 33bfa666-f704-4905-b5d8-30be7abd894f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"33bfa666-f704-4905-b5d8-30be7abd894f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb\", \"title\": \"Redis\", \"tokens\": 2363, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2aa7946d-a00f-4f81-bdd6-e48e11274ff7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb\", \"title\": \"Redis\", \"tokens\": 2363, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"b08d2ce2199cb0ef05b32554508f11d39da0689c732dce096a9cbf3dd901eca0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"968cd1e8-9d08-41c1-a291-1f6c231698b0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb\", \"title\": \"Redis\", \"tokens\": 2363, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"bd34788ffe83d6511cd0a9ff2503d8fab821af0fb6a3c6c53271ad57793cbc8d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bbad043e-7781-46ab-b542-079d2921150e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dc5a05be55dbfc5b2b58c5484e6582f2da7e8cd8fdf126f134a2b850ab78a20c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2504, \"end_char_idx\": 6474, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2aa7946d-a00f-4f81-bdd6-e48e11274ff7", "doc_id": "2aa7946d-a00f-4f81-bdd6-e48e11274ff7", "ref_doc_id": "2aa7946d-a00f-4f81-bdd6-e48e11274ff7", "chroma:document": "### Features\n\nRediSearch uses compressed, inverted indexes for fast indexing with a low memory footprint. RediSearch indexes enhance Redis by providing exact-phrase matching, fuzzy search, and numeric filtering, among many other features. Such as:\n\n* Full-Text indexing of multiple fields in Redis hashes\n* Incremental indexing without performance loss\n* Vector similarity search\n* Document ranking (using [tf-idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf), with optional user-provided weights)\n* Field weighting\n* Complex boolean queries with AND, OR, and NOT operators\n* Prefix matching, fuzzy matching, and exact-phrase queries\n* Support for [double-metaphone phonetic matching](https://redis.io/docs/stack/search/reference/phonetic_matching/)\n* Auto-complete suggestions (with fuzzy prefix suggestions)\n* Stemming-based query expansion in [many languages](https://redis.io/docs/stack/search/reference/stemming/) (using [Snowball](http://snowballstem.org/))\n* Support for Chinese-language tokenization and querying (using [Friso](https://github.com/lionsoul2014/friso))\n* Numeric filters and ranges\n* Geospatial searches using [Redis geospatial indexing](/commands/georadius)\n* A powerful aggregations engine\n* Supports for all utf-8 encoded text\n* Retrieve full documents, selected fields, or only the document IDs\n* Sorting results (for example, by creation date)\n* JSON support through RedisJSON\n\n\n### Clients\n\nGiven the large ecosystem around Redis, there are most likely client libraries in the language you need. You can use any standard Redis client library to run RediSearch commands, but it's easiest to use a library that wraps the RediSearch API. Below are a few examples, but you can find more client libraries [here](https://redis.io/resources/clients/).\n\n| Project | Language | License | Author | Stars |\n|----------|---------|--------|---------|-------|\n| [jedis][jedis-url] | Java | MIT | [Redis][redis-url] | ![Stars][jedis-stars] |\n| [redis-py][redis-py-url] | Python | MIT | [Redis][redis-url] | ![Stars][redis-py-stars] |\n| [node-redis][node-redis-url] | Node.js | MIT | [Redis][redis-url] | ![Stars][node-redis-stars] |\n| [nredisstack][nredisstack-url] | .NET | MIT | [Redis][redis-url] | ![Stars][nredisstack-stars] |\n\n[redis-url]: https://redis.com\n\n[redis-py-url]: https://github.com/redis/redis-py\n[redis-py-stars]: https://img.shields.io/github/stars/redis/redis-py.svg?style=social&amp;label=Star&amp;maxAge=2592000\n[redis-py-package]: https://pypi.python.org/pypi/redis\n\n[jedis-url]: https://github.com/redis/jedis\n[jedis-stars]: https://img.shields.io/github/stars/redis/jedis.svg?style=social&amp;label=Star&amp;maxAge=2592000\n[Jedis-package]: https://search.maven.org/artifact/redis.clients/jedis\n\n[nredisstack-url]: https://github.com/redis/nredisstack\n[nredisstack-stars]: https://img.shields.io/github/stars/redis/nredisstack.svg?style=social&amp;label=Star&amp;maxAge=2592000\n[nredisstack-package]: https://www.nuget.org/packages/nredisstack/\n\n[node-redis-url]: https://github.com/redis/node-redis\n[node-redis-stars]: https://img.shields.io/github/stars/redis/node-redis.svg?style=social&amp;label=Star&amp;maxAge=2592000\n[node-redis-package]: https://www.npmjs.com/package/redis\n\n[redis-om-python-url]: https://github.com/redis/redis-om-python\n[redis-om-python-author]: https://redis.com\n[redis-om-python-stars]: https://img.shields.io/github/stars/redis/redis-om-python.svg?style=social&amp;label=Star&amp;maxAge=2592000\n\n[redisearch-go-url]: https://github.com/RediSearch/redisearch-go\n[redisearch-go-author]: https://redis.com\n[redisearch-go-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-go.svg?style=social&amp;label=Star&amp;maxAge=2592000\n\n[redisearch-api-rs-url]: https://github.com/RediSearch/redisearch-api-rs\n[redisearch-api-rs-author]: https://redis.com\n[redisearch-api-rs-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-api-rs.svg?style=social&amp;label=Star&amp;maxAge=2592000"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: bbad043e-7781-46ab-b542-079d2921150e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"bbad043e-7781-46ab-b542-079d2921150e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb\", \"title\": \"Redis\", \"tokens\": 2363, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2aa7946d-a00f-4f81-bdd6-e48e11274ff7\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb\", \"title\": \"Redis\", \"tokens\": 2363, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"b08d2ce2199cb0ef05b32554508f11d39da0689c732dce096a9cbf3dd901eca0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"33bfa666-f704-4905-b5d8-30be7abd894f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb\", \"title\": \"Redis\", \"tokens\": 2363, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"9b21f1fabb3bcd7cfc5b7c63fb1a4bee3ce73426fe0a6272cc4f44e2fa77c13e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6477, \"end_char_idx\": 9609, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2aa7946d-a00f-4f81-bdd6-e48e11274ff7", "doc_id": "2aa7946d-a00f-4f81-bdd6-e48e11274ff7", "ref_doc_id": "2aa7946d-a00f-4f81-bdd6-e48e11274ff7", "chroma:document": "### Deployment Options\n\nThere are many ways to deploy Redis with RediSearch. The easiest way to get started is to use Docker, but there are are many potential options for deployment such as\n\n- [Redis Cloud](https://redis.com/redis-enterprise-cloud/overview/)\n- Cloud marketplaces: [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-e6y7ork67pjwg?sr=0-2&ref_=beagle&applicationId=AWSMPContessa), [Google Marketplace](https://console.cloud.google.com/marketplace/details/redislabs-public/redis-enterprise?pli=1), or [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/garantiadata.redis_enterprise_1sp_public_preview?tab=Overview)\n- On-premise: [Redis Enterprise Software](https://redis.com/redis-enterprise-software/overview/)\n- Kubernetes: [Redis Enterprise Software on Kubernetes](https://docs.redis.com/latest/kubernetes/)\n- [Docker (RediSearch)](https://hub.docker.com/r/redislabs/redisearch)\n- [Docker (Redis Stack)](https://hub.docker.com/r/redis/redis-stack)\n\n\n### Cluster support\n\nRediSearch has a distributed cluster version that scales to billions of documents across hundreds of servers. At the moment, distributed RediSearch is available as part of [Redis Enterprise Cloud](https://redis.com/redis-enterprise-cloud/overview/) and [Redis Enterprise Software](https://redis.com/redis-enterprise-software/overview/).\n\nSee [RediSearch on Redis Enterprise](https://redis.com/modules/redisearch/) for more information.\n\n### Examples\n\n- [Product Search](https://github.com/RedisVentures/redis-product-search) - eCommerce product search (with image and text)\n- [Product Recommendations with DocArray / Jina](https://github.com/jina-ai/product-recommendation-redis-docarray) - Content-based product recommendations example with Redis and DocArray.\n- [Redis VSS in RecSys](https://github.com/RedisVentures/Redis-Recsys) - 3 end-to-end Redis & NVIDIA Merlin Recommendation System Architectures.\n- [Azure OpenAI Embeddings Q&A](https://github.com/ruoccofabrizio/azure-open-ai-embeddings-qna) - OpenAI and Redis as a Q&A service on Azure.\n- [ArXiv Paper Search](https://github.com/RedisVentures/redis-arXiv-search) - Semantic search over arXiv scholarly papers\n\n\n### More Resources\n\nFor more information on how to use Redis as a vector database, check out the following resources:\n\n- [Redis Vector Similarity Docs](https://redis.io/docs/stack/search/reference/vectors/) - Redis official docs for Vector Search.\n- [Redis-py Search Docs](https://redis.readthedocs.io/en/latest/redismodules.html#redisearch-commands) - Redis-py client library docs for RediSearch.\n- [Vector Similarity Search: From Basics to Production](https://mlops.community/vector-similarity-search-from-basics-to-production/) - Introductory blog post to VSS and Redis as a VectorDB.\n- [AI-Powered Document Search](https://datasciencedojo.com/blog/ai-powered-document-search/) - Blog post covering AI Powered Document Search Use Cases & Architectures.\n- [Vector Database Benchmarks](https://jina.ai/news/benchmark-vector-search-databases-with-one-million-data/) - Jina AI VectorDB benchmarks comparing Redis against others."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 0b710ae6-bb0b-41de-8014-33a33a86d175
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0b710ae6-bb0b-41de-8014-33a33a86d175\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e0f3c006-85d6-4d74-90c3-543c7afc102d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1ea90e9d-d264-451b-82cf-59f0972d5d1b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d570b3617e31f41433e5b1d8fd413e18a61b6b06e39832d89f4fc2cfe4ddd3fe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3027, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "ref_doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "chroma:document": "# Using Redis as a Vector Database with OpenAI\n\nThis notebook provides an introduction to using Redis as a vector database with OpenAI embeddings. Redis is a scalable, real-time database that can be used as a vector database when using the [RediSearch Module](https://oss.redislabs.com/redisearch/). The RediSearch module allows you to index and search for vectors in Redis. This notebook will show you how to use the RediSearch module to index and search for vectors created by using the OpenAI API and stored in Redis.\n\n### What is Redis?\n\nMost developers from a web services background are probably familiar with Redis. At it's core, Redis is an open-source key-value store that can be used as a cache, message broker, and database. Developers choice Redis because it is fast, has a large ecosystem of client libraries, and has been deployed by major enterprises for years.\n\nIn addition to the traditional uses of Redis. Redis also provides [Redis Modules](https://redis.io/modules) which are a way to extend Redis with new data types and commands. Example modules include [RedisJSON](https://redis.io/docs/stack/json/), [RedisTimeSeries](https://redis.io/docs/stack/timeseries/), [RedisBloom](https://redis.io/docs/stack/bloom/) and [RediSearch](https://redis.io/docs/stack/search/).\n\n### What is RediSearch?\n\nRediSearch is a [Redis module](https://redis.io/modules) that provides querying, secondary indexing, full-text search and vector search for Redis. To use RediSearch, you first declare indexes on your Redis data. You can then use the RediSearch clients to query that data. For more information on the feature set of RediSearch, see the [README](./README.md) or the [RediSearch documentation](https://redis.io/docs/stack/search/).\n\n### Deployment options\n\nThere are a number of ways to deploy Redis. For local development, the quickest method is to use the [Redis Stack docker container](https://hub.docker.com/r/redis/redis-stack) which we will use here. Redis Stack contains a number of Redis modules that can be used together to create a fast, multi-model data store and query engine.\n\nFor production use cases, The easiest way to get started is to use the [Redis Cloud](https://redislabs.com/redis-enterprise-cloud/overview/) service. Redis Cloud is a fully managed Redis service. You can also deploy Redis on your own infrastructure using [Redis Enterprise](https://redislabs.com/redis-enterprise/overview/). Redis Enterprise is a fully managed Redis service that can be deployed in kubernetes, on-premises or in the cloud.\n\nAdditionally, every major cloud provider ([AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-e6y7ork67pjwg?sr=0-2&ref_=beagle&applicationId=AWSMPContessa), [Google Marketplace](https://console.cloud.google.com/marketplace/details/redislabs-public/redis-enterprise?pli=1), or [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/garantiadata.redis_enterprise_1sp_public_preview?tab=Overview)) offers Redis Enterprise in a marketplace offering."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 1ea90e9d-d264-451b-82cf-59f0972d5d1b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"1ea90e9d-d264-451b-82cf-59f0972d5d1b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e0f3c006-85d6-4d74-90c3-543c7afc102d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0b710ae6-bb0b-41de-8014-33a33a86d175\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"f12a9bd3e27a18166f0027474e77a1d5d03b7a5c1c41d4a3e85aefd6888a96b3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dd124470-5e0f-48ca-8856-95a2b0fea4fa\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2a79989f01b54851ba761c4f72e140fa361cc4be1f7d2f59f6bff5623f6332ec\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3031, \"end_char_idx\": 5765, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "ref_doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "chroma:document": "## Prerequisites\n\nBefore we start this project, we need to set up the following:\n\n* start a Redis database with RediSearch (redis-stack)\n* install libraries\n    * [Redis-py](https://github.com/redis/redis-py)\n* get your [OpenAI API key](https://beta.openai.com/account/api-keys)\n\n===========================================================\n\n### Start Redis\n\nTo keep this example simple, we will use the Redis Stack docker container which we can start as follows\n\n```bash\n$ docker-compose up -d\n```\n\nThis also includes the [RedisInsight](https://redis.com/redis-enterprise/redis-insight/) GUI for managing your Redis database which you can view at [http://localhost:8001](http://localhost:8001) once you start the docker container.\n\nYou're all set up and ready to go! Next, we import and create our client for communicating with the Redis database we just created.\n\n## Install Requirements\n\nRedis-Py is the python client for communicating with Redis. We will use this to communicate with our Redis-stack database. \n\n\n```python\n! pip install redis wget pandas openai\n```\n\n===========================================================\n## Prepare your OpenAI API key\n\nThe `OpenAI API key` is used for vectorization of query data.\n\nIf you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n\nOnce you get your key, please add it to your environment variables as `OPENAI_API_KEY` by using following command:\n\n\n```python\n! export OPENAI_API_KEY=\"your API key\"\n```\n\n\n```python\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\nimport os\nimport openai\n\n# Note. alternatively you can set a temporary env variable like this:\n# os.environ[\"OPENAI_API_KEY\"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    openai.api_key = os.getenv(\"OPENAI_API_KEY\")\n    print (\"OPENAI_API_KEY is ready\")\nelse:\n    print (\"OPENAI_API_KEY environment variable not found\")\n```\n\n    OPENAI_API_KEY is ready\n\n\n## Load data\n\nIn this section we'll load embedded data that has already been converted into vectors. We'll use this data to create an index in Redis and then search for similar vectors.\n\n\n```python\nimport sys\nimport numpy as np\nimport pandas as pd\nfrom typing import List\n\n# use helper function in nbutils.py to download and read the data\n# this should take from 5-10 min to run\nif os.getcwd() not in sys.path:\n    sys.path.append(os.getcwd())\nimport nbutils\n\nnbutils.download_wikipedia_data()\ndata = nbutils.read_wikipedia_data()\n\ndata.head()\n```\n\n    File Downloaded"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: dd124470-5e0f-48ca-8856-95a2b0fea4fa
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"dd124470-5e0f-48ca-8856-95a2b0fea4fa\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e0f3c006-85d6-4d74-90c3-543c7afc102d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1ea90e9d-d264-451b-82cf-59f0972d5d1b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"9f09fccf3f16dd31a15ad373413be01a4367092852784baecb0265e294d78a38\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dea63753-3038-4d5f-a0ba-86a5a1250f95\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8cd0a9c9c8bed21afa4c25e5abe679fb75283551f60ad360179fa48e2e466a3e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4518, \"end_char_idx\": 6782, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "ref_doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "chroma:document": "```python\n! export OPENAI_API_KEY=\"your API key\"\n```\n\n\n```python\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\nimport os\nimport openai\n\n# Note. alternatively you can set a temporary env variable like this:\n# os.environ[\"OPENAI_API_KEY\"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    openai.api_key = os.getenv(\"OPENAI_API_KEY\")\n    print (\"OPENAI_API_KEY is ready\")\nelse:\n    print (\"OPENAI_API_KEY environment variable not found\")\n```\n\n    OPENAI_API_KEY is ready\n\n\n## Load data\n\nIn this section we'll load embedded data that has already been converted into vectors. We'll use this data to create an index in Redis and then search for similar vectors.\n\n\n```python\nimport sys\nimport numpy as np\nimport pandas as pd\nfrom typing import List\n\n# use helper function in nbutils.py to download and read the data\n# this should take from 5-10 min to run\nif os.getcwd() not in sys.path:\n    sys.path.append(os.getcwd())\nimport nbutils\n\nnbutils.download_wikipedia_data()\ndata = nbutils.read_wikipedia_data()\n\ndata.head()\n```\n\n    File Downloaded\n\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>url</th>\n      <th>title</th>\n      <th>text</th>\n      <th>title_vector</th>\n      <th>content_vector</th>\n      <th>vector_id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>https://simple.wikipedia.org/wiki/April</td>\n      <td>April</td>\n      <td>April is the fourth month of the year in the J...</td>\n      <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n      <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>https://simple.wikipedia.org/wiki/August</td>\n      <td>August</td>\n      <td>August (Aug.)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: dea63753-3038-4d5f-a0ba-86a5a1250f95
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"dea63753-3038-4d5f-a0ba-86a5a1250f95\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e0f3c006-85d6-4d74-90c3-543c7afc102d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dd124470-5e0f-48ca-8856-95a2b0fea4fa\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"04631ed923641cbd6b693a87c9df5d6bd071760be3fd82d5cbe7553c01255924\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"17015c72-b4a3-4065-8476-80f181acc532\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a38d440fd46e3d8ec15186b41e85bf1941379cf88d54866f98815a4eabb1d814\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5771, \"end_char_idx\": 7813, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "ref_doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "chroma:document": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>url</th>\n      <th>title</th>\n      <th>text</th>\n      <th>title_vector</th>\n      <th>content_vector</th>\n      <th>vector_id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>https://simple.wikipedia.org/wiki/April</td>\n      <td>April</td>\n      <td>April is the fourth month of the year in the J...</td>\n      <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n      <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>https://simple.wikipedia.org/wiki/August</td>\n      <td>August</td>\n      <td>August (Aug.) is the eighth month of the year ...</td>\n      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>6</td>\n      <td>https://simple.wikipedia.org/wiki/Art</td>\n      <td>Art</td>\n      <td>Art is a creative activity that expresses imag...</td>\n      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>8</td>\n      <td>https://simple.wikipedia.org/wiki/A</td>\n      <td>A</td>\n      <td>A or a is the first letter of the English alph...</td>\n      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n      <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>9</td>\n      <td>https://simple.wikipedia.org/wiki/Air</td>\n      <td>Air</td>\n      <td>Air refers to the Earth's atmosphere."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 17015c72-b4a3-4065-8476-80f181acc532
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"17015c72-b4a3-4065-8476-80f181acc532\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e0f3c006-85d6-4d74-90c3-543c7afc102d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dea63753-3038-4d5f-a0ba-86a5a1250f95\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"d90c715b264776998131e75bb9ab0ca7a5ba3cccd1fef786e7d83411ca4327d6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2fddba5b-ffde-45bf-8a4b-f1270ff11663\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a608461bed49331890805a388d9965aafc86c90bcef890b0aa3e983daf0e5a7a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6783, \"end_char_idx\": 8997, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "ref_doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "chroma:document": "is the eighth month of the year ...</td>\n      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>6</td>\n      <td>https://simple.wikipedia.org/wiki/Art</td>\n      <td>Art</td>\n      <td>Art is a creative activity that expresses imag...</td>\n      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>8</td>\n      <td>https://simple.wikipedia.org/wiki/A</td>\n      <td>A</td>\n      <td>A or a is the first letter of the English alph...</td>\n      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n      <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>9</td>\n      <td>https://simple.wikipedia.org/wiki/Air</td>\n      <td>Air</td>\n      <td>Air refers to the Earth's atmosphere. Air is a...</td>\n      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n      <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n## Connect to Redis\n\nNow that we have our Redis database running, we can connect to it using the Redis-py client. We will use the default host and port for the Redis database which is `localhost:6379`.\n\n\n\n\n```python\nimport redis\nfrom redis.commands.search.indexDefinition import (\n    IndexDefinition,\n    IndexType\n)\nfrom redis.commands.search.query import Query\nfrom redis.commands.search.field import (\n    TextField,\n    VectorField\n)\n\nREDIS_HOST =  \"localhost\"\nREDIS_PORT = 6379\nREDIS_PASSWORD = \"\" # default for passwordless Redis\n\n# Connect to Redis\nredis_client = redis.Redis(\n    host=REDIS_HOST,\n    port=REDIS_PORT,\n    password=REDIS_PASSWORD\n)\nredis_client.ping()\n```\n\n\n\n\n    True\n\n\n\n## Creating a Search Index in Redis\n\nThe below cells will show how to specify and create a search index in Redis. We will:\n\n1. Set some constants for defining our index like the distance metric and the index name\n2. Define the index schema with RediSearch fields\n3. Create the index"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 2fddba5b-ffde-45bf-8a4b-f1270ff11663
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"2fddba5b-ffde-45bf-8a4b-f1270ff11663\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e0f3c006-85d6-4d74-90c3-543c7afc102d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"17015c72-b4a3-4065-8476-80f181acc532\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"57e2fb0a9a329b51ab6508b9ccae3e2d0b863a3afb9dafffdb8c8f2ff91376e5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7f977bc2-aba7-41e1-9748-73c0965650d2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2aef452339ec5229e17ec0e24316586af693499bb31ac7542bce86a3639e450d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7814, \"end_char_idx\": 10752, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "ref_doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "chroma:document": "Air is a...</td>\n      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n      <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n## Connect to Redis\n\nNow that we have our Redis database running, we can connect to it using the Redis-py client. We will use the default host and port for the Redis database which is `localhost:6379`.\n\n\n\n\n```python\nimport redis\nfrom redis.commands.search.indexDefinition import (\n    IndexDefinition,\n    IndexType\n)\nfrom redis.commands.search.query import Query\nfrom redis.commands.search.field import (\n    TextField,\n    VectorField\n)\n\nREDIS_HOST =  \"localhost\"\nREDIS_PORT = 6379\nREDIS_PASSWORD = \"\" # default for passwordless Redis\n\n# Connect to Redis\nredis_client = redis.Redis(\n    host=REDIS_HOST,\n    port=REDIS_PORT,\n    password=REDIS_PASSWORD\n)\nredis_client.ping()\n```\n\n\n\n\n    True\n\n\n\n## Creating a Search Index in Redis\n\nThe below cells will show how to specify and create a search index in Redis. We will:\n\n1. Set some constants for defining our index like the distance metric and the index name\n2. Define the index schema with RediSearch fields\n3. Create the index\n\n\n```python\n# Constants\nVECTOR_DIM = len(data['title_vector'][0]) # length of the vectors\nVECTOR_NUMBER = len(data)                 # initial number of vectors\nINDEX_NAME = \"embeddings-index\"           # name of the search index\nPREFIX = \"doc\"                            # prefix for the document keys\nDISTANCE_METRIC = \"COSINE\"                # distance metric for the vectors (ex. COSINE, IP, L2)\n```\n\n\n```python\n# Define RediSearch fields for each of the columns in the dataset\ntitle = TextField(name=\"title\")\nurl = TextField(name=\"url\")\ntext = TextField(name=\"text\")\ntitle_embedding = VectorField(\"title_vector\",\n    \"FLAT\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER,\n    }\n)\ntext_embedding = VectorField(\"content_vector\",\n    \"FLAT\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER,\n    }\n)\nfields = [title, url, text, title_embedding, text_embedding]\n```\n\n\n```python\n# Check if index exists\ntry:\n    redis_client.ft(INDEX_NAME).info()\n    print(\"Index already exists\")\nexcept:\n    # Create RediSearch Index\n    redis_client.ft(INDEX_NAME).create_index(\n        fields = fields,\n        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n)\n```\n\n## Load Documents into the Index\n\nNow that we have a search index, we can load documents into it. We will use the same documents we used in the previous examples. In Redis, either the HASH or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The below cells will show how to load documents into the index."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 7f977bc2-aba7-41e1-9748-73c0965650d2
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"7f977bc2-aba7-41e1-9748-73c0965650d2\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e0f3c006-85d6-4d74-90c3-543c7afc102d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2fddba5b-ffde-45bf-8a4b-f1270ff11663\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"e1132ed715785cd6b2613ba802c4e18e1417baabdde0c713f7f0e56e917b3ce2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"12d86870-d5cf-49c9-bfa3-7a0f422f7735\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4a0d2119b71c60a4b5306fee9f383594b7e682068d2543563c32df6bf2bd7628\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9403, \"end_char_idx\": 11963, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "ref_doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "chroma:document": "```python\n# Define RediSearch fields for each of the columns in the dataset\ntitle = TextField(name=\"title\")\nurl = TextField(name=\"url\")\ntext = TextField(name=\"text\")\ntitle_embedding = VectorField(\"title_vector\",\n    \"FLAT\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER,\n    }\n)\ntext_embedding = VectorField(\"content_vector\",\n    \"FLAT\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER,\n    }\n)\nfields = [title, url, text, title_embedding, text_embedding]\n```\n\n\n```python\n# Check if index exists\ntry:\n    redis_client.ft(INDEX_NAME).info()\n    print(\"Index already exists\")\nexcept:\n    # Create RediSearch Index\n    redis_client.ft(INDEX_NAME).create_index(\n        fields = fields,\n        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n)\n```\n\n## Load Documents into the Index\n\nNow that we have a search index, we can load documents into it. We will use the same documents we used in the previous examples. In Redis, either the HASH or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The below cells will show how to load documents into the index.\n\n\n```python\ndef index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):\n    records = documents.to_dict(\"records\")\n    for doc in records:\n        key = f\"{prefix}:{str(doc['id'])}\"\n\n        # create byte vectors for title and content\n        title_embedding = np.array(doc[\"title_vector\"], dtype=np.float32).tobytes()\n        content_embedding = np.array(doc[\"content_vector\"], dtype=np.float32).tobytes()\n\n        # replace list of floats with byte vectors\n        doc[\"title_vector\"] = title_embedding\n        doc[\"content_vector\"] = content_embedding\n\n        client.hset(key, mapping = doc)\n```\n\n\n```python\nindex_documents(redis_client, PREFIX, data)\nprint(f\"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}\")\n```\n\n    Loaded 25000 documents in Redis search index with name: embeddings-index\n\n\n## Simple Vector Search Queries with OpenAI Query Embeddings\n\nNow that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 12d86870-d5cf-49c9-bfa3-7a0f422f7735
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"12d86870-d5cf-49c9-bfa3-7a0f422f7735\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e0f3c006-85d6-4d74-90c3-543c7afc102d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7f977bc2-aba7-41e1-9748-73c0965650d2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"0d347f4355c6b4eedb1ffc51149d5c237b5cc4d80796b7aa1925d1ee41b7dc1d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"eba201ef-4444-40c6-9cb6-9e40fc0e8a0d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0bb1516120e00d61d536bb174f6c12726c16166080eef2402122c96f1832e3ab\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10755, \"end_char_idx\": 13727, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "ref_doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "chroma:document": "```python\ndef index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):\n    records = documents.to_dict(\"records\")\n    for doc in records:\n        key = f\"{prefix}:{str(doc['id'])}\"\n\n        # create byte vectors for title and content\n        title_embedding = np.array(doc[\"title_vector\"], dtype=np.float32).tobytes()\n        content_embedding = np.array(doc[\"content_vector\"], dtype=np.float32).tobytes()\n\n        # replace list of floats with byte vectors\n        doc[\"title_vector\"] = title_embedding\n        doc[\"content_vector\"] = content_embedding\n\n        client.hset(key, mapping = doc)\n```\n\n\n```python\nindex_documents(redis_client, PREFIX, data)\nprint(f\"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}\")\n```\n\n    Loaded 25000 documents in Redis search index with name: embeddings-index\n\n\n## Simple Vector Search Queries with OpenAI Query Embeddings\n\nNow that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database.\n\n\n```python\ndef search_redis(\n    redis_client: redis.Redis,\n    user_query: str,\n    index_name: str = \"embeddings-index\",\n    vector_field: str = \"title_vector\",\n    return_fields: list = [\"title\", \"url\", \"text\", \"vector_score\"],\n    hybrid_fields = \"*\",\n    k: int = 20,\n    print_results: bool = True,\n) -> List[dict]:\n\n    # Creates embedding vector from user query\n    embedded_query = openai.Embedding.create(input=user_query,\n                                            model=\"text-embedding-3-small\",\n                                            )[\"data\"][0]['embedding']\n\n    # Prepare the Query\n    base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'\n    query = (\n        Query(base_query)\n         .return_fields(*return_fields)\n         .sort_by(\"vector_score\")\n         .paging(0, k)\n         .dialect(2)\n    )\n    params_dict = {\"vector\": np.array(embedded_query).astype(dtype=np.float32).tobytes()}\n\n    # perform vector search\n    results = redis_client.ft(index_name).search(query, params_dict)\n    if print_results:\n        for i, article in enumerate(results.docs):\n            score = 1 - float(article.vector_score)\n            print(f\"{i}. {article.title} (Score: {round(score ,3) })\")\n    return results.docs\n```\n\n\n```python\n# For using OpenAI to generate query embedding\nresults = search_redis(redis_client, 'modern art in Europe', k=10)\n```\n\n    0. Museum of Modern Art (Score: 0.875)\n    1. Western Europe (Score: 0.868)\n    2. Renaissance art (Score: 0.864)\n    3. Pop art (Score: 0.86)\n    4. Northern Europe (Score: 0.855)\n    5. Hellenistic art (Score: 0.853)\n    6. Modernist literature (Score: 0.847)\n    7. Art film (Score: 0.843)\n    8. Central Europe (Score: 0.843)\n    9. European (Score: 0.841)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: eba201ef-4444-40c6-9cb6-9e40fc0e8a0d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"eba201ef-4444-40c6-9cb6-9e40fc0e8a0d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e0f3c006-85d6-4d74-90c3-543c7afc102d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"12d86870-d5cf-49c9-bfa3-7a0f422f7735\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6b75a9d29ddd5a7b652a6f859b49e020105eeb98194063047412457a444b936d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"979144aa-e737-4c29-bb31-60f1386e3fa2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"559d12af48b71ced4c4154ceb046b72a950acd30db6df770a4129b5e21b40cc3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13234, \"end_char_idx\": 15300, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "ref_doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "chroma:document": "```python\n# For using OpenAI to generate query embedding\nresults = search_redis(redis_client, 'modern art in Europe', k=10)\n```\n\n    0. Museum of Modern Art (Score: 0.875)\n    1. Western Europe (Score: 0.868)\n    2. Renaissance art (Score: 0.864)\n    3. Pop art (Score: 0.86)\n    4. Northern Europe (Score: 0.855)\n    5. Hellenistic art (Score: 0.853)\n    6. Modernist literature (Score: 0.847)\n    7. Art film (Score: 0.843)\n    8. Central Europe (Score: 0.843)\n    9. European (Score: 0.841)\n\n\n\n```python\nresults = search_redis(redis_client, 'Famous battles in Scottish history', vector_field='content_vector', k=10)\n```\n\n    0. Battle of Bannockburn (Score: 0.869)\n    1. Wars of Scottish Independence (Score: 0.861)\n    2. 1651 (Score: 0.853)\n    3. First War of Scottish Independence (Score: 0.85)\n    4. Robert I of Scotland (Score: 0.846)\n    5. 841 (Score: 0.844)\n    6. 1716 (Score: 0.844)\n    7. 1314 (Score: 0.837)\n    8. 1263 (Score: 0.836)\n    9. William Wallace (Score: 0.835)\n\n\n## Hybrid Queries with Redis\n\nThe previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the below example, we will combine vector search with full text search.\n\n\n```python\ndef create_hybrid_field(field_name: str, value: str) -> str:\n    return f'@{field_name}:\"{value}\"'\n\n# search the content vector for articles about famous battles in Scottish history and only include results with Scottish in the title\nresults = search_redis(redis_client,\n                       \"Famous battles in Scottish history\",\n                       vector_field=\"title_vector\",\n                       k=5,\n                       hybrid_fields=create_hybrid_field(\"title\", \"Scottish\")\n                       )\n```\n\n    0. First War of Scottish Independence (Score: 0.892)\n    1. Wars of Scottish Independence (Score: 0.889)\n    2. Second War of Scottish Independence (Score: 0.879)\n    3. List of Scottish monarchs (Score: 0.873)\n    4. Scottish Borders (Score: 0.863)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 979144aa-e737-4c29-bb31-60f1386e3fa2
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"979144aa-e737-4c29-bb31-60f1386e3fa2\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e0f3c006-85d6-4d74-90c3-543c7afc102d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"eba201ef-4444-40c6-9cb6-9e40fc0e8a0d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"5707566de7a33eb1fdb41d81cafaaf0426027e4ac6f0d64985a0d55f326d5dca\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"047a5976-c7ea-4c8e-ad9c-321e735451e8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3f934ea4c249ac489b58c50e639a9dec48926fc9d23d9da862fa9e7d9a05eab7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14227, \"end_char_idx\": 17343, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "ref_doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "chroma:document": "## Hybrid Queries with Redis\n\nThe previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the below example, we will combine vector search with full text search.\n\n\n```python\ndef create_hybrid_field(field_name: str, value: str) -> str:\n    return f'@{field_name}:\"{value}\"'\n\n# search the content vector for articles about famous battles in Scottish history and only include results with Scottish in the title\nresults = search_redis(redis_client,\n                       \"Famous battles in Scottish history\",\n                       vector_field=\"title_vector\",\n                       k=5,\n                       hybrid_fields=create_hybrid_field(\"title\", \"Scottish\")\n                       )\n```\n\n    0. First War of Scottish Independence (Score: 0.892)\n    1. Wars of Scottish Independence (Score: 0.889)\n    2. Second War of Scottish Independence (Score: 0.879)\n    3. List of Scottish monarchs (Score: 0.873)\n    4. Scottish Borders (Score: 0.863)\n\n\n\n```python\n# run a hybrid query for articles about Art in the title vector and only include results with the phrase \"Leonardo da Vinci\" in the text\nresults = search_redis(redis_client,\n                       \"Art\",\n                       vector_field=\"title_vector\",\n                       k=5,\n                       hybrid_fields=create_hybrid_field(\"text\", \"Leonardo da Vinci\")\n                       )\n\n# find specific mention of Leonardo da Vinci in the text that our full-text-search query returned\nmention = [sentence for sentence in results[0].text.split(\"\\n\") if \"Leonardo da Vinci\" in sentence][0]\nmention\n```\n\n    0. Art (Score: 1.0)\n    1. Paint (Score: 0.896)\n    2. Renaissance art (Score: 0.88)\n    3. Painting (Score: 0.874)\n    4. Renaissance (Score: 0.846)\n\n\n\n\n\n    'In Europe, after the Middle Ages, there was a \"Renaissance\" which means \"rebirth\". People rediscovered science and artists were allowed to paint subjects other than religious subjects. People like Michelangelo and Leonardo da Vinci still painted religious pictures, but they also now could paint mythological pictures too. These artists also invented perspective where things in the distance look smaller in the picture. This was new because in the Middle Ages people would paint all the figures close up and just overlapping each other. These artists used nudity regularly in their art.'\n\n\n\n## HNSW Index\n\nUp until now, we've been using the ``FLAT`` or \"brute-force\" index to run our queries. Redis also supports the ``HNSW`` index which is a fast, approximate index. The ``HNSW`` index is a graph-based index that uses a hierarchical navigable small world graph to store vectors. The ``HNSW`` index is a good choice for large datasets where you want to run approximate queries.\n\n``HNSW`` will take longer to build and consume more memory for most cases than ``FLAT`` but will be faster to run queries on, especially for large datasets.\n\nThe following cells will show how to create an ``HNSW`` index and run queries with it using the same data as before."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 047a5976-c7ea-4c8e-ad9c-321e735451e8
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"047a5976-c7ea-4c8e-ad9c-321e735451e8\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e0f3c006-85d6-4d74-90c3-543c7afc102d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"979144aa-e737-4c29-bb31-60f1386e3fa2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"58f452b3738cc433fa12b11e9929d538fb1cd110c791f3bbb358041a2563983c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1033a569-4384-4cc4-aee7-13f0d7e2a51c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"351b8cb5df5b075c6e22471bf20d5a9100892b9360be4b27030167d3e9e7b9d0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16087, \"end_char_idx\": 18745, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "ref_doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "chroma:document": "'In Europe, after the Middle Ages, there was a \"Renaissance\" which means \"rebirth\". People rediscovered science and artists were allowed to paint subjects other than religious subjects. People like Michelangelo and Leonardo da Vinci still painted religious pictures, but they also now could paint mythological pictures too. These artists also invented perspective where things in the distance look smaller in the picture. This was new because in the Middle Ages people would paint all the figures close up and just overlapping each other. These artists used nudity regularly in their art.'\n\n\n\n## HNSW Index\n\nUp until now, we've been using the ``FLAT`` or \"brute-force\" index to run our queries. Redis also supports the ``HNSW`` index which is a fast, approximate index. The ``HNSW`` index is a graph-based index that uses a hierarchical navigable small world graph to store vectors. The ``HNSW`` index is a good choice for large datasets where you want to run approximate queries.\n\n``HNSW`` will take longer to build and consume more memory for most cases than ``FLAT`` but will be faster to run queries on, especially for large datasets.\n\nThe following cells will show how to create an ``HNSW`` index and run queries with it using the same data as before.\n\n\n```python\n# re-define RediSearch vector fields to use HNSW index\ntitle_embedding = VectorField(\"title_vector\",\n    \"HNSW\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER\n    }\n)\ntext_embedding = VectorField(\"content_vector\",\n    \"HNSW\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER\n    }\n)\nfields = [title, url, text, title_embedding, text_embedding]\n```\n\n\n```python\nimport time\n# Check if index exists\nHNSW_INDEX_NAME = INDEX_NAME+ \"_HNSW\"\n\ntry:\n    redis_client.ft(HNSW_INDEX_NAME).info()\n    print(\"Index already exists\")\nexcept:\n    # Create RediSearch Index\n    redis_client.ft(HNSW_INDEX_NAME).create_index(\n        fields = fields,\n        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n    )\n\n# since RediSearch creates the index in the background for existing documents, we will wait until\n# indexing is complete before running our queries. Although this is not necessary for the first query,\n# some queries may take longer to run if the index is not fully built. In general, Redis will perform\n# best when adding new documents to existing indices rather than new indices on existing documents.\nwhile redis_client.ft(HNSW_INDEX_NAME).info()[\"indexing\"] == \"1\":\n    time.sleep(5)\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 1033a569-4384-4cc4-aee7-13f0d7e2a51c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"1033a569-4384-4cc4-aee7-13f0d7e2a51c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e0f3c006-85d6-4d74-90c3-543c7afc102d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"047a5976-c7ea-4c8e-ad9c-321e735451e8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"5fd1d54c4034a33ea4948fff888d957f1222c4beb545833fc730d76ab80db802\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5baaa791-caac-44c5-b4ef-cbed9455f3eb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"10d82b05be97afebf2ff236a04b28e8635da2b57e558f661bf4e3eb3f1ebfbd8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17346, \"end_char_idx\": 19231, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "ref_doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "chroma:document": "```python\n# re-define RediSearch vector fields to use HNSW index\ntitle_embedding = VectorField(\"title_vector\",\n    \"HNSW\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER\n    }\n)\ntext_embedding = VectorField(\"content_vector\",\n    \"HNSW\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER\n    }\n)\nfields = [title, url, text, title_embedding, text_embedding]\n```\n\n\n```python\nimport time\n# Check if index exists\nHNSW_INDEX_NAME = INDEX_NAME+ \"_HNSW\"\n\ntry:\n    redis_client.ft(HNSW_INDEX_NAME).info()\n    print(\"Index already exists\")\nexcept:\n    # Create RediSearch Index\n    redis_client.ft(HNSW_INDEX_NAME).create_index(\n        fields = fields,\n        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n    )\n\n# since RediSearch creates the index in the background for existing documents, we will wait until\n# indexing is complete before running our queries. Although this is not necessary for the first query,\n# some queries may take longer to run if the index is not fully built. In general, Redis will perform\n# best when adding new documents to existing indices rather than new indices on existing documents.\nwhile redis_client.ft(HNSW_INDEX_NAME).info()[\"indexing\"] == \"1\":\n    time.sleep(5)\n```\n\n\n```python\nresults = search_redis(redis_client, 'modern art in Europe', index_name=HNSW_INDEX_NAME, k=10)\n```\n\n    0. Western Europe (Score: 0.868)\n    1. Northern Europe (Score: 0.855)\n    2. Central Europe (Score: 0.843)\n    3. European (Score: 0.841)\n    4. Eastern Europe (Score: 0.839)\n    5. Europe (Score: 0.839)\n    6. Western European Union (Score: 0.837)\n    7. Southern Europe (Score: 0.831)\n    8. Western civilization (Score: 0.83)\n    9. Council of Europe (Score: 0.827)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 5baaa791-caac-44c5-b4ef-cbed9455f3eb
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"5baaa791-caac-44c5-b4ef-cbed9455f3eb\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e0f3c006-85d6-4d74-90c3-543c7afc102d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1033a569-4384-4cc4-aee7-13f0d7e2a51c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"14e964cd6f8306781b21d904a8ca1fdc0f100b32727f22168a88be4a47dd63d4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cd8cf02d-f35c-42dd-9c4f-939e8cd95a34\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5b0d7e8666a5b807e3bbea3400b6775cad72d42af45bacc0f92b52f9d959d8db\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17885, \"end_char_idx\": 21190, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "ref_doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "chroma:document": "```python\nimport time\n# Check if index exists\nHNSW_INDEX_NAME = INDEX_NAME+ \"_HNSW\"\n\ntry:\n    redis_client.ft(HNSW_INDEX_NAME).info()\n    print(\"Index already exists\")\nexcept:\n    # Create RediSearch Index\n    redis_client.ft(HNSW_INDEX_NAME).create_index(\n        fields = fields,\n        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n    )\n\n# since RediSearch creates the index in the background for existing documents, we will wait until\n# indexing is complete before running our queries. Although this is not necessary for the first query,\n# some queries may take longer to run if the index is not fully built. In general, Redis will perform\n# best when adding new documents to existing indices rather than new indices on existing documents.\nwhile redis_client.ft(HNSW_INDEX_NAME).info()[\"indexing\"] == \"1\":\n    time.sleep(5)\n```\n\n\n```python\nresults = search_redis(redis_client, 'modern art in Europe', index_name=HNSW_INDEX_NAME, k=10)\n```\n\n    0. Western Europe (Score: 0.868)\n    1. Northern Europe (Score: 0.855)\n    2. Central Europe (Score: 0.843)\n    3. European (Score: 0.841)\n    4. Eastern Europe (Score: 0.839)\n    5. Europe (Score: 0.839)\n    6. Western European Union (Score: 0.837)\n    7. Southern Europe (Score: 0.831)\n    8. Western civilization (Score: 0.83)\n    9. Council of Europe (Score: 0.827)\n\n\n\n```python\n# compare the results of the HNSW index to the FLAT index and time both queries\ndef time_queries(iterations: int = 10):\n    print(\" ----- Flat Index ----- \")\n    t0 = time.time()\n    for i in range(iterations):\n        results_flat = search_redis(redis_client, 'modern art in Europe', k=10, print_results=False)\n    t0 = (time.time() - t0) / iterations\n    results_flat = search_redis(redis_client, 'modern art in Europe', k=10, print_results=True)\n    print(f\"Flat index query time: {round(t0, 3)} seconds\\n\")\n    time.sleep(1)\n    print(\" ----- HNSW Index ------ \")\n    t1 = time.time()\n    for i in range(iterations):\n        results_hnsw = search_redis(redis_client, 'modern art in Europe', index_name=HNSW_INDEX_NAME, k=10, print_results=False)\n    t1 = (time.time() - t1) / iterations\n    results_hnsw = search_redis(redis_client, 'modern art in Europe', index_name=HNSW_INDEX_NAME, k=10, print_results=True)\n    print(f\"HNSW index query time: {round(t1, 3)} seconds\")\n    print(\" ------------------------ \")\ntime_queries()\n```\n\n     ----- Flat Index ----- \n    0. Museum of Modern Art (Score: 0.875)\n    1. Western Europe (Score: 0.867)\n    2. Renaissance art (Score: 0.864)\n    3. Pop art (Score: 0.861)\n    4. Northern Europe (Score: 0.855)\n    5. Hellenistic art (Score: 0.853)\n    6. Modernist literature (Score: 0.847)\n    7. Art film (Score: 0.843)\n    8. Central Europe (Score: 0.843)\n    9. Art (Score: 0.842)\n    Flat index query time: 0.263 seconds\n    \n     ----- HNSW Index ------ \n    0. Western Europe (Score: 0.867)\n    1. Northern Europe (Score: 0.855)\n    2. Central Europe (Score: 0.843)\n    3. European (Score: 0.841)\n    4. Eastern Europe (Score: 0.839)\n    5. Europe (Score: 0.839)\n    6. Western European Union (Score: 0.837)\n    7. Southern Europe (Score: 0.831)\n    8. Western civilization (Score: 0.83)\n    9. Council of Europe (Score: 0.827)\n    HNSW index query time: 0.129 seconds\n     ------------------------"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: cd8cf02d-f35c-42dd-9c4f-939e8cd95a34
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"cd8cf02d-f35c-42dd-9c4f-939e8cd95a34\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e0f3c006-85d6-4d74-90c3-543c7afc102d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5baaa791-caac-44c5-b4ef-cbed9455f3eb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"4f6925f7dae29ce25a580ccd831642b52ee7b2c90aeefef5e0ad0e0893a662d3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21195, \"end_char_idx\": 21209, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "ref_doc_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "chroma:document": "```python\n\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 9244bddd-3360-4744-a991-864322a572d9
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"9244bddd-3360-4744-a991-864322a572d9\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b5843b81-291b-40bc-b969-8753ef31e079\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ffe37e45-f015-476f-ad6b-d69d47540510\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"48771e548cd855d4c03fd7f47a090609cdb397dbaf2db6a8493b3df0c8026bfe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3212, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b5843b81-291b-40bc-b969-8753ef31e079", "doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "ref_doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "chroma:document": "# Running Hybrid VSS Queries with Redis and OpenAI\n\nThis notebook provides an introduction to using Redis as a vector database with OpenAI embeddings and running hybrid queries that combine VSS and lexical search using Redis Query and Search capability. Redis is a scalable, real-time database that can be used as a vector database when using the [RediSearch Module](https://oss.redislabs.com/redisearch/). The Redis Query and Search capability allows you to index and search for vectors in Redis. This notebook will show you how to use the Redis Query and Search to index and search for vectors created by using the OpenAI API and stored in Redis.\n\nHybrid queries combine vector similarity with traditional Redis Query and Search filtering capabilities on GEO, NUMERIC, TAG or TEXT data simplifying application code. A common example of a hybrid query in an e-commerce use case is to find items visually similar to a given query image limited to items available in a GEO location and within a price range.\n\n## Prerequisites\n\nBefore we start this project, we need to set up the following:\n\n* start a Redis database with RediSearch (redis-stack)\n* install libraries\n    * [Redis-py](https://github.com/redis/redis-py)\n* get your [OpenAI API key](https://beta.openai.com/account/api-keys)\n\n===========================================================\n\n### Start Redis\n\nTo keep this example simple, we will use the Redis Stack docker container which we can start as follows\n\n```bash\n$ docker-compose up -d\n```\n\nThis also includes the [RedisInsight](https://redis.com/redis-enterprise/redis-insight/) GUI for managing your Redis database which you can view at [http://localhost:8001](http://localhost:8001) once you start the docker container.\n\nYou're all set up and ready to go! Next, we import and create our client for communicating with the Redis database we just created.\n\n## Install Requirements\n\nRedis-Py is the python client for communicating with Redis. We will use this to communicate with our Redis-stack database. \n\n\n```python\n! pip install redis pandas openai\n\n```\n\n    Defaulting to user installation because normal site-packages is not writeable\n    Requirement already satisfied: redis in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (4.5.4)\n    Requirement already satisfied: pandas in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (2.0.1)\n    Requirement already satisfied: openai in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (0.27.6)\n    Requirement already satisfied: async-timeout>=4.0.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from redis) (4.0.2)\n    Requirement already satisfied: python-dateutil>=2.8.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2.8.2)\n    Requirement already satisfied: pytz>=2020.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2023.3)\n    Requirement already satisfied: tzdata>=2022.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2023.3)\n    Requirement already satisfied: numpy>=1.20.3 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (1.23."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: ffe37e45-f015-476f-ad6b-d69d47540510
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"ffe37e45-f015-476f-ad6b-d69d47540510\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b5843b81-291b-40bc-b969-8753ef31e079\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9244bddd-3360-4744-a991-864322a572d9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"5fe4391cf140269d5190fb08948899485054c02937a5d473a8137e5eb960c365\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"12c53a10-67c9-44f2-90ac-aeded8d4afa7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b5bbc484afc986c9a18fe4c73a4cb013ff3346a538ef8364957199a81b3e8c43\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2024, \"end_char_idx\": 4613, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b5843b81-291b-40bc-b969-8753ef31e079", "doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "ref_doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "chroma:document": "```python\n! pip install redis pandas openai\n\n```\n\n    Defaulting to user installation because normal site-packages is not writeable\n    Requirement already satisfied: redis in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (4.5.4)\n    Requirement already satisfied: pandas in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (2.0.1)\n    Requirement already satisfied: openai in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (0.27.6)\n    Requirement already satisfied: async-timeout>=4.0.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from redis) (4.0.2)\n    Requirement already satisfied: python-dateutil>=2.8.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2.8.2)\n    Requirement already satisfied: pytz>=2020.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2023.3)\n    Requirement already satisfied: tzdata>=2022.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2023.3)\n    Requirement already satisfied: numpy>=1.20.3 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (1.23.4)\n    Requirement already satisfied: requests>=2.20 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (2.28.1)\n    Requirement already satisfied: tqdm in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (4.64.1)\n    Requirement already satisfied: aiohttp in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (3.8.4)\n    Requirement already satisfied: six>=1.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)\n    Requirement already satisfied: charset-normalizer<3,>=2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2.1.1)\n    Requirement already satisfied: idna<4,>=2.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (3.4)\n    Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (1.26.12)\n    Requirement already satisfied: certifi>=2017.4.17 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2022.9.24)\n    Requirement already satisfied: attrs>=17.3.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (23.1.0)\n    Requirement already satisfied: multidict<7."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 12c53a10-67c9-44f2-90ac-aeded8d4afa7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"12c53a10-67c9-44f2-90ac-aeded8d4afa7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b5843b81-291b-40bc-b969-8753ef31e079\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ffe37e45-f015-476f-ad6b-d69d47540510\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"744a50b963670a7a5df564127bba56bd8a4e063cd0ffc345c62100d8c7ba9f6d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2b0c37c8-8365-4d42-a287-214b22ed14f8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ac5d72e261ea856563d0f3e6c0751e53d40e3f5887c8bb35bd939b6770c12dd3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3349, \"end_char_idx\": 5584, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b5843b81-291b-40bc-b969-8753ef31e079", "doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "ref_doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "chroma:document": "28.1)\n    Requirement already satisfied: tqdm in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (4.64.1)\n    Requirement already satisfied: aiohttp in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (3.8.4)\n    Requirement already satisfied: six>=1.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)\n    Requirement already satisfied: charset-normalizer<3,>=2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2.1.1)\n    Requirement already satisfied: idna<4,>=2.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (3.4)\n    Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (1.26.12)\n    Requirement already satisfied: certifi>=2017.4.17 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2022.9.24)\n    Requirement already satisfied: attrs>=17.3.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (23.1.0)\n    Requirement already satisfied: multidict<7.0,>=4.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (6.0.4)\n    Requirement already satisfied: yarl<2.0,>=1.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.9.2)\n    Requirement already satisfied: frozenlist>=1.1.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.3)\n    Requirement already satisfied: aiosignal>=1.1.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.1)\n\n\n===========================================================\n## Prepare your OpenAI API key\n\nThe `OpenAI API key` is used for vectorization of query data.\n\nIf you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n\nOnce you get your key, please add it to your environment variables as `OPENAI_API_KEY` by using following command:"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 2b0c37c8-8365-4d42-a287-214b22ed14f8
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"2b0c37c8-8365-4d42-a287-214b22ed14f8\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b5843b81-291b-40bc-b969-8753ef31e079\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"12c53a10-67c9-44f2-90ac-aeded8d4afa7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"9578ab9e6236f83de7fa47ba03515812c76a4c810c3b8d428fc89e0e4e5377b6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"53dbc9fb-40c8-4841-8fe5-6c7e939d7379\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"54d7a83cada1ca318c4286170ec41e73ccba6ca043acce6f116fe4b991a1a070\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4233, \"end_char_idx\": 6312, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b5843b81-291b-40bc-b969-8753ef31e079", "doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "ref_doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "chroma:document": "20->openai) (1.26.12)\n    Requirement already satisfied: certifi>=2017.4.17 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2022.9.24)\n    Requirement already satisfied: attrs>=17.3.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (23.1.0)\n    Requirement already satisfied: multidict<7.0,>=4.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (6.0.4)\n    Requirement already satisfied: yarl<2.0,>=1.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.9.2)\n    Requirement already satisfied: frozenlist>=1.1.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.3)\n    Requirement already satisfied: aiosignal>=1.1.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.1)\n\n\n===========================================================\n## Prepare your OpenAI API key\n\nThe `OpenAI API key` is used for vectorization of query data.\n\nIf you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n\nOnce you get your key, please add it to your environment variables as `OPENAI_API_KEY` by using following command:\n\n\n```python\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\nimport os\nimport openai\n\nos.environ[\"OPENAI_API_KEY\"] = '<YOUR_OPENAI_API_KEY>'\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    openai.api_key = os.getenv(\"OPENAI_API_KEY\")\n    print (\"OPENAI_API_KEY is ready\")\nelse:\n    print (\"OPENAI_API_KEY environment variable not found\")\n\n```\n\n    OPENAI_API_KEY is ready\n\n\n## Load data\n\nIn this section we'll load and clean an ecommerce dataset. We'll generate embeddings using OpenAI and use this data to create an index in Redis and then search for similar vectors."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 53dbc9fb-40c8-4841-8fe5-6c7e939d7379
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"53dbc9fb-40c8-4841-8fe5-6c7e939d7379\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b5843b81-291b-40bc-b969-8753ef31e079\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2b0c37c8-8365-4d42-a287-214b22ed14f8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"06c8e9ab77553ba89a1202b8133db929cd1eff6c8af40ef7192183c2bdfa8535\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"545e5dae-826e-4ade-9f6c-b30391c0a264\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dd753ee1a3c82a2e076ed454a1897d74cbf852bbf64769b6524519050158bef7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4761, \"end_char_idx\": 7685, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b5843b81-291b-40bc-b969-8753ef31e079", "doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "ref_doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "chroma:document": "0,>=1.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.9.2)\n    Requirement already satisfied: frozenlist>=1.1.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.3)\n    Requirement already satisfied: aiosignal>=1.1.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.1)\n\n\n===========================================================\n## Prepare your OpenAI API key\n\nThe `OpenAI API key` is used for vectorization of query data.\n\nIf you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n\nOnce you get your key, please add it to your environment variables as `OPENAI_API_KEY` by using following command:\n\n\n```python\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\nimport os\nimport openai\n\nos.environ[\"OPENAI_API_KEY\"] = '<YOUR_OPENAI_API_KEY>'\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    openai.api_key = os.getenv(\"OPENAI_API_KEY\")\n    print (\"OPENAI_API_KEY is ready\")\nelse:\n    print (\"OPENAI_API_KEY environment variable not found\")\n\n```\n\n    OPENAI_API_KEY is ready\n\n\n## Load data\n\nIn this section we'll load and clean an ecommerce dataset. We'll generate embeddings using OpenAI and use this data to create an index in Redis and then search for similar vectors.\n\n\n```python\nimport pandas as pd\nimport numpy as np\nfrom typing import List\n\nfrom utils.embeddings_utils import (\n    get_embeddings,\n    distances_from_embeddings,\n    tsne_components_from_embeddings,\n    chart_from_components,\n    indices_of_nearest_neighbors_from_distances,\n)\n\nEMBEDDING_MODEL = \"text-embedding-3-small\"\n\n# load in data and clean data types and drop null rows\ndf = pd.read_csv(\"../../data/styles_2k.csv\", on_bad_lines='skip')\ndf.dropna(inplace=True)\ndf[\"year\"] = df[\"year\"].astype(int)\ndf.info()\n\n# print dataframe\nn_examples = 5\ndf.head(n_examples)\n\n```\n\n    <class 'pandas.core.frame.DataFrame'>\n    Index: 1978 entries, 0 to 1998\n    Data columns (total 10 columns):\n     #   Column              Non-Null Count  Dtype \n    ---  ------              --------------  ----- \n     0   id                  1978 non-null   int64 \n     1   gender              1978 non-null   object\n     2   masterCategory      1978 non-null   object\n     3   subCategory         1978 non-null   object\n     4   articleType         1978 non-null   object\n     5   baseColour          1978 non-null   object\n     6   season              1978 non-null   object\n     7   year                1978 non-null   int64 \n     8   usage               1978 non-null   object\n     9   productDisplayName  1978 non-null   object\n    dtypes: int64(2), object(8)\n    memory usage: 170.0+ KB"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 545e5dae-826e-4ade-9f6c-b30391c0a264
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"545e5dae-826e-4ade-9f6c-b30391c0a264\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b5843b81-291b-40bc-b969-8753ef31e079\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"53dbc9fb-40c8-4841-8fe5-6c7e939d7379\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"30ae67437e33a6d7035398517d20fdec8b1741cec4c93e4501ac7dc277bdf4d0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e56ef2bc-a8ee-4934-b4ad-b45c2857ec9e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8a1838f53f0ebecc073805ab5138940bc791f72e8342f728f05fa6c5cae09e4e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6315, \"end_char_idx\": 9727, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b5843b81-291b-40bc-b969-8753ef31e079", "doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "ref_doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "chroma:document": "```python\nimport pandas as pd\nimport numpy as np\nfrom typing import List\n\nfrom utils.embeddings_utils import (\n    get_embeddings,\n    distances_from_embeddings,\n    tsne_components_from_embeddings,\n    chart_from_components,\n    indices_of_nearest_neighbors_from_distances,\n)\n\nEMBEDDING_MODEL = \"text-embedding-3-small\"\n\n# load in data and clean data types and drop null rows\ndf = pd.read_csv(\"../../data/styles_2k.csv\", on_bad_lines='skip')\ndf.dropna(inplace=True)\ndf[\"year\"] = df[\"year\"].astype(int)\ndf.info()\n\n# print dataframe\nn_examples = 5\ndf.head(n_examples)\n\n```\n\n    <class 'pandas.core.frame.DataFrame'>\n    Index: 1978 entries, 0 to 1998\n    Data columns (total 10 columns):\n     #   Column              Non-Null Count  Dtype \n    ---  ------              --------------  ----- \n     0   id                  1978 non-null   int64 \n     1   gender              1978 non-null   object\n     2   masterCategory      1978 non-null   object\n     3   subCategory         1978 non-null   object\n     4   articleType         1978 non-null   object\n     5   baseColour          1978 non-null   object\n     6   season              1978 non-null   object\n     7   year                1978 non-null   int64 \n     8   usage               1978 non-null   object\n     9   productDisplayName  1978 non-null   object\n    dtypes: int64(2), object(8)\n    memory usage: 170.0+ KB\n\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>gender</th>\n      <th>masterCategory</th>\n      <th>subCategory</th>\n      <th>articleType</th>\n      <th>baseColour</th>\n      <th>season</th>\n      <th>year</th>\n      <th>usage</th>\n      <th>productDisplayName</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>15970</td>\n      <td>Men</td>\n      <td>Apparel</td>\n      <td>Topwear</td>\n      <td>Shirts</td>\n      <td>Navy Blue</td>\n      <td>Fall</td>\n      <td>2011</td>\n      <td>Casual</td>\n      <td>Turtle Check Men Navy Blue Shirt</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>39386</td>\n      <td>Men</td>\n      <td>Apparel</td>\n      <td>Bottomwear</td>\n      <td>Jeans</td>\n      <td>Blue</td>\n      <td>Summer</td>\n      <td>2012</td>\n      <td>Casual</td>\n      <td>Peter England Men Party Blue Jeans</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>59263</td>\n      <td>Women</td>\n      <td>Accessories</td>\n      <td>Watches</td>\n      <td>Watches</td>\n      <td>Silver</td>\n      <td>Winter</td>\n      <td>2016</td>\n      <td>Casual</td>\n      <td>Titan Women Silver Watch</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>21379</td>\n      <td>Men</td>\n      <td>Apparel</td>\n      <td>Bottomwear</td>\n      <td>Track Pants</td>\n      <td>Black</td>\n      <td>Fall</td>\n      <td>2011</td>\n      <td>Casual</td>\n      <td>Manchester United Men Solid Black Track Pants</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>53759</td>\n      <td>Men</td>\n      <td>Apparel</td>\n      <td>Topwear</td>\n      <td>Tshirts</td>\n      <td>Grey</td>\n      <td>Summer</td>\n      <td>2012</td>\n      <td>Casual</td>\n      <td>Puma Men Grey T-shirt</td>\n    </tr>\n  </tbody>\n</table>\n</div>"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: e56ef2bc-a8ee-4934-b4ad-b45c2857ec9e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e56ef2bc-a8ee-4934-b4ad-b45c2857ec9e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b5843b81-291b-40bc-b969-8753ef31e079\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"545e5dae-826e-4ade-9f6c-b30391c0a264\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"9e0660a48e3bbab420a5fef8734244693b8114f102a002d8bfb8e036f58e5410\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"821a352b-4613-4062-933a-39ea0ea10b12\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fbc02fb4f6fbee58d7cafd7b61e743d803df335815c61765236654adeb6db225\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9732, \"end_char_idx\": 12404, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b5843b81-291b-40bc-b969-8753ef31e079", "doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "ref_doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "chroma:document": "```python\ndf[\"product_text\"] = df.apply(lambda row: f\"name {row['productDisplayName']} category {row['masterCategory']} subcategory {row['subCategory']} color {row['baseColour']} gender {row['gender']}\".lower(), axis=1)\ndf.rename({\"id\":\"product_id\"}, inplace=True, axis=1)\n\ndf.info()\n\n```\n\n    <class 'pandas.core.frame.DataFrame'>\n    Index: 1978 entries, 0 to 1998\n    Data columns (total 11 columns):\n     #   Column              Non-Null Count  Dtype \n    ---  ------              --------------  ----- \n     0   product_id          1978 non-null   int64 \n     1   gender              1978 non-null   object\n     2   masterCategory      1978 non-null   object\n     3   subCategory         1978 non-null   object\n     4   articleType         1978 non-null   object\n     5   baseColour          1978 non-null   object\n     6   season              1978 non-null   object\n     7   year                1978 non-null   int64 \n     8   usage               1978 non-null   object\n     9   productDisplayName  1978 non-null   object\n     10  product_text        1978 non-null   object\n    dtypes: int64(2), object(9)\n    memory usage: 185.4+ KB\n\n\n\n```python\n# check out one of the texts we will use to create semantic embeddings\ndf[\"product_text\"][0]\n\n```\n\n\n\n\n    'name turtle check men navy blue shirt category apparel subcategory topwear color navy blue gender men'\n\n\n\n## Connect to Redis\n\nNow that we have our Redis database running, we can connect to it using the Redis-py client. We will use the default host and port for the Redis database which is `localhost:6379`.\n\n\n\n\n```python\nimport redis\nfrom redis.commands.search.indexDefinition import (\n    IndexDefinition,\n    IndexType\n)\nfrom redis.commands.search.query import Query\nfrom redis.commands.search.field import (\n    TagField,\n    NumericField,\n    TextField,\n    VectorField\n)\n\nREDIS_HOST =  \"localhost\"\nREDIS_PORT = 6379\nREDIS_PASSWORD = \"\" # default for passwordless Redis\n\n# Connect to Redis\nredis_client = redis.Redis(\n    host=REDIS_HOST,\n    port=REDIS_PORT,\n    password=REDIS_PASSWORD\n)\nredis_client.ping()\n\n```\n\n\n\n\n    True\n\n\n\n## Creating a Search Index in Redis\n\nThe below cells will show how to specify and create a search index in Redis. We will:\n\n1. Set some constants for defining our index like the distance metric and the index name\n2. Define the index schema with RediSearch fields\n3. Create the index\n\n\n```python\n# Constants\nINDEX_NAME = \"product_embeddings\"           # name of the search index\nPREFIX = \"doc\"                            # prefix for the document keys\nDISTANCE_METRIC = \"L2\"                # distance metric for the vectors (ex. COSINE, IP, L2)\nNUMBER_OF_VECTORS = len(df)\n\n```"}


