# embeddings_queue

created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: d761063e-44e7-41a8-922e-a4d7cd301ad8
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"d761063e-44e7-41a8-922e-a4d7cd301ad8\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"63e6f2b4-c9ee-4985-9d20-03dbcba64098\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"841bbb31bd2c6b12b8e2a0a815f36c672d806f70c2cea477eb765e0078d5d30f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e348c590-62ec-4748-a350-ed4c3e69ac09\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb\", \"title\": \"Using Chroma for Embeddings Search\", \"tokens\": 10702, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7c2b129ad8b2741e27748339c013df87f6bf81982c02f9df61f2ca009c360a53\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 30517, \"end_char_idx\": 32346, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "63e6f2b4-c9ee-4985-9d20-03dbcba64098", "doc_id": "63e6f2b4-c9ee-4985-9d20-03dbcba64098", "ref_doc_id": "63e6f2b4-c9ee-4985-9d20-03dbcba64098", "chroma:document": "```python\ncontent_query_result = query_collection(\n    collection=wikipedia_content_collection,\n    query=\"Famous battles in Scottish history\",\n    max_results=10,\n    dataframe=article_df\n)\ncontent_query_result.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>score</th>\n      <th>title</th>\n      <th>content</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>2923</th>\n      <td>13135</td>\n      <td>0.261328</td>\n      <td>1651</td>\n      <td>\\n\\nEvents \\n January 1 \u2013 Charles II crowned K...</td>\n    </tr>\n    <tr>\n      <th>3694</th>\n      <td>13571</td>\n      <td>0.277058</td>\n      <td>Stirling</td>\n      <td>Stirling () is a city in the middle of Scotlan...</td>\n    </tr>\n    <tr>\n      <th>6248</th>\n      <td>2923</td>\n      <td>0.294823</td>\n      <td>841</td>\n      <td>\\n\\nEvents \\n June 25: Battle of Fontenay \u2013 Lo...</td>\n    </tr>\n    <tr>\n      <th>6297</th>\n      <td>13568</td>\n      <td>0.300756</td>\n      <td>1746</td>\n      <td>\\n\\nEvents \\n January 8 \u2013 Bonnie Prince Charli...</td>\n    </tr>\n    <tr>\n      <th>11702</th>\n      <td>11708</td>\n      <td>0.307572</td>\n      <td>William Wallace</td>\n      <td>William Wallace was a Scottish knight who foug...</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\nNow that you've got a basic embeddings search running, you can [hop over to the Chroma docs](https://docs.trychroma.com/usage-guide#using-where-filters) to learn more about how to add filters to your query, update/delete data in your collections, and deploy Chroma."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 8790a711-f766-4be3-b00a-cbbdc17e2f53
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"8790a711-f766-4be3-b00a-cbbdc17e2f53\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a9e4f44b-54ee-4495-a500-d34c826303db\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"25c95edc7e5dad0c1b9078e99e3bd2dc77ff2d16024dcf330f1e6d3273aea014\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1c0cd859-e48b-487c-9b94-e8f02cdc13f1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2a683a846ba84360e06402d30d14238dd58a09a52d66cff465ff09bba1bf294b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3231, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "ref_doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "chroma:document": "# Robust Question Answering with Chroma and OpenAI \n\nThis notebook guides you step-by-step through answering questions about a collection of data, using [Chroma](https://trychroma.com), an open-source embeddings database, along with OpenAI's [text embeddings](https://platform.openai.com/docs/guides/embeddings/use-cases) and [chat completion](https://platform.openai.com/docs/guides/chat) API's. \n\nAdditionally, this notebook demonstrates some of the tradeoffs in making a question answering system more robust. As we shall see, *simple querying doesn't always create the best results*! \n\n## Question Answering with LLMs\n\nLarge language models (LLMs) like OpenAI's ChatGPT can be used to answer questions about data that the model may not have been trained on, or have access to. For example;\n\n- Personal data like e-mails and notes\n- Highly specialized data like archival or legal documents\n- Newly created data like recent news stories\n\nIn order to overcome this limitation, we can use a data store which is amenable to querying in natural language, just like the LLM itself. An embeddings store like Chroma represents documents as [embeddings](https://openai.com/blog/introducing-text-and-code-embeddings), alongside the documents themselves. \n\nBy embedding a text query, Chroma can find relevant documents, which we can then pass to the LLM to answer our question. We'll show detailed examples and variants of this approach. \n\n# Setup and preliminaries\n\nFirst we make sure the python dependencies we need are installed. \n\n\n```python\n%pip install -qU openai chromadb pandas\n```\n\n    Note: you may need to restart the kernel to use updated packages.\n\n\nWe use OpenAI's API's throughout this notebook. You can get an API key from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys)\n\nYou can add your API key as an environment variable by executing the command `export OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx` in a terminal. Note that you will need to reload the notebook if the environment variable wasn't set yet. Alternatively, you can set it in the notebook, see below. \n\n\n```python\nimport os\n\n# Uncomment the following line to set the environment variable in the notebook\n# os.environ[\"OPENAI_API_KEY\"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    print(\"OPENAI_API_KEY is ready\")\n    import openai\n    openai.api_key = os.getenv(\"OPENAI_API_KEY\")\nelse:\n    print(\"OPENAI_API_KEY environment variable not found\")\n```\n\n    OPENAI_API_KEY is ready\n\n\n# Dataset\n\nThroughout this notebook, we use the [SciFact dataset](https://github.com/allenai/scifact). This is a curated dataset of expert annotated scientific claims, with an accompanying text corpus of paper titles and abstracts. Each claim may be supported, contradicted, or not have enough evidence either way, according to the documents in the corpus. \n\nHaving the corpus available as ground-truth allows us to investigate how well the following approaches to LLM question answering perform. \n\n\n```python\n# Load the claim dataset\nimport pandas as pd\n\ndata_path = '../../data'\n\nclaim_df = pd.read_json(f'{data_path}/scifact_claims.jsonl', lines=True)\nclaim_df.head()\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 1c0cd859-e48b-487c-9b94-e8f02cdc13f1
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"1c0cd859-e48b-487c-9b94-e8f02cdc13f1\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a9e4f44b-54ee-4495-a500-d34c826303db\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"25c95edc7e5dad0c1b9078e99e3bd2dc77ff2d16024dcf330f1e6d3273aea014\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8790a711-f766-4be3-b00a-cbbdc17e2f53\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1920ee204472f814b61e9250a0fc6daf50d1527159b4ca1e11930fb8ab9cc597\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"621f3261-e332-4b94-aaac-4f8a5ade6519\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"266c03c3164aa015d5aee257ec93c88a5e2583aa158dfe4fe1195110e1268966\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1528, \"end_char_idx\": 4626, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "ref_doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "chroma:document": "```python\n%pip install -qU openai chromadb pandas\n```\n\n    Note: you may need to restart the kernel to use updated packages.\n\n\nWe use OpenAI's API's throughout this notebook. You can get an API key from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys)\n\nYou can add your API key as an environment variable by executing the command `export OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx` in a terminal. Note that you will need to reload the notebook if the environment variable wasn't set yet. Alternatively, you can set it in the notebook, see below. \n\n\n```python\nimport os\n\n# Uncomment the following line to set the environment variable in the notebook\n# os.environ[\"OPENAI_API_KEY\"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    print(\"OPENAI_API_KEY is ready\")\n    import openai\n    openai.api_key = os.getenv(\"OPENAI_API_KEY\")\nelse:\n    print(\"OPENAI_API_KEY environment variable not found\")\n```\n\n    OPENAI_API_KEY is ready\n\n\n# Dataset\n\nThroughout this notebook, we use the [SciFact dataset](https://github.com/allenai/scifact). This is a curated dataset of expert annotated scientific claims, with an accompanying text corpus of paper titles and abstracts. Each claim may be supported, contradicted, or not have enough evidence either way, according to the documents in the corpus. \n\nHaving the corpus available as ground-truth allows us to investigate how well the following approaches to LLM question answering perform. \n\n\n```python\n# Load the claim dataset\nimport pandas as pd\n\ndata_path = '../../data'\n\nclaim_df = pd.read_json(f'{data_path}/scifact_claims.jsonl', lines=True)\nclaim_df.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>claim</th>\n      <th>evidence</th>\n      <th>cited_doc_ids</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>0-dimensional biomaterials show inductive prop...</td>\n      <td>{}</td>\n      <td>[31715818]</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>3</td>\n      <td>1,000 genomes project enables mapping of genet...</td>\n      <td>{'14717500': [{'sentences': [2, 5], 'label': '...</td>\n      <td>[14717500]</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>5</td>\n      <td>1/2000 in UK have abnormal PrP positivity.</td>\n      <td>{'13734012': [{'sentences': [4], 'label': 'SUP...</td>\n      <td>[13734012]</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>13</td>\n      <td>5% of perinatal mortality is due to low birth ...</td>\n      <td>{}</td>\n      <td>[1606628]</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>36</td>\n      <td>A deficiency of vitamin B12 increases blood le...</td>\n      <td>{}</td>\n      <td>[5152028, 11705328]</td>\n    </tr>\n  </tbody>\n</table>\n</div>"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 621f3261-e332-4b94-aaac-4f8a5ade6519
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"621f3261-e332-4b94-aaac-4f8a5ade6519\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a9e4f44b-54ee-4495-a500-d34c826303db\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"25c95edc7e5dad0c1b9078e99e3bd2dc77ff2d16024dcf330f1e6d3273aea014\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1c0cd859-e48b-487c-9b94-e8f02cdc13f1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"dc963caea472ea55f64757756babc9624d15d60ec1a30b51b586da334e59eb5a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6dbbdc9b-e8ef-446f-a1cc-3e09ec5475b2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"355fbc1e7a71331fb5247d3623d9b8fb0538274ed8dba5cc363dcba2fd1c0cbf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4630, \"end_char_idx\": 7216, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "ref_doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "chroma:document": "# Just asking the model\n\nGPT-3.5 was trained on a large amount of scientific information. As a baseline, we'd like to understand what the model already knows without any further context. This will allow us to calibrate overall performance. \n\nWe construct an appropriate prompt, with some example facts, then query the model with each claim in the dataset. We ask the model to assess a claim as 'True', 'False', or 'NEE' if there is not enough evidence one way or the other. \n\n\n```python\ndef build_prompt(claim):\n    return [\n        {\"role\": \"system\", \"content\": \"I will ask you to assess a scientific claim. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence.\"},\n        {\"role\": \"user\", \"content\": f\"\"\"        \nExample:\n\nClaim:\n0-dimensional biomaterials show inductive properties.\n\nAssessment:\nFalse\n\nClaim:\n1/2000 in UK have abnormal PrP positivity.\n\nAssessment:\nTrue\n\nClaim:\nAspirin inhibits the production of PGE2.\n\nAssessment:\nFalse\n\nEnd of examples. Assess the following claim:\n\nClaim:\n{claim}\n\nAssessment:\n\"\"\"}\n    ]\n\n\ndef assess_claims(claims):\n    responses = []\n    # Query the OpenAI API\n    for claim in claims:\n        response = openai.ChatCompletion.create(\n            model='gpt-3.5-turbo',\n            messages=build_prompt(claim),\n            max_tokens=3,\n        )\n        # Strip any punctuation or whitespace from the response\n        responses.append(response.choices[0].message.content.strip('., '))\n\n    return responses\n```\n\nWe sample 100 claims from the dataset\n\n\n```python\n# Let's take a look at 100 claims\nsamples = claim_df.sample(50)\n\nclaims = samples['claim'].tolist() \n\n```\n\nWe evaluate the ground-truth according to the dataset. From the dataset description, each claim is either supported or contradicted by the evidence, or else there isn't enough evidence either way. \n\n\n```python\ndef get_groundtruth(evidence):\n    groundtruth = []\n    for e in evidence:\n        # Evidence is empty \n        if len(e) == 0:\n            groundtruth.append('NEE')\n        else:\n            # In this dataset, all evidence for a given claim is consistent, either SUPPORT or CONTRADICT\n            if list(e.values())[0][0]['label'] == 'SUPPORT':\n                groundtruth.append('True')\n            else:\n                groundtruth.append('False')\n    return groundtruth\n```\n\n\n```python\nevidence = samples['evidence'].tolist()\ngroundtruth = get_groundtruth(evidence)\n```\n\nWe also output the confusion matrix, comparing the model's assessments with the ground truth, in an easy to read table."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 6dbbdc9b-e8ef-446f-a1cc-3e09ec5475b2
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"6dbbdc9b-e8ef-446f-a1cc-3e09ec5475b2\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a9e4f44b-54ee-4495-a500-d34c826303db\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"25c95edc7e5dad0c1b9078e99e3bd2dc77ff2d16024dcf330f1e6d3273aea014\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"621f3261-e332-4b94-aaac-4f8a5ade6519\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"c9ad8b9d602b4b156d548a9d9b74cb0db1802330fb4ae042b09294caeae56f5f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ea8d92ef-6d3d-4fa0-b240-eeab0f11e84c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2585e3d072f75550210dd85dba3e086b3331ed428f25dc567392fffc97703a33\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5728, \"end_char_idx\": 8574, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "ref_doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "chroma:document": "def assess_claims(claims):\n    responses = []\n    # Query the OpenAI API\n    for claim in claims:\n        response = openai.ChatCompletion.create(\n            model='gpt-3.5-turbo',\n            messages=build_prompt(claim),\n            max_tokens=3,\n        )\n        # Strip any punctuation or whitespace from the response\n        responses.append(response.choices[0].message.content.strip('., '))\n\n    return responses\n```\n\nWe sample 100 claims from the dataset\n\n\n```python\n# Let's take a look at 100 claims\nsamples = claim_df.sample(50)\n\nclaims = samples['claim'].tolist() \n\n```\n\nWe evaluate the ground-truth according to the dataset. From the dataset description, each claim is either supported or contradicted by the evidence, or else there isn't enough evidence either way. \n\n\n```python\ndef get_groundtruth(evidence):\n    groundtruth = []\n    for e in evidence:\n        # Evidence is empty \n        if len(e) == 0:\n            groundtruth.append('NEE')\n        else:\n            # In this dataset, all evidence for a given claim is consistent, either SUPPORT or CONTRADICT\n            if list(e.values())[0][0]['label'] == 'SUPPORT':\n                groundtruth.append('True')\n            else:\n                groundtruth.append('False')\n    return groundtruth\n```\n\n\n```python\nevidence = samples['evidence'].tolist()\ngroundtruth = get_groundtruth(evidence)\n```\n\nWe also output the confusion matrix, comparing the model's assessments with the ground truth, in an easy to read table. \n\n\n```python\ndef confusion_matrix(inferred, groundtruth):\n    assert len(inferred) == len(groundtruth)\n    confusion = {\n        'True': {'True': 0, 'False': 0, 'NEE': 0},\n        'False': {'True': 0, 'False': 0, 'NEE': 0},\n        'NEE': {'True': 0, 'False': 0, 'NEE': 0},\n    }\n    for i, g in zip(inferred, groundtruth):\n        confusion[i][g] += 1\n\n    # Pretty print the confusion matrix\n    print('\\tGroundtruth')\n    print('\\tTrue\\tFalse\\tNEE')\n    for i in confusion:\n        print(i, end='\\t')\n        for g in confusion[i]:\n            print(confusion[i][g], end='\\t')\n        print()\n\n    return confusion\n```\n\nWe ask the model to directly assess the claims, without additional context. \n\n\n```python\ngpt_inferred = assess_claims(claims)\nconfusion_matrix(gpt_inferred, groundtruth)\n```\n\n    \tGroundtruth\n    \tTrue\tFalse\tNEE\n    True\t15\t5\t14\t\n    False\t0\t2\t1\t\n    NEE\t3\t3\t7\t\n\n\n\n\n\n    {'True': {'True': 15, 'False': 5, 'NEE': 14},\n     'False': {'True': 0, 'False': 2, 'NEE': 1},\n     'NEE': {'True': 3, 'False': 3, 'NEE': 7}}\n\n\n\n## Results\n\nFrom these results we see that the LLM is strongly biased to assess claims as true, even when they are false, and also tends to assess false claims as not having enough evidence. Note that 'not enough evidence' is with respect to the model's assessment of the claim in a vacuum, without additional context."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: ea8d92ef-6d3d-4fa0-b240-eeab0f11e84c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"ea8d92ef-6d3d-4fa0-b240-eeab0f11e84c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a9e4f44b-54ee-4495-a500-d34c826303db\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"25c95edc7e5dad0c1b9078e99e3bd2dc77ff2d16024dcf330f1e6d3273aea014\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6dbbdc9b-e8ef-446f-a1cc-3e09ec5475b2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"6c78b61cedbcc8368de574ff9ffbb87393104364aea950242f11e56dc545c0ab\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5106c423-ec4a-4e8c-819a-ed3802bcb75f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dbcd26f0518ab002681e4ec6edc0956a9caf6d5ea8419af30b57c73648576cf6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7919, \"end_char_idx\": 8951, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "ref_doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "chroma:document": "```python\ngpt_inferred = assess_claims(claims)\nconfusion_matrix(gpt_inferred, groundtruth)\n```\n\n    \tGroundtruth\n    \tTrue\tFalse\tNEE\n    True\t15\t5\t14\t\n    False\t0\t2\t1\t\n    NEE\t3\t3\t7\t\n\n\n\n\n\n    {'True': {'True': 15, 'False': 5, 'NEE': 14},\n     'False': {'True': 0, 'False': 2, 'NEE': 1},\n     'NEE': {'True': 3, 'False': 3, 'NEE': 7}}\n\n\n\n## Results\n\nFrom these results we see that the LLM is strongly biased to assess claims as true, even when they are false, and also tends to assess false claims as not having enough evidence. Note that 'not enough evidence' is with respect to the model's assessment of the claim in a vacuum, without additional context.\n\n\n# Adding context \n\nWe now add the additional context available from the corpus of paper titles and abstracts. This section shows how to load a text corpus into Chroma, using OpenAI text embeddings. \n\nFirst, we load the text corpus. \n\n\n```python\n# Load the corpus into a dataframe\ncorpus_df = pd.read_json(f'{data_path}/scifact_corpus.jsonl', lines=True)\ncorpus_df.head()\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 5106c423-ec4a-4e8c-819a-ed3802bcb75f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"5106c423-ec4a-4e8c-819a-ed3802bcb75f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a9e4f44b-54ee-4495-a500-d34c826303db\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"25c95edc7e5dad0c1b9078e99e3bd2dc77ff2d16024dcf330f1e6d3273aea014\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ea8d92ef-6d3d-4fa0-b240-eeab0f11e84c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b7290c0ccd1416931bfd3ca0e89e350b2f7deb9cfff181e5b4976345c000a921\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f88febbc-f8f9-4549-8753-317e88ec35f7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"76197a26079d0781a448b68ca74f4b47cd67db1631ef0f0f6dc0a516ac4e0d45\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7919, \"end_char_idx\": 10477, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "ref_doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "chroma:document": "```python\ngpt_inferred = assess_claims(claims)\nconfusion_matrix(gpt_inferred, groundtruth)\n```\n\n    \tGroundtruth\n    \tTrue\tFalse\tNEE\n    True\t15\t5\t14\t\n    False\t0\t2\t1\t\n    NEE\t3\t3\t7\t\n\n\n\n\n\n    {'True': {'True': 15, 'False': 5, 'NEE': 14},\n     'False': {'True': 0, 'False': 2, 'NEE': 1},\n     'NEE': {'True': 3, 'False': 3, 'NEE': 7}}\n\n\n\n## Results\n\nFrom these results we see that the LLM is strongly biased to assess claims as true, even when they are false, and also tends to assess false claims as not having enough evidence. Note that 'not enough evidence' is with respect to the model's assessment of the claim in a vacuum, without additional context.\n\n\n# Adding context \n\nWe now add the additional context available from the corpus of paper titles and abstracts. This section shows how to load a text corpus into Chroma, using OpenAI text embeddings. \n\nFirst, we load the text corpus. \n\n\n```python\n# Load the corpus into a dataframe\ncorpus_df = pd.read_json(f'{data_path}/scifact_corpus.jsonl', lines=True)\ncorpus_df.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>doc_id</th>\n      <th>title</th>\n      <th>abstract</th>\n      <th>structured</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>4983</td>\n      <td>Microstructural development of human newborn c...</td>\n      <td>[Alterations of the architecture of cerebral w...</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>5836</td>\n      <td>Induction of myelodysplasia by myeloid-derived...</td>\n      <td>[Myelodysplastic syndromes (MDS) are age-depen...</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>7912</td>\n      <td>BC1 RNA, the transcript from a master gene for...</td>\n      <td>[ID elements are short interspersed elements (...</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>18670</td>\n      <td>The DNA Methylome of Human Peripheral Blood Mo...</td>\n      <td>[DNA methylation plays an important role in bi...</td>\n      <td>False</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>19238</td>\n      <td>The human myelin basic protein gene is include...</td>\n      <td>[Two human Golli (for gene expressed in the ol...</td>\n      <td>False</td>\n    </tr>\n  </tbody>\n</table>\n</div>"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: f88febbc-f8f9-4549-8753-317e88ec35f7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"f88febbc-f8f9-4549-8753-317e88ec35f7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a9e4f44b-54ee-4495-a500-d34c826303db\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"25c95edc7e5dad0c1b9078e99e3bd2dc77ff2d16024dcf330f1e6d3273aea014\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5106c423-ec4a-4e8c-819a-ed3802bcb75f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"93cd7fabc4ca2dc9aaf35e854d15e72818afded70c272d2fdfd8708775bf12b6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bf7ec4c7-f3cb-4275-ad45-b443d4e9a480\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4141e1ba657640c16d1e19319fd2889088058dddfcd42bf528197d0f5098e009\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10481, \"end_char_idx\": 13640, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "ref_doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "chroma:document": "## Loading the corpus into Chroma\n\nThe next step is to load the corpus into Chroma. Given an embedding function, Chroma will automatically handle embedding each document, and will store it alongside its text and metadata, making it simple to query.\n\nWe instantiate a (ephemeral) Chroma client, and create a collection for the SciFact title and abstract corpus. \nChroma can also be instantiated in a persisted configuration; learn more at the [Chroma docs](https://docs.trychroma.com/usage-guide?lang=py). \n\n\n```python\nimport chromadb\nfrom chromadb.utils.embedding_functions import OpenAIEmbeddingFunction\n\n# We initialize an embedding function, and provide it to the collection.\nembedding_function = OpenAIEmbeddingFunction(api_key=os.getenv(\"OPENAI_API_KEY\"))\n\nchroma_client = chromadb.Client() # Ephemeral by default\nscifact_corpus_collection = chroma_client.create_collection(name='scifact_corpus', embedding_function=embedding_function)\n```\n\n    Running Chroma using direct local API.\n    Using DuckDB in-memory for database. Data will be transient.\n\n\nNext we load the corpus into Chroma. Because this data loading is memory intensive, we recommend using a batched loading scheme in batches of 50-1000. For this example it should take just over one minute for the entire corpus. It's being embedded in the background, automatically, using the `embedding_function` we specified earlier.\n\n\n```python\nbatch_size = 100\n\nfor i in range(0, len(corpus_df), batch_size):\n    batch_df = corpus_df[i:i+batch_size]\n    scifact_corpus_collection.add(\n        ids=batch_df['doc_id'].apply(lambda x: str(x)).tolist(), # Chroma takes string IDs.\n        documents=(batch_df['title'] + '. ' + batch_df['abstract'].apply(lambda x: ' '.join(x))).to_list(), # We concatenate the title and abstract.\n        metadatas=[{\"structured\": structured} for structured in batch_df['structured'].to_list()] # We also store the metadata, though we don't use it in this example.\n    )\n```\n\n## Retrieving context\n\nNext we retrieve documents from the corpus which may be relevant to each claim in our sample. We want to provide these as context to the LLM for evaluating the claims. We retrieve the 3 most relevant documents for each claim, according to the embedding distance. \n\n\n```python\nclaim_query_result = scifact_corpus_collection.query(query_texts=claims, include=['documents', 'distances'], n_results=3)\n```\n\nWe create a new prompt, this time taking into account the additional context we retrieve from the corpus. \n\n\n```python\ndef build_prompt_with_context(claim, context):\n    return [{'role': 'system', 'content': \"I will ask you to assess whether a particular scientific claim, based on evidence provided. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence.\"}, \n            {'role': 'user', 'content': f\"\"\"\"\nThe evidence is the following:\n\n{' '.join(context)}\n\nAssess the following claim on the basis of the evidence. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence. Do not output any other text. \n\nClaim:\n{claim}\n\nAssessment:\n\"\"\"}]"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: bf7ec4c7-f3cb-4275-ad45-b443d4e9a480
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"bf7ec4c7-f3cb-4275-ad45-b443d4e9a480\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a9e4f44b-54ee-4495-a500-d34c826303db\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"25c95edc7e5dad0c1b9078e99e3bd2dc77ff2d16024dcf330f1e6d3273aea014\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f88febbc-f8f9-4549-8753-317e88ec35f7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"45122bce3534621076baf8e20f26c356195b96506d5e8f8e32946581a692ba83\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c243ab37-7743-43ca-a28a-b5c317266a28\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0628ac2ec49633ccfa9cdc83edaaf9ad092c43126c21199ecdd621dbcda51b1d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12733, \"end_char_idx\": 15739, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "ref_doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "chroma:document": "```python\nclaim_query_result = scifact_corpus_collection.query(query_texts=claims, include=['documents', 'distances'], n_results=3)\n```\n\nWe create a new prompt, this time taking into account the additional context we retrieve from the corpus. \n\n\n```python\ndef build_prompt_with_context(claim, context):\n    return [{'role': 'system', 'content': \"I will ask you to assess whether a particular scientific claim, based on evidence provided. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence.\"}, \n            {'role': 'user', 'content': f\"\"\"\"\nThe evidence is the following:\n\n{' '.join(context)}\n\nAssess the following claim on the basis of the evidence. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence. Do not output any other text. \n\nClaim:\n{claim}\n\nAssessment:\n\"\"\"}]\n\n\ndef assess_claims_with_context(claims, contexts):\n    responses = []\n    # Query the OpenAI API\n    for claim, context in zip(claims, contexts):\n        # If no evidence is provided, return NEE\n        if len(context) == 0:\n            responses.append('NEE')\n            continue\n        response = openai.ChatCompletion.create(\n            model='gpt-3.5-turbo',\n            messages=build_prompt_with_context(claim=claim, context=context),\n            max_tokens=3,\n        )\n        # Strip any punctuation or whitespace from the response\n        responses.append(response.choices[0].message.content.strip('., '))\n\n    return responses\n```\n\nThen ask the model to evaluate the claims with the retrieved context. \n\n\n```python\ngpt_with_context_evaluation = assess_claims_with_context(claims, claim_query_result['documents'])\nconfusion_matrix(gpt_with_context_evaluation, groundtruth)\n```\n\n    \tGroundtruth\n    \tTrue\tFalse\tNEE\n    True\t16\t2\t8\t\n    False\t1\t6\t5\t\n    NEE\t1\t2\t9\t\n\n\n\n\n\n    {'True': {'True': 16, 'False': 2, 'NEE': 8},\n     'False': {'True': 1, 'False': 6, 'NEE': 5},\n     'NEE': {'True': 1, 'False': 2, 'NEE': 9}}\n\n\n\n## Results\n\nWe see that the model is a lot less likely to evaluate a False claim as true (2 instances VS 5 previously), but that claims without enough evidence are still often assessed as True or False.\n\nTaking a look at the retrieved documents, we see that they are sometimes not relevant to the claim - this causes the model to be confused by the extra information, and it may decide that sufficient evidence is present, even when the information is irrelevant. This happens because we always ask for the 3 'most' relevant documents, but these might not be relevant at all beyond a certain point.  \n\n## Filtering context on relevance\n\nAlong with the documents themselves, Chroma returns a distance score. We can try thresholding on distance, so that fewer irrelevant documents make it into the context we provide the model. \n\nIf, after filtering on the threshold, no context documents remain, we bypass the model and simply return that there is not enough evidence."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: c243ab37-7743-43ca-a28a-b5c317266a28
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"c243ab37-7743-43ca-a28a-b5c317266a28\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a9e4f44b-54ee-4495-a500-d34c826303db\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"25c95edc7e5dad0c1b9078e99e3bd2dc77ff2d16024dcf330f1e6d3273aea014\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bf7ec4c7-f3cb-4275-ad45-b443d4e9a480\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3729fe140c9a3a0567e885c86c59e2cf562add074666d3915a31e47b36e0fae6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b6eed232-7249-443e-a73a-92c2e95c7c2d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0debb42270b184f1c2c79545a873540128782d62b2012be14f939979e8a2e7a5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14361, \"end_char_idx\": 16808, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "ref_doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "chroma:document": "```python\ngpt_with_context_evaluation = assess_claims_with_context(claims, claim_query_result['documents'])\nconfusion_matrix(gpt_with_context_evaluation, groundtruth)\n```\n\n    \tGroundtruth\n    \tTrue\tFalse\tNEE\n    True\t16\t2\t8\t\n    False\t1\t6\t5\t\n    NEE\t1\t2\t9\t\n\n\n\n\n\n    {'True': {'True': 16, 'False': 2, 'NEE': 8},\n     'False': {'True': 1, 'False': 6, 'NEE': 5},\n     'NEE': {'True': 1, 'False': 2, 'NEE': 9}}\n\n\n\n## Results\n\nWe see that the model is a lot less likely to evaluate a False claim as true (2 instances VS 5 previously), but that claims without enough evidence are still often assessed as True or False.\n\nTaking a look at the retrieved documents, we see that they are sometimes not relevant to the claim - this causes the model to be confused by the extra information, and it may decide that sufficient evidence is present, even when the information is irrelevant. This happens because we always ask for the 3 'most' relevant documents, but these might not be relevant at all beyond a certain point.  \n\n## Filtering context on relevance\n\nAlong with the documents themselves, Chroma returns a distance score. We can try thresholding on distance, so that fewer irrelevant documents make it into the context we provide the model. \n\nIf, after filtering on the threshold, no context documents remain, we bypass the model and simply return that there is not enough evidence. \n\n\n```python\ndef filter_query_result(query_result, distance_threshold=0.25):\n# For each query result, retain only the documents whose distance is below the threshold\n    for ids, docs, distances in zip(query_result['ids'], query_result['documents'], query_result['distances']):\n        for i in range(len(ids)-1, -1, -1):\n            if distances[i] > distance_threshold:\n                ids.pop(i)\n                docs.pop(i)\n                distances.pop(i)\n    return query_result\n\n```\n\n\n```python\nfiltered_claim_query_result = filter_query_result(claim_query_result)\n```\n\nNow we assess the claims using this cleaner context. \n\n\n```python\ngpt_with_filtered_context_evaluation = assess_claims_with_context(claims, filtered_claim_query_result['documents'])\nconfusion_matrix(gpt_with_filtered_context_evaluation, groundtruth)\n```\n\n    \tGroundtruth\n    \tTrue\tFalse\tNEE\n    True\t10\t2\t1\t\n    False\t0\t2\t1\t\n    NEE\t8\t6\t20\t\n\n\n\n\n\n    {'True': {'True': 10, 'False': 2, 'NEE': 1},\n     'False': {'True': 0, 'False': 2, 'NEE': 1},\n     'NEE': {'True': 8, 'False': 6, 'NEE': 20}}"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: b6eed232-7249-443e-a73a-92c2e95c7c2d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"b6eed232-7249-443e-a73a-92c2e95c7c2d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a9e4f44b-54ee-4495-a500-d34c826303db\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"25c95edc7e5dad0c1b9078e99e3bd2dc77ff2d16024dcf330f1e6d3273aea014\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c243ab37-7743-43ca-a28a-b5c317266a28\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"4983f20d5b7bd7727ce4f8ec059dddbf2966fc7f5326586feadc74a2c0497731\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"242d1d2f-c5b8-4321-b6b0-1e83866c8507\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"725e8f3d5074fb077f7affdec9d8e9d486e4728c381e176262d779f22aaa5afd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15743, \"end_char_idx\": 18984, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "ref_doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "chroma:document": "```python\ndef filter_query_result(query_result, distance_threshold=0.25):\n# For each query result, retain only the documents whose distance is below the threshold\n    for ids, docs, distances in zip(query_result['ids'], query_result['documents'], query_result['distances']):\n        for i in range(len(ids)-1, -1, -1):\n            if distances[i] > distance_threshold:\n                ids.pop(i)\n                docs.pop(i)\n                distances.pop(i)\n    return query_result\n\n```\n\n\n```python\nfiltered_claim_query_result = filter_query_result(claim_query_result)\n```\n\nNow we assess the claims using this cleaner context. \n\n\n```python\ngpt_with_filtered_context_evaluation = assess_claims_with_context(claims, filtered_claim_query_result['documents'])\nconfusion_matrix(gpt_with_filtered_context_evaluation, groundtruth)\n```\n\n    \tGroundtruth\n    \tTrue\tFalse\tNEE\n    True\t10\t2\t1\t\n    False\t0\t2\t1\t\n    NEE\t8\t6\t20\t\n\n\n\n\n\n    {'True': {'True': 10, 'False': 2, 'NEE': 1},\n     'False': {'True': 0, 'False': 2, 'NEE': 1},\n     'NEE': {'True': 8, 'False': 6, 'NEE': 20}}\n\n\n\n## Results\n\nThe model now assesses many fewer claims as True or False when there is not enough evidence present. However, it now biases away from certainty. Most claims are now assessed as having not enough evidence, because a large fraction of them are filtered out by the distance threshold. It's possible to tune the distance threshold to find the optimal operating point, but this can be difficult, and is dataset and embedding model dependent. \n\n# Hypothetical Document Embeddings: Using hallucinations productively\n\nWe want to be able to retrieve relevant documents, without retrieving less relevant ones which might confuse the model. One way to accomplish this is to improve the retrieval query. \n\nUntil now, we have queried the dataset using _claims_ which are single sentence statements, while the corpus contains _abstracts_ describing a scientific paper. Intuitively, while these might be related, there are significant differences in their structure and meaning. These differences are encoded by the embedding model, and so influence the distances between the query and the most relevant results. \n\nWe can overcome this by leveraging the power of LLMs to generate relevant text. While the facts might be hallucinated, the content and structure of the documents the models generate is more similar to the documents in our corpus, than the queries are. This could lead to better queries and hence better results. \n\nThis approach is called [Hypothetical Document Embeddings (HyDE)](https://arxiv.org/abs/2212.10496), and has been shown to be quite good at the retrieval task. It should help us bring more relevant information into the context, without polluting it.  \n\nTL;DR:\n- you get much better matches when you embed whole abstracts rather than single sentences\n- but claims are usually single sentences\n- So HyDE shows that using GPT3 to expand claims into hallucinated abstracts and then searching based on those abstracts works (claims -> abstracts -> results) better than searching directly (claims -> results)\n\nFirst, we use in-context examples to prompt the model to generate documents similar to what's in the corpus, for each claim we want to assess."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 242d1d2f-c5b8-4321-b6b0-1e83866c8507
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"242d1d2f-c5b8-4321-b6b0-1e83866c8507\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a9e4f44b-54ee-4495-a500-d34c826303db\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"25c95edc7e5dad0c1b9078e99e3bd2dc77ff2d16024dcf330f1e6d3273aea014\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b6eed232-7249-443e-a73a-92c2e95c7c2d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"ad01de2b74880929ba2dc575b2e28d2fab37d6dbc8e7851c8667d26935e525be\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1c89a6f9-8e77-43c9-80b6-aca55ed943c2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3ce52a35ffd3ed9ac882558ad576d3512cb63f01ed926b3e9164054a92403d5f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18988, \"end_char_idx\": 22163, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "ref_doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "chroma:document": "```python\ndef build_hallucination_prompt(claim):\n    return [{'role': 'system', 'content': \"\"\"I will ask you to write an abstract for a scientific paper which supports or refutes a given claim. It should be written in scientific language, include a title. Output only one abstract, then stop.\n    \n    An Example:\n\n    Claim:\n    A high microerythrocyte count raises vulnerability to severe anemia in homozygous alpha (+)- thalassemia trait subjects.\n\n    Abstract:\n    BACKGROUND The heritable haemoglobinopathy alpha(+)-thalassaemia is caused by the reduced synthesis of alpha-globin chains that form part of normal adult haemoglobin (Hb). Individuals homozygous for alpha(+)-thalassaemia have microcytosis and an increased erythrocyte count. Alpha(+)-thalassaemia homozygosity confers considerable protection against severe malaria, including severe malarial anaemia (SMA) (Hb concentration < 50 g/l), but does not influence parasite count. We tested the hypothesis that the erythrocyte indices associated with alpha(+)-thalassaemia homozygosity provide a haematological benefit during acute malaria.   \n    METHODS AND FINDINGS Data from children living on the north coast of Papua New Guinea who had participated in a case-control study of the protection afforded by alpha(+)-thalassaemia against severe malaria were reanalysed to assess the genotype-specific reduction in erythrocyte count and Hb levels associated with acute malarial disease. We observed a reduction in median erythrocyte count of approximately 1.5 x 10(12)/l in all children with acute falciparum malaria relative to values in community children (p < 0.001). We developed a simple mathematical model of the linear relationship between Hb concentration and erythrocyte count. This model predicted that children homozygous for alpha(+)-thalassaemia lose less Hb than children of normal genotype for a reduction in erythrocyte count of >1.1 x 10(12)/l as a result of the reduced mean cell Hb in homozygous alpha(+)-thalassaemia. In addition, children homozygous for alpha(+)-thalassaemia require a 10% greater reduction in erythrocyte count than children of normal genotype (p = 0.02) for Hb concentration to fall to 50 g/l, the cutoff for SMA. We estimated that the haematological profile in children homozygous for alpha(+)-thalassaemia reduces the risk of SMA during acute malaria compared to children of normal genotype (relative risk 0.52; 95% confidence interval [CI] 0.24-1.12, p = 0.09).   \n    CONCLUSIONS The increased erythrocyte count and microcytosis in children homozygous for alpha(+)-thalassaemia may contribute substantially to their protection against SMA. A lower concentration of Hb per erythrocyte and a larger population of erythrocytes may be a biologically advantageous strategy against the significant reduction in erythrocyte count that occurs during acute infection with the malaria parasite Plasmodium falciparum. This haematological profile may reduce the risk of anaemia by other Plasmodium species, as well as other causes of anaemia. Other host polymorphisms that induce an increased erythrocyte count and microcytosis may confer a similar advantage.\n\n    End of example."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 1c89a6f9-8e77-43c9-80b6-aca55ed943c2
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"1c89a6f9-8e77-43c9-80b6-aca55ed943c2\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a9e4f44b-54ee-4495-a500-d34c826303db\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"25c95edc7e5dad0c1b9078e99e3bd2dc77ff2d16024dcf330f1e6d3273aea014\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"242d1d2f-c5b8-4321-b6b0-1e83866c8507\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"04066ad3375562bd5888110b2f7b53149065aa04965af8c2c6da5f45c463c0ec\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"68da7890-9d6f-49bf-8c54-d40b1c02c421\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"89ee72a4c1a935544a9770be131bcf7707e09d9833f115e340c7d1adc505a3f5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 20622, \"end_char_idx\": 23668, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "ref_doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "chroma:document": "We developed a simple mathematical model of the linear relationship between Hb concentration and erythrocyte count. This model predicted that children homozygous for alpha(+)-thalassaemia lose less Hb than children of normal genotype for a reduction in erythrocyte count of >1.1 x 10(12)/l as a result of the reduced mean cell Hb in homozygous alpha(+)-thalassaemia. In addition, children homozygous for alpha(+)-thalassaemia require a 10% greater reduction in erythrocyte count than children of normal genotype (p = 0.02) for Hb concentration to fall to 50 g/l, the cutoff for SMA. We estimated that the haematological profile in children homozygous for alpha(+)-thalassaemia reduces the risk of SMA during acute malaria compared to children of normal genotype (relative risk 0.52; 95% confidence interval [CI] 0.24-1.12, p = 0.09).   \n    CONCLUSIONS The increased erythrocyte count and microcytosis in children homozygous for alpha(+)-thalassaemia may contribute substantially to their protection against SMA. A lower concentration of Hb per erythrocyte and a larger population of erythrocytes may be a biologically advantageous strategy against the significant reduction in erythrocyte count that occurs during acute infection with the malaria parasite Plasmodium falciparum. This haematological profile may reduce the risk of anaemia by other Plasmodium species, as well as other causes of anaemia. Other host polymorphisms that induce an increased erythrocyte count and microcytosis may confer a similar advantage.\n\n    End of example. \n    \n    \"\"\"}, {'role': 'user', 'content': f\"\"\"\"\n    Perform the task for the following claim.\n\n    Claim:\n    {claim}\n\n    Abstract:\n    \"\"\"}]\n\n\ndef hallucinate_evidence(claims):\n    # Query the OpenAI API\n    responses = []\n    # Query the OpenAI API\n    for claim in claims:\n        response = openai.ChatCompletion.create(\n            model='gpt-3.5-turbo',\n            messages=build_hallucination_prompt(claim),\n        )\n        responses.append(response.choices[0].message.content)\n    return responses\n```\n\nWe hallucinate a document for each claim.\n\n*NB: This can take a while, about 30m for 100 claims*. You can reduce the number of claims we want to assess to get results more quickly. \n\n\n```python\nhallucinated_evidence = hallucinate_evidence(claims)\n```\n\nWe use the hallucinated documents as queries into the corpus, and filter the results using the same distance threshold. \n\n\n```python\nhallucinated_query_result = scifact_corpus_collection.query(query_texts=hallucinated_evidence, include=['documents', 'distances'], n_results=3)\nfiltered_hallucinated_query_result = filter_query_result(hallucinated_query_result)\n```\n\nWe then ask the model to assess the claims, using the new context. \n\n\n```python\ngpt_with_hallucinated_context_evaluation = assess_claims_with_context(claims, filtered_hallucinated_query_result['documents'])\nconfusion_matrix(gpt_with_hallucinated_context_evaluation, groundtruth)\n```\n\n    \tGroundtruth\n    \tTrue\tFalse\tNEE\n    True\t15\t2\t5\t\n    False\t1\t5\t4\t\n    NEE\t2\t3\t13"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 68da7890-9d6f-49bf-8c54-d40b1c02c421
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"68da7890-9d6f-49bf-8c54-d40b1c02c421\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"a9e4f44b-54ee-4495-a500-d34c826303db\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"25c95edc7e5dad0c1b9078e99e3bd2dc77ff2d16024dcf330f1e6d3273aea014\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1c89a6f9-8e77-43c9-80b6-aca55ed943c2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb\", \"title\": \"Robust Question Answering with Chroma and OpenAI\", \"tokens\": 6044, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9077edbc114af176fa6b7231741bffbde42726d0e37b9e61c18bd5c343755042\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 22148, \"end_char_idx\": 24502, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "ref_doc_id": "a9e4f44b-54ee-4495-a500-d34c826303db", "chroma:document": "End of example. \n    \n    \"\"\"}, {'role': 'user', 'content': f\"\"\"\"\n    Perform the task for the following claim.\n\n    Claim:\n    {claim}\n\n    Abstract:\n    \"\"\"}]\n\n\ndef hallucinate_evidence(claims):\n    # Query the OpenAI API\n    responses = []\n    # Query the OpenAI API\n    for claim in claims:\n        response = openai.ChatCompletion.create(\n            model='gpt-3.5-turbo',\n            messages=build_hallucination_prompt(claim),\n        )\n        responses.append(response.choices[0].message.content)\n    return responses\n```\n\nWe hallucinate a document for each claim.\n\n*NB: This can take a while, about 30m for 100 claims*. You can reduce the number of claims we want to assess to get results more quickly. \n\n\n```python\nhallucinated_evidence = hallucinate_evidence(claims)\n```\n\nWe use the hallucinated documents as queries into the corpus, and filter the results using the same distance threshold. \n\n\n```python\nhallucinated_query_result = scifact_corpus_collection.query(query_texts=hallucinated_evidence, include=['documents', 'distances'], n_results=3)\nfiltered_hallucinated_query_result = filter_query_result(hallucinated_query_result)\n```\n\nWe then ask the model to assess the claims, using the new context. \n\n\n```python\ngpt_with_hallucinated_context_evaluation = assess_claims_with_context(claims, filtered_hallucinated_query_result['documents'])\nconfusion_matrix(gpt_with_hallucinated_context_evaluation, groundtruth)\n```\n\n    \tGroundtruth\n    \tTrue\tFalse\tNEE\n    True\t15\t2\t5\t\n    False\t1\t5\t4\t\n    NEE\t2\t3\t13\t\n\n\n\n\n\n    {'True': {'True': 15, 'False': 2, 'NEE': 5},\n     'False': {'True': 1, 'False': 5, 'NEE': 4},\n     'NEE': {'True': 2, 'False': 3, 'NEE': 13}}\n\n\n\n## Results\n\nCombining HyDE with a simple distance threshold leads to a significant improvement. The model no longer biases assessing claims as True, nor toward their not being enough evidence. It also correctly assesses when there isn't enough evidence more often.\n\n# Conclusion\n\nEquipping LLMs with a context based on a corpus of documents is a powerful technique for bringing the general reasoning and natural language interactions of LLMs to your own data. However, it's important to know that naive query and retrieval may not produce the best possible results! Ultimately understanding the data will help get the most out of the retrieval based question-answering approach."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 6612b9f5-1d90-4cdc-9b31-d181c0283258
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"6612b9f5-1d90-4cdc-9b31-d181c0283258\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3adbfd21-8657-4768-9e3c-962d88c32a76\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3bdefec7716901ce665401ca02cd191586bb259932bb6115ff0b3a7de65a8825\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"188d3e2f-52d7-4e49-8c9c-4f07a222d691\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"088acc09ac6d00e2f69d60f7eaec540a731a22f958b8c7d25f609591117eb096\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3626, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "ref_doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "chroma:document": "# Using Redis for Embeddings Search\n\nThis notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.\n\n### What is a Vector Database\n\nA vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.\n\n### Why use a Vector Database\n\nVector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.\n\n\n### Demo Flow\nThe demo flow is:\n- **Setup**: Import packages and set any required variables\n- **Load data**: Load a dataset and embed it using OpenAI embeddings\n- **Redis**\n    - *Setup*: Set up the Redis-Py client. For more details go [here](https://github.com/redis/redis-py)\n    - *Index Data*: Create the search index for vector search and hybrid search (vector + full-text search) on all available fields.\n    - *Search Data*: Run a few example queries with various goals in mind.\n\nOnce you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.\n\n## Setup\n\nImport the required libraries and set the embedding model that we'd like to use.\n\n\n```python\n# We'll need to install the Redis client\n!pip install redis\n\n#Install wget to pull zip file\n!pip install wget\n```\n\n\n```python\nimport openai\n\nfrom typing import List, Iterator\nimport pandas as pd\nimport numpy as np\nimport os\nimport wget\nfrom ast import literal_eval\n\n# Redis client library for Python\nimport redis\n\n# I've set this to our new embeddings model, this can be changed to the embedding model of your choice\nEMBEDDING_MODEL = \"text-embedding-3-small\"\n\n# Ignore unclosed SSL socket warnings - optional in case you get these errors\nimport warnings\n\nwarnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning) \n```\n\n## Load data\n\nIn this section we'll load embedded data that we've prepared previous to this session.\n\n\n```python\nembeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'\n\n# The file is ~700 MB so this will take some time\nwget.download(embeddings_url)\n```\n\n\n```python\nimport zipfile\nwith zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n    zip_ref.extractall(\"../data\")\n```\n\n\n```python\narticle_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')\n```\n\n\n```python\narticle_df.head()\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 188d3e2f-52d7-4e49-8c9c-4f07a222d691
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"188d3e2f-52d7-4e49-8c9c-4f07a222d691\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3adbfd21-8657-4768-9e3c-962d88c32a76\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3bdefec7716901ce665401ca02cd191586bb259932bb6115ff0b3a7de65a8825\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6612b9f5-1d90-4cdc-9b31-d181c0283258\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a30a3a6b953d69365ff59f0682c39dcc13ded46d51028c64b1cea2cbbc62c699\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7e528539-7bf6-4860-bb7e-9a5f932a5dcd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"55d75d734c6c465942a4bbe99c9e2550ca1dde90d5af183d906f3fc23ae149c3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2312, \"end_char_idx\": 4642, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "ref_doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "chroma:document": "```python\n# We'll need to install the Redis client\n!pip install redis\n\n#Install wget to pull zip file\n!pip install wget\n```\n\n\n```python\nimport openai\n\nfrom typing import List, Iterator\nimport pandas as pd\nimport numpy as np\nimport os\nimport wget\nfrom ast import literal_eval\n\n# Redis client library for Python\nimport redis\n\n# I've set this to our new embeddings model, this can be changed to the embedding model of your choice\nEMBEDDING_MODEL = \"text-embedding-3-small\"\n\n# Ignore unclosed SSL socket warnings - optional in case you get these errors\nimport warnings\n\nwarnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning) \n```\n\n## Load data\n\nIn this section we'll load embedded data that we've prepared previous to this session.\n\n\n```python\nembeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'\n\n# The file is ~700 MB so this will take some time\nwget.download(embeddings_url)\n```\n\n\n```python\nimport zipfile\nwith zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n    zip_ref.extractall(\"../data\")\n```\n\n\n```python\narticle_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')\n```\n\n\n```python\narticle_df.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>url</th>\n      <th>title</th>\n      <th>text</th>\n      <th>title_vector</th>\n      <th>content_vector</th>\n      <th>vector_id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>https://simple.wikipedia.org/wiki/April</td>\n      <td>April</td>\n      <td>April is the fourth month of the year in the J...</td>\n      <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n      <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>https://simple.wikipedia.org/wiki/August</td>\n      <td>August</td>\n      <td>August (Aug.)"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 7e528539-7bf6-4860-bb7e-9a5f932a5dcd
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"7e528539-7bf6-4860-bb7e-9a5f932a5dcd\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3adbfd21-8657-4768-9e3c-962d88c32a76\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3bdefec7716901ce665401ca02cd191586bb259932bb6115ff0b3a7de65a8825\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"188d3e2f-52d7-4e49-8c9c-4f07a222d691\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"e476af5894888abcb5f1bd0a2fd3bc086e400bf81f3467c9b4768f685dc2e46d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"75fc6cae-7cc4-4f60-899e-104b03b8ea17\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"18e7b60db5538910c0b015b09df347c6a085ba812f72bb29fa3eeccb225ebb90\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3495, \"end_char_idx\": 5673, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "ref_doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "chroma:document": "```python\narticle_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')\n```\n\n\n```python\narticle_df.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>url</th>\n      <th>title</th>\n      <th>text</th>\n      <th>title_vector</th>\n      <th>content_vector</th>\n      <th>vector_id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>https://simple.wikipedia.org/wiki/April</td>\n      <td>April</td>\n      <td>April is the fourth month of the year in the J...</td>\n      <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n      <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>https://simple.wikipedia.org/wiki/August</td>\n      <td>August</td>\n      <td>August (Aug.) is the eighth month of the year ...</td>\n      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>6</td>\n      <td>https://simple.wikipedia.org/wiki/Art</td>\n      <td>Art</td>\n      <td>Art is a creative activity that expresses imag...</td>\n      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>8</td>\n      <td>https://simple.wikipedia.org/wiki/A</td>\n      <td>A</td>\n      <td>A or a is the first letter of the English alph...</td>\n      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n      <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>9</td>\n      <td>https://simple.wikipedia.org/wiki/Air</td>\n      <td>Air</td>\n      <td>Air refers to the Earth's atmosphere."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 75fc6cae-7cc4-4f60-899e-104b03b8ea17
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"75fc6cae-7cc4-4f60-899e-104b03b8ea17\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3adbfd21-8657-4768-9e3c-962d88c32a76\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3bdefec7716901ce665401ca02cd191586bb259932bb6115ff0b3a7de65a8825\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7e528539-7bf6-4860-bb7e-9a5f932a5dcd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"e04f7b4560675863f7fc0d9d6ed26ae14bef1cea93b210fb462e81c1de5ff7d0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dd0f910d-d710-48b9-9ca5-a34c77f83bb8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"647a53a8136e9b26b4b4b4cefb65311c24b5a2c31beb16c74dc5582c5bf676be\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4643, \"end_char_idx\": 6972, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "ref_doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "chroma:document": "is the eighth month of the year ...</td>\n      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>6</td>\n      <td>https://simple.wikipedia.org/wiki/Art</td>\n      <td>Art</td>\n      <td>Art is a creative activity that expresses imag...</td>\n      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>8</td>\n      <td>https://simple.wikipedia.org/wiki/A</td>\n      <td>A</td>\n      <td>A or a is the first letter of the English alph...</td>\n      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n      <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>9</td>\n      <td>https://simple.wikipedia.org/wiki/Air</td>\n      <td>Air</td>\n      <td>Air refers to the Earth's atmosphere. Air is a...</td>\n      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n      <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n\n```python\n# Read vectors from strings back into a list\narticle_df['title_vector'] = article_df.title_vector.apply(literal_eval)\narticle_df['content_vector'] = article_df.content_vector.apply(literal_eval)\n\n# Set vector_id to be a string\narticle_df['vector_id'] = article_df['vector_id'].apply(str)\n```\n\n\n```python\narticle_df.info(show_counts=True)\n```\n\n    <class 'pandas.core.frame.DataFrame'>\n    RangeIndex: 25000 entries, 0 to 24999\n    Data columns (total 7 columns):\n     #   Column          Non-Null Count  Dtype \n    ---  ------          --------------  ----- \n     0   id              25000 non-null  int64 \n     1   url             25000 non-null  object\n     2   title           25000 non-null  object\n     3   text            25000 non-null  object\n     4   title_vector    25000 non-null  object\n     5   content_vector  25000 non-null  object\n     6   vector_id       25000 non-null  object\n    dtypes: int64(1), object(6)\n    memory usage: 1.3+ MB\n\n\n# Redis\n\nThe next vector database covered in this tutorial is **[Redis](https://redis.io)**. You most likely already know Redis."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: dd0f910d-d710-48b9-9ca5-a34c77f83bb8
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"dd0f910d-d710-48b9-9ca5-a34c77f83bb8\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3adbfd21-8657-4768-9e3c-962d88c32a76\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3bdefec7716901ce665401ca02cd191586bb259932bb6115ff0b3a7de65a8825\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"75fc6cae-7cc4-4f60-899e-104b03b8ea17\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"08e33b8ec000596a573bb0170811045acdd37a4d6dd19d2571f032bdf07ebcb9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"31b5c796-f3fe-43d5-b17e-b850195e505c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"06777b422b8a962b9e3d3037f35aa5facb415c0cb6155bf970990a0f6ca82fbf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5674, \"end_char_idx\": 8400, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "ref_doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "chroma:document": "Air is a...</td>\n      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n      <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n\n```python\n# Read vectors from strings back into a list\narticle_df['title_vector'] = article_df.title_vector.apply(literal_eval)\narticle_df['content_vector'] = article_df.content_vector.apply(literal_eval)\n\n# Set vector_id to be a string\narticle_df['vector_id'] = article_df['vector_id'].apply(str)\n```\n\n\n```python\narticle_df.info(show_counts=True)\n```\n\n    <class 'pandas.core.frame.DataFrame'>\n    RangeIndex: 25000 entries, 0 to 24999\n    Data columns (total 7 columns):\n     #   Column          Non-Null Count  Dtype \n    ---  ------          --------------  ----- \n     0   id              25000 non-null  int64 \n     1   url             25000 non-null  object\n     2   title           25000 non-null  object\n     3   text            25000 non-null  object\n     4   title_vector    25000 non-null  object\n     5   content_vector  25000 non-null  object\n     6   vector_id       25000 non-null  object\n    dtypes: int64(1), object(6)\n    memory usage: 1.3+ MB\n\n\n# Redis\n\nThe next vector database covered in this tutorial is **[Redis](https://redis.io)**. You most likely already know Redis. What you might not be aware of is the [RediSearch module](https://github.com/RediSearch/RediSearch). Enterprises have been using Redis with the RediSearch module for years now across all major cloud providers, Redis Cloud, and on premise. Recently, the Redis team added vector storage and search capability to this module in addition to the features RediSearch already had.\n\nGiven the large ecosystem around Redis, there are most likely client libraries in the language you need. You can use any standard Redis client library to run RediSearch commands, but it's easiest to use a library that wraps the RediSearch API. Below are a few examples, but you can find more client libraries [here](https://redis.io/resources/clients/).\n\n| Project | Language | License | Author | Stars |\n|----------|---------|--------|---------|-------|\n| [jedis][jedis-url] | Java | MIT | [Redis][redis-url] | ![Stars][jedis-stars] |\n| [redis-py][redis-py-url] | Python | MIT | [Redis][redis-url] | ![Stars][redis-py-stars] |\n| [node-redis][node-redis-url] | Node.js | MIT | [Redis][redis-url] | ![Stars][node-redis-stars] |\n| [nredisstack][nredisstack-url] | .NET | MIT | [Redis][redis-url] | ![Stars][nredisstack-stars] |\n| [redisearch-go][redisearch-go-url] | Go | BSD | [Redis][redisearch-go-author] | [![redisearch-go-stars]][redisearch-go-url] |\n| [redisearch-api-rs][redisearch-api-rs-url] | Rust | BSD | [Redis][redisearch-api-rs-author] | [!"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 31b5c796-f3fe-43d5-b17e-b850195e505c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"31b5c796-f3fe-43d5-b17e-b850195e505c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3adbfd21-8657-4768-9e3c-962d88c32a76\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3bdefec7716901ce665401ca02cd191586bb259932bb6115ff0b3a7de65a8825\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dd0f910d-d710-48b9-9ca5-a34c77f83bb8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7a0c60b065020ea12ccfc129569d6abc3f269da1a329b5b73b62c9c99350b136\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c22885d1-d3ff-4fad-93e8-8aca5cba6235\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"845a012f52e37a98b828c0f17f31fdfd1a68f9a6186ab60e3678e5e6f272f198\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6937, \"end_char_idx\": 10175, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "ref_doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "chroma:document": "You most likely already know Redis. What you might not be aware of is the [RediSearch module](https://github.com/RediSearch/RediSearch). Enterprises have been using Redis with the RediSearch module for years now across all major cloud providers, Redis Cloud, and on premise. Recently, the Redis team added vector storage and search capability to this module in addition to the features RediSearch already had.\n\nGiven the large ecosystem around Redis, there are most likely client libraries in the language you need. You can use any standard Redis client library to run RediSearch commands, but it's easiest to use a library that wraps the RediSearch API. Below are a few examples, but you can find more client libraries [here](https://redis.io/resources/clients/).\n\n| Project | Language | License | Author | Stars |\n|----------|---------|--------|---------|-------|\n| [jedis][jedis-url] | Java | MIT | [Redis][redis-url] | ![Stars][jedis-stars] |\n| [redis-py][redis-py-url] | Python | MIT | [Redis][redis-url] | ![Stars][redis-py-stars] |\n| [node-redis][node-redis-url] | Node.js | MIT | [Redis][redis-url] | ![Stars][node-redis-stars] |\n| [nredisstack][nredisstack-url] | .NET | MIT | [Redis][redis-url] | ![Stars][nredisstack-stars] |\n| [redisearch-go][redisearch-go-url] | Go | BSD | [Redis][redisearch-go-author] | [![redisearch-go-stars]][redisearch-go-url] |\n| [redisearch-api-rs][redisearch-api-rs-url] | Rust | BSD | [Redis][redisearch-api-rs-author] | [![redisearch-api-rs-stars]][redisearch-api-rs-url] |\n\n[redis-url]: https://redis.com\n\n[redis-py-url]: https://github.com/redis/redis-py\n[redis-py-stars]: https://img.shields.io/github/stars/redis/redis-py.svg?style=social&amp;label=Star&amp;maxAge=2592000\n[redis-py-package]: https://pypi.python.org/pypi/redis\n\n[jedis-url]: https://github.com/redis/jedis\n[jedis-stars]: https://img.shields.io/github/stars/redis/jedis.svg?style=social&amp;label=Star&amp;maxAge=2592000\n[Jedis-package]: https://search.maven.org/artifact/redis.clients/jedis\n\n[nredisstack-url]: https://github.com/redis/nredisstack\n[nredisstack-stars]: https://img.shields.io/github/stars/redis/nredisstack.svg?style=social&amp;label=Star&amp;maxAge=2592000\n[nredisstack-package]: https://www.nuget.org/packages/nredisstack/\n\n[node-redis-url]: https://github.com/redis/node-redis\n[node-redis-stars]: https://img.shields.io/github/stars/redis/node-redis.svg?style=social&amp;label=Star&amp;maxAge=2592000\n[node-redis-package]: https://www.npmjs.com/package/redis\n\n[redis-om-python-url]: https://github.com/redis/redis-om-python\n[redis-om-python-author]: https://redis.com\n[redis-om-python-stars]: https://img.shields.io/github/stars/redis/redis-om-python.svg?style=social&amp;label=Star&amp;maxAge=2592000\n\n[redisearch-go-url]: https://github.com/RediSearch/redisearch-go\n[redisearch-go-author]: https://redis.com\n[redisearch-go-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-go.svg?style=social&amp;label=Star&amp;maxAge=2592000\n\n[redisearch-api-rs-url]: https://github.com/RediSearch/redisearch-api-rs\n[redisearch-api-rs-author]: https://redis.com\n[redisearch-api-rs-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-api-rs.svg?style=social&amp;label=Star&amp;maxAge=2592000"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: c22885d1-d3ff-4fad-93e8-8aca5cba6235
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"c22885d1-d3ff-4fad-93e8-8aca5cba6235\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3adbfd21-8657-4768-9e3c-962d88c32a76\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3bdefec7716901ce665401ca02cd191586bb259932bb6115ff0b3a7de65a8825\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"31b5c796-f3fe-43d5-b17e-b850195e505c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7afc5350feeeb783aa150eafa100e6bf52c13c23fa8f78e06b3402945235a9db\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"df249d63-808e-4ff7-96ca-ffb6eaa9acf3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4b9ab54977f54ba8b00795c25cff3079b45023950e8f5130d45a560a1d613543\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10178, \"end_char_idx\": 13003, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "ref_doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "chroma:document": "In the below cells, we will walk you through using Redis as a vector database. Since many of you are likely already used to the Redis API, this should be familiar to most.\n\n## Setup\n\nThere are many ways to deploy Redis with RediSearch. The easiest way to get started is to use Docker, but there are are many potential options for deployment. For other deployment options, see the [redis directory](./redis) in this repo.\n\nFor this tutorial, we will use Redis Stack on Docker.\n\nStart a version of Redis with RediSearch (Redis Stack) by running the following docker command\n\n```bash\n$ cd redis\n$ docker compose up -d\n```\nThis also includes the [RedisInsight](https://redis.com/redis-enterprise/redis-insight/) GUI for managing your Redis database which you can view at [http://localhost:8001](http://localhost:8001) once you start the docker container.\n\nYou're all set up and ready to go! Next, we import and create our client for communicating with the Redis database we just created.\n\n\n```python\nimport redis\nfrom redis.commands.search.indexDefinition import (\n    IndexDefinition,\n    IndexType\n)\nfrom redis.commands.search.query import Query\nfrom redis.commands.search.field import (\n    TextField,\n    VectorField\n)\n\nREDIS_HOST =  \"localhost\"\nREDIS_PORT = 6379\nREDIS_PASSWORD = \"\" # default for passwordless Redis\n\n# Connect to Redis\nredis_client = redis.Redis(\n    host=REDIS_HOST,\n    port=REDIS_PORT,\n    password=REDIS_PASSWORD\n)\nredis_client.ping()\n```\n\n\n\n\n    True\n\n\n\n## Creating a Search Index\n\nThe below cells will show how to specify and create a search index in Redis. We will\n\n1. Set some constants for defining our index like the distance metric and the index name\n2. Define the index schema with RediSearch fields\n3. Create the index\n\n\n\n```python\n# Constants\nVECTOR_DIM = len(article_df['title_vector'][0]) # length of the vectors\nVECTOR_NUMBER = len(article_df)                 # initial number of vectors\nINDEX_NAME = \"embeddings-index\"                 # name of the search index\nPREFIX = \"doc\"                                  # prefix for the document keys\nDISTANCE_METRIC = \"COSINE\"                      # distance metric for the vectors (ex. COSINE, IP, L2)\n```\n\n\n```python\n# Define RediSearch fields for each of the columns in the dataset\ntitle = TextField(name=\"title\")\nurl = TextField(name=\"url\")\ntext = TextField(name=\"text\")\ntitle_embedding = VectorField(\"title_vector\",\n    \"FLAT\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER,\n    }\n)\ntext_embedding = VectorField(\"content_vector\",\n    \"FLAT\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER,\n    }\n)\nfields = [title, url, text, title_embedding, text_embedding]\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: df249d63-808e-4ff7-96ca-ffb6eaa9acf3
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"df249d63-808e-4ff7-96ca-ffb6eaa9acf3\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3adbfd21-8657-4768-9e3c-962d88c32a76\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3bdefec7716901ce665401ca02cd191586bb259932bb6115ff0b3a7de65a8825\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c22885d1-d3ff-4fad-93e8-8aca5cba6235\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"64fbe990c43a77e9a9e8371638cbf21d7b754abac300134d294e8da0575c0942\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3835003c-5d4e-4cd0-b1e3-a195a7ab80a1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"39ab50975f0a16a65a7b2763f6d87b336d84a7e1a0abc07e558b57eddd8b6425\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11647, \"end_char_idx\": 14591, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "ref_doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "chroma:document": "True\n\n\n\n## Creating a Search Index\n\nThe below cells will show how to specify and create a search index in Redis. We will\n\n1. Set some constants for defining our index like the distance metric and the index name\n2. Define the index schema with RediSearch fields\n3. Create the index\n\n\n\n```python\n# Constants\nVECTOR_DIM = len(article_df['title_vector'][0]) # length of the vectors\nVECTOR_NUMBER = len(article_df)                 # initial number of vectors\nINDEX_NAME = \"embeddings-index\"                 # name of the search index\nPREFIX = \"doc\"                                  # prefix for the document keys\nDISTANCE_METRIC = \"COSINE\"                      # distance metric for the vectors (ex. COSINE, IP, L2)\n```\n\n\n```python\n# Define RediSearch fields for each of the columns in the dataset\ntitle = TextField(name=\"title\")\nurl = TextField(name=\"url\")\ntext = TextField(name=\"text\")\ntitle_embedding = VectorField(\"title_vector\",\n    \"FLAT\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER,\n    }\n)\ntext_embedding = VectorField(\"content_vector\",\n    \"FLAT\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER,\n    }\n)\nfields = [title, url, text, title_embedding, text_embedding]\n```\n\n\n```python\n# Check if index exists\ntry:\n    redis_client.ft(INDEX_NAME).info()\n    print(\"Index already exists\")\nexcept:\n    # Create RediSearch Index\n    redis_client.ft(INDEX_NAME).create_index(\n        fields = fields,\n        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n    )\n```\n\n## Load Documents into the Index\n\nNow that we have a search index, we can load documents into it. We will use the same documents we used in the previous examples. In Redis, either the Hash or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The below cells will show how to load documents into the index.\n\n\n```python\ndef index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):\n    records = documents.to_dict(\"records\")\n    for doc in records:\n        key = f\"{prefix}:{str(doc['id'])}\"\n\n        # create byte vectors for title and content\n        title_embedding = np.array(doc[\"title_vector\"], dtype=np.float32).tobytes()\n        content_embedding = np.array(doc[\"content_vector\"], dtype=np.float32).tobytes()\n\n        # replace list of floats with byte vectors\n        doc[\"title_vector\"] = title_embedding\n        doc[\"content_vector\"] = content_embedding\n\n        client.hset(key, mapping = doc)\n```\n\n\n```python\nindex_documents(redis_client, PREFIX, article_df)\nprint(f\"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}\")\n```\n\n    Loaded 25000 documents in Redis search index with name: embeddings-index"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 3835003c-5d4e-4cd0-b1e3-a195a7ab80a1
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"3835003c-5d4e-4cd0-b1e3-a195a7ab80a1\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3adbfd21-8657-4768-9e3c-962d88c32a76\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3bdefec7716901ce665401ca02cd191586bb259932bb6115ff0b3a7de65a8825\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"df249d63-808e-4ff7-96ca-ffb6eaa9acf3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"c28097a506d63bd6c325b1f5fff14c0e174c3163d64074abcbcc52509f15ba9a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cd1fb819-8e73-4a9f-a379-31e5e9e9e4db\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"14c41fb6388a9b347e8d726c9e619ceb3216ba48c7a37f4be7844bc2417bedb4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13006, \"end_char_idx\": 15568, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "ref_doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "chroma:document": "```python\n# Check if index exists\ntry:\n    redis_client.ft(INDEX_NAME).info()\n    print(\"Index already exists\")\nexcept:\n    # Create RediSearch Index\n    redis_client.ft(INDEX_NAME).create_index(\n        fields = fields,\n        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n    )\n```\n\n## Load Documents into the Index\n\nNow that we have a search index, we can load documents into it. We will use the same documents we used in the previous examples. In Redis, either the Hash or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The below cells will show how to load documents into the index.\n\n\n```python\ndef index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):\n    records = documents.to_dict(\"records\")\n    for doc in records:\n        key = f\"{prefix}:{str(doc['id'])}\"\n\n        # create byte vectors for title and content\n        title_embedding = np.array(doc[\"title_vector\"], dtype=np.float32).tobytes()\n        content_embedding = np.array(doc[\"content_vector\"], dtype=np.float32).tobytes()\n\n        # replace list of floats with byte vectors\n        doc[\"title_vector\"] = title_embedding\n        doc[\"content_vector\"] = content_embedding\n\n        client.hset(key, mapping = doc)\n```\n\n\n```python\nindex_documents(redis_client, PREFIX, article_df)\nprint(f\"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}\")\n```\n\n    Loaded 25000 documents in Redis search index with name: embeddings-index\n\n\n## Running Search Queries\n\nNow that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database. Each example will demonstrate specific features to keep in mind when developing your search application with Redis.\n\n1. **Return Fields**: You can specify which fields you want to return in the search results. This is useful if you only want to return a subset of the fields in your documents and doesn't require a separate call to retrieve documents. In the below example, we will only return the `title` field in the search results.\n2. **Hybrid Search**: You can combine vector search with any of the other RediSearch fields for hybrid search such as full text search, tag, geo, and numeric. In the below example, we will combine vector search with full text search."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: cd1fb819-8e73-4a9f-a379-31e5e9e9e4db
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"cd1fb819-8e73-4a9f-a379-31e5e9e9e4db\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3adbfd21-8657-4768-9e3c-962d88c32a76\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3bdefec7716901ce665401ca02cd191586bb259932bb6115ff0b3a7de65a8825\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3835003c-5d4e-4cd0-b1e3-a195a7ab80a1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"16c5f4e6f8ab2fd5594e5cdb8743dee485b49664bb6c0aa75c4460f3417f13f4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9ea578c5-e988-4938-9cc4-6228953ca51c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5765ce945710b5e3b8c7b151bd19436eaaff407dedd1df6a8bf9fdfd431e2c4a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13720, \"end_char_idx\": 16762, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "ref_doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "chroma:document": "```python\ndef index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):\n    records = documents.to_dict(\"records\")\n    for doc in records:\n        key = f\"{prefix}:{str(doc['id'])}\"\n\n        # create byte vectors for title and content\n        title_embedding = np.array(doc[\"title_vector\"], dtype=np.float32).tobytes()\n        content_embedding = np.array(doc[\"content_vector\"], dtype=np.float32).tobytes()\n\n        # replace list of floats with byte vectors\n        doc[\"title_vector\"] = title_embedding\n        doc[\"content_vector\"] = content_embedding\n\n        client.hset(key, mapping = doc)\n```\n\n\n```python\nindex_documents(redis_client, PREFIX, article_df)\nprint(f\"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}\")\n```\n\n    Loaded 25000 documents in Redis search index with name: embeddings-index\n\n\n## Running Search Queries\n\nNow that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database. Each example will demonstrate specific features to keep in mind when developing your search application with Redis.\n\n1. **Return Fields**: You can specify which fields you want to return in the search results. This is useful if you only want to return a subset of the fields in your documents and doesn't require a separate call to retrieve documents. In the below example, we will only return the `title` field in the search results.\n2. **Hybrid Search**: You can combine vector search with any of the other RediSearch fields for hybrid search such as full text search, tag, geo, and numeric. In the below example, we will combine vector search with full text search.\n\n\n\n```python\ndef search_redis(\n    redis_client: redis.Redis,\n    user_query: str,\n    index_name: str = \"embeddings-index\",\n    vector_field: str = \"title_vector\",\n    return_fields: list = [\"title\", \"url\", \"text\", \"vector_score\"],\n    hybrid_fields = \"*\",\n    k: int = 20,\n) -> List[dict]:\n\n    # Creates embedding vector from user query\n    embedded_query = openai.Embedding.create(input=user_query,\n                                            model=EMBEDDING_MODEL,\n                                            )[\"data\"][0]['embedding']\n\n    # Prepare the Query\n    base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'\n    query = (\n        Query(base_query)\n         .return_fields(*return_fields)\n         .sort_by(\"vector_score\")\n         .paging(0, k)\n         .dialect(2)\n    )\n    params_dict = {\"vector\": np.array(embedded_query).astype(dtype=np.float32).tobytes()}\n\n    # perform vector search\n    results = redis_client.ft(index_name).search(query, params_dict)\n    for i, article in enumerate(results.docs):\n        score = 1 - float(article.vector_score)\n        print(f\"{i}. {article.title} (Score: {round(score ,3) })\")\n    return results.docs\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 9ea578c5-e988-4938-9cc4-6228953ca51c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"9ea578c5-e988-4938-9cc4-6228953ca51c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3adbfd21-8657-4768-9e3c-962d88c32a76\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3bdefec7716901ce665401ca02cd191586bb259932bb6115ff0b3a7de65a8825\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cd1fb819-8e73-4a9f-a379-31e5e9e9e4db\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"66ccaa68872203cd5f25ae0df1b218940f77990e71f8cf489dc1bc4462c939d9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8aff7058-32c5-4219-a7cf-26e23d3860f9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b87ee3e2b78c1e1e0d083d238aa850d7def6c3e19e6b872ed42bcf58c8d1f8d0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15572, \"end_char_idx\": 18252, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "ref_doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "chroma:document": "```python\ndef search_redis(\n    redis_client: redis.Redis,\n    user_query: str,\n    index_name: str = \"embeddings-index\",\n    vector_field: str = \"title_vector\",\n    return_fields: list = [\"title\", \"url\", \"text\", \"vector_score\"],\n    hybrid_fields = \"*\",\n    k: int = 20,\n) -> List[dict]:\n\n    # Creates embedding vector from user query\n    embedded_query = openai.Embedding.create(input=user_query,\n                                            model=EMBEDDING_MODEL,\n                                            )[\"data\"][0]['embedding']\n\n    # Prepare the Query\n    base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'\n    query = (\n        Query(base_query)\n         .return_fields(*return_fields)\n         .sort_by(\"vector_score\")\n         .paging(0, k)\n         .dialect(2)\n    )\n    params_dict = {\"vector\": np.array(embedded_query).astype(dtype=np.float32).tobytes()}\n\n    # perform vector search\n    results = redis_client.ft(index_name).search(query, params_dict)\n    for i, article in enumerate(results.docs):\n        score = 1 - float(article.vector_score)\n        print(f\"{i}. {article.title} (Score: {round(score ,3) })\")\n    return results.docs\n```\n\n\n```python\n# For using OpenAI to generate query embedding\nopenai.api_key = os.getenv(\"OPENAI_API_KEY\", \"sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\")\nresults = search_redis(redis_client, 'modern art in Europe', k=10)\n```\n\n    0. Museum of Modern Art (Score: 0.875)\n    1. Western Europe (Score: 0.867)\n    2. Renaissance art (Score: 0.864)\n    3. Pop art (Score: 0.86)\n    4. Northern Europe (Score: 0.855)\n    5. Hellenistic art (Score: 0.853)\n    6. Modernist literature (Score: 0.847)\n    7. Art film (Score: 0.843)\n    8. Central Europe (Score: 0.843)\n    9. European (Score: 0.841)\n\n\n\n```python\nresults = search_redis(redis_client, 'Famous battles in Scottish history', vector_field='content_vector', k=10)\n```\n\n    0. Battle of Bannockburn (Score: 0.869)\n    1. Wars of Scottish Independence (Score: 0.861)\n    2. 1651 (Score: 0.853)\n    3. First War of Scottish Independence (Score: 0.85)\n    4. Robert I of Scotland (Score: 0.846)\n    5. 841 (Score: 0.844)\n    6. 1716 (Score: 0.844)\n    7. 1314 (Score: 0.837)\n    8. 1263 (Score: 0.836)\n    9. William Wallace (Score: 0.835)\n\n\n## Hybrid Queries with Redis\n\nThe previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the below example, we will combine vector search with full text search.\n\n\n```python\ndef create_hybrid_field(field_name: str, value: str) -> str:\n    return f'@{field_name}:\"{value}\"'\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 8aff7058-32c5-4219-a7cf-26e23d3860f9
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"8aff7058-32c5-4219-a7cf-26e23d3860f9\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3adbfd21-8657-4768-9e3c-962d88c32a76\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3bdefec7716901ce665401ca02cd191586bb259932bb6115ff0b3a7de65a8825\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9ea578c5-e988-4938-9cc4-6228953ca51c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"609e3b9e27ad7680c07683f42dd24f8a9da715c50b0ac06c39597165828e6718\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"590e602d-c88a-420b-aa98-f465dd82c155\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6cb263833ce2db73fc6ea420ea725838d71a38887aa9f8c5fbb0f7d90ed6e5ce\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17354, \"end_char_idx\": 20462, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "ref_doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "chroma:document": "```python\nresults = search_redis(redis_client, 'Famous battles in Scottish history', vector_field='content_vector', k=10)\n```\n\n    0. Battle of Bannockburn (Score: 0.869)\n    1. Wars of Scottish Independence (Score: 0.861)\n    2. 1651 (Score: 0.853)\n    3. First War of Scottish Independence (Score: 0.85)\n    4. Robert I of Scotland (Score: 0.846)\n    5. 841 (Score: 0.844)\n    6. 1716 (Score: 0.844)\n    7. 1314 (Score: 0.837)\n    8. 1263 (Score: 0.836)\n    9. William Wallace (Score: 0.835)\n\n\n## Hybrid Queries with Redis\n\nThe previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the below example, we will combine vector search with full text search.\n\n\n```python\ndef create_hybrid_field(field_name: str, value: str) -> str:\n    return f'@{field_name}:\"{value}\"'\n```\n\n\n```python\n# search the content vector for articles about famous battles in Scottish history and only include results with Scottish in the title\nresults = search_redis(redis_client,\n                       \"Famous battles in Scottish history\",\n                       vector_field=\"title_vector\",\n                       k=5,\n                       hybrid_fields=create_hybrid_field(\"title\", \"Scottish\")\n                       )\n```\n\n    0. First War of Scottish Independence (Score: 0.892)\n    1. Wars of Scottish Independence (Score: 0.889)\n    2. Second War of Scottish Independence (Score: 0.879)\n    3. List of Scottish monarchs (Score: 0.873)\n    4. Scottish Borders (Score: 0.863)\n\n\n\n```python\n# run a hybrid query for articles about Art in the title vector and only include results with the phrase \"Leonardo da Vinci\" in the text\nresults = search_redis(redis_client,\n                       \"Art\",\n                       vector_field=\"title_vector\",\n                       k=5,\n                       hybrid_fields=create_hybrid_field(\"text\", \"Leonardo da Vinci\")\n                       )\n\n# find specific mention of Leonardo da Vinci in the text that our full-text-search query returned\nmention = [sentence for sentence in results[0].text.split(\"\\n\") if \"Leonardo da Vinci\" in sentence][0]\nmention\n```\n\n    0. Art (Score: 1.0)\n    1. Paint (Score: 0.896)\n    2. Renaissance art (Score: 0.88)\n    3. Painting (Score: 0.874)\n    4. Renaissance (Score: 0.846)\n\n\n\n\n\n    'In Europe, after the Middle Ages, there was a \"Renaissance\" which means \"rebirth\". People rediscovered science and artists were allowed to paint subjects other than religious subjects. People like Michelangelo and Leonardo da Vinci still painted religious pictures, but they also now could paint mythological pictures too. These artists also invented perspective where things in the distance look smaller in the picture. This was new because in the Middle Ages people would paint all the figures close up and just overlapping each other. These artists used nudity regularly in their art.'\n\n\n\nFor more example with Redis as a vector database, see the README and examples within the ``vector_databases/redis`` directory of this repository"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 590e602d-c88a-420b-aa98-f465dd82c155
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"590e602d-c88a-420b-aa98-f465dd82c155\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"3adbfd21-8657-4768-9e3c-962d88c32a76\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3bdefec7716901ce665401ca02cd191586bb259932bb6115ff0b3a7de65a8825\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8aff7058-32c5-4219-a7cf-26e23d3860f9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb\", \"title\": \"Using Redis for Embeddings Search\", \"tokens\": 5244, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9b51aabfa4d7d36dfafc3f88d870c49ac69ca3d0502473002c8cf298eef2b4a4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18942, \"end_char_idx\": 20479, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "ref_doc_id": "3adbfd21-8657-4768-9e3c-962d88c32a76", "chroma:document": "```python\n# run a hybrid query for articles about Art in the title vector and only include results with the phrase \"Leonardo da Vinci\" in the text\nresults = search_redis(redis_client,\n                       \"Art\",\n                       vector_field=\"title_vector\",\n                       k=5,\n                       hybrid_fields=create_hybrid_field(\"text\", \"Leonardo da Vinci\")\n                       )\n\n# find specific mention of Leonardo da Vinci in the text that our full-text-search query returned\nmention = [sentence for sentence in results[0].text.split(\"\\n\") if \"Leonardo da Vinci\" in sentence][0]\nmention\n```\n\n    0. Art (Score: 1.0)\n    1. Paint (Score: 0.896)\n    2. Renaissance art (Score: 0.88)\n    3. Painting (Score: 0.874)\n    4. Renaissance (Score: 0.846)\n\n\n\n\n\n    'In Europe, after the Middle Ages, there was a \"Renaissance\" which means \"rebirth\". People rediscovered science and artists were allowed to paint subjects other than religious subjects. People like Michelangelo and Leonardo da Vinci still painted religious pictures, but they also now could paint mythological pictures too. These artists also invented perspective where things in the distance look smaller in the picture. This was new because in the Middle Ages people would paint all the figures close up and just overlapping each other. These artists used nudity regularly in their art.'\n\n\n\nFor more example with Redis as a vector database, see the README and examples within the ``vector_databases/redis`` directory of this repository\n\n\n```python\n\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 730980dc-faa2-40a4-8477-452ea790992b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"730980dc-faa2-40a4-8477-452ea790992b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb\", \"title\": \"Redis\", \"tokens\": 2363, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9e104ea9-de02-4b11-9b1b-8b3ceb8701a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb\", \"title\": \"Redis\", \"tokens\": 2363, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"dd9bdac203d53c40ee396b7bdfc544c39019ebfc1ee54b54944d9ed8a8a26207\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b363c602-e9c5-4203-9a06-257493cb8571\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"60252f48545fffa75c59a4eb516fd84fc4c1ac8dcd6a08004bbf9cef8f9bfd57\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2501, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9e104ea9-de02-4b11-9b1b-8b3ceb8701a4", "doc_id": "9e104ea9-de02-4b11-9b1b-8b3ceb8701a4", "ref_doc_id": "9e104ea9-de02-4b11-9b1b-8b3ceb8701a4", "chroma:document": "# Redis\n\n### What is Redis?\n\nMost developers from a web services background are probably familiar with Redis. At it's core, Redis is an open-source key-value store that can be used as a cache, message broker, and database. Developers choice Redis because it is fast, has a large ecosystem of client libraries, and has been deployed by major enterprises for years.\n\nIn addition to the traditional uses of Redis. Redis also provides [Redis Modules](https://redis.io/modules) which are a way to extend Redis with new capabilities, commands and data types. Example modules include [RedisJSON](https://redis.io/docs/stack/json/), [RedisTimeSeries](https://redis.io/docs/stack/timeseries/), [RedisBloom](https://redis.io/docs/stack/bloom/) and [RediSearch](https://redis.io/docs/stack/search/).\n\n\n### Deployment options\n\nThere are a number of ways to deploy Redis. For local development, the quickest method is to use the [Redis Stack docker container](https://hub.docker.com/r/redis/redis-stack) which we will use here. Redis Stack contains a number of Redis modules that can be used together to create a fast, multi-model data store and query engine.\n\nFor production use cases, The easiest way to get started is to use the [Redis Cloud](https://redislabs.com/redis-enterprise-cloud/overview/) service. Redis Cloud is a fully managed Redis service. You can also deploy Redis on your own infrastructure using [Redis Enterprise](https://redislabs.com/redis-enterprise/overview/). Redis Enterprise is a fully managed Redis service that can be deployed in kubernetes, on-premises or in the cloud.\n\nAdditionally, every major cloud provider ([AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-e6y7ork67pjwg?sr=0-2&ref_=beagle&applicationId=AWSMPContessa), [Google Marketplace](https://console.cloud.google.com/marketplace/details/redislabs-public/redis-enterprise?pli=1), or [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/garantiadata.redis_enterprise_1sp_public_preview?tab=Overview)) offers Redis Enterprise in a marketplace offering.\n\n\n### What is RediSearch?\n\nRediSearch is a [Redis module](https://redis.io/modules) that provides querying, secondary indexing, full-text search and vector search for Redis. To use RediSearch, you first declare indexes on your Redis data. You can then use the RediSearch clients to query that data. For more information on the feature set of RediSearch, see the [RediSearch documentation](https://redis.io/docs/stack/search/)."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: b363c602-e9c5-4203-9a06-257493cb8571
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"b363c602-e9c5-4203-9a06-257493cb8571\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb\", \"title\": \"Redis\", \"tokens\": 2363, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9e104ea9-de02-4b11-9b1b-8b3ceb8701a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb\", \"title\": \"Redis\", \"tokens\": 2363, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"dd9bdac203d53c40ee396b7bdfc544c39019ebfc1ee54b54944d9ed8a8a26207\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"730980dc-faa2-40a4-8477-452ea790992b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb\", \"title\": \"Redis\", \"tokens\": 2363, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"68ec2c580913a9ed1c819d974c9cc1d5d8583cb73cb1c7eed281c5d9e87045a8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a1fcd0bd-9b4a-4c6d-b101-ea6f12f74061\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cb7257b683082a9e2a155f60bb6de86a717353be30ded39f860e2f3d5be40b32\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 791, \"end_char_idx\": 3909, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9e104ea9-de02-4b11-9b1b-8b3ceb8701a4", "doc_id": "9e104ea9-de02-4b11-9b1b-8b3ceb8701a4", "ref_doc_id": "9e104ea9-de02-4b11-9b1b-8b3ceb8701a4", "chroma:document": "### Deployment options\n\nThere are a number of ways to deploy Redis. For local development, the quickest method is to use the [Redis Stack docker container](https://hub.docker.com/r/redis/redis-stack) which we will use here. Redis Stack contains a number of Redis modules that can be used together to create a fast, multi-model data store and query engine.\n\nFor production use cases, The easiest way to get started is to use the [Redis Cloud](https://redislabs.com/redis-enterprise-cloud/overview/) service. Redis Cloud is a fully managed Redis service. You can also deploy Redis on your own infrastructure using [Redis Enterprise](https://redislabs.com/redis-enterprise/overview/). Redis Enterprise is a fully managed Redis service that can be deployed in kubernetes, on-premises or in the cloud.\n\nAdditionally, every major cloud provider ([AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-e6y7ork67pjwg?sr=0-2&ref_=beagle&applicationId=AWSMPContessa), [Google Marketplace](https://console.cloud.google.com/marketplace/details/redislabs-public/redis-enterprise?pli=1), or [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/garantiadata.redis_enterprise_1sp_public_preview?tab=Overview)) offers Redis Enterprise in a marketplace offering.\n\n\n### What is RediSearch?\n\nRediSearch is a [Redis module](https://redis.io/modules) that provides querying, secondary indexing, full-text search and vector search for Redis. To use RediSearch, you first declare indexes on your Redis data. You can then use the RediSearch clients to query that data. For more information on the feature set of RediSearch, see the [RediSearch documentation](https://redis.io/docs/stack/search/).\n\n\n### Features\n\nRediSearch uses compressed, inverted indexes for fast indexing with a low memory footprint. RediSearch indexes enhance Redis by providing exact-phrase matching, fuzzy search, and numeric filtering, among many other features. Such as:\n\n* Full-Text indexing of multiple fields in Redis hashes\n* Incremental indexing without performance loss\n* Vector similarity search\n* Document ranking (using [tf-idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf), with optional user-provided weights)\n* Field weighting\n* Complex boolean queries with AND, OR, and NOT operators\n* Prefix matching, fuzzy matching, and exact-phrase queries\n* Support for [double-metaphone phonetic matching](https://redis.io/docs/stack/search/reference/phonetic_matching/)\n* Auto-complete suggestions (with fuzzy prefix suggestions)\n* Stemming-based query expansion in [many languages](https://redis.io/docs/stack/search/reference/stemming/) (using [Snowball](http://snowballstem.org/))\n* Support for Chinese-language tokenization and querying (using [Friso](https://github.com/lionsoul2014/friso))\n* Numeric filters and ranges\n* Geospatial searches using [Redis geospatial indexing](/commands/georadius)\n* A powerful aggregations engine\n* Supports for all utf-8 encoded text\n* Retrieve full documents, selected fields, or only the document IDs\n* Sorting results (for example, by creation date)\n* JSON support through RedisJSON"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: a1fcd0bd-9b4a-4c6d-b101-ea6f12f74061
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"a1fcd0bd-9b4a-4c6d-b101-ea6f12f74061\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb\", \"title\": \"Redis\", \"tokens\": 2363, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9e104ea9-de02-4b11-9b1b-8b3ceb8701a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb\", \"title\": \"Redis\", \"tokens\": 2363, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"dd9bdac203d53c40ee396b7bdfc544c39019ebfc1ee54b54944d9ed8a8a26207\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b363c602-e9c5-4203-9a06-257493cb8571\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb\", \"title\": \"Redis\", \"tokens\": 2363, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"08c6b497044c08cec949d92f1e5db01f87dbf5c3ca18abe78a2addb16d693272\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9cefccf1-367f-4753-9429-4e834db1cfc4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dc5a05be55dbfc5b2b58c5484e6582f2da7e8cd8fdf126f134a2b850ab78a20c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2504, \"end_char_idx\": 6474, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9e104ea9-de02-4b11-9b1b-8b3ceb8701a4", "doc_id": "9e104ea9-de02-4b11-9b1b-8b3ceb8701a4", "ref_doc_id": "9e104ea9-de02-4b11-9b1b-8b3ceb8701a4", "chroma:document": "### Features\n\nRediSearch uses compressed, inverted indexes for fast indexing with a low memory footprint. RediSearch indexes enhance Redis by providing exact-phrase matching, fuzzy search, and numeric filtering, among many other features. Such as:\n\n* Full-Text indexing of multiple fields in Redis hashes\n* Incremental indexing without performance loss\n* Vector similarity search\n* Document ranking (using [tf-idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf), with optional user-provided weights)\n* Field weighting\n* Complex boolean queries with AND, OR, and NOT operators\n* Prefix matching, fuzzy matching, and exact-phrase queries\n* Support for [double-metaphone phonetic matching](https://redis.io/docs/stack/search/reference/phonetic_matching/)\n* Auto-complete suggestions (with fuzzy prefix suggestions)\n* Stemming-based query expansion in [many languages](https://redis.io/docs/stack/search/reference/stemming/) (using [Snowball](http://snowballstem.org/))\n* Support for Chinese-language tokenization and querying (using [Friso](https://github.com/lionsoul2014/friso))\n* Numeric filters and ranges\n* Geospatial searches using [Redis geospatial indexing](/commands/georadius)\n* A powerful aggregations engine\n* Supports for all utf-8 encoded text\n* Retrieve full documents, selected fields, or only the document IDs\n* Sorting results (for example, by creation date)\n* JSON support through RedisJSON\n\n\n### Clients\n\nGiven the large ecosystem around Redis, there are most likely client libraries in the language you need. You can use any standard Redis client library to run RediSearch commands, but it's easiest to use a library that wraps the RediSearch API. Below are a few examples, but you can find more client libraries [here](https://redis.io/resources/clients/).\n\n| Project | Language | License | Author | Stars |\n|----------|---------|--------|---------|-------|\n| [jedis][jedis-url] | Java | MIT | [Redis][redis-url] | ![Stars][jedis-stars] |\n| [redis-py][redis-py-url] | Python | MIT | [Redis][redis-url] | ![Stars][redis-py-stars] |\n| [node-redis][node-redis-url] | Node.js | MIT | [Redis][redis-url] | ![Stars][node-redis-stars] |\n| [nredisstack][nredisstack-url] | .NET | MIT | [Redis][redis-url] | ![Stars][nredisstack-stars] |\n\n[redis-url]: https://redis.com\n\n[redis-py-url]: https://github.com/redis/redis-py\n[redis-py-stars]: https://img.shields.io/github/stars/redis/redis-py.svg?style=social&amp;label=Star&amp;maxAge=2592000\n[redis-py-package]: https://pypi.python.org/pypi/redis\n\n[jedis-url]: https://github.com/redis/jedis\n[jedis-stars]: https://img.shields.io/github/stars/redis/jedis.svg?style=social&amp;label=Star&amp;maxAge=2592000\n[Jedis-package]: https://search.maven.org/artifact/redis.clients/jedis\n\n[nredisstack-url]: https://github.com/redis/nredisstack\n[nredisstack-stars]: https://img.shields.io/github/stars/redis/nredisstack.svg?style=social&amp;label=Star&amp;maxAge=2592000\n[nredisstack-package]: https://www.nuget.org/packages/nredisstack/\n\n[node-redis-url]: https://github.com/redis/node-redis\n[node-redis-stars]: https://img.shields.io/github/stars/redis/node-redis.svg?style=social&amp;label=Star&amp;maxAge=2592000\n[node-redis-package]: https://www.npmjs.com/package/redis\n\n[redis-om-python-url]: https://github.com/redis/redis-om-python\n[redis-om-python-author]: https://redis.com\n[redis-om-python-stars]: https://img.shields.io/github/stars/redis/redis-om-python.svg?style=social&amp;label=Star&amp;maxAge=2592000\n\n[redisearch-go-url]: https://github.com/RediSearch/redisearch-go\n[redisearch-go-author]: https://redis.com\n[redisearch-go-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-go.svg?style=social&amp;label=Star&amp;maxAge=2592000\n\n[redisearch-api-rs-url]: https://github.com/RediSearch/redisearch-api-rs\n[redisearch-api-rs-author]: https://redis.com\n[redisearch-api-rs-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-api-rs.svg?style=social&amp;label=Star&amp;maxAge=2592000"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 9cefccf1-367f-4753-9429-4e834db1cfc4
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"9cefccf1-367f-4753-9429-4e834db1cfc4\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb\", \"title\": \"Redis\", \"tokens\": 2363, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"9e104ea9-de02-4b11-9b1b-8b3ceb8701a4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb\", \"title\": \"Redis\", \"tokens\": 2363, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"dd9bdac203d53c40ee396b7bdfc544c39019ebfc1ee54b54944d9ed8a8a26207\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a1fcd0bd-9b4a-4c6d-b101-ea6f12f74061\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb\", \"title\": \"Redis\", \"tokens\": 2363, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"48676b29323bf31b682a1633f05b7302eb1f87f1ec29e3af7cdf9746be318cfe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6477, \"end_char_idx\": 9609, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "9e104ea9-de02-4b11-9b1b-8b3ceb8701a4", "doc_id": "9e104ea9-de02-4b11-9b1b-8b3ceb8701a4", "ref_doc_id": "9e104ea9-de02-4b11-9b1b-8b3ceb8701a4", "chroma:document": "### Deployment Options\n\nThere are many ways to deploy Redis with RediSearch. The easiest way to get started is to use Docker, but there are are many potential options for deployment such as\n\n- [Redis Cloud](https://redis.com/redis-enterprise-cloud/overview/)\n- Cloud marketplaces: [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-e6y7ork67pjwg?sr=0-2&ref_=beagle&applicationId=AWSMPContessa), [Google Marketplace](https://console.cloud.google.com/marketplace/details/redislabs-public/redis-enterprise?pli=1), or [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/garantiadata.redis_enterprise_1sp_public_preview?tab=Overview)\n- On-premise: [Redis Enterprise Software](https://redis.com/redis-enterprise-software/overview/)\n- Kubernetes: [Redis Enterprise Software on Kubernetes](https://docs.redis.com/latest/kubernetes/)\n- [Docker (RediSearch)](https://hub.docker.com/r/redislabs/redisearch)\n- [Docker (Redis Stack)](https://hub.docker.com/r/redis/redis-stack)\n\n\n### Cluster support\n\nRediSearch has a distributed cluster version that scales to billions of documents across hundreds of servers. At the moment, distributed RediSearch is available as part of [Redis Enterprise Cloud](https://redis.com/redis-enterprise-cloud/overview/) and [Redis Enterprise Software](https://redis.com/redis-enterprise-software/overview/).\n\nSee [RediSearch on Redis Enterprise](https://redis.com/modules/redisearch/) for more information.\n\n### Examples\n\n- [Product Search](https://github.com/RedisVentures/redis-product-search) - eCommerce product search (with image and text)\n- [Product Recommendations with DocArray / Jina](https://github.com/jina-ai/product-recommendation-redis-docarray) - Content-based product recommendations example with Redis and DocArray.\n- [Redis VSS in RecSys](https://github.com/RedisVentures/Redis-Recsys) - 3 end-to-end Redis & NVIDIA Merlin Recommendation System Architectures.\n- [Azure OpenAI Embeddings Q&A](https://github.com/ruoccofabrizio/azure-open-ai-embeddings-qna) - OpenAI and Redis as a Q&A service on Azure.\n- [ArXiv Paper Search](https://github.com/RedisVentures/redis-arXiv-search) - Semantic search over arXiv scholarly papers\n\n\n### More Resources\n\nFor more information on how to use Redis as a vector database, check out the following resources:\n\n- [Redis Vector Similarity Docs](https://redis.io/docs/stack/search/reference/vectors/) - Redis official docs for Vector Search.\n- [Redis-py Search Docs](https://redis.readthedocs.io/en/latest/redismodules.html#redisearch-commands) - Redis-py client library docs for RediSearch.\n- [Vector Similarity Search: From Basics to Production](https://mlops.community/vector-similarity-search-from-basics-to-production/) - Introductory blog post to VSS and Redis as a VectorDB.\n- [AI-Powered Document Search](https://datasciencedojo.com/blog/ai-powered-document-search/) - Blog post covering AI Powered Document Search Use Cases & Architectures.\n- [Vector Database Benchmarks](https://jina.ai/news/benchmark-vector-search-databases-with-one-million-data/) - Jina AI VectorDB benchmarks comparing Redis against others."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 210b92bd-9b65-4f09-9cb5-88957eac2a07
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"210b92bd-9b65-4f09-9cb5-88957eac2a07\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7ac803a-183f-427d-b80e-e80c95d8b29c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a06dbbc18a21404ee251fbad7dcb97a77785448211aec31d3d9fde97033df4f8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"39ad06e1-f683-4a8a-ac4b-e7ed7791d9bf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d570b3617e31f41433e5b1d8fd413e18a61b6b06e39832d89f4fc2cfe4ddd3fe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3027, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "ref_doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "chroma:document": "# Using Redis as a Vector Database with OpenAI\n\nThis notebook provides an introduction to using Redis as a vector database with OpenAI embeddings. Redis is a scalable, real-time database that can be used as a vector database when using the [RediSearch Module](https://oss.redislabs.com/redisearch/). The RediSearch module allows you to index and search for vectors in Redis. This notebook will show you how to use the RediSearch module to index and search for vectors created by using the OpenAI API and stored in Redis.\n\n### What is Redis?\n\nMost developers from a web services background are probably familiar with Redis. At it's core, Redis is an open-source key-value store that can be used as a cache, message broker, and database. Developers choice Redis because it is fast, has a large ecosystem of client libraries, and has been deployed by major enterprises for years.\n\nIn addition to the traditional uses of Redis. Redis also provides [Redis Modules](https://redis.io/modules) which are a way to extend Redis with new data types and commands. Example modules include [RedisJSON](https://redis.io/docs/stack/json/), [RedisTimeSeries](https://redis.io/docs/stack/timeseries/), [RedisBloom](https://redis.io/docs/stack/bloom/) and [RediSearch](https://redis.io/docs/stack/search/).\n\n### What is RediSearch?\n\nRediSearch is a [Redis module](https://redis.io/modules) that provides querying, secondary indexing, full-text search and vector search for Redis. To use RediSearch, you first declare indexes on your Redis data. You can then use the RediSearch clients to query that data. For more information on the feature set of RediSearch, see the [README](./README.md) or the [RediSearch documentation](https://redis.io/docs/stack/search/).\n\n### Deployment options\n\nThere are a number of ways to deploy Redis. For local development, the quickest method is to use the [Redis Stack docker container](https://hub.docker.com/r/redis/redis-stack) which we will use here. Redis Stack contains a number of Redis modules that can be used together to create a fast, multi-model data store and query engine.\n\nFor production use cases, The easiest way to get started is to use the [Redis Cloud](https://redislabs.com/redis-enterprise-cloud/overview/) service. Redis Cloud is a fully managed Redis service. You can also deploy Redis on your own infrastructure using [Redis Enterprise](https://redislabs.com/redis-enterprise/overview/). Redis Enterprise is a fully managed Redis service that can be deployed in kubernetes, on-premises or in the cloud.\n\nAdditionally, every major cloud provider ([AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-e6y7ork67pjwg?sr=0-2&ref_=beagle&applicationId=AWSMPContessa), [Google Marketplace](https://console.cloud.google.com/marketplace/details/redislabs-public/redis-enterprise?pli=1), or [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/garantiadata.redis_enterprise_1sp_public_preview?tab=Overview)) offers Redis Enterprise in a marketplace offering."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 39ad06e1-f683-4a8a-ac4b-e7ed7791d9bf
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"39ad06e1-f683-4a8a-ac4b-e7ed7791d9bf\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7ac803a-183f-427d-b80e-e80c95d8b29c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a06dbbc18a21404ee251fbad7dcb97a77785448211aec31d3d9fde97033df4f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"210b92bd-9b65-4f09-9cb5-88957eac2a07\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"cc674e46d46a9186ef07aaa6de525839208453c5a0bb5c93cd5e1468a8441038\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8bd1e8a5-6dc4-4bf7-8b47-f05f3fce88ac\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2a79989f01b54851ba761c4f72e140fa361cc4be1f7d2f59f6bff5623f6332ec\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3031, \"end_char_idx\": 5765, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "ref_doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "chroma:document": "## Prerequisites\n\nBefore we start this project, we need to set up the following:\n\n* start a Redis database with RediSearch (redis-stack)\n* install libraries\n    * [Redis-py](https://github.com/redis/redis-py)\n* get your [OpenAI API key](https://beta.openai.com/account/api-keys)\n\n===========================================================\n\n### Start Redis\n\nTo keep this example simple, we will use the Redis Stack docker container which we can start as follows\n\n```bash\n$ docker-compose up -d\n```\n\nThis also includes the [RedisInsight](https://redis.com/redis-enterprise/redis-insight/) GUI for managing your Redis database which you can view at [http://localhost:8001](http://localhost:8001) once you start the docker container.\n\nYou're all set up and ready to go! Next, we import and create our client for communicating with the Redis database we just created.\n\n## Install Requirements\n\nRedis-Py is the python client for communicating with Redis. We will use this to communicate with our Redis-stack database. \n\n\n```python\n! pip install redis wget pandas openai\n```\n\n===========================================================\n## Prepare your OpenAI API key\n\nThe `OpenAI API key` is used for vectorization of query data.\n\nIf you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n\nOnce you get your key, please add it to your environment variables as `OPENAI_API_KEY` by using following command:\n\n\n```python\n! export OPENAI_API_KEY=\"your API key\"\n```\n\n\n```python\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\nimport os\nimport openai\n\n# Note. alternatively you can set a temporary env variable like this:\n# os.environ[\"OPENAI_API_KEY\"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    openai.api_key = os.getenv(\"OPENAI_API_KEY\")\n    print (\"OPENAI_API_KEY is ready\")\nelse:\n    print (\"OPENAI_API_KEY environment variable not found\")\n```\n\n    OPENAI_API_KEY is ready\n\n\n## Load data\n\nIn this section we'll load embedded data that has already been converted into vectors. We'll use this data to create an index in Redis and then search for similar vectors.\n\n\n```python\nimport sys\nimport numpy as np\nimport pandas as pd\nfrom typing import List\n\n# use helper function in nbutils.py to download and read the data\n# this should take from 5-10 min to run\nif os.getcwd() not in sys.path:\n    sys.path.append(os.getcwd())\nimport nbutils\n\nnbutils.download_wikipedia_data()\ndata = nbutils.read_wikipedia_data()\n\ndata.head()\n```\n\n    File Downloaded"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 8bd1e8a5-6dc4-4bf7-8b47-f05f3fce88ac
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"8bd1e8a5-6dc4-4bf7-8b47-f05f3fce88ac\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7ac803a-183f-427d-b80e-e80c95d8b29c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a06dbbc18a21404ee251fbad7dcb97a77785448211aec31d3d9fde97033df4f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"39ad06e1-f683-4a8a-ac4b-e7ed7791d9bf\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b0c3d470216607edcacb5c3f6fe276978668d83b04bbb91e31860b347ff01b5d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cc864081-23aa-44d7-babb-1bce7c9ad3c7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8cd0a9c9c8bed21afa4c25e5abe679fb75283551f60ad360179fa48e2e466a3e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4518, \"end_char_idx\": 6782, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "ref_doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "chroma:document": "```python\n! export OPENAI_API_KEY=\"your API key\"\n```\n\n\n```python\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\nimport os\nimport openai\n\n# Note. alternatively you can set a temporary env variable like this:\n# os.environ[\"OPENAI_API_KEY\"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    openai.api_key = os.getenv(\"OPENAI_API_KEY\")\n    print (\"OPENAI_API_KEY is ready\")\nelse:\n    print (\"OPENAI_API_KEY environment variable not found\")\n```\n\n    OPENAI_API_KEY is ready\n\n\n## Load data\n\nIn this section we'll load embedded data that has already been converted into vectors. We'll use this data to create an index in Redis and then search for similar vectors.\n\n\n```python\nimport sys\nimport numpy as np\nimport pandas as pd\nfrom typing import List\n\n# use helper function in nbutils.py to download and read the data\n# this should take from 5-10 min to run\nif os.getcwd() not in sys.path:\n    sys.path.append(os.getcwd())\nimport nbutils\n\nnbutils.download_wikipedia_data()\ndata = nbutils.read_wikipedia_data()\n\ndata.head()\n```\n\n    File Downloaded\n\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>url</th>\n      <th>title</th>\n      <th>text</th>\n      <th>title_vector</th>\n      <th>content_vector</th>\n      <th>vector_id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>https://simple.wikipedia.org/wiki/April</td>\n      <td>April</td>\n      <td>April is the fourth month of the year in the J...</td>\n      <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n      <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>https://simple.wikipedia.org/wiki/August</td>\n      <td>August</td>\n      <td>August (Aug.)"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: cc864081-23aa-44d7-babb-1bce7c9ad3c7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"cc864081-23aa-44d7-babb-1bce7c9ad3c7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7ac803a-183f-427d-b80e-e80c95d8b29c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a06dbbc18a21404ee251fbad7dcb97a77785448211aec31d3d9fde97033df4f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8bd1e8a5-6dc4-4bf7-8b47-f05f3fce88ac\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"d4cb6ff89e4b682ae4194fc7436923b2b3ec03127942d4afc3c13730ae31a19f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"99558b46-a06a-4ebc-9fa6-0e691dcc8f3b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a38d440fd46e3d8ec15186b41e85bf1941379cf88d54866f98815a4eabb1d814\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5771, \"end_char_idx\": 7813, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "ref_doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "chroma:document": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>url</th>\n      <th>title</th>\n      <th>text</th>\n      <th>title_vector</th>\n      <th>content_vector</th>\n      <th>vector_id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>https://simple.wikipedia.org/wiki/April</td>\n      <td>April</td>\n      <td>April is the fourth month of the year in the J...</td>\n      <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n      <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>https://simple.wikipedia.org/wiki/August</td>\n      <td>August</td>\n      <td>August (Aug.) is the eighth month of the year ...</td>\n      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>6</td>\n      <td>https://simple.wikipedia.org/wiki/Art</td>\n      <td>Art</td>\n      <td>Art is a creative activity that expresses imag...</td>\n      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>8</td>\n      <td>https://simple.wikipedia.org/wiki/A</td>\n      <td>A</td>\n      <td>A or a is the first letter of the English alph...</td>\n      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n      <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>9</td>\n      <td>https://simple.wikipedia.org/wiki/Air</td>\n      <td>Air</td>\n      <td>Air refers to the Earth's atmosphere."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 99558b46-a06a-4ebc-9fa6-0e691dcc8f3b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"99558b46-a06a-4ebc-9fa6-0e691dcc8f3b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7ac803a-183f-427d-b80e-e80c95d8b29c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a06dbbc18a21404ee251fbad7dcb97a77785448211aec31d3d9fde97033df4f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cc864081-23aa-44d7-babb-1bce7c9ad3c7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"aeaf459cfba1154701312e90778778d05839d50242770f076e88f0791b143d33\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b9b8e2ca-cf48-41bc-bc5e-a7e957628654\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a608461bed49331890805a388d9965aafc86c90bcef890b0aa3e983daf0e5a7a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6783, \"end_char_idx\": 8997, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "ref_doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "chroma:document": "is the eighth month of the year ...</td>\n      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>6</td>\n      <td>https://simple.wikipedia.org/wiki/Art</td>\n      <td>Art</td>\n      <td>Art is a creative activity that expresses imag...</td>\n      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>8</td>\n      <td>https://simple.wikipedia.org/wiki/A</td>\n      <td>A</td>\n      <td>A or a is the first letter of the English alph...</td>\n      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n      <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>9</td>\n      <td>https://simple.wikipedia.org/wiki/Air</td>\n      <td>Air</td>\n      <td>Air refers to the Earth's atmosphere. Air is a...</td>\n      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n      <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n## Connect to Redis\n\nNow that we have our Redis database running, we can connect to it using the Redis-py client. We will use the default host and port for the Redis database which is `localhost:6379`.\n\n\n\n\n```python\nimport redis\nfrom redis.commands.search.indexDefinition import (\n    IndexDefinition,\n    IndexType\n)\nfrom redis.commands.search.query import Query\nfrom redis.commands.search.field import (\n    TextField,\n    VectorField\n)\n\nREDIS_HOST =  \"localhost\"\nREDIS_PORT = 6379\nREDIS_PASSWORD = \"\" # default for passwordless Redis\n\n# Connect to Redis\nredis_client = redis.Redis(\n    host=REDIS_HOST,\n    port=REDIS_PORT,\n    password=REDIS_PASSWORD\n)\nredis_client.ping()\n```\n\n\n\n\n    True\n\n\n\n## Creating a Search Index in Redis\n\nThe below cells will show how to specify and create a search index in Redis. We will:\n\n1. Set some constants for defining our index like the distance metric and the index name\n2. Define the index schema with RediSearch fields\n3. Create the index"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: b9b8e2ca-cf48-41bc-bc5e-a7e957628654
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"b9b8e2ca-cf48-41bc-bc5e-a7e957628654\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7ac803a-183f-427d-b80e-e80c95d8b29c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a06dbbc18a21404ee251fbad7dcb97a77785448211aec31d3d9fde97033df4f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"99558b46-a06a-4ebc-9fa6-0e691dcc8f3b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"549afef9e20ba0fa2770b41f533cd2252e986f9b0758481c04b9a3319ea08887\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"86325a13-15e7-4019-94b2-c8c87761df24\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2aef452339ec5229e17ec0e24316586af693499bb31ac7542bce86a3639e450d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7814, \"end_char_idx\": 10752, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "ref_doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "chroma:document": "Air is a...</td>\n      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n      <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n## Connect to Redis\n\nNow that we have our Redis database running, we can connect to it using the Redis-py client. We will use the default host and port for the Redis database which is `localhost:6379`.\n\n\n\n\n```python\nimport redis\nfrom redis.commands.search.indexDefinition import (\n    IndexDefinition,\n    IndexType\n)\nfrom redis.commands.search.query import Query\nfrom redis.commands.search.field import (\n    TextField,\n    VectorField\n)\n\nREDIS_HOST =  \"localhost\"\nREDIS_PORT = 6379\nREDIS_PASSWORD = \"\" # default for passwordless Redis\n\n# Connect to Redis\nredis_client = redis.Redis(\n    host=REDIS_HOST,\n    port=REDIS_PORT,\n    password=REDIS_PASSWORD\n)\nredis_client.ping()\n```\n\n\n\n\n    True\n\n\n\n## Creating a Search Index in Redis\n\nThe below cells will show how to specify and create a search index in Redis. We will:\n\n1. Set some constants for defining our index like the distance metric and the index name\n2. Define the index schema with RediSearch fields\n3. Create the index\n\n\n```python\n# Constants\nVECTOR_DIM = len(data['title_vector'][0]) # length of the vectors\nVECTOR_NUMBER = len(data)                 # initial number of vectors\nINDEX_NAME = \"embeddings-index\"           # name of the search index\nPREFIX = \"doc\"                            # prefix for the document keys\nDISTANCE_METRIC = \"COSINE\"                # distance metric for the vectors (ex. COSINE, IP, L2)\n```\n\n\n```python\n# Define RediSearch fields for each of the columns in the dataset\ntitle = TextField(name=\"title\")\nurl = TextField(name=\"url\")\ntext = TextField(name=\"text\")\ntitle_embedding = VectorField(\"title_vector\",\n    \"FLAT\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER,\n    }\n)\ntext_embedding = VectorField(\"content_vector\",\n    \"FLAT\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER,\n    }\n)\nfields = [title, url, text, title_embedding, text_embedding]\n```\n\n\n```python\n# Check if index exists\ntry:\n    redis_client.ft(INDEX_NAME).info()\n    print(\"Index already exists\")\nexcept:\n    # Create RediSearch Index\n    redis_client.ft(INDEX_NAME).create_index(\n        fields = fields,\n        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n)\n```\n\n## Load Documents into the Index\n\nNow that we have a search index, we can load documents into it. We will use the same documents we used in the previous examples. In Redis, either the HASH or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The below cells will show how to load documents into the index."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 86325a13-15e7-4019-94b2-c8c87761df24
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"86325a13-15e7-4019-94b2-c8c87761df24\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7ac803a-183f-427d-b80e-e80c95d8b29c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a06dbbc18a21404ee251fbad7dcb97a77785448211aec31d3d9fde97033df4f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b9b8e2ca-cf48-41bc-bc5e-a7e957628654\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1142df6defb1f33d8fdb1bede372d5c06e2b0d99bf0945ad0cd6d22f81615e11\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"97247e6f-3b41-4ac5-bb03-49123c470a24\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4a0d2119b71c60a4b5306fee9f383594b7e682068d2543563c32df6bf2bd7628\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9403, \"end_char_idx\": 11963, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "ref_doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "chroma:document": "```python\n# Define RediSearch fields for each of the columns in the dataset\ntitle = TextField(name=\"title\")\nurl = TextField(name=\"url\")\ntext = TextField(name=\"text\")\ntitle_embedding = VectorField(\"title_vector\",\n    \"FLAT\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER,\n    }\n)\ntext_embedding = VectorField(\"content_vector\",\n    \"FLAT\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER,\n    }\n)\nfields = [title, url, text, title_embedding, text_embedding]\n```\n\n\n```python\n# Check if index exists\ntry:\n    redis_client.ft(INDEX_NAME).info()\n    print(\"Index already exists\")\nexcept:\n    # Create RediSearch Index\n    redis_client.ft(INDEX_NAME).create_index(\n        fields = fields,\n        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n)\n```\n\n## Load Documents into the Index\n\nNow that we have a search index, we can load documents into it. We will use the same documents we used in the previous examples. In Redis, either the HASH or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The below cells will show how to load documents into the index.\n\n\n```python\ndef index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):\n    records = documents.to_dict(\"records\")\n    for doc in records:\n        key = f\"{prefix}:{str(doc['id'])}\"\n\n        # create byte vectors for title and content\n        title_embedding = np.array(doc[\"title_vector\"], dtype=np.float32).tobytes()\n        content_embedding = np.array(doc[\"content_vector\"], dtype=np.float32).tobytes()\n\n        # replace list of floats with byte vectors\n        doc[\"title_vector\"] = title_embedding\n        doc[\"content_vector\"] = content_embedding\n\n        client.hset(key, mapping = doc)\n```\n\n\n```python\nindex_documents(redis_client, PREFIX, data)\nprint(f\"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}\")\n```\n\n    Loaded 25000 documents in Redis search index with name: embeddings-index\n\n\n## Simple Vector Search Queries with OpenAI Query Embeddings\n\nNow that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 97247e6f-3b41-4ac5-bb03-49123c470a24
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"97247e6f-3b41-4ac5-bb03-49123c470a24\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7ac803a-183f-427d-b80e-e80c95d8b29c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a06dbbc18a21404ee251fbad7dcb97a77785448211aec31d3d9fde97033df4f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"86325a13-15e7-4019-94b2-c8c87761df24\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"134c2f2d063ee1a9a3ca91a8bfdd1b0db01bd8e10f82a86d7b4875e0e1274426\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b606b79e-c545-4422-9c5e-7e7d994d0ccb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0bb1516120e00d61d536bb174f6c12726c16166080eef2402122c96f1832e3ab\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10755, \"end_char_idx\": 13727, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "ref_doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "chroma:document": "```python\ndef index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):\n    records = documents.to_dict(\"records\")\n    for doc in records:\n        key = f\"{prefix}:{str(doc['id'])}\"\n\n        # create byte vectors for title and content\n        title_embedding = np.array(doc[\"title_vector\"], dtype=np.float32).tobytes()\n        content_embedding = np.array(doc[\"content_vector\"], dtype=np.float32).tobytes()\n\n        # replace list of floats with byte vectors\n        doc[\"title_vector\"] = title_embedding\n        doc[\"content_vector\"] = content_embedding\n\n        client.hset(key, mapping = doc)\n```\n\n\n```python\nindex_documents(redis_client, PREFIX, data)\nprint(f\"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}\")\n```\n\n    Loaded 25000 documents in Redis search index with name: embeddings-index\n\n\n## Simple Vector Search Queries with OpenAI Query Embeddings\n\nNow that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database.\n\n\n```python\ndef search_redis(\n    redis_client: redis.Redis,\n    user_query: str,\n    index_name: str = \"embeddings-index\",\n    vector_field: str = \"title_vector\",\n    return_fields: list = [\"title\", \"url\", \"text\", \"vector_score\"],\n    hybrid_fields = \"*\",\n    k: int = 20,\n    print_results: bool = True,\n) -> List[dict]:\n\n    # Creates embedding vector from user query\n    embedded_query = openai.Embedding.create(input=user_query,\n                                            model=\"text-embedding-3-small\",\n                                            )[\"data\"][0]['embedding']\n\n    # Prepare the Query\n    base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'\n    query = (\n        Query(base_query)\n         .return_fields(*return_fields)\n         .sort_by(\"vector_score\")\n         .paging(0, k)\n         .dialect(2)\n    )\n    params_dict = {\"vector\": np.array(embedded_query).astype(dtype=np.float32).tobytes()}\n\n    # perform vector search\n    results = redis_client.ft(index_name).search(query, params_dict)\n    if print_results:\n        for i, article in enumerate(results.docs):\n            score = 1 - float(article.vector_score)\n            print(f\"{i}. {article.title} (Score: {round(score ,3) })\")\n    return results.docs\n```\n\n\n```python\n# For using OpenAI to generate query embedding\nresults = search_redis(redis_client, 'modern art in Europe', k=10)\n```\n\n    0. Museum of Modern Art (Score: 0.875)\n    1. Western Europe (Score: 0.868)\n    2. Renaissance art (Score: 0.864)\n    3. Pop art (Score: 0.86)\n    4. Northern Europe (Score: 0.855)\n    5. Hellenistic art (Score: 0.853)\n    6. Modernist literature (Score: 0.847)\n    7. Art film (Score: 0.843)\n    8. Central Europe (Score: 0.843)\n    9. European (Score: 0.841)"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: b606b79e-c545-4422-9c5e-7e7d994d0ccb
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"b606b79e-c545-4422-9c5e-7e7d994d0ccb\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7ac803a-183f-427d-b80e-e80c95d8b29c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a06dbbc18a21404ee251fbad7dcb97a77785448211aec31d3d9fde97033df4f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"97247e6f-3b41-4ac5-bb03-49123c470a24\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1255aeae23352436c91b24b44a2ef2e1418b6c4349c99ec77257e8dadd9be0be\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"048bd0fd-8881-42c5-9522-48aa0ab5dc47\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"559d12af48b71ced4c4154ceb046b72a950acd30db6df770a4129b5e21b40cc3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13234, \"end_char_idx\": 15300, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "ref_doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "chroma:document": "```python\n# For using OpenAI to generate query embedding\nresults = search_redis(redis_client, 'modern art in Europe', k=10)\n```\n\n    0. Museum of Modern Art (Score: 0.875)\n    1. Western Europe (Score: 0.868)\n    2. Renaissance art (Score: 0.864)\n    3. Pop art (Score: 0.86)\n    4. Northern Europe (Score: 0.855)\n    5. Hellenistic art (Score: 0.853)\n    6. Modernist literature (Score: 0.847)\n    7. Art film (Score: 0.843)\n    8. Central Europe (Score: 0.843)\n    9. European (Score: 0.841)\n\n\n\n```python\nresults = search_redis(redis_client, 'Famous battles in Scottish history', vector_field='content_vector', k=10)\n```\n\n    0. Battle of Bannockburn (Score: 0.869)\n    1. Wars of Scottish Independence (Score: 0.861)\n    2. 1651 (Score: 0.853)\n    3. First War of Scottish Independence (Score: 0.85)\n    4. Robert I of Scotland (Score: 0.846)\n    5. 841 (Score: 0.844)\n    6. 1716 (Score: 0.844)\n    7. 1314 (Score: 0.837)\n    8. 1263 (Score: 0.836)\n    9. William Wallace (Score: 0.835)\n\n\n## Hybrid Queries with Redis\n\nThe previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the below example, we will combine vector search with full text search.\n\n\n```python\ndef create_hybrid_field(field_name: str, value: str) -> str:\n    return f'@{field_name}:\"{value}\"'\n\n# search the content vector for articles about famous battles in Scottish history and only include results with Scottish in the title\nresults = search_redis(redis_client,\n                       \"Famous battles in Scottish history\",\n                       vector_field=\"title_vector\",\n                       k=5,\n                       hybrid_fields=create_hybrid_field(\"title\", \"Scottish\")\n                       )\n```\n\n    0. First War of Scottish Independence (Score: 0.892)\n    1. Wars of Scottish Independence (Score: 0.889)\n    2. Second War of Scottish Independence (Score: 0.879)\n    3. List of Scottish monarchs (Score: 0.873)\n    4. Scottish Borders (Score: 0.863)"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 048bd0fd-8881-42c5-9522-48aa0ab5dc47
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"048bd0fd-8881-42c5-9522-48aa0ab5dc47\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7ac803a-183f-427d-b80e-e80c95d8b29c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a06dbbc18a21404ee251fbad7dcb97a77785448211aec31d3d9fde97033df4f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b606b79e-c545-4422-9c5e-7e7d994d0ccb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"32bf47c21f53fb34bbd00248f18375e47ad2a6f5fc34302746f87f4269624066\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f120c215-2446-4f86-ac02-ab8685e15a05\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3f934ea4c249ac489b58c50e639a9dec48926fc9d23d9da862fa9e7d9a05eab7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14227, \"end_char_idx\": 17343, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "ref_doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "chroma:document": "## Hybrid Queries with Redis\n\nThe previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the below example, we will combine vector search with full text search.\n\n\n```python\ndef create_hybrid_field(field_name: str, value: str) -> str:\n    return f'@{field_name}:\"{value}\"'\n\n# search the content vector for articles about famous battles in Scottish history and only include results with Scottish in the title\nresults = search_redis(redis_client,\n                       \"Famous battles in Scottish history\",\n                       vector_field=\"title_vector\",\n                       k=5,\n                       hybrid_fields=create_hybrid_field(\"title\", \"Scottish\")\n                       )\n```\n\n    0. First War of Scottish Independence (Score: 0.892)\n    1. Wars of Scottish Independence (Score: 0.889)\n    2. Second War of Scottish Independence (Score: 0.879)\n    3. List of Scottish monarchs (Score: 0.873)\n    4. Scottish Borders (Score: 0.863)\n\n\n\n```python\n# run a hybrid query for articles about Art in the title vector and only include results with the phrase \"Leonardo da Vinci\" in the text\nresults = search_redis(redis_client,\n                       \"Art\",\n                       vector_field=\"title_vector\",\n                       k=5,\n                       hybrid_fields=create_hybrid_field(\"text\", \"Leonardo da Vinci\")\n                       )\n\n# find specific mention of Leonardo da Vinci in the text that our full-text-search query returned\nmention = [sentence for sentence in results[0].text.split(\"\\n\") if \"Leonardo da Vinci\" in sentence][0]\nmention\n```\n\n    0. Art (Score: 1.0)\n    1. Paint (Score: 0.896)\n    2. Renaissance art (Score: 0.88)\n    3. Painting (Score: 0.874)\n    4. Renaissance (Score: 0.846)\n\n\n\n\n\n    'In Europe, after the Middle Ages, there was a \"Renaissance\" which means \"rebirth\". People rediscovered science and artists were allowed to paint subjects other than religious subjects. People like Michelangelo and Leonardo da Vinci still painted religious pictures, but they also now could paint mythological pictures too. These artists also invented perspective where things in the distance look smaller in the picture. This was new because in the Middle Ages people would paint all the figures close up and just overlapping each other. These artists used nudity regularly in their art.'\n\n\n\n## HNSW Index\n\nUp until now, we've been using the ``FLAT`` or \"brute-force\" index to run our queries. Redis also supports the ``HNSW`` index which is a fast, approximate index. The ``HNSW`` index is a graph-based index that uses a hierarchical navigable small world graph to store vectors. The ``HNSW`` index is a good choice for large datasets where you want to run approximate queries.\n\n``HNSW`` will take longer to build and consume more memory for most cases than ``FLAT`` but will be faster to run queries on, especially for large datasets.\n\nThe following cells will show how to create an ``HNSW`` index and run queries with it using the same data as before."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: f120c215-2446-4f86-ac02-ab8685e15a05
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"f120c215-2446-4f86-ac02-ab8685e15a05\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7ac803a-183f-427d-b80e-e80c95d8b29c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a06dbbc18a21404ee251fbad7dcb97a77785448211aec31d3d9fde97033df4f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"048bd0fd-8881-42c5-9522-48aa0ab5dc47\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"77b60f153759ece6106d715a8f8abc231eddadcf18a06faab06da4a6eaf7f557\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"16c5c583-b0f7-4df9-9dd6-c3d59f924af7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"351b8cb5df5b075c6e22471bf20d5a9100892b9360be4b27030167d3e9e7b9d0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16087, \"end_char_idx\": 18745, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "ref_doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "chroma:document": "'In Europe, after the Middle Ages, there was a \"Renaissance\" which means \"rebirth\". People rediscovered science and artists were allowed to paint subjects other than religious subjects. People like Michelangelo and Leonardo da Vinci still painted religious pictures, but they also now could paint mythological pictures too. These artists also invented perspective where things in the distance look smaller in the picture. This was new because in the Middle Ages people would paint all the figures close up and just overlapping each other. These artists used nudity regularly in their art.'\n\n\n\n## HNSW Index\n\nUp until now, we've been using the ``FLAT`` or \"brute-force\" index to run our queries. Redis also supports the ``HNSW`` index which is a fast, approximate index. The ``HNSW`` index is a graph-based index that uses a hierarchical navigable small world graph to store vectors. The ``HNSW`` index is a good choice for large datasets where you want to run approximate queries.\n\n``HNSW`` will take longer to build and consume more memory for most cases than ``FLAT`` but will be faster to run queries on, especially for large datasets.\n\nThe following cells will show how to create an ``HNSW`` index and run queries with it using the same data as before.\n\n\n```python\n# re-define RediSearch vector fields to use HNSW index\ntitle_embedding = VectorField(\"title_vector\",\n    \"HNSW\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER\n    }\n)\ntext_embedding = VectorField(\"content_vector\",\n    \"HNSW\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER\n    }\n)\nfields = [title, url, text, title_embedding, text_embedding]\n```\n\n\n```python\nimport time\n# Check if index exists\nHNSW_INDEX_NAME = INDEX_NAME+ \"_HNSW\"\n\ntry:\n    redis_client.ft(HNSW_INDEX_NAME).info()\n    print(\"Index already exists\")\nexcept:\n    # Create RediSearch Index\n    redis_client.ft(HNSW_INDEX_NAME).create_index(\n        fields = fields,\n        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n    )\n\n# since RediSearch creates the index in the background for existing documents, we will wait until\n# indexing is complete before running our queries. Although this is not necessary for the first query,\n# some queries may take longer to run if the index is not fully built. In general, Redis will perform\n# best when adding new documents to existing indices rather than new indices on existing documents.\nwhile redis_client.ft(HNSW_INDEX_NAME).info()[\"indexing\"] == \"1\":\n    time.sleep(5)\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 16c5c583-b0f7-4df9-9dd6-c3d59f924af7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"16c5c583-b0f7-4df9-9dd6-c3d59f924af7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7ac803a-183f-427d-b80e-e80c95d8b29c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a06dbbc18a21404ee251fbad7dcb97a77785448211aec31d3d9fde97033df4f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f120c215-2446-4f86-ac02-ab8685e15a05\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"6a887d9d22ce196b9d6259b32923564ad8bb64074f9185b043004c62b2c636c3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8208aaf2-99ed-4d73-9674-76b7a3ae000a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"10d82b05be97afebf2ff236a04b28e8635da2b57e558f661bf4e3eb3f1ebfbd8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17346, \"end_char_idx\": 19231, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "ref_doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "chroma:document": "```python\n# re-define RediSearch vector fields to use HNSW index\ntitle_embedding = VectorField(\"title_vector\",\n    \"HNSW\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER\n    }\n)\ntext_embedding = VectorField(\"content_vector\",\n    \"HNSW\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": VECTOR_DIM,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": VECTOR_NUMBER\n    }\n)\nfields = [title, url, text, title_embedding, text_embedding]\n```\n\n\n```python\nimport time\n# Check if index exists\nHNSW_INDEX_NAME = INDEX_NAME+ \"_HNSW\"\n\ntry:\n    redis_client.ft(HNSW_INDEX_NAME).info()\n    print(\"Index already exists\")\nexcept:\n    # Create RediSearch Index\n    redis_client.ft(HNSW_INDEX_NAME).create_index(\n        fields = fields,\n        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n    )\n\n# since RediSearch creates the index in the background for existing documents, we will wait until\n# indexing is complete before running our queries. Although this is not necessary for the first query,\n# some queries may take longer to run if the index is not fully built. In general, Redis will perform\n# best when adding new documents to existing indices rather than new indices on existing documents.\nwhile redis_client.ft(HNSW_INDEX_NAME).info()[\"indexing\"] == \"1\":\n    time.sleep(5)\n```\n\n\n```python\nresults = search_redis(redis_client, 'modern art in Europe', index_name=HNSW_INDEX_NAME, k=10)\n```\n\n    0. Western Europe (Score: 0.868)\n    1. Northern Europe (Score: 0.855)\n    2. Central Europe (Score: 0.843)\n    3. European (Score: 0.841)\n    4. Eastern Europe (Score: 0.839)\n    5. Europe (Score: 0.839)\n    6. Western European Union (Score: 0.837)\n    7. Southern Europe (Score: 0.831)\n    8. Western civilization (Score: 0.83)\n    9. Council of Europe (Score: 0.827)"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 8208aaf2-99ed-4d73-9674-76b7a3ae000a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"8208aaf2-99ed-4d73-9674-76b7a3ae000a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7ac803a-183f-427d-b80e-e80c95d8b29c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a06dbbc18a21404ee251fbad7dcb97a77785448211aec31d3d9fde97033df4f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"16c5c583-b0f7-4df9-9dd6-c3d59f924af7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"aeef900ab914868bd5c4c1ff556aa2b916e6658780a98b17b79b26a61bc82e5b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"293d4327-5456-4f47-b074-3a6c0b2e751f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5b0d7e8666a5b807e3bbea3400b6775cad72d42af45bacc0f92b52f9d959d8db\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17885, \"end_char_idx\": 21190, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "ref_doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "chroma:document": "```python\nimport time\n# Check if index exists\nHNSW_INDEX_NAME = INDEX_NAME+ \"_HNSW\"\n\ntry:\n    redis_client.ft(HNSW_INDEX_NAME).info()\n    print(\"Index already exists\")\nexcept:\n    # Create RediSearch Index\n    redis_client.ft(HNSW_INDEX_NAME).create_index(\n        fields = fields,\n        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n    )\n\n# since RediSearch creates the index in the background for existing documents, we will wait until\n# indexing is complete before running our queries. Although this is not necessary for the first query,\n# some queries may take longer to run if the index is not fully built. In general, Redis will perform\n# best when adding new documents to existing indices rather than new indices on existing documents.\nwhile redis_client.ft(HNSW_INDEX_NAME).info()[\"indexing\"] == \"1\":\n    time.sleep(5)\n```\n\n\n```python\nresults = search_redis(redis_client, 'modern art in Europe', index_name=HNSW_INDEX_NAME, k=10)\n```\n\n    0. Western Europe (Score: 0.868)\n    1. Northern Europe (Score: 0.855)\n    2. Central Europe (Score: 0.843)\n    3. European (Score: 0.841)\n    4. Eastern Europe (Score: 0.839)\n    5. Europe (Score: 0.839)\n    6. Western European Union (Score: 0.837)\n    7. Southern Europe (Score: 0.831)\n    8. Western civilization (Score: 0.83)\n    9. Council of Europe (Score: 0.827)\n\n\n\n```python\n# compare the results of the HNSW index to the FLAT index and time both queries\ndef time_queries(iterations: int = 10):\n    print(\" ----- Flat Index ----- \")\n    t0 = time.time()\n    for i in range(iterations):\n        results_flat = search_redis(redis_client, 'modern art in Europe', k=10, print_results=False)\n    t0 = (time.time() - t0) / iterations\n    results_flat = search_redis(redis_client, 'modern art in Europe', k=10, print_results=True)\n    print(f\"Flat index query time: {round(t0, 3)} seconds\\n\")\n    time.sleep(1)\n    print(\" ----- HNSW Index ------ \")\n    t1 = time.time()\n    for i in range(iterations):\n        results_hnsw = search_redis(redis_client, 'modern art in Europe', index_name=HNSW_INDEX_NAME, k=10, print_results=False)\n    t1 = (time.time() - t1) / iterations\n    results_hnsw = search_redis(redis_client, 'modern art in Europe', index_name=HNSW_INDEX_NAME, k=10, print_results=True)\n    print(f\"HNSW index query time: {round(t1, 3)} seconds\")\n    print(\" ------------------------ \")\ntime_queries()\n```\n\n     ----- Flat Index ----- \n    0. Museum of Modern Art (Score: 0.875)\n    1. Western Europe (Score: 0.867)\n    2. Renaissance art (Score: 0.864)\n    3. Pop art (Score: 0.861)\n    4. Northern Europe (Score: 0.855)\n    5. Hellenistic art (Score: 0.853)\n    6. Modernist literature (Score: 0.847)\n    7. Art film (Score: 0.843)\n    8. Central Europe (Score: 0.843)\n    9. Art (Score: 0.842)\n    Flat index query time: 0.263 seconds\n    \n     ----- HNSW Index ------ \n    0. Western Europe (Score: 0.867)\n    1. Northern Europe (Score: 0.855)\n    2. Central Europe (Score: 0.843)\n    3. European (Score: 0.841)\n    4. Eastern Europe (Score: 0.839)\n    5. Europe (Score: 0.839)\n    6. Western European Union (Score: 0.837)\n    7. Southern Europe (Score: 0.831)\n    8. Western civilization (Score: 0.83)\n    9. Council of Europe (Score: 0.827)\n    HNSW index query time: 0.129 seconds\n     ------------------------"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 293d4327-5456-4f47-b074-3a6c0b2e751f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"293d4327-5456-4f47-b074-3a6c0b2e751f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d7ac803a-183f-427d-b80e-e80c95d8b29c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a06dbbc18a21404ee251fbad7dcb97a77785448211aec31d3d9fde97033df4f8\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8208aaf2-99ed-4d73-9674-76b7a3ae000a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb\", \"title\": \"Using Redis as a Vector Database with OpenAI\", \"tokens\": 5520, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"736356085d18a191c5c9900dfd76b785f54c75e2fda6d02407c2ae52ca068d11\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21195, \"end_char_idx\": 21209, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "ref_doc_id": "d7ac803a-183f-427d-b80e-e80c95d8b29c", "chroma:document": "```python\n\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: f8aa1d4b-d198-4947-9acf-501b4c7a359d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"f8aa1d4b-d198-4947-9acf-501b4c7a359d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60d79eba-f7df-4e3d-a0e3-5be420e13a39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1a1e9c86a0c7e0dd7950d0417fc95d0b24cfef8382d201a54d7fa0cebf30ede5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a172a12f-491a-423d-80fc-ea8b0a3cd2f8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"48771e548cd855d4c03fd7f47a090609cdb397dbaf2db6a8493b3df0c8026bfe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3212, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "ref_doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "chroma:document": "# Running Hybrid VSS Queries with Redis and OpenAI\n\nThis notebook provides an introduction to using Redis as a vector database with OpenAI embeddings and running hybrid queries that combine VSS and lexical search using Redis Query and Search capability. Redis is a scalable, real-time database that can be used as a vector database when using the [RediSearch Module](https://oss.redislabs.com/redisearch/). The Redis Query and Search capability allows you to index and search for vectors in Redis. This notebook will show you how to use the Redis Query and Search to index and search for vectors created by using the OpenAI API and stored in Redis.\n\nHybrid queries combine vector similarity with traditional Redis Query and Search filtering capabilities on GEO, NUMERIC, TAG or TEXT data simplifying application code. A common example of a hybrid query in an e-commerce use case is to find items visually similar to a given query image limited to items available in a GEO location and within a price range.\n\n## Prerequisites\n\nBefore we start this project, we need to set up the following:\n\n* start a Redis database with RediSearch (redis-stack)\n* install libraries\n    * [Redis-py](https://github.com/redis/redis-py)\n* get your [OpenAI API key](https://beta.openai.com/account/api-keys)\n\n===========================================================\n\n### Start Redis\n\nTo keep this example simple, we will use the Redis Stack docker container which we can start as follows\n\n```bash\n$ docker-compose up -d\n```\n\nThis also includes the [RedisInsight](https://redis.com/redis-enterprise/redis-insight/) GUI for managing your Redis database which you can view at [http://localhost:8001](http://localhost:8001) once you start the docker container.\n\nYou're all set up and ready to go! Next, we import and create our client for communicating with the Redis database we just created.\n\n## Install Requirements\n\nRedis-Py is the python client for communicating with Redis. We will use this to communicate with our Redis-stack database. \n\n\n```python\n! pip install redis pandas openai\n\n```\n\n    Defaulting to user installation because normal site-packages is not writeable\n    Requirement already satisfied: redis in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (4.5.4)\n    Requirement already satisfied: pandas in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (2.0.1)\n    Requirement already satisfied: openai in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (0.27.6)\n    Requirement already satisfied: async-timeout>=4.0.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from redis) (4.0.2)\n    Requirement already satisfied: python-dateutil>=2.8.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2.8.2)\n    Requirement already satisfied: pytz>=2020.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2023.3)\n    Requirement already satisfied: tzdata>=2022.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2023.3)\n    Requirement already satisfied: numpy>=1.20.3 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (1.23."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: a172a12f-491a-423d-80fc-ea8b0a3cd2f8
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"a172a12f-491a-423d-80fc-ea8b0a3cd2f8\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60d79eba-f7df-4e3d-a0e3-5be420e13a39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1a1e9c86a0c7e0dd7950d0417fc95d0b24cfef8382d201a54d7fa0cebf30ede5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f8aa1d4b-d198-4947-9acf-501b4c7a359d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"2cc06a9e95f21a5fb44e77d7082f19207668ef122b7bdac5af56b012b52a5432\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"557a10ea-a4d2-46fa-ab3a-af3128560db4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b5bbc484afc986c9a18fe4c73a4cb013ff3346a538ef8364957199a81b3e8c43\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2024, \"end_char_idx\": 4613, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "ref_doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "chroma:document": "```python\n! pip install redis pandas openai\n\n```\n\n    Defaulting to user installation because normal site-packages is not writeable\n    Requirement already satisfied: redis in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (4.5.4)\n    Requirement already satisfied: pandas in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (2.0.1)\n    Requirement already satisfied: openai in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (0.27.6)\n    Requirement already satisfied: async-timeout>=4.0.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from redis) (4.0.2)\n    Requirement already satisfied: python-dateutil>=2.8.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2.8.2)\n    Requirement already satisfied: pytz>=2020.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2023.3)\n    Requirement already satisfied: tzdata>=2022.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2023.3)\n    Requirement already satisfied: numpy>=1.20.3 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (1.23.4)\n    Requirement already satisfied: requests>=2.20 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (2.28.1)\n    Requirement already satisfied: tqdm in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (4.64.1)\n    Requirement already satisfied: aiohttp in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (3.8.4)\n    Requirement already satisfied: six>=1.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)\n    Requirement already satisfied: charset-normalizer<3,>=2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2.1.1)\n    Requirement already satisfied: idna<4,>=2.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (3.4)\n    Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (1.26.12)\n    Requirement already satisfied: certifi>=2017.4.17 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2022.9.24)\n    Requirement already satisfied: attrs>=17.3.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (23.1.0)\n    Requirement already satisfied: multidict<7."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 557a10ea-a4d2-46fa-ab3a-af3128560db4
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"557a10ea-a4d2-46fa-ab3a-af3128560db4\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60d79eba-f7df-4e3d-a0e3-5be420e13a39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1a1e9c86a0c7e0dd7950d0417fc95d0b24cfef8382d201a54d7fa0cebf30ede5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a172a12f-491a-423d-80fc-ea8b0a3cd2f8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"40339a3d0de714fb05d4d88fcfdf887037f65e21016132889ebc2b4a15b7d7d0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6cbd0544-4178-459a-b95f-21aefb294123\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ac5d72e261ea856563d0f3e6c0751e53d40e3f5887c8bb35bd939b6770c12dd3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3349, \"end_char_idx\": 5584, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "ref_doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "chroma:document": "28.1)\n    Requirement already satisfied: tqdm in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (4.64.1)\n    Requirement already satisfied: aiohttp in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (3.8.4)\n    Requirement already satisfied: six>=1.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)\n    Requirement already satisfied: charset-normalizer<3,>=2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2.1.1)\n    Requirement already satisfied: idna<4,>=2.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (3.4)\n    Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (1.26.12)\n    Requirement already satisfied: certifi>=2017.4.17 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2022.9.24)\n    Requirement already satisfied: attrs>=17.3.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (23.1.0)\n    Requirement already satisfied: multidict<7.0,>=4.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (6.0.4)\n    Requirement already satisfied: yarl<2.0,>=1.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.9.2)\n    Requirement already satisfied: frozenlist>=1.1.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.3)\n    Requirement already satisfied: aiosignal>=1.1.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.1)\n\n\n===========================================================\n## Prepare your OpenAI API key\n\nThe `OpenAI API key` is used for vectorization of query data.\n\nIf you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n\nOnce you get your key, please add it to your environment variables as `OPENAI_API_KEY` by using following command:"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 6cbd0544-4178-459a-b95f-21aefb294123
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"6cbd0544-4178-459a-b95f-21aefb294123\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60d79eba-f7df-4e3d-a0e3-5be420e13a39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1a1e9c86a0c7e0dd7950d0417fc95d0b24cfef8382d201a54d7fa0cebf30ede5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"557a10ea-a4d2-46fa-ab3a-af3128560db4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f9a571971dc7309d32ef930d12209f48f411bac6c43bdfe4c2f077090d77cf85\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7c419296-216d-4093-a4f8-52b2643358ed\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"54d7a83cada1ca318c4286170ec41e73ccba6ca043acce6f116fe4b991a1a070\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4233, \"end_char_idx\": 6312, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "ref_doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "chroma:document": "20->openai) (1.26.12)\n    Requirement already satisfied: certifi>=2017.4.17 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2022.9.24)\n    Requirement already satisfied: attrs>=17.3.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (23.1.0)\n    Requirement already satisfied: multidict<7.0,>=4.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (6.0.4)\n    Requirement already satisfied: yarl<2.0,>=1.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.9.2)\n    Requirement already satisfied: frozenlist>=1.1.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.3)\n    Requirement already satisfied: aiosignal>=1.1.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.1)\n\n\n===========================================================\n## Prepare your OpenAI API key\n\nThe `OpenAI API key` is used for vectorization of query data.\n\nIf you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n\nOnce you get your key, please add it to your environment variables as `OPENAI_API_KEY` by using following command:\n\n\n```python\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\nimport os\nimport openai\n\nos.environ[\"OPENAI_API_KEY\"] = '<YOUR_OPENAI_API_KEY>'\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    openai.api_key = os.getenv(\"OPENAI_API_KEY\")\n    print (\"OPENAI_API_KEY is ready\")\nelse:\n    print (\"OPENAI_API_KEY environment variable not found\")\n\n```\n\n    OPENAI_API_KEY is ready\n\n\n## Load data\n\nIn this section we'll load and clean an ecommerce dataset. We'll generate embeddings using OpenAI and use this data to create an index in Redis and then search for similar vectors."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 7c419296-216d-4093-a4f8-52b2643358ed
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"7c419296-216d-4093-a4f8-52b2643358ed\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60d79eba-f7df-4e3d-a0e3-5be420e13a39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1a1e9c86a0c7e0dd7950d0417fc95d0b24cfef8382d201a54d7fa0cebf30ede5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6cbd0544-4178-459a-b95f-21aefb294123\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1c6db14290c86fbee931b5b11ad8f26eb0f32f7a598bc285f4532b2363197b15\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5cac430a-b25b-4b73-b785-30d7a2b65903\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dd753ee1a3c82a2e076ed454a1897d74cbf852bbf64769b6524519050158bef7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4761, \"end_char_idx\": 7685, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "ref_doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "chroma:document": "0,>=1.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.9.2)\n    Requirement already satisfied: frozenlist>=1.1.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.3)\n    Requirement already satisfied: aiosignal>=1.1.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.1)\n\n\n===========================================================\n## Prepare your OpenAI API key\n\nThe `OpenAI API key` is used for vectorization of query data.\n\nIf you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n\nOnce you get your key, please add it to your environment variables as `OPENAI_API_KEY` by using following command:\n\n\n```python\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\nimport os\nimport openai\n\nos.environ[\"OPENAI_API_KEY\"] = '<YOUR_OPENAI_API_KEY>'\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    openai.api_key = os.getenv(\"OPENAI_API_KEY\")\n    print (\"OPENAI_API_KEY is ready\")\nelse:\n    print (\"OPENAI_API_KEY environment variable not found\")\n\n```\n\n    OPENAI_API_KEY is ready\n\n\n## Load data\n\nIn this section we'll load and clean an ecommerce dataset. We'll generate embeddings using OpenAI and use this data to create an index in Redis and then search for similar vectors.\n\n\n```python\nimport pandas as pd\nimport numpy as np\nfrom typing import List\n\nfrom utils.embeddings_utils import (\n    get_embeddings,\n    distances_from_embeddings,\n    tsne_components_from_embeddings,\n    chart_from_components,\n    indices_of_nearest_neighbors_from_distances,\n)\n\nEMBEDDING_MODEL = \"text-embedding-3-small\"\n\n# load in data and clean data types and drop null rows\ndf = pd.read_csv(\"../../data/styles_2k.csv\", on_bad_lines='skip')\ndf.dropna(inplace=True)\ndf[\"year\"] = df[\"year\"].astype(int)\ndf.info()\n\n# print dataframe\nn_examples = 5\ndf.head(n_examples)\n\n```\n\n    <class 'pandas.core.frame.DataFrame'>\n    Index: 1978 entries, 0 to 1998\n    Data columns (total 10 columns):\n     #   Column              Non-Null Count  Dtype \n    ---  ------              --------------  ----- \n     0   id                  1978 non-null   int64 \n     1   gender              1978 non-null   object\n     2   masterCategory      1978 non-null   object\n     3   subCategory         1978 non-null   object\n     4   articleType         1978 non-null   object\n     5   baseColour          1978 non-null   object\n     6   season              1978 non-null   object\n     7   year                1978 non-null   int64 \n     8   usage               1978 non-null   object\n     9   productDisplayName  1978 non-null   object\n    dtypes: int64(2), object(8)\n    memory usage: 170.0+ KB"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 5cac430a-b25b-4b73-b785-30d7a2b65903
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"5cac430a-b25b-4b73-b785-30d7a2b65903\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60d79eba-f7df-4e3d-a0e3-5be420e13a39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1a1e9c86a0c7e0dd7950d0417fc95d0b24cfef8382d201a54d7fa0cebf30ede5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7c419296-216d-4093-a4f8-52b2643358ed\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"dc37c0dec1d0bfe41cca903cb9a07958424f5fcfb6c4996093008e4cbf139cbe\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4ba9d095-3ed3-4e54-a1ec-10a1dfe35a98\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8a1838f53f0ebecc073805ab5138940bc791f72e8342f728f05fa6c5cae09e4e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6315, \"end_char_idx\": 9727, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "ref_doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "chroma:document": "```python\nimport pandas as pd\nimport numpy as np\nfrom typing import List\n\nfrom utils.embeddings_utils import (\n    get_embeddings,\n    distances_from_embeddings,\n    tsne_components_from_embeddings,\n    chart_from_components,\n    indices_of_nearest_neighbors_from_distances,\n)\n\nEMBEDDING_MODEL = \"text-embedding-3-small\"\n\n# load in data and clean data types and drop null rows\ndf = pd.read_csv(\"../../data/styles_2k.csv\", on_bad_lines='skip')\ndf.dropna(inplace=True)\ndf[\"year\"] = df[\"year\"].astype(int)\ndf.info()\n\n# print dataframe\nn_examples = 5\ndf.head(n_examples)\n\n```\n\n    <class 'pandas.core.frame.DataFrame'>\n    Index: 1978 entries, 0 to 1998\n    Data columns (total 10 columns):\n     #   Column              Non-Null Count  Dtype \n    ---  ------              --------------  ----- \n     0   id                  1978 non-null   int64 \n     1   gender              1978 non-null   object\n     2   masterCategory      1978 non-null   object\n     3   subCategory         1978 non-null   object\n     4   articleType         1978 non-null   object\n     5   baseColour          1978 non-null   object\n     6   season              1978 non-null   object\n     7   year                1978 non-null   int64 \n     8   usage               1978 non-null   object\n     9   productDisplayName  1978 non-null   object\n    dtypes: int64(2), object(8)\n    memory usage: 170.0+ KB\n\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>gender</th>\n      <th>masterCategory</th>\n      <th>subCategory</th>\n      <th>articleType</th>\n      <th>baseColour</th>\n      <th>season</th>\n      <th>year</th>\n      <th>usage</th>\n      <th>productDisplayName</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>15970</td>\n      <td>Men</td>\n      <td>Apparel</td>\n      <td>Topwear</td>\n      <td>Shirts</td>\n      <td>Navy Blue</td>\n      <td>Fall</td>\n      <td>2011</td>\n      <td>Casual</td>\n      <td>Turtle Check Men Navy Blue Shirt</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>39386</td>\n      <td>Men</td>\n      <td>Apparel</td>\n      <td>Bottomwear</td>\n      <td>Jeans</td>\n      <td>Blue</td>\n      <td>Summer</td>\n      <td>2012</td>\n      <td>Casual</td>\n      <td>Peter England Men Party Blue Jeans</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>59263</td>\n      <td>Women</td>\n      <td>Accessories</td>\n      <td>Watches</td>\n      <td>Watches</td>\n      <td>Silver</td>\n      <td>Winter</td>\n      <td>2016</td>\n      <td>Casual</td>\n      <td>Titan Women Silver Watch</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>21379</td>\n      <td>Men</td>\n      <td>Apparel</td>\n      <td>Bottomwear</td>\n      <td>Track Pants</td>\n      <td>Black</td>\n      <td>Fall</td>\n      <td>2011</td>\n      <td>Casual</td>\n      <td>Manchester United Men Solid Black Track Pants</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>53759</td>\n      <td>Men</td>\n      <td>Apparel</td>\n      <td>Topwear</td>\n      <td>Tshirts</td>\n      <td>Grey</td>\n      <td>Summer</td>\n      <td>2012</td>\n      <td>Casual</td>\n      <td>Puma Men Grey T-shirt</td>\n    </tr>\n  </tbody>\n</table>\n</div>"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 4ba9d095-3ed3-4e54-a1ec-10a1dfe35a98
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"4ba9d095-3ed3-4e54-a1ec-10a1dfe35a98\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60d79eba-f7df-4e3d-a0e3-5be420e13a39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1a1e9c86a0c7e0dd7950d0417fc95d0b24cfef8382d201a54d7fa0cebf30ede5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5cac430a-b25b-4b73-b785-30d7a2b65903\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7707ffb8976b55b04baa826e458040f0b8b20ad58ef24d6adb10175b91db40b0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3b339025-9fc0-4936-93ac-f844d72c7c87\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fbc02fb4f6fbee58d7cafd7b61e743d803df335815c61765236654adeb6db225\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9732, \"end_char_idx\": 12404, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "ref_doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "chroma:document": "```python\ndf[\"product_text\"] = df.apply(lambda row: f\"name {row['productDisplayName']} category {row['masterCategory']} subcategory {row['subCategory']} color {row['baseColour']} gender {row['gender']}\".lower(), axis=1)\ndf.rename({\"id\":\"product_id\"}, inplace=True, axis=1)\n\ndf.info()\n\n```\n\n    <class 'pandas.core.frame.DataFrame'>\n    Index: 1978 entries, 0 to 1998\n    Data columns (total 11 columns):\n     #   Column              Non-Null Count  Dtype \n    ---  ------              --------------  ----- \n     0   product_id          1978 non-null   int64 \n     1   gender              1978 non-null   object\n     2   masterCategory      1978 non-null   object\n     3   subCategory         1978 non-null   object\n     4   articleType         1978 non-null   object\n     5   baseColour          1978 non-null   object\n     6   season              1978 non-null   object\n     7   year                1978 non-null   int64 \n     8   usage               1978 non-null   object\n     9   productDisplayName  1978 non-null   object\n     10  product_text        1978 non-null   object\n    dtypes: int64(2), object(9)\n    memory usage: 185.4+ KB\n\n\n\n```python\n# check out one of the texts we will use to create semantic embeddings\ndf[\"product_text\"][0]\n\n```\n\n\n\n\n    'name turtle check men navy blue shirt category apparel subcategory topwear color navy blue gender men'\n\n\n\n## Connect to Redis\n\nNow that we have our Redis database running, we can connect to it using the Redis-py client. We will use the default host and port for the Redis database which is `localhost:6379`.\n\n\n\n\n```python\nimport redis\nfrom redis.commands.search.indexDefinition import (\n    IndexDefinition,\n    IndexType\n)\nfrom redis.commands.search.query import Query\nfrom redis.commands.search.field import (\n    TagField,\n    NumericField,\n    TextField,\n    VectorField\n)\n\nREDIS_HOST =  \"localhost\"\nREDIS_PORT = 6379\nREDIS_PASSWORD = \"\" # default for passwordless Redis\n\n# Connect to Redis\nredis_client = redis.Redis(\n    host=REDIS_HOST,\n    port=REDIS_PORT,\n    password=REDIS_PASSWORD\n)\nredis_client.ping()\n\n```\n\n\n\n\n    True\n\n\n\n## Creating a Search Index in Redis\n\nThe below cells will show how to specify and create a search index in Redis. We will:\n\n1. Set some constants for defining our index like the distance metric and the index name\n2. Define the index schema with RediSearch fields\n3. Create the index\n\n\n```python\n# Constants\nINDEX_NAME = \"product_embeddings\"           # name of the search index\nPREFIX = \"doc\"                            # prefix for the document keys\nDISTANCE_METRIC = \"L2\"                # distance metric for the vectors (ex. COSINE, IP, L2)\nNUMBER_OF_VECTORS = len(df)\n\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 3b339025-9fc0-4936-93ac-f844d72c7c87
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"3b339025-9fc0-4936-93ac-f844d72c7c87\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60d79eba-f7df-4e3d-a0e3-5be420e13a39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1a1e9c86a0c7e0dd7950d0417fc95d0b24cfef8382d201a54d7fa0cebf30ede5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4ba9d095-3ed3-4e54-a1ec-10a1dfe35a98\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"6f32c40107bcb4fa1b7d3c5cdeb65a716fb0238e96ce81a210840f4cea635289\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b7399859-0fef-4dd0-a47e-27b30ccf7c2f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e9dcab3fc94bd79bc76d8419f0f6229bdb90ebe00c4b7b45ecd1a5172bdbd1c7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10875, \"end_char_idx\": 13801, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "ref_doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "chroma:document": "```python\n# check out one of the texts we will use to create semantic embeddings\ndf[\"product_text\"][0]\n\n```\n\n\n\n\n    'name turtle check men navy blue shirt category apparel subcategory topwear color navy blue gender men'\n\n\n\n## Connect to Redis\n\nNow that we have our Redis database running, we can connect to it using the Redis-py client. We will use the default host and port for the Redis database which is `localhost:6379`.\n\n\n\n\n```python\nimport redis\nfrom redis.commands.search.indexDefinition import (\n    IndexDefinition,\n    IndexType\n)\nfrom redis.commands.search.query import Query\nfrom redis.commands.search.field import (\n    TagField,\n    NumericField,\n    TextField,\n    VectorField\n)\n\nREDIS_HOST =  \"localhost\"\nREDIS_PORT = 6379\nREDIS_PASSWORD = \"\" # default for passwordless Redis\n\n# Connect to Redis\nredis_client = redis.Redis(\n    host=REDIS_HOST,\n    port=REDIS_PORT,\n    password=REDIS_PASSWORD\n)\nredis_client.ping()\n\n```\n\n\n\n\n    True\n\n\n\n## Creating a Search Index in Redis\n\nThe below cells will show how to specify and create a search index in Redis. We will:\n\n1. Set some constants for defining our index like the distance metric and the index name\n2. Define the index schema with RediSearch fields\n3. Create the index\n\n\n```python\n# Constants\nINDEX_NAME = \"product_embeddings\"           # name of the search index\nPREFIX = \"doc\"                            # prefix for the document keys\nDISTANCE_METRIC = \"L2\"                # distance metric for the vectors (ex. COSINE, IP, L2)\nNUMBER_OF_VECTORS = len(df)\n\n```\n\n\n```python\n# Define RediSearch fields for each of the columns in the dataset\nname = TextField(name=\"productDisplayName\")\ncategory = TagField(name=\"masterCategory\")\narticleType = TagField(name=\"articleType\")\ngender = TagField(name=\"gender\")\nseason = TagField(name=\"season\")\nyear = NumericField(name=\"year\")\ntext_embedding = VectorField(\"product_vector\",\n    \"FLAT\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": 1536,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": NUMBER_OF_VECTORS,\n    }\n)\nfields = [name, category, articleType, gender, season, year, text_embedding]\n\n```\n\n\n```python\n# Check if index exists\ntry:\n    redis_client.ft(INDEX_NAME).info()\n    print(\"Index already exists\")\nexcept:\n    # Create RediSearch Index\n    redis_client.ft(INDEX_NAME).create_index(\n        fields = fields,\n        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n)\n\n```\n\n## Generate OpenAI Embeddings and Load Documents into the Index\n\nNow that we have a search index, we can load documents into it. We will use the dataframe containing the styles dataset loaded previously. In Redis, either the HASH or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The cells below will show how to get OpenAI embeddings for the different products and load documents into the index."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: b7399859-0fef-4dd0-a47e-27b30ccf7c2f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"b7399859-0fef-4dd0-a47e-27b30ccf7c2f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60d79eba-f7df-4e3d-a0e3-5be420e13a39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1a1e9c86a0c7e0dd7950d0417fc95d0b24cfef8382d201a54d7fa0cebf30ede5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3b339025-9fc0-4936-93ac-f844d72c7c87\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"0508acc78ccea3c2745e162ab93f8ba7c7e0518e06201f9df19ddedb176b3c52\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"adff5a9b-fb91-434b-873b-2ae793037e55\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0cc7cfab9ef3a6ccc1ae9dd760ea15783bb28972036b85c80f10369ce4a5c891\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12113, \"end_char_idx\": 15336, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "ref_doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "chroma:document": "```python\n# Constants\nINDEX_NAME = \"product_embeddings\"           # name of the search index\nPREFIX = \"doc\"                            # prefix for the document keys\nDISTANCE_METRIC = \"L2\"                # distance metric for the vectors (ex. COSINE, IP, L2)\nNUMBER_OF_VECTORS = len(df)\n\n```\n\n\n```python\n# Define RediSearch fields for each of the columns in the dataset\nname = TextField(name=\"productDisplayName\")\ncategory = TagField(name=\"masterCategory\")\narticleType = TagField(name=\"articleType\")\ngender = TagField(name=\"gender\")\nseason = TagField(name=\"season\")\nyear = NumericField(name=\"year\")\ntext_embedding = VectorField(\"product_vector\",\n    \"FLAT\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": 1536,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": NUMBER_OF_VECTORS,\n    }\n)\nfields = [name, category, articleType, gender, season, year, text_embedding]\n\n```\n\n\n```python\n# Check if index exists\ntry:\n    redis_client.ft(INDEX_NAME).info()\n    print(\"Index already exists\")\nexcept:\n    # Create RediSearch Index\n    redis_client.ft(INDEX_NAME).create_index(\n        fields = fields,\n        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n)\n\n```\n\n## Generate OpenAI Embeddings and Load Documents into the Index\n\nNow that we have a search index, we can load documents into it. We will use the dataframe containing the styles dataset loaded previously. In Redis, either the HASH or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The cells below will show how to get OpenAI embeddings for the different products and load documents into the index.\n\n\n```python\n# Use OpenAI get_embeddings batch requests to speed up embedding creation\ndef embeddings_batch_request(documents: pd.DataFrame):\n    records = documents.to_dict(\"records\")\n    print(\"Records to process: \", len(records))\n    product_vectors = []\n    docs = []\n    batchsize = 1000\n\n    for idx,doc in enumerate(records,start=1):\n        # create byte vectors\n        docs.append(doc[\"product_text\"])\n        if idx % batchsize == 0:\n            product_vectors += get_embeddings(docs, EMBEDDING_MODEL)\n            docs.clear()\n            print(\"Vectors processed \", len(product_vectors), end='\\r')\n    product_vectors += get_embeddings(docs, EMBEDDING_MODEL)\n    print(\"Vectors processed \", len(product_vectors), end='\\r')\n    return product_vectors\n\n```\n\n\n```python\ndef index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):\n    product_vectors = embeddings_batch_request(documents)\n    records = documents.to_dict(\"records\")\n    batchsize = 500\n\n    # Use Redis pipelines to batch calls and save on round trip network communication\n    pipe = client.pipeline()\n    for idx,doc in enumerate(records,start=1):\n        key = f\"{prefix}:{str(doc['product_id'])}\"\n\n        # create byte vectors\n        text_embedding = np.array((product_vectors[idx-1]), dtype=np.float32).tobytes()\n\n        # replace list of floats with byte vectors\n        doc[\"product_vector\"] = text_embedding\n\n        pipe.hset(key, mapping = doc)\n        if idx % batchsize == 0:\n            pipe.execute()\n    pipe.execute()\n\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: adff5a9b-fb91-434b-873b-2ae793037e55
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"adff5a9b-fb91-434b-873b-2ae793037e55\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60d79eba-f7df-4e3d-a0e3-5be420e13a39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1a1e9c86a0c7e0dd7950d0417fc95d0b24cfef8382d201a54d7fa0cebf30ede5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b7399859-0fef-4dd0-a47e-27b30ccf7c2f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b126dca29fc257b0dff762c6e55fe1f24f5501949267a04d9f3d5adf0834dde6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"60b44ce2-1c72-4725-897e-3d98515ef59a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"84210844c53e5d61d8ed50964d4be8ebf098a4d84e7d0e7bbbafc30037758387\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13804, \"end_char_idx\": 16042, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "ref_doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "chroma:document": "```python\n# Use OpenAI get_embeddings batch requests to speed up embedding creation\ndef embeddings_batch_request(documents: pd.DataFrame):\n    records = documents.to_dict(\"records\")\n    print(\"Records to process: \", len(records))\n    product_vectors = []\n    docs = []\n    batchsize = 1000\n\n    for idx,doc in enumerate(records,start=1):\n        # create byte vectors\n        docs.append(doc[\"product_text\"])\n        if idx % batchsize == 0:\n            product_vectors += get_embeddings(docs, EMBEDDING_MODEL)\n            docs.clear()\n            print(\"Vectors processed \", len(product_vectors), end='\\r')\n    product_vectors += get_embeddings(docs, EMBEDDING_MODEL)\n    print(\"Vectors processed \", len(product_vectors), end='\\r')\n    return product_vectors\n\n```\n\n\n```python\ndef index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):\n    product_vectors = embeddings_batch_request(documents)\n    records = documents.to_dict(\"records\")\n    batchsize = 500\n\n    # Use Redis pipelines to batch calls and save on round trip network communication\n    pipe = client.pipeline()\n    for idx,doc in enumerate(records,start=1):\n        key = f\"{prefix}:{str(doc['product_id'])}\"\n\n        # create byte vectors\n        text_embedding = np.array((product_vectors[idx-1]), dtype=np.float32).tobytes()\n\n        # replace list of floats with byte vectors\n        doc[\"product_vector\"] = text_embedding\n\n        pipe.hset(key, mapping = doc)\n        if idx % batchsize == 0:\n            pipe.execute()\n    pipe.execute()\n\n```\n\n\n```python\n%%time\nindex_documents(redis_client, PREFIX, df)\nprint(f\"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}\")\n\n```\n\n    Records to process:  1978\n    Loaded 1978 documents in Redis search index with name: product_embeddings\n    CPU times: user 619 ms, sys: 78.9 ms, total: 698 ms\n    Wall time: 3.34 s\n\n\n## Simple Vector Search Queries with OpenAI Query Embeddings\n\nNow that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 60b44ce2-1c72-4725-897e-3d98515ef59a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"60b44ce2-1c72-4725-897e-3d98515ef59a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60d79eba-f7df-4e3d-a0e3-5be420e13a39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1a1e9c86a0c7e0dd7950d0417fc95d0b24cfef8382d201a54d7fa0cebf30ede5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"adff5a9b-fb91-434b-873b-2ae793037e55\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"32dd92a228b6de464e11d9ec2e4a9f6109f93c3f83f535e16d97447abb5e6423\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2fb68a1d-092c-48dd-abd6-de94d5ce12e7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"427afe109f8013c071a1ccf83d173d235e01dba5add7f8ea8e5834734dbe578c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14571, \"end_char_idx\": 17385, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "ref_doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "chroma:document": "```python\ndef index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):\n    product_vectors = embeddings_batch_request(documents)\n    records = documents.to_dict(\"records\")\n    batchsize = 500\n\n    # Use Redis pipelines to batch calls and save on round trip network communication\n    pipe = client.pipeline()\n    for idx,doc in enumerate(records,start=1):\n        key = f\"{prefix}:{str(doc['product_id'])}\"\n\n        # create byte vectors\n        text_embedding = np.array((product_vectors[idx-1]), dtype=np.float32).tobytes()\n\n        # replace list of floats with byte vectors\n        doc[\"product_vector\"] = text_embedding\n\n        pipe.hset(key, mapping = doc)\n        if idx % batchsize == 0:\n            pipe.execute()\n    pipe.execute()\n\n```\n\n\n```python\n%%time\nindex_documents(redis_client, PREFIX, df)\nprint(f\"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}\")\n\n```\n\n    Records to process:  1978\n    Loaded 1978 documents in Redis search index with name: product_embeddings\n    CPU times: user 619 ms, sys: 78.9 ms, total: 698 ms\n    Wall time: 3.34 s\n\n\n## Simple Vector Search Queries with OpenAI Query Embeddings\n\nNow that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database.\n\n\n```python\ndef search_redis(\n    redis_client: redis.Redis,\n    user_query: str,\n    index_name: str = \"product_embeddings\",\n    vector_field: str = \"product_vector\",\n    return_fields: list = [\"productDisplayName\", \"masterCategory\", \"gender\", \"season\", \"year\", \"vector_score\"],\n    hybrid_fields = \"*\",\n    k: int = 20,\n    print_results: bool = True,\n) -> List[dict]:\n\n    # Use OpenAI to create embedding vector from user query\n    embedded_query = openai.Embedding.create(input=user_query,\n                                            model=\"text-embedding-3-small\",\n                                            )[\"data\"][0]['embedding']\n\n    # Prepare the Query\n    base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'\n    query = (\n        Query(base_query)\n         .return_fields(*return_fields)\n         .sort_by(\"vector_score\")\n         .paging(0, k)\n         .dialect(2)\n    )\n    params_dict = {\"vector\": np.array(embedded_query).astype(dtype=np.float32).tobytes()}\n\n    # perform vector search\n    results = redis_client.ft(index_name).search(query, params_dict)\n    if print_results:\n        for i, product in enumerate(results.docs):\n            score = 1 - float(product.vector_score)\n            print(f\"{i}. {product.productDisplayName} (Score: {round(score ,3) })\")\n    return results.docs\n\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 2fb68a1d-092c-48dd-abd6-de94d5ce12e7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"2fb68a1d-092c-48dd-abd6-de94d5ce12e7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60d79eba-f7df-4e3d-a0e3-5be420e13a39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1a1e9c86a0c7e0dd7950d0417fc95d0b24cfef8382d201a54d7fa0cebf30ede5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"60b44ce2-1c72-4725-897e-3d98515ef59a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"66cb3dec06f3fbd8ab69b2cc02977a133900246a0ad9cb87bd926e954a445c89\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8fa7a490-2893-4f55-8511-9e71139a8f36\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e98b9963d8996e68db99fea6328ba8f9af5d69fb5bd4690d91718a7e655aae6c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15702, \"end_char_idx\": 18333, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "ref_doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "chroma:document": "## Simple Vector Search Queries with OpenAI Query Embeddings\n\nNow that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database.\n\n\n```python\ndef search_redis(\n    redis_client: redis.Redis,\n    user_query: str,\n    index_name: str = \"product_embeddings\",\n    vector_field: str = \"product_vector\",\n    return_fields: list = [\"productDisplayName\", \"masterCategory\", \"gender\", \"season\", \"year\", \"vector_score\"],\n    hybrid_fields = \"*\",\n    k: int = 20,\n    print_results: bool = True,\n) -> List[dict]:\n\n    # Use OpenAI to create embedding vector from user query\n    embedded_query = openai.Embedding.create(input=user_query,\n                                            model=\"text-embedding-3-small\",\n                                            )[\"data\"][0]['embedding']\n\n    # Prepare the Query\n    base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'\n    query = (\n        Query(base_query)\n         .return_fields(*return_fields)\n         .sort_by(\"vector_score\")\n         .paging(0, k)\n         .dialect(2)\n    )\n    params_dict = {\"vector\": np.array(embedded_query).astype(dtype=np.float32).tobytes()}\n\n    # perform vector search\n    results = redis_client.ft(index_name).search(query, params_dict)\n    if print_results:\n        for i, product in enumerate(results.docs):\n            score = 1 - float(product.vector_score)\n            print(f\"{i}. {product.productDisplayName} (Score: {round(score ,3) })\")\n    return results.docs\n\n```\n\n\n```python\n# Execute a simple vector search in Redis\nresults = search_redis(redis_client, 'man blue jeans', k=10)\n\n```\n\n    0. John Players Men Blue Jeans (Score: 0.791)\n    1. Lee Men Tino Blue Jeans (Score: 0.775)\n    2. Peter England Men Party Blue Jeans (Score: 0.763)\n    3. Lee Men Blue Chicago Fit Jeans (Score: 0.761)\n    4. Lee Men Blue Chicago Fit Jeans (Score: 0.761)\n    5. French Connection Men Blue Jeans (Score: 0.74)\n    6. Locomotive Men Washed Blue Jeans (Score: 0.739)\n    7. Locomotive Men Washed Blue Jeans (Score: 0.739)\n    8. Do U Speak Green Men Blue Shorts (Score: 0.736)\n    9. Palm Tree Kids Boy Washed Blue Jeans (Score: 0.732)\n\n\n## Hybrid Queries with Redis\n\nThe previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the example below, we will combine vector search with full text search."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 8fa7a490-2893-4f55-8511-9e71139a8f36
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"8fa7a490-2893-4f55-8511-9e71139a8f36\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60d79eba-f7df-4e3d-a0e3-5be420e13a39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1a1e9c86a0c7e0dd7950d0417fc95d0b24cfef8382d201a54d7fa0cebf30ede5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2fb68a1d-092c-48dd-abd6-de94d5ce12e7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"c9ccb5648f159e81637ff10710e65ea328103ee6bafd0a40f4463f1d498936dd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5a6bfd67-140b-4124-ad93-2f1325e5d550\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"78139ef8cfd93f539797b1e3f18d032888c1a0f0ac35a4e06eea55ce811870b3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17388, \"end_char_idx\": 20144, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "ref_doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "chroma:document": "```python\n# Execute a simple vector search in Redis\nresults = search_redis(redis_client, 'man blue jeans', k=10)\n\n```\n\n    0. John Players Men Blue Jeans (Score: 0.791)\n    1. Lee Men Tino Blue Jeans (Score: 0.775)\n    2. Peter England Men Party Blue Jeans (Score: 0.763)\n    3. Lee Men Blue Chicago Fit Jeans (Score: 0.761)\n    4. Lee Men Blue Chicago Fit Jeans (Score: 0.761)\n    5. French Connection Men Blue Jeans (Score: 0.74)\n    6. Locomotive Men Washed Blue Jeans (Score: 0.739)\n    7. Locomotive Men Washed Blue Jeans (Score: 0.739)\n    8. Do U Speak Green Men Blue Shorts (Score: 0.736)\n    9. Palm Tree Kids Boy Washed Blue Jeans (Score: 0.732)\n\n\n## Hybrid Queries with Redis\n\nThe previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the example below, we will combine vector search with full text search.\n\n\n```python\n# improve search quality by adding hybrid query for \"man blue jeans\" in the product vector combined with a phrase search for \"blue jeans\"\nresults = search_redis(redis_client,\n                       \"man blue jeans\",\n                       vector_field=\"product_vector\",\n                       k=10,\n                       hybrid_fields='@productDisplayName:\"blue jeans\"'\n                       )\n\n```\n\n    0. John Players Men Blue Jeans (Score: 0.791)\n    1. Lee Men Tino Blue Jeans (Score: 0.775)\n    2. Peter England Men Party Blue Jeans (Score: 0.763)\n    3. French Connection Men Blue Jeans (Score: 0.74)\n    4. Locomotive Men Washed Blue Jeans (Score: 0.739)\n    5. Locomotive Men Washed Blue Jeans (Score: 0.739)\n    6. Palm Tree Kids Boy Washed Blue Jeans (Score: 0.732)\n    7. Denizen Women Blue Jeans (Score: 0.725)\n    8. Jealous 21 Women Washed Blue Jeans (Score: 0.713)\n    9. Jealous 21 Women Washed Blue Jeans (Score: 0.713)\n\n\n\n```python\n# hybrid query for shirt in the product vector and only include results with the phrase \"slim fit\" in the title\nresults = search_redis(redis_client,\n                       \"shirt\",\n                       vector_field=\"product_vector\",\n                       k=10,\n                       hybrid_fields='@productDisplayName:\"slim fit\"'\n                       )\n\n```\n\n    0. Basics Men White Slim Fit Striped Shirt (Score: 0.633)\n    1. ADIDAS Men's Slim Fit White T-shirt (Score: 0.628)\n    2. Basics Men Blue Slim Fit Checked Shirt (Score: 0.627)\n    3. Basics Men Blue Slim Fit Checked Shirt (Score: 0.627)\n    4. Basics Men Red Slim Fit Checked Shirt (Score: 0.623)\n    5. Basics Men Navy Slim Fit Checked Shirt (Score: 0.613)\n    6. Lee Rinse Navy Blue Slim Fit Jeans (Score: 0.558)\n    7. Tokyo Talkies Women Navy Slim Fit Jeans (Score: 0.552)"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 5a6bfd67-140b-4124-ad93-2f1325e5d550
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"5a6bfd67-140b-4124-ad93-2f1325e5d550\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60d79eba-f7df-4e3d-a0e3-5be420e13a39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1a1e9c86a0c7e0dd7950d0417fc95d0b24cfef8382d201a54d7fa0cebf30ede5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8fa7a490-2893-4f55-8511-9e71139a8f36\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"32b3a6b01e1a416b139c815169249bd53c8046cf35ca1960b8c28815b116a079\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b5d24f36-075a-422f-9331-b9ae16990241\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6542bcbadd9014a90bbf608532b3c81be147d74bc42ba28447573347a75ab055\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19288, \"end_char_idx\": 21168, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "ref_doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "chroma:document": "```python\n# hybrid query for shirt in the product vector and only include results with the phrase \"slim fit\" in the title\nresults = search_redis(redis_client,\n                       \"shirt\",\n                       vector_field=\"product_vector\",\n                       k=10,\n                       hybrid_fields='@productDisplayName:\"slim fit\"'\n                       )\n\n```\n\n    0. Basics Men White Slim Fit Striped Shirt (Score: 0.633)\n    1. ADIDAS Men's Slim Fit White T-shirt (Score: 0.628)\n    2. Basics Men Blue Slim Fit Checked Shirt (Score: 0.627)\n    3. Basics Men Blue Slim Fit Checked Shirt (Score: 0.627)\n    4. Basics Men Red Slim Fit Checked Shirt (Score: 0.623)\n    5. Basics Men Navy Slim Fit Checked Shirt (Score: 0.613)\n    6. Lee Rinse Navy Blue Slim Fit Jeans (Score: 0.558)\n    7. Tokyo Talkies Women Navy Slim Fit Jeans (Score: 0.552)\n\n\n\n```python\n# hybrid query for watch in the product vector and only include results with the tag \"Accessories\" in the masterCategory field\nresults = search_redis(redis_client,\n                       \"watch\",\n                       vector_field=\"product_vector\",\n                       k=10,\n                       hybrid_fields='@masterCategory:{Accessories}'\n                       )\n\n```\n\n    0. Titan Women Gold Watch (Score: 0.544)\n    1. Being Human Men Grey Dial Blue Strap Watch (Score: 0.544)\n    2. Police Men Black Dial Watch PL12170JSB (Score: 0.544)\n    3. Titan Men Black Watch (Score: 0.543)\n    4. Police Men Black Dial Chronograph Watch PL12777JS-02M (Score: 0.542)\n    5. CASIO Youth Series Digital Men Black Small Dial Digital Watch W-210-1CVDF I065 (Score: 0.542)\n    6. Titan Women Silver Watch (Score: 0.542)\n    7. Police Men Black Dial Watch PL12778MSU-61 (Score: 0.541)\n    8. Titan Raga Women Gold Watch (Score: 0.539)\n    9. ADIDAS Original Men Black Dial Chronograph Watch ADH2641 (Score: 0.539)"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: b5d24f36-075a-422f-9331-b9ae16990241
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"b5d24f36-075a-422f-9331-b9ae16990241\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60d79eba-f7df-4e3d-a0e3-5be420e13a39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1a1e9c86a0c7e0dd7950d0417fc95d0b24cfef8382d201a54d7fa0cebf30ede5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5a6bfd67-140b-4124-ad93-2f1325e5d550\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"e1034bcd49914cd2cd6dc0ba2368dd905c06667d1d4c103c632a32cf5ea02afd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8ad14b5d-180c-4397-9d03-c26f8430e4f1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"02816a573c7830234a50742354c3079de7e0a8d731e89b6d8555aa2988d7a767\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 20148, \"end_char_idx\": 22057, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "ref_doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "chroma:document": "```python\n# hybrid query for watch in the product vector and only include results with the tag \"Accessories\" in the masterCategory field\nresults = search_redis(redis_client,\n                       \"watch\",\n                       vector_field=\"product_vector\",\n                       k=10,\n                       hybrid_fields='@masterCategory:{Accessories}'\n                       )\n\n```\n\n    0. Titan Women Gold Watch (Score: 0.544)\n    1. Being Human Men Grey Dial Blue Strap Watch (Score: 0.544)\n    2. Police Men Black Dial Watch PL12170JSB (Score: 0.544)\n    3. Titan Men Black Watch (Score: 0.543)\n    4. Police Men Black Dial Chronograph Watch PL12777JS-02M (Score: 0.542)\n    5. CASIO Youth Series Digital Men Black Small Dial Digital Watch W-210-1CVDF I065 (Score: 0.542)\n    6. Titan Women Silver Watch (Score: 0.542)\n    7. Police Men Black Dial Watch PL12778MSU-61 (Score: 0.541)\n    8. Titan Raga Women Gold Watch (Score: 0.539)\n    9. ADIDAS Original Men Black Dial Chronograph Watch ADH2641 (Score: 0.539)\n\n\n\n```python\n# hybrid query for sandals in the product vector and only include results within the 2011-2012 year range\nresults = search_redis(redis_client,\n                       \"sandals\",\n                       vector_field=\"product_vector\",\n                       k=10,\n                       hybrid_fields='@year:[2011 2012]'\n                       )\n\n```\n\n    0. Enroute Teens Orange Sandals (Score: 0.701)\n    1. Fila Men Camper Brown Sandals (Score: 0.692)\n    2. Clarks Men Black Leather Closed Sandals (Score: 0.691)\n    3. Coolers Men Black Sandals (Score: 0.69)\n    4. Coolers Men Black Sandals (Score: 0.69)\n    5. Enroute Teens Brown Sandals (Score: 0.69)\n    6. Crocs Dora Boots Pink Sandals (Score: 0.69)\n    7. Enroute Men Leather Black Sandals (Score: 0.685)\n    8. ADIDAS Men Navy Blue Benton Sandals (Score: 0.684)\n    9. Coolers Men Black Sports Sandals (Score: 0.684)"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 8ad14b5d-180c-4397-9d03-c26f8430e4f1
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"8ad14b5d-180c-4397-9d03-c26f8430e4f1\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"60d79eba-f7df-4e3d-a0e3-5be420e13a39\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1a1e9c86a0c7e0dd7950d0417fc95d0b24cfef8382d201a54d7fa0cebf30ede5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b5d24f36-075a-422f-9331-b9ae16990241\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"5a2fc793bc4c0f3659196dbf2cfb0fc342d3549a89e89b356952899ffa72b85a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21172, \"end_char_idx\": 23790, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "ref_doc_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "chroma:document": "```python\n# hybrid query for sandals in the product vector and only include results within the 2011-2012 year range\nresults = search_redis(redis_client,\n                       \"sandals\",\n                       vector_field=\"product_vector\",\n                       k=10,\n                       hybrid_fields='@year:[2011 2012]'\n                       )\n\n```\n\n    0. Enroute Teens Orange Sandals (Score: 0.701)\n    1. Fila Men Camper Brown Sandals (Score: 0.692)\n    2. Clarks Men Black Leather Closed Sandals (Score: 0.691)\n    3. Coolers Men Black Sandals (Score: 0.69)\n    4. Coolers Men Black Sandals (Score: 0.69)\n    5. Enroute Teens Brown Sandals (Score: 0.69)\n    6. Crocs Dora Boots Pink Sandals (Score: 0.69)\n    7. Enroute Men Leather Black Sandals (Score: 0.685)\n    8. ADIDAS Men Navy Blue Benton Sandals (Score: 0.684)\n    9. Coolers Men Black Sports Sandals (Score: 0.684)\n\n\n\n```python\n# hybrid query for sandals in the product vector and only include results within the 2011-2012 year range from the summer season\nresults = search_redis(redis_client,\n                       \"blue sandals\",\n                       vector_field=\"product_vector\",\n                       k=10,\n                       hybrid_fields='(@year:[2011 2012] @season:{Summer})'\n                       )\n\n```\n\n    0. ADIDAS Men Navy Blue Benton Sandals (Score: 0.691)\n    1. Enroute Teens Brown Sandals (Score: 0.681)\n    2. ADIDAS Women's Adi Groove Blue Flip Flop (Score: 0.672)\n    3. Enroute Women Turquoise Blue Flats (Score: 0.671)\n    4. Red Tape Men Black Sandals (Score: 0.67)\n    5. Enroute Teens Orange Sandals (Score: 0.661)\n    6. Vans Men Blue Era Scilla Plaid Shoes (Score: 0.658)\n    7. FILA Men Aruba Navy Blue Sandal (Score: 0.657)\n    8. Quiksilver Men Blue Flip Flops (Score: 0.656)\n    9. Reebok Men Navy Twist Sandals (Score: 0.656)\n\n\n\n```python\n# hybrid query for a brown belt filtering results by a year (NUMERIC) with a specific article types (TAG) and with a brand name (TEXT)\nresults = search_redis(redis_client,\n                       \"brown belt\",\n                       vector_field=\"product_vector\",\n                       k=10,\n                       hybrid_fields='(@year:[2012 2012] @articleType:{Shirts | Belts} @productDisplayName:\"Wrangler\")'\n                       )\n\n```\n\n    0. Wrangler Men Leather Brown Belt (Score: 0.67)\n    1. Wrangler Women Black Belt (Score: 0.639)\n    2. Wrangler Men Green Striped Shirt (Score: 0.575)\n    3. Wrangler Men Purple Striped Shirt (Score: 0.549)\n    4. Wrangler Men Griffith White Shirt (Score: 0.543)\n    5. Wrangler Women Stella Green Shirt (Score: 0.542)"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 845cb823-0152-473f-97fd-547a4c7c0398
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"845cb823-0152-473f-97fd-547a4c7c0398\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"73e09a74-6398-44e1-955c-76b6a0839554\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9eec624cd3ebfae27701b1260b8fc26541da89d72f061e152021165c691d897c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"39ff86fd-e87f-4d32-8876-bed104b3a574\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9577e2f6f7b9db4edb4ac6f9c29af58f8407f3420a92bfa98d400fc6e6a1e7eb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3175, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "73e09a74-6398-44e1-955c-76b6a0839554", "doc_id": "73e09a74-6398-44e1-955c-76b6a0839554", "ref_doc_id": "73e09a74-6398-44e1-955c-76b6a0839554", "chroma:document": "# Redis as a Context Store with OpenAI Chat\nThis notebook demonstrates how to use Redis as high-speed context memory with ChatGPT.\n\n## Prerequisites\n* Redis instance with the Redis Search and Redis JSON modules\n* Redis-py client lib\n* OpenAI Python client lib\n* OpenAI API key\n\n## Installation\nInstall Python modules necessary for the examples.\n\n\n```python\n! pip install redis openai python-dotenv openai[datalib]\n```\n\n## OpenAI API Key\nCreate a .env file and add your OpenAI key to it\n\n\n```python\nOPENAI_API_KEY=your_key\n```\n\n## OpenAI Setup\nKey load + helper function for chat completion\n\n\n```python\nimport openai\nimport os\nfrom dotenv import load_dotenv\n\nload_dotenv()\nopenai.api_key = os.getenv(\"OPENAI_API_KEY\")\n\ndef get_completion(prompt, model=\"gpt-3.5-turbo\"):\n    messages = [{\"role\": \"user\", \"content\": prompt}]\n    response = openai.ChatCompletion.create(\n        model=model,\n        messages=messages,\n        temperature=0, \n    )\n    return response.choices[0].message[\"content\"]\n```\n\n## Experiment - Chat Completion on a Topic outside of the Model's Knowledge Cutoff Date\nGpt-3.5-turbo was trained on data up to Sep 2021.  Let's ask it a question about something that is beyond that date.  In this case, the FTX/Sam Bankman-Fried scandal.\n\n\n```python\nprompt = \"Is Sam Bankman-Fried's company, FTX, considered a well-managed company?\"\nresponse = get_completion(prompt)\nprint(response)\n```\n\n## Incomplete Information\nAn unfortunate behavior of these AI systems is the system will provide a confident-sounding response - even when the system is not confident with its result.  One way to mitigate this is prompt re-engineering, as seen below.\n\n\n```python\nprompt =\"Is Sam Bankman-Fried's company, FTX, considered a well-managed company?  If you don't know for certain, say unknown.\"\nresponse = get_completion(prompt)\nprint(response)\n```\n\n## Additional Context\nAnother way to combat incomplete information is to give the system more information such that it can make intelligent decisions vs guessing.  We'll use Redis as the source for that additional context.  We'll pull in business news articles from after the GPT knowledge cut-off date such that the system will have a better understanding of how FTX was actually managed. \n\n## Start the Redis Stack Docker container\n\n\n```python\n! docker compose up -d\n```\n\n## Connect Redis client\n\n\n```python\nfrom redis import from_url\n\nREDIS_URL = 'redis://localhost:6379'\nclient = from_url(REDIS_URL)\nclient.ping()\n```\n\n\n\n\n    True\n\n\n\n## Create Index\n[FT.CREATE](https://redis.io/commands/ft.create/)\n\n\n```python\nfrom redis.commands.search.field import TextField, VectorField\nfrom redis.commands.search.indexDefinition import IndexDefinition, IndexType\n\nschema = [ VectorField('$.vector', \n            \"FLAT\", \n            {   \"TYPE\": 'FLOAT32', \n                \"DIM\": 1536, \n                \"DISTANCE_METRIC\": \"COSINE\"\n            },  as_name='vector' ),\n            TextField('$.content', as_name='content')\n        ]\nidx_def = IndexDefinition(index_type=IndexType.JSON, prefix=['doc:'])\ntry: \n    client.ft('idx').dropindex()\nexcept:\n    pass\nclient.ft('idx').create_index(schema, definition=idx_def)\n```\n\n\n\n\n    b'OK'"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 39ff86fd-e87f-4d32-8876-bed104b3a574
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"39ff86fd-e87f-4d32-8876-bed104b3a574\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"73e09a74-6398-44e1-955c-76b6a0839554\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9eec624cd3ebfae27701b1260b8fc26541da89d72f061e152021165c691d897c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"845cb823-0152-473f-97fd-547a4c7c0398\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7094249ebfbcb0436b8f10a3d8010d9f21ffb426d6191d7805d1c6e8f51a129a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4ab80487-d602-41e0-9afd-63ece9967ac0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6f8b77ba77826d25bb17bac139a8a74e96a27b2ea3b399b86bb720d397d6930f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1658, \"end_char_idx\": 4564, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "73e09a74-6398-44e1-955c-76b6a0839554", "doc_id": "73e09a74-6398-44e1-955c-76b6a0839554", "ref_doc_id": "73e09a74-6398-44e1-955c-76b6a0839554", "chroma:document": "```python\nprompt =\"Is Sam Bankman-Fried's company, FTX, considered a well-managed company?  If you don't know for certain, say unknown.\"\nresponse = get_completion(prompt)\nprint(response)\n```\n\n## Additional Context\nAnother way to combat incomplete information is to give the system more information such that it can make intelligent decisions vs guessing.  We'll use Redis as the source for that additional context.  We'll pull in business news articles from after the GPT knowledge cut-off date such that the system will have a better understanding of how FTX was actually managed. \n\n## Start the Redis Stack Docker container\n\n\n```python\n! docker compose up -d\n```\n\n## Connect Redis client\n\n\n```python\nfrom redis import from_url\n\nREDIS_URL = 'redis://localhost:6379'\nclient = from_url(REDIS_URL)\nclient.ping()\n```\n\n\n\n\n    True\n\n\n\n## Create Index\n[FT.CREATE](https://redis.io/commands/ft.create/)\n\n\n```python\nfrom redis.commands.search.field import TextField, VectorField\nfrom redis.commands.search.indexDefinition import IndexDefinition, IndexType\n\nschema = [ VectorField('$.vector', \n            \"FLAT\", \n            {   \"TYPE\": 'FLOAT32', \n                \"DIM\": 1536, \n                \"DISTANCE_METRIC\": \"COSINE\"\n            },  as_name='vector' ),\n            TextField('$.content', as_name='content')\n        ]\nidx_def = IndexDefinition(index_type=IndexType.JSON, prefix=['doc:'])\ntry: \n    client.ft('idx').dropindex()\nexcept:\n    pass\nclient.ft('idx').create_index(schema, definition=idx_def)\n```\n\n\n\n\n    b'OK'\n\n\n\n## Load Data Files into Redis as JSON Objects with Text and Vector Fields\n[Redis JSON](https://redis.io/docs/stack/json/)\n\n\n```python\nimport os\nimport openai\n\ndirectory = './assets/'\nmodel='text-embedding-3-small'\ni = 1\nfor file in os.listdir(directory):\n    with open(os.path.join(directory, file)) as f:\n        content = f.read()\n        vector = openai.Embedding.create(input = [content], model = model)['data'][0]['embedding']\n        client.json().set(f'doc:{i}', '$', {'content': content, 'vector': vector})\n    i += 1\n```\n\n## Embed the Question and Perform VSS to find the most relevant document\n[KNN Search](https://redis.io/docs/stack/search/reference/vectors/#knn-search)\n\n\n```python\nfrom redis.commands.search.query import Query\nimport numpy as np\n\nvec = np.array(openai.Embedding.create(input = [prompt], model = model)['data'][0]['embedding'], dtype=np.float32).tobytes()\nq = Query('*=>[KNN 1 @vector $query_vec AS vector_score]')\\\n    .sort_by('vector_score')\\\n    .return_fields('content')\\\n    .dialect(2)    \nparams = {\"query_vec\": vec}\n\ncontext = client.ft('idx').search(q, query_params=params).docs[0].content\nprint(context)\n```\n\n    Embattled Crypto Exchange FTX Files for Bankruptcy\n    \n    Nov. 11, 2022\n    On Monday, Sam Bankman-Fried, the chief executive of the cryptocurrency exchange FTX, took to Twitter to reassure his customers: \u201cFTX is fine,\u201d he wrote."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 4ab80487-d602-41e0-9afd-63ece9967ac0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"4ab80487-d602-41e0-9afd-63ece9967ac0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"73e09a74-6398-44e1-955c-76b6a0839554\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9eec624cd3ebfae27701b1260b8fc26541da89d72f061e152021165c691d897c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"39ff86fd-e87f-4d32-8876-bed104b3a574\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"e869331b83b94b7193f8230745a040de1f8612142621a37e9b3972dd504280e9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9fec01b2-2394-48f7-904e-b6743777d905\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a70e33542e45a8ab605868decd6c71edd5f94f50b9e50309a56f0b591bcebcbd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3170, \"end_char_idx\": 6457, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "73e09a74-6398-44e1-955c-76b6a0839554", "doc_id": "73e09a74-6398-44e1-955c-76b6a0839554", "ref_doc_id": "73e09a74-6398-44e1-955c-76b6a0839554", "chroma:document": "b'OK'\n\n\n\n## Load Data Files into Redis as JSON Objects with Text and Vector Fields\n[Redis JSON](https://redis.io/docs/stack/json/)\n\n\n```python\nimport os\nimport openai\n\ndirectory = './assets/'\nmodel='text-embedding-3-small'\ni = 1\nfor file in os.listdir(directory):\n    with open(os.path.join(directory, file)) as f:\n        content = f.read()\n        vector = openai.Embedding.create(input = [content], model = model)['data'][0]['embedding']\n        client.json().set(f'doc:{i}', '$', {'content': content, 'vector': vector})\n    i += 1\n```\n\n## Embed the Question and Perform VSS to find the most relevant document\n[KNN Search](https://redis.io/docs/stack/search/reference/vectors/#knn-search)\n\n\n```python\nfrom redis.commands.search.query import Query\nimport numpy as np\n\nvec = np.array(openai.Embedding.create(input = [prompt], model = model)['data'][0]['embedding'], dtype=np.float32).tobytes()\nq = Query('*=>[KNN 1 @vector $query_vec AS vector_score]')\\\n    .sort_by('vector_score')\\\n    .return_fields('content')\\\n    .dialect(2)    \nparams = {\"query_vec\": vec}\n\ncontext = client.ft('idx').search(q, query_params=params).docs[0].content\nprint(context)\n```\n\n    Embattled Crypto Exchange FTX Files for Bankruptcy\n    \n    Nov. 11, 2022\n    On Monday, Sam Bankman-Fried, the chief executive of the cryptocurrency exchange FTX, took to Twitter to reassure his customers: \u201cFTX is fine,\u201d he wrote. \u201cAssets are fine.\u201d\n    \n    On Friday, FTX announced that it was filing for bankruptcy, capping an extraordinary week of corporate drama that has upended crypto markets, sent shock waves through an industry struggling to gain mainstream credibility and sparked government investigations that could lead to more damaging revelations or even criminal charges.\n    \n    In a statement on Twitter, the company said that Mr. Bankman-Fried had resigned, with John J. Ray III, a corporate turnaround specialist, taking over as chief executive.\n    \n    The speed of FTX\u2019s downfall has left crypto insiders stunned. Just days ago, Mr. Bankman-Fried was considered one of the smartest leaders in the crypto industry, an influential figure in Washington who was lobbying to shape regulations. And FTX was widely viewed as one of the most stable and responsible companies in the freewheeling, loosely regulated crypto industry.\n    \n    \u201cHere we are, with one of the richest people in the world, his net worth dropping to zero, his business dropping to zero,\u201d said Jared Ellias, a bankruptcy professor at Harvard Law School. \u201cThe velocity of this failure is just unbelievable.\u201d\n    \n    Now, the bankruptcy has set up a rush among investors and customers to salvage funds from what remains of FTX. A surge of customers tried to withdraw funds from the platform this week, and the company couldn\u2019t meet the demand. The exchange owes as much as $8 billion, according to people familiar with its finances.\n    \n    FTX\u2019s collapse has destabilized the crypto industry, which was already reeling from a crash in the spring that drained $1 trillion from the market. The prices of the leading cryptocurrencies, Bitcoin and Ether, have plummeted. The crypto lender BlockFi, which was closely entangled with FTX, announced on Thursday that it was suspending operations as a result of FTX\u2019s collapse.\n    \n    Mr."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 9fec01b2-2394-48f7-904e-b6743777d905
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"9fec01b2-2394-48f7-904e-b6743777d905\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"73e09a74-6398-44e1-955c-76b6a0839554\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9eec624cd3ebfae27701b1260b8fc26541da89d72f061e152021165c691d897c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4ab80487-d602-41e0-9afd-63ece9967ac0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"d8822cdbf4806eacde0372165e2ff29e9de4a2b1b7405a1c7ba98512dcc6a9eb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"219880c5-75ed-4cee-bd07-807f22773840\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b2153be666f5197b61503fa31fcdd4b832fd34f27f3c69be0b1bde62eaba416f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4565, \"end_char_idx\": 8177, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "73e09a74-6398-44e1-955c-76b6a0839554", "doc_id": "73e09a74-6398-44e1-955c-76b6a0839554", "ref_doc_id": "73e09a74-6398-44e1-955c-76b6a0839554", "chroma:document": "\u201cAssets are fine.\u201d\n    \n    On Friday, FTX announced that it was filing for bankruptcy, capping an extraordinary week of corporate drama that has upended crypto markets, sent shock waves through an industry struggling to gain mainstream credibility and sparked government investigations that could lead to more damaging revelations or even criminal charges.\n    \n    In a statement on Twitter, the company said that Mr. Bankman-Fried had resigned, with John J. Ray III, a corporate turnaround specialist, taking over as chief executive.\n    \n    The speed of FTX\u2019s downfall has left crypto insiders stunned. Just days ago, Mr. Bankman-Fried was considered one of the smartest leaders in the crypto industry, an influential figure in Washington who was lobbying to shape regulations. And FTX was widely viewed as one of the most stable and responsible companies in the freewheeling, loosely regulated crypto industry.\n    \n    \u201cHere we are, with one of the richest people in the world, his net worth dropping to zero, his business dropping to zero,\u201d said Jared Ellias, a bankruptcy professor at Harvard Law School. \u201cThe velocity of this failure is just unbelievable.\u201d\n    \n    Now, the bankruptcy has set up a rush among investors and customers to salvage funds from what remains of FTX. A surge of customers tried to withdraw funds from the platform this week, and the company couldn\u2019t meet the demand. The exchange owes as much as $8 billion, according to people familiar with its finances.\n    \n    FTX\u2019s collapse has destabilized the crypto industry, which was already reeling from a crash in the spring that drained $1 trillion from the market. The prices of the leading cryptocurrencies, Bitcoin and Ether, have plummeted. The crypto lender BlockFi, which was closely entangled with FTX, announced on Thursday that it was suspending operations as a result of FTX\u2019s collapse.\n    \n    Mr. Bankman-Fried was backed by some of the highest-profile venture capital investors in Silicon Valley, including Sequoia Capital and Lightspeed Venture Partners. Some of those investors, facing questions about how closely they scrutinized FTX before they put money into it, have said that their nine-figure investments in the crypto exchange are now essentially worthless.\n    \n    The company\u2019s demise has also set off a reckoning over risky practices that have become pervasive in crypto, an industry that was founded partly as a corrective to the type of dangerous financial engineering that caused the 2008 economic crisis.\n    \n    \u201cI\u2019m really sorry, again, that we ended up here,\u201d Mr. Bankman-Fried said on Twitter on Friday. \u201cHopefully this can bring some amount of transparency, trust, and governance.\u201d\n    \n    The bankruptcy filing marks the start of what will probably be months or even years of legal fallout, as lawyers try to work out whether the exchange can ever continue to operate in some form and customers demand compensation. FTX is already the target of investigations by the Securities and Exchange Commission and the Justice Department, with investigators focused on whether the company improperly used customer funds to prop up Alameda Research, a trading firm that Mr. Bankman-Fried also founded.\n    \n    The bankruptcy filing included FTX, its U.S. arm and Alameda. According to a bare-bones legal filing in U.S. Bankruptcy Court in Delaware, FTX has assets valued between $10 billion and $50 billion, with the size of its liabilities in the same range. The company has more than 100,000 creditors, the filing said.\n    \n    The bankruptcy is a stunning fall from grace for the 30-year-old Mr."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 219880c5-75ed-4cee-bd07-807f22773840
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"219880c5-75ed-4cee-bd07-807f22773840\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"73e09a74-6398-44e1-955c-76b6a0839554\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9eec624cd3ebfae27701b1260b8fc26541da89d72f061e152021165c691d897c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9fec01b2-2394-48f7-904e-b6743777d905\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b8ad5c5bcf7bcabedd32ee64167098d9dec7b44d42fab23c5a17be6d2749b229\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8a08a232-fbfc-4cfd-9cfd-a06cbe48e204\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"12e28056c6f8a916c4f0b7be06f8de47e4a3bcb67333d3d611275c27ff046d25\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6293, \"end_char_idx\": 9656, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "73e09a74-6398-44e1-955c-76b6a0839554", "doc_id": "73e09a74-6398-44e1-955c-76b6a0839554", "ref_doc_id": "73e09a74-6398-44e1-955c-76b6a0839554", "chroma:document": "The crypto lender BlockFi, which was closely entangled with FTX, announced on Thursday that it was suspending operations as a result of FTX\u2019s collapse.\n    \n    Mr. Bankman-Fried was backed by some of the highest-profile venture capital investors in Silicon Valley, including Sequoia Capital and Lightspeed Venture Partners. Some of those investors, facing questions about how closely they scrutinized FTX before they put money into it, have said that their nine-figure investments in the crypto exchange are now essentially worthless.\n    \n    The company\u2019s demise has also set off a reckoning over risky practices that have become pervasive in crypto, an industry that was founded partly as a corrective to the type of dangerous financial engineering that caused the 2008 economic crisis.\n    \n    \u201cI\u2019m really sorry, again, that we ended up here,\u201d Mr. Bankman-Fried said on Twitter on Friday. \u201cHopefully this can bring some amount of transparency, trust, and governance.\u201d\n    \n    The bankruptcy filing marks the start of what will probably be months or even years of legal fallout, as lawyers try to work out whether the exchange can ever continue to operate in some form and customers demand compensation. FTX is already the target of investigations by the Securities and Exchange Commission and the Justice Department, with investigators focused on whether the company improperly used customer funds to prop up Alameda Research, a trading firm that Mr. Bankman-Fried also founded.\n    \n    The bankruptcy filing included FTX, its U.S. arm and Alameda. According to a bare-bones legal filing in U.S. Bankruptcy Court in Delaware, FTX has assets valued between $10 billion and $50 billion, with the size of its liabilities in the same range. The company has more than 100,000 creditors, the filing said.\n    \n    The bankruptcy is a stunning fall from grace for the 30-year-old Mr. Bankman-Fried, who cultivated a reputation as a boy genius with a host of endearing quirks, including a habit of sleeping on a beanbag at the office. At one point, he was one of the richest people in the industry, with an estimated fortune of $24 billion. He hobnobbed with actors, professional athletes and former world leaders.\n    \n    Mr. Bankman-Fried\u2019s crypto empire had an elaborate structure. The bankruptcy filing lists more than 130 corporate entities affiliated with FTX and Alameda. But as of June, FTX had only about 300 employees, a point of pride for Mr. Bankman-Fried, who said he had resisted calls from venture investors to hire more staff.\n    \n    \u201cWe told them additional employees added too quickly were net negative,\u201d Mr. Bankman-Fried said on Twitter in June. \u201cThey could take it or leave it.\u201d\n    \n    Unusually for a major start-up, none of FTX\u2019s investors had seats on the board, which instead consisted of Mr. Bankman-Fried, another FTX executive and a lawyer in Antigua and Barbuda.\n    \n    FTX and Alameda were based in the Bahamas, where Mr. Bankman-Fried and a small circle of top executives called most of the shots and lived together in a luxury resort. Officially, Alameda was run by Caroline Ellison, a former trader for the hedge fund Jane Street, but Mr. Bankman-Fried was heavily involved, contributing to the decision-making on big trades, according to a person familiar with the matter.\n    \n    In addition to Mr. Bankman-Fried and Ms."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 8a08a232-fbfc-4cfd-9cfd-a06cbe48e204
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"8a08a232-fbfc-4cfd-9cfd-a06cbe48e204\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"73e09a74-6398-44e1-955c-76b6a0839554\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9eec624cd3ebfae27701b1260b8fc26541da89d72f061e152021165c691d897c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"219880c5-75ed-4cee-bd07-807f22773840\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1322f9f4969fe5bfcbdb6f71769ca418e476f5ff61ffc39c370db6a52f0fc178\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fc55b6a9-2edf-4992-bad9-095a27dda8dd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f0785849910c8ffbd3a0e71ac91ea9380511878168d243920912b8a520b60862\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8038, \"end_char_idx\": 11334, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "73e09a74-6398-44e1-955c-76b6a0839554", "doc_id": "73e09a74-6398-44e1-955c-76b6a0839554", "ref_doc_id": "73e09a74-6398-44e1-955c-76b6a0839554", "chroma:document": "The company has more than 100,000 creditors, the filing said.\n    \n    The bankruptcy is a stunning fall from grace for the 30-year-old Mr. Bankman-Fried, who cultivated a reputation as a boy genius with a host of endearing quirks, including a habit of sleeping on a beanbag at the office. At one point, he was one of the richest people in the industry, with an estimated fortune of $24 billion. He hobnobbed with actors, professional athletes and former world leaders.\n    \n    Mr. Bankman-Fried\u2019s crypto empire had an elaborate structure. The bankruptcy filing lists more than 130 corporate entities affiliated with FTX and Alameda. But as of June, FTX had only about 300 employees, a point of pride for Mr. Bankman-Fried, who said he had resisted calls from venture investors to hire more staff.\n    \n    \u201cWe told them additional employees added too quickly were net negative,\u201d Mr. Bankman-Fried said on Twitter in June. \u201cThey could take it or leave it.\u201d\n    \n    Unusually for a major start-up, none of FTX\u2019s investors had seats on the board, which instead consisted of Mr. Bankman-Fried, another FTX executive and a lawyer in Antigua and Barbuda.\n    \n    FTX and Alameda were based in the Bahamas, where Mr. Bankman-Fried and a small circle of top executives called most of the shots and lived together in a luxury resort. Officially, Alameda was run by Caroline Ellison, a former trader for the hedge fund Jane Street, but Mr. Bankman-Fried was heavily involved, contributing to the decision-making on big trades, according to a person familiar with the matter.\n    \n    In addition to Mr. Bankman-Fried and Ms. Ellison, the circle of executives running FTX included Nishad Singh, FTX\u2019s director of engineering, and Gary Wang, the chief technology officer. Few others had visibility into how the company was run: When the firm collapsed this week, lower-ranking employees were left confused and blindsided, according to people familiar with the matter. Mr. Singh and Ms. Ellison did not respond to requests for comment; Mr. Wang could not immediately be reached.\n    \n    As a crypto exchange, FTX provided a marketplace for customers to buy, sell and store a wide range of digital currencies. Most of its revenue stemmed from a risky type of trade \u2014 in which crypto investors borrowed money to make huge bets on the future prices of cryptocurrencies \u2014 that remains illegal in the United States. But Mr. Bankman-Fried also ran a smaller U.S. affiliate that offered more basic trading options.\n    \n    Mr. Bankman-Fried\u2019s problems started over the weekend, when the chief executive of Binance, the largest crypto exchange, suggested publicly that FTX might be on shaky financial footing. A rush of customers tried to withdraw their crypto holdings from the platform, and FTX was unable to meet the demand.\n    \n    On Tuesday, Mr. Bankman-Fried said he had struck a deal to sell FTX to Binance. But after reviewing the company\u2019s financial documents, Binance\u2019s chief executive, Changpeng Zhao, pulled out of the agreement, leaving Mr. Bankman-Fried with limited options.\n    \n    In calls with investors and messages to employees this week, he apologized repeatedly and stressed that he was working hard to raise money and resolve the situation. But the hole was ultimately too big to fill."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: fc55b6a9-2edf-4992-bad9-095a27dda8dd
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"fc55b6a9-2edf-4992-bad9-095a27dda8dd\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"73e09a74-6398-44e1-955c-76b6a0839554\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9eec624cd3ebfae27701b1260b8fc26541da89d72f061e152021165c691d897c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8a08a232-fbfc-4cfd-9cfd-a06cbe48e204\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"0e859605259b640dfbd69c91e876fa8cc3e2e7b0126cd8d807e72c3427333292\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a81fc8c1-bb84-4d46-9dd0-5186c244d765\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9ca998b635bf1b5ffd80e15f3d78bca5560b3ef1b0c58578bf8f7095e46337a9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9616, \"end_char_idx\": 13136, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "73e09a74-6398-44e1-955c-76b6a0839554", "doc_id": "73e09a74-6398-44e1-955c-76b6a0839554", "ref_doc_id": "73e09a74-6398-44e1-955c-76b6a0839554", "chroma:document": "In addition to Mr. Bankman-Fried and Ms. Ellison, the circle of executives running FTX included Nishad Singh, FTX\u2019s director of engineering, and Gary Wang, the chief technology officer. Few others had visibility into how the company was run: When the firm collapsed this week, lower-ranking employees were left confused and blindsided, according to people familiar with the matter. Mr. Singh and Ms. Ellison did not respond to requests for comment; Mr. Wang could not immediately be reached.\n    \n    As a crypto exchange, FTX provided a marketplace for customers to buy, sell and store a wide range of digital currencies. Most of its revenue stemmed from a risky type of trade \u2014 in which crypto investors borrowed money to make huge bets on the future prices of cryptocurrencies \u2014 that remains illegal in the United States. But Mr. Bankman-Fried also ran a smaller U.S. affiliate that offered more basic trading options.\n    \n    Mr. Bankman-Fried\u2019s problems started over the weekend, when the chief executive of Binance, the largest crypto exchange, suggested publicly that FTX might be on shaky financial footing. A rush of customers tried to withdraw their crypto holdings from the platform, and FTX was unable to meet the demand.\n    \n    On Tuesday, Mr. Bankman-Fried said he had struck a deal to sell FTX to Binance. But after reviewing the company\u2019s financial documents, Binance\u2019s chief executive, Changpeng Zhao, pulled out of the agreement, leaving Mr. Bankman-Fried with limited options.\n    \n    In calls with investors and messages to employees this week, he apologized repeatedly and stressed that he was working hard to raise money and resolve the situation. But the hole was ultimately too big to fill.\n    \n    FTX\u2019s bankruptcy is the latest \u2014 and by far the biggest \u2014 in a series of bankruptcies that have shaken the crypto world this year. After a market crash in the spring, two crypto lending companies, Celsius Network and Voyager Digital, filed for bankruptcy, kicking off months of legal maneuvering over how their remaining assets should be divided. In an ironic twist, FTX had recently won an auction to buy Voyager\u2019s remaining assets.\n    \n    As it enters its own bankruptcy process, FTX will be led by Mr. Ray, who has ample experience managing distressed situations. He helped manage Enron after the collapse of its business in an accounting fraud scandal in 2001. And he helped liquidate the trust of the subprime mortgage company ResCap after its 2012 bankruptcy.\n    \n    The bankruptcy proceedings may be only the beginning of Mr. Bankman-Fried\u2019s legal troubles. Federal investigators are examining the relationship between FTX and Alameda, and customers are likely to file lawsuits.\n    \n    Mr. Bankman-Fried\u2019s old allies have quickly abandoned him. On Thursday night, the team running the FTX Future Fund, a charitable group that Mr. Bankman-Fried bankrolled, announced that they were resigning.\n    \n    \u201cWe were shocked and immensely saddened to learn of the recent events at FTX,\u201d they wrote in a statement. \u201cWe have fundamental questions about the legitimacy and integrity of the business operations that were funding the FTX Foundation and the Future Fund.\u201d\n    \n    Not long ago, Mr. Bankman-Fried was performing a comedy routine onstage at a conference with Anthony Scaramucci, the former White House communications director and a business partner of FTX.\n    \n    \u201cI\u2019m disappointed,\u201d Mr. Scaramucci said in an interview on CNBC on Friday. \u201cDuped, I guess, is the right word.\u201d"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: a81fc8c1-bb84-4d46-9dd0-5186c244d765
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"a81fc8c1-bb84-4d46-9dd0-5186c244d765\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"73e09a74-6398-44e1-955c-76b6a0839554\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9eec624cd3ebfae27701b1260b8fc26541da89d72f061e152021165c691d897c\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fc55b6a9-2edf-4992-bad9-095a27dda8dd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"24ca6164e71f071a7a47e5ec8b01515360edbffd85fa2b87d552948f3bb38530\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11290, \"end_char_idx\": 14107, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "73e09a74-6398-44e1-955c-76b6a0839554", "doc_id": "73e09a74-6398-44e1-955c-76b6a0839554", "ref_doc_id": "73e09a74-6398-44e1-955c-76b6a0839554", "chroma:document": "But the hole was ultimately too big to fill.\n    \n    FTX\u2019s bankruptcy is the latest \u2014 and by far the biggest \u2014 in a series of bankruptcies that have shaken the crypto world this year. After a market crash in the spring, two crypto lending companies, Celsius Network and Voyager Digital, filed for bankruptcy, kicking off months of legal maneuvering over how their remaining assets should be divided. In an ironic twist, FTX had recently won an auction to buy Voyager\u2019s remaining assets.\n    \n    As it enters its own bankruptcy process, FTX will be led by Mr. Ray, who has ample experience managing distressed situations. He helped manage Enron after the collapse of its business in an accounting fraud scandal in 2001. And he helped liquidate the trust of the subprime mortgage company ResCap after its 2012 bankruptcy.\n    \n    The bankruptcy proceedings may be only the beginning of Mr. Bankman-Fried\u2019s legal troubles. Federal investigators are examining the relationship between FTX and Alameda, and customers are likely to file lawsuits.\n    \n    Mr. Bankman-Fried\u2019s old allies have quickly abandoned him. On Thursday night, the team running the FTX Future Fund, a charitable group that Mr. Bankman-Fried bankrolled, announced that they were resigning.\n    \n    \u201cWe were shocked and immensely saddened to learn of the recent events at FTX,\u201d they wrote in a statement. \u201cWe have fundamental questions about the legitimacy and integrity of the business operations that were funding the FTX Foundation and the Future Fund.\u201d\n    \n    Not long ago, Mr. Bankman-Fried was performing a comedy routine onstage at a conference with Anthony Scaramucci, the former White House communications director and a business partner of FTX.\n    \n    \u201cI\u2019m disappointed,\u201d Mr. Scaramucci said in an interview on CNBC on Friday. \u201cDuped, I guess, is the right word.\u201d\n    \n\n\n## Repeat the Question to OpenAI with context\nNow that we have relevant context, add that to the prompt to OpenAI and get a very different response.\n\n\n```python\nprompt = f\"\"\"\nUsing the information delimited by triple backticks, answer this question: Is Sam Bankman-Fried's company, FTX, considered a well-managed company?\n\nContext: ```{context}```\n\"\"\"\n\nresponse = get_completion(prompt)\nprint(response)\n```\n\n    No, Sam Bankman-Fried's company FTX is not considered a well-managed company as it has filed for bankruptcy and owes as much as $8 billion to its creditors. The collapse of FTX has destabilized the crypto industry, and the company is already the target of investigations by the Securities and Exchange Commission and the Justice Department. FTX was widely viewed as one of the most stable and responsible companies in the freewheeling, loosely regulated crypto industry, but its risky practices have become pervasive in crypto, leading to a reckoning."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: f5f916bf-98fb-4b91-a510-5046868f4bfb
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"f5f916bf-98fb-4b91-a510-5046868f4bfb\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"08e115f5-0f6b-41c0-8aa1-a201735de5bc\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"59ff14ffeee6b7e47ae59ae97b4b178f7ca83621d6168206707ca95cbea61f95\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6e838e90-90ae-4998-8315-298334c945b3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8d2d0ca97243dde3ea8c9044ae310d031613be0ed046441b15d692edd9f715e9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 709, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "doc_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "ref_doc_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "chroma:document": "# Redis Vectors as JSON with OpenAI\nThis notebook expands on the other Redis OpenAI-cookbook examples with examples of how to use JSON with vectors.\n[Storing Vectors in JSON](https://redis.io/docs/stack/search/reference/vectors/#storing-vectors-in-json)\n\n## Prerequisites\n* Redis instance with the Redis Search and Redis JSON modules\n* Redis-py client lib\n* OpenAI API key\n\n## Installation\nInstall Python modules necessary for the examples.\n\n\n```python\n! pip install redis openai python-dotenv openai[datalib]\n```\n\n## OpenAI API Key\nCreate a .env file and add your OpenAI key to it\n\n\n```python\nOPENAI_API_KEY=your_key\n```\n\n## Create Text Vectors\nCreate embeddings (array of floats) of the news excerpts below."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 6e838e90-90ae-4998-8315-298334c945b3
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"6e838e90-90ae-4998-8315-298334c945b3\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"08e115f5-0f6b-41c0-8aa1-a201735de5bc\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"59ff14ffeee6b7e47ae59ae97b4b178f7ca83621d6168206707ca95cbea61f95\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f5f916bf-98fb-4b91-a510-5046868f4bfb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"348ddf8544ecf5da8aa39d931d3b7b382d73f867aed87216bdf5582a88a7244c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6109e86a-35a8-47e8-9247-316235354563\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b52d340c57370ee932a3f11059eb95a1125a10f80eaf12ffe964ec2d6c67af04\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3176, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "doc_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "ref_doc_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "chroma:document": "# Redis Vectors as JSON with OpenAI\nThis notebook expands on the other Redis OpenAI-cookbook examples with examples of how to use JSON with vectors.\n[Storing Vectors in JSON](https://redis.io/docs/stack/search/reference/vectors/#storing-vectors-in-json)\n\n## Prerequisites\n* Redis instance with the Redis Search and Redis JSON modules\n* Redis-py client lib\n* OpenAI API key\n\n## Installation\nInstall Python modules necessary for the examples.\n\n\n```python\n! pip install redis openai python-dotenv openai[datalib]\n```\n\n## OpenAI API Key\nCreate a .env file and add your OpenAI key to it\n\n\n```python\nOPENAI_API_KEY=your_key\n```\n\n## Create Text Vectors\nCreate embeddings (array of floats) of the news excerpts below.\n\n\n```python\nimport openai\nimport os\nfrom dotenv import load_dotenv\n\nload_dotenv()\nopenai.api_key = os.getenv(\"OPENAI_API_KEY\")\n\ndef get_vector(text, model=\"text-embedding-3-small\"):\n    text = text.replace(\"\\n\", \" \")\n    return openai.Embedding.create(input = [text], model = model)['data'][0]['embedding']\n\ntext_1 = \"\"\"Japan narrowly escapes recession\n\nJapan's economy teetered on the brink of a technical recession in the three months to September, figures show.\n\nRevised figures indicated growth of just 0.1% - and a similar-sized contraction in the previous quarter. On an annual basis, the data suggests annual growth of just 0.2%, suggesting a much more hesitant recovery than had previously been thought. A common technical definition of a recession is two successive quarters of negative growth.\nThe government was keen to play down the worrying implications of the data. \"I maintain the view that Japan's economy remains in a minor adjustment phase in an upward climb, and we will monitor developments carefully,\" said economy minister Heizo Takenaka. But in the face of the strengthening yen making exports less competitive and indications of weakening economic conditions ahead, observers were less sanguine. \"It's painting a picture of a recovery... much patchier than previously thought,\" said Paul Sheard, economist at Lehman Brothers in Tokyo. Improvements in the job market apparently have yet to feed through to domestic demand, with private consumption up just 0.2% in the third quarter.\n\"\"\"\n\ntext_2 = \"\"\"Dibaba breaks 5,000m world record\n\nEthiopia's Tirunesh Dibaba set a new world record in winning the women's 5,000m at the Boston Indoor Games.\n\nDibaba won in 14 minutes 32.93 seconds to erase the previous world indoor mark of 14:39.29 set by another Ethiopian, Berhane Adera, in Stuttgart last year. But compatriot Kenenisa Bekele's record hopes were dashed when he miscounted his laps in the men's 3,000m and staged his sprint finish a lap too soon. Ireland's Alistair Cragg won in 7:39.89 as Bekele battled to second in 7:41.42. \"I didn't want to sit back and get out-kicked,\" said Cragg. \"So I kept on the pace. The plan was to go with 500m to go no matter what, but when Bekele made the mistake that was it. The race was mine.\" Sweden's Carolina Kluft, the Olympic heptathlon champion, and Slovenia's Jolanda Ceplak had winning performances, too. Kluft took the long jump at 6.63m, while Ceplak easily won the women's 800m in 2:01.52. \n\"\"\""}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 6109e86a-35a8-47e8-9247-316235354563
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"6109e86a-35a8-47e8-9247-316235354563\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"08e115f5-0f6b-41c0-8aa1-a201735de5bc\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"59ff14ffeee6b7e47ae59ae97b4b178f7ca83621d6168206707ca95cbea61f95\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6e838e90-90ae-4998-8315-298334c945b3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"adf90edc5a7283a25aa69c87a8dc7f0186fde47d3c6b6530d3f5bcc4d047327f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e642379a-83fd-41a5-9252-fecf05eb91a4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8aa3fa81465b5cd8bfb62fd6cb659f9a29efc7516604b196434a77fe9e194f48\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3179, \"end_char_idx\": 6356, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "doc_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "ref_doc_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "chroma:document": "text_3 = \"\"\"Google's toolbar sparks concern\n\nSearch engine firm Google has released a trial tool which is concerning some net users because it directs people to pre-selected commercial websites.\n\nThe AutoLink feature comes with Google's latest toolbar and provides links in a webpage to Amazon.com if it finds a book's ISBN number on the site. It also links to Google's map service, if there is an address, or to car firm Carfax, if there is a licence plate. Google said the feature, available only in the US, \"adds useful links\". But some users are concerned that Google's dominant position in the search engine market place could mean it would be giving a competitive edge to firms like Amazon.\n\nAutoLink works by creating a link to a website based on information contained in a webpage - even if there is no link specified and whether or not the publisher of the page has given permission.\n\nIf a user clicks the AutoLink feature in the Google toolbar then a webpage with a book's unique ISBN number would link directly to Amazon's website. It could mean online libraries that list ISBN book numbers find they are directing users to Amazon.com whether they like it or not. Websites which have paid for advertising on their pages may also be directing people to rival services. Dan Gillmor, founder of Grassroots Media, which supports citizen-based media, said the tool was a \"bad idea, and an unfortunate move by a company that is looking to continue its hypergrowth\". In a statement Google said the feature was still only in beta, ie trial, stage and that the company welcomed feedback from users. It said: \"The user can choose never to click on the AutoLink button, and web pages she views will never be modified. \"In addition, the user can choose to disable the AutoLink feature entirely at any time.\"\n\nThe new tool has been compared to the Smart Tags feature from Microsoft by some users. It was widely criticised by net users and later dropped by Microsoft after concerns over trademark use were raised. Smart Tags allowed Microsoft to link any word on a web page to another site chosen by the company. Google said none of the companies which received AutoLinks had paid for the service. Some users said AutoLink would only be fair if websites had to sign up to allow the feature to work on their pages or if they received revenue for any \"click through\" to a commercial site. Cory Doctorow, European outreach coordinator for digital civil liberties group Electronic Fronter Foundation, said that Google should not be penalised for its market dominance. \"Of course Google should be allowed to direct people to whatever proxies it chooses. \"But as an end user I would want to know - 'Can I choose to use this service?, 'How much is Google being paid?', 'Can I substitute my own companies for the ones chosen by Google?'.\" Mr Doctorow said the only objection would be if users were forced into using AutoLink or \"tricked into using the service\".\n\"\"\"\n\ndoc_1 = {\"content\": text_1, \"vector\": get_vector(text_1)}\ndoc_2 = {\"content\": text_2, \"vector\": get_vector(text_2)}\ndoc_3 = {\"content\": text_3, \"vector\": get_vector(text_3)}\n\n```\n\n## Start the Redis Stack Docker container"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: e642379a-83fd-41a5-9252-fecf05eb91a4
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"e642379a-83fd-41a5-9252-fecf05eb91a4\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"08e115f5-0f6b-41c0-8aa1-a201735de5bc\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"59ff14ffeee6b7e47ae59ae97b4b178f7ca83621d6168206707ca95cbea61f95\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6109e86a-35a8-47e8-9247-316235354563\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"8b21fde02b0bd82bbfe5baf341271a51a0e4519f36a6245b1628383fc826b519\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7cf8afae-6c9b-4741-92ea-ffccdaf522d3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b9286c8669040e5a17b1e8e84fda5b717dfcc7b4eb9245ee5613eb8a88da4958\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6359, \"end_char_idx\": 8920, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "doc_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "ref_doc_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "chroma:document": "```python\n! docker compose up -d\n```\n\n    \u001b[1A\u001b[1B\u001b[0G\u001b[?25l[+] Running 0/0\n     \u283f Container redisjson-redis-1  Starting                                   \u001b[34m0.1s \u001b[0m\n    \u001b[?25h\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 0/1\n     \u283f Container redisjson-redis-1  Starting                                   \u001b[34m0.2s \u001b[0m\n    \u001b[?25h\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 0/1\n     \u283f Container redisjson-redis-1  Starting                                   \u001b[34m0.3s \u001b[0m\n    \u001b[?25h\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 0/1\n     \u283f Container redisjson-redis-1  Starting                                   \u001b[34m0.4s \u001b[0m\n    \u001b[?25h\u001b[1A\u001b[1A\u001b[0G\u001b[?25l\u001b[34m[+] Running 1/1\u001b[0m\n     \u001b[32m\u2714\u001b[0m Container redisjson-redis-1  \u001b[32mStarted\u001b[0m                                    \u001b[34m0.4s \u001b[0m\n    \u001b[?25h\n\n## Connect Redis client\n\n\n```python\nfrom redis import from_url\n\nREDIS_URL = 'redis://localhost:6379'\nclient = from_url(REDIS_URL)\nclient.ping()\n```\n\n\n\n\n    True\n\n\n\n## Create Index\n[FT.CREATE](https://redis.io/commands/ft.create/)\n\n\n```python\nfrom redis.commands.search.field import TextField, VectorField\nfrom redis.commands.search.indexDefinition import IndexDefinition, IndexType\n\nschema = [ VectorField('$.vector', \n            \"FLAT\", \n            {   \"TYPE\": 'FLOAT32', \n                \"DIM\": len(doc_1['vector']), \n                \"DISTANCE_METRIC\": \"COSINE\"\n            },  as_name='vector' ),\n            TextField('$.content', as_name='content')\n        ]\nidx_def = IndexDefinition(index_type=IndexType.JSON, prefix=['doc:'])\ntry: \n    client.ft('idx').dropindex()\nexcept:\n    pass\nclient.ft('idx').create_index(schema, definition=idx_def)\n```\n\n\n\n\n    b'OK'\n\n\n\n## Load Data into Redis as JSON objects\n[Redis JSON](https://redis.io/docs/stack/json/)\n\n\n```python\nclient.json().set('doc:1', '$', doc_1)\nclient.json().set('doc:2', '$', doc_2)\nclient.json().set('doc:3', '$', doc_3)\n```\n\n\n\n\n    True\n\n\n\n# Semantic Search\nGiven a sports-related article, search Redis via Vector Similarity Search (VSS) for similar articles.  \n[KNN Search](https://redis.io/docs/stack/search/reference/vectors/#knn-search)\n\n\n```python\nfrom redis.commands.search.query import Query\nimport numpy as np\n\ntext_4 = \"\"\"Radcliffe yet to answer GB call\n\nPaula Radcliffe has been granted extra time to decide whether to compete in the World Cross-Country Championships.\n\nThe 31-year-old is concerned the event, which starts on 19 March in France, could upset her preparations for the London Marathon on 17 April. \"There is no question that Paula would be a huge asset to the GB team,\" said Zara Hyde Peters of UK Athletics."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 7cf8afae-6c9b-4741-92ea-ffccdaf522d3
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"7cf8afae-6c9b-4741-92ea-ffccdaf522d3\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"08e115f5-0f6b-41c0-8aa1-a201735de5bc\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"59ff14ffeee6b7e47ae59ae97b4b178f7ca83621d6168206707ca95cbea61f95\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e642379a-83fd-41a5-9252-fecf05eb91a4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"fd5fd94403a097a4a4fafc9b86338ce408558cdf1bd78e12479b0815f0b47e1a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d6b59611-6bca-4882-813f-6d77b2851e80\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"431278fcc6f7763266c02451d5e8c6b681ddd24f0951f25d768679f2244321cb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7352, \"end_char_idx\": 10431, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "doc_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "ref_doc_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "chroma:document": "```python\nfrom redis.commands.search.field import TextField, VectorField\nfrom redis.commands.search.indexDefinition import IndexDefinition, IndexType\n\nschema = [ VectorField('$.vector', \n            \"FLAT\", \n            {   \"TYPE\": 'FLOAT32', \n                \"DIM\": len(doc_1['vector']), \n                \"DISTANCE_METRIC\": \"COSINE\"\n            },  as_name='vector' ),\n            TextField('$.content', as_name='content')\n        ]\nidx_def = IndexDefinition(index_type=IndexType.JSON, prefix=['doc:'])\ntry: \n    client.ft('idx').dropindex()\nexcept:\n    pass\nclient.ft('idx').create_index(schema, definition=idx_def)\n```\n\n\n\n\n    b'OK'\n\n\n\n## Load Data into Redis as JSON objects\n[Redis JSON](https://redis.io/docs/stack/json/)\n\n\n```python\nclient.json().set('doc:1', '$', doc_1)\nclient.json().set('doc:2', '$', doc_2)\nclient.json().set('doc:3', '$', doc_3)\n```\n\n\n\n\n    True\n\n\n\n# Semantic Search\nGiven a sports-related article, search Redis via Vector Similarity Search (VSS) for similar articles.  \n[KNN Search](https://redis.io/docs/stack/search/reference/vectors/#knn-search)\n\n\n```python\nfrom redis.commands.search.query import Query\nimport numpy as np\n\ntext_4 = \"\"\"Radcliffe yet to answer GB call\n\nPaula Radcliffe has been granted extra time to decide whether to compete in the World Cross-Country Championships.\n\nThe 31-year-old is concerned the event, which starts on 19 March in France, could upset her preparations for the London Marathon on 17 April. \"There is no question that Paula would be a huge asset to the GB team,\" said Zara Hyde Peters of UK Athletics. \"But she is working out whether she can accommodate the worlds without too much compromise in her marathon training.\" Radcliffe must make a decision by Tuesday - the deadline for team nominations. British team member Hayley Yelling said the team would understand if Radcliffe opted out of the event. \"It would be fantastic to have Paula in the team,\" said the European cross-country champion. \"But you have to remember that athletics is basically an individual sport and anything achieved for the team is a bonus. \"She is not messing us around. We all understand the problem.\" Radcliffe was world cross-country champion in 2001 and 2002 but missed last year's event because of injury. In her absence, the GB team won bronze in Brussels.\n\"\"\"\n\nvec = np.array(get_vector(text_4), dtype=np.float32).tobytes()\nq = Query('*=>[KNN 3 @vector $query_vec AS vector_score]')\\\n    .sort_by('vector_score')\\\n    .return_fields('vector_score', 'content')\\\n    .dialect(2)    \nparams = {\"query_vec\": vec}\n\nresults = client.ft('idx').search(q, query_params=params)\nfor doc in results.docs:\n    print(f\"distance:{round(float(doc['vector_score']),3)} content:{doc['content']}\\n\")\n\n```\n\n    distance:0.188 content:Dibaba breaks 5,000m world record\n    \n    Ethiopia's Tirunesh Dibaba set a new world record in winning the women's 5,000m at the Boston Indoor Games.\n    \n    Dibaba won in 14 minutes 32.93 seconds to erase the previous world indoor mark of 14:39.29 set by another Ethiopian, Berhane Adera, in Stuttgart last year."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: d6b59611-6bca-4882-813f-6d77b2851e80
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"d6b59611-6bca-4882-813f-6d77b2851e80\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"08e115f5-0f6b-41c0-8aa1-a201735de5bc\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"59ff14ffeee6b7e47ae59ae97b4b178f7ca83621d6168206707ca95cbea61f95\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7cf8afae-6c9b-4741-92ea-ffccdaf522d3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"838cb0ce7446c5b956afbbce3f40a0bdaa122004cc68f3fd09234decbdb42e4d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2db36b52-60f6-47ca-873a-6624a100765f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5559c858b990357d3a95ded10a53e69b4f160ff1ee6e0a09f415b29c5a2f59b6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8810, \"end_char_idx\": 11869, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "doc_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "ref_doc_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "chroma:document": "\"There is no question that Paula would be a huge asset to the GB team,\" said Zara Hyde Peters of UK Athletics. \"But she is working out whether she can accommodate the worlds without too much compromise in her marathon training.\" Radcliffe must make a decision by Tuesday - the deadline for team nominations. British team member Hayley Yelling said the team would understand if Radcliffe opted out of the event. \"It would be fantastic to have Paula in the team,\" said the European cross-country champion. \"But you have to remember that athletics is basically an individual sport and anything achieved for the team is a bonus. \"She is not messing us around. We all understand the problem.\" Radcliffe was world cross-country champion in 2001 and 2002 but missed last year's event because of injury. In her absence, the GB team won bronze in Brussels.\n\"\"\"\n\nvec = np.array(get_vector(text_4), dtype=np.float32).tobytes()\nq = Query('*=>[KNN 3 @vector $query_vec AS vector_score]')\\\n    .sort_by('vector_score')\\\n    .return_fields('vector_score', 'content')\\\n    .dialect(2)    \nparams = {\"query_vec\": vec}\n\nresults = client.ft('idx').search(q, query_params=params)\nfor doc in results.docs:\n    print(f\"distance:{round(float(doc['vector_score']),3)} content:{doc['content']}\\n\")\n\n```\n\n    distance:0.188 content:Dibaba breaks 5,000m world record\n    \n    Ethiopia's Tirunesh Dibaba set a new world record in winning the women's 5,000m at the Boston Indoor Games.\n    \n    Dibaba won in 14 minutes 32.93 seconds to erase the previous world indoor mark of 14:39.29 set by another Ethiopian, Berhane Adera, in Stuttgart last year. But compatriot Kenenisa Bekele's record hopes were dashed when he miscounted his laps in the men's 3,000m and staged his sprint finish a lap too soon. Ireland's Alistair Cragg won in 7:39.89 as Bekele battled to second in 7:41.42. \"I didn't want to sit back and get out-kicked,\" said Cragg. \"So I kept on the pace. The plan was to go with 500m to go no matter what, but when Bekele made the mistake that was it. The race was mine.\" Sweden's Carolina Kluft, the Olympic heptathlon champion, and Slovenia's Jolanda Ceplak had winning performances, too. Kluft took the long jump at 6.63m, while Ceplak easily won the women's 800m in 2:01.52. \n    \n    \n    distance:0.268 content:Japan narrowly escapes recession\n    \n    Japan's economy teetered on the brink of a technical recession in the three months to September, figures show.\n    \n    Revised figures indicated growth of just 0.1% - and a similar-sized contraction in the previous quarter. On an annual basis, the data suggests annual growth of just 0.2%, suggesting a much more hesitant recovery than had previously been thought. A common technical definition of a recession is two successive quarters of negative growth.\n    The government was keen to play down the worrying implications of the data. \"I maintain the view that Japan's economy remains in a minor adjustment phase in an upward climb, and we will monitor developments carefully,\" said economy minister Heizo Takenaka."}


