# embedding_metadata

key: _node_content
string_value:
{"id_": "dfafe846-7612-4dd1-a479-b0939958cda3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e7bb1333-0665-4aae-87f0-682a6c6ac76d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3ebf0fc03bb8cb0cda019036db1abc158d03013b99b40bda7543918043e0a2b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "72608f29-db99-4f3a-a66b-e9948823a582", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "56ced9361d6401f6762c2b895c88b3e6929d2152ee6edc7caf79fb66348093fd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9fed994e-4367-4ece-8947-c7aa7f921f29", "node_type": "1", "metadata": {}, "hash": "dcb5fdf35242faba40e9d7263ad8f6db55a2fd6b0f845c50e155f89f47c218fe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5095, "end_char_idx": 7839, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import wget

embeddings_url = "https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip"

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```

The downloaded file has to be then extracted:


```python
import zipfile
import os
import re
import tempfile

current_directory = os.getcwd()
zip_file_path = os.path.join(current_directory, "vector_database_wikipedia_articles_embedded.zip")
output_directory = os.path.join(current_directory, "../../data")

with zipfile.ZipFile(zip_file_path, "r") as zip_ref:
    zip_ref.extractall(output_directory)


# check the csv file exist
file_name = "vector_database_wikipedia_articles_embedded.csv"
data_directory = os.path.join(current_directory, "../../data")
file_path = os.path.join(data_directory, file_name)


if os.path.exists(file_path):
    print(f"The file {file_name} exists in the data directory.")
else:
    print(f"The file {file_name} does not exist in the data directory.")

```

    The file vector_database_wikipedia_articles_embedded.csv exists in the data directory.


## Load data

In this section we are going to load the data prepared previous to this session, so you don't have to recompute the embeddings of Wikipedia articles with your own credits.


```python
!unzip -n vector_database_wikipedia_articles_embedded.zip
!ls -lh vector_database_wikipedia_articles_embedded.csv
```

    Archive:  vector_database_wikipedia_articles_embedded.zip
    -rw-r--r--@ 1 geng  staff   1.7G Jan 31 01:19 vector_database_wikipedia_articles_embedded.csv


Take a look at the data.


```python
import pandas, json
data = pandas.read_csv('../../data/vector_database_wikipedia_articles_embedded.csv')
data
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>url</th>
      <th>title</th>
      <th>text</th>
      <th>title_vector</th>
      <th>content_vector</th>
      <th>vector_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>https://simple.wikipedia.org/wiki/April</td>
      <td>April</td>
      <td>April is the fourth month of the year in the J...</td>
      <td>[0.001009464613161981, -0.020700545981526375, ...</td>
      <td>[-0.011253940872848034, -0.013491976074874401,...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>https://simple.wikipedia.org/wiki/August</td>
      <td>August</td>
      <td>August (Aug.)


key: _node_content
string_value:
{"id_": "9fed994e-4367-4ece-8947-c7aa7f921f29", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e7bb1333-0665-4aae-87f0-682a6c6ac76d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3ebf0fc03bb8cb0cda019036db1abc158d03013b99b40bda7543918043e0a2b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dfafe846-7612-4dd1-a479-b0939958cda3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c71d0254c86be16a95be8d4eaa32a68f2b4ccf27caf54a336d731fdc3c6f4d58", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c4c41c69-6d07-4e35-a6e4-fd18bc492587", "node_type": "1", "metadata": {}, "hash": "011b0d32711c90c16319018a035728a3c0ca2a04dfb1efed79bd335d8a6f47c0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6673, "end_char_idx": 8870, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Take a look at the data.


```python
import pandas, json
data = pandas.read_csv('../../data/vector_database_wikipedia_articles_embedded.csv')
data
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>url</th>
      <th>title</th>
      <th>text</th>
      <th>title_vector</th>
      <th>content_vector</th>
      <th>vector_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>https://simple.wikipedia.org/wiki/April</td>
      <td>April</td>
      <td>April is the fourth month of the year in the J...</td>
      <td>[0.001009464613161981, -0.020700545981526375, ...</td>
      <td>[-0.011253940872848034, -0.013491976074874401,...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>https://simple.wikipedia.org/wiki/August</td>
      <td>August</td>
      <td>August (Aug.) is the eighth month of the year ...</td>
      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>
      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>6</td>
      <td>https://simple.wikipedia.org/wiki/Art</td>
      <td>Art</td>
      <td>Art is a creative activity that expresses imag...</td>
      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>
      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>
      <td>2</td>
    </tr>
    <tr>
      <th>3</th>
      <td>8</td>
      <td>https://simple.wikipedia.org/wiki/A</td>
      <td>A</td>
      <td>A or a is the first letter of the English alph...</td>
      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>
      <td>[0.024894846603274345, -0.022186409682035446, ...</td>
      <td>3</td>
    </tr>
    <tr>
      <th>4</th>
      <td>9</td>
      <td>https://simple.wikipedia.org/wiki/Air</td>
      <td>Air</td>
      <td>Air refers to the Earth's atmosphere.


key: _node_content
string_value:
{"id_": "c4c41c69-6d07-4e35-a6e4-fd18bc492587", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e7bb1333-0665-4aae-87f0-682a6c6ac76d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3ebf0fc03bb8cb0cda019036db1abc158d03013b99b40bda7543918043e0a2b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9fed994e-4367-4ece-8947-c7aa7f921f29", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e817952481f58ba0aaead98384419120b6411207e9b77231f49273822ff6ce2c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6e0f2a96-4840-449d-8032-327b922f5932", "node_type": "1", "metadata": {}, "hash": "470d5828e8cd532d739e910589b771a421cb39c440084aff18376f3ca8bd37bc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
is the eighth month of the year ...</td>
      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>
      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>6</td>
      <td>https://simple.wikipedia.org/wiki/Art</td>
      <td>Art</td>
      <td>Art is a creative activity that expresses imag...</td>
      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>
      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>
      <td>2</td>
    </tr>
    <tr>
      <th>3</th>
      <td>8</td>
      <td>https://simple.wikipedia.org/wiki/A</td>
      <td>A</td>
      <td>A or a is the first letter of the English alph...</td>
      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>
      <td>[0.024894846603274345, -0.022186409682035446, ...</td>
      <td>3</td>
    </tr>
    <tr>
      <th>4</th>
      <td>9</td>
      <td>https://simple.wikipedia.org/wiki/Air</td>
      <td>Air</td>
      <td>Air refers to the Earth's atmosphere. Air is a.</td>
      <td>[0.02224554680287838, -0.02044147066771984, -0.</td>
      <td>[0.021524671465158463, 0.018522677943110466, -.</td>
      <td>4</td>
    </tr>
    <tr>
      <th>.</th>
      <td>.</td>
      <td>.</td>
      <td>.</td>
      <td>.</td>
      <td>.</td>
      <td>.</td>
      <td>.</td>
    </tr>
    <tr>
      <th>24995</th>
      <td>98295</td>
      <td>https://simple.wikipedia.org/wiki/Geneva</td>
      <td>Geneva</td>
      <td>Geneva (,  ,  ,  , ) is the second biggest cit.</td>
      <td>[-0.015773078426718712, 0.01737344264984131, 0.</td>
      <td>[0.008000412955880165, 0.02008531428873539, 0.</td>
      <td>24995</td>
    </tr>
    <tr>
      <th>24996</th>
      <td>98316</td>
      <td>https://simple.wikipedia.org/wiki/Concubinage</td>
      <td>Concubinage</td>
      <td>Concubinage is the state of a woman in a relat.</td>
      <td>[-0.00519518880173564,


key: _node_content
string_value:
{"id_": "6e0f2a96-4840-449d-8032-327b922f5932", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e7bb1333-0665-4aae-87f0-682a6c6ac76d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3ebf0fc03bb8cb0cda019036db1abc158d03013b99b40bda7543918043e0a2b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c4c41c69-6d07-4e35-a6e4-fd18bc492587", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8046964a1921a60e9cdd03ea8ab33010c7555e2f9f5869909e7df0f2387120ba", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0b552788-bc1d-45ea-bd6d-9376c892d3f0", "node_type": "1", "metadata": {}, "hash": "831cab4bd3222fceba166e0d8e4406d5e7d8e5a5cd14fad71ed69eab3d0c360f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Air is a.</td>
      <td>[0.02224554680287838, -0.02044147066771984, -0.</td>
      <td>[0.021524671465158463, 0.018522677943110466, -.</td>
      <td>4</td>
    </tr>
    <tr>
      <th>.</th>
      <td>.</td>
      <td>.</td>
      <td>.</td>
      <td>.</td>
      <td>.</td>
      <td>.</td>
      <td>.</td>
    </tr>
    <tr>
      <th>24995</th>
      <td>98295</td>
      <td>https://simple.wikipedia.org/wiki/Geneva</td>
      <td>Geneva</td>
      <td>Geneva (,  ,  ,  , ) is the second biggest cit.</td>
      <td>[-0.015773078426718712, 0.01737344264984131, 0.</td>
      <td>[0.008000412955880165, 0.02008531428873539, 0.</td>
      <td>24995</td>
    </tr>
    <tr>
      <th>24996</th>
      <td>98316</td>
      <td>https://simple.wikipedia.org/wiki/Concubinage</td>
      <td>Concubinage</td>
      <td>Concubinage is the state of a woman in a relat.</td>
      <td>[-0.00519518880173564, 0.005898841191083193, 0.</td>
      <td>[-0.01736736111342907, -0.002740012714639306, .</td>
      <td>24996</td>
    </tr>
    <tr>
      <th>24997</th>
      <td>98318</td>
      <td>https://simple.wikipedia.org/wiki/Mistress%20%.</td>
      <td>Mistress (lover)</td>
      <td>A mistress is a man's long term female sexual .</td>
      <td>[-0.023164259269833565, -0.02052430994808674, .</td>
      <td>[-0.017878392711281776, -0.0004517830966506153.</td>
      <td>24997</td>
    </tr>
    <tr>
      <th>24998</th>
      <td>98326</td>
      <td>https://simple.wikipedia.org/wiki/Eastern%20Front</td>
      <td>Eastern Front</td>
      <td>Eastern Front can be one of the following:\n\n.</td>
      <td>[-0.00681863259524107, 0.002171179046854377, 8.</td>
      <td>[-0.0019235472427681088, -0.004023272544145584.</td>
      <td>24998</td>
    </tr>
    <tr>
      <th>24999</th>
      <td>98327</td>
      <td>https://simple.wikipedia.org/wiki/Italian%20Ca.


key: _node_content
string_value:
{"id_": "0b552788-bc1d-45ea-bd6d-9376c892d3f0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e7bb1333-0665-4aae-87f0-682a6c6ac76d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3ebf0fc03bb8cb0cda019036db1abc158d03013b99b40bda7543918043e0a2b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6e0f2a96-4840-449d-8032-327b922f5932", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "cca9a0e32d42e6f392e7542bb98702c4ae613ccbe6ed270dd4da796f40f77aa4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "26ec2d2a-93b7-4a84-b0e3-7406012e6487", "node_type": "1", "metadata": {}, "hash": "65f8d3530eee2759b889e843ca6fe6dc21293706f007647d80f094916678e9a9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</td>
      <td>[-0.00519518880173564, 0.005898841191083193, 0.</td>
      <td>[-0.01736736111342907, -0.002740012714639306, .</td>
      <td>24996</td>
    </tr>
    <tr>
      <th>24997</th>
      <td>98318</td>
      <td>https://simple.wikipedia.org/wiki/Mistress%20%.</td>
      <td>Mistress (lover)</td>
      <td>A mistress is a man's long term female sexual .</td>
      <td>[-0.023164259269833565, -0.02052430994808674, .</td>
      <td>[-0.017878392711281776, -0.0004517830966506153.</td>
      <td>24997</td>
    </tr>
    <tr>
      <th>24998</th>
      <td>98326</td>
      <td>https://simple.wikipedia.org/wiki/Eastern%20Front</td>
      <td>Eastern Front</td>
      <td>Eastern Front can be one of the following:\n\n.</td>
      <td>[-0.00681863259524107, 0.002171179046854377, 8.</td>
      <td>[-0.0019235472427681088, -0.004023272544145584.</td>
      <td>24998</td>
    </tr>
    <tr>
      <th>24999</th>
      <td>98327</td>
      <td>https://simple.wikipedia.org/wiki/Italian%20Ca.</td>
      <td>Italian Campaign</td>
      <td>Italian Campaign can mean the following:\n\nTh.</td>
      <td>[-0.014151256531476974, -0.008553029969334602,</td>
      <td>[-0.011758845299482346, -0.01346028596162796, .</td>
      <td>24999</td>
    </tr>
  </tbody>
</table>
<p>25000 rows × 7 columns</p>
</div>




```python
title_vector_length = len(json.loads(data['title_vector'].iloc[0]))
content_vector_length = len(json.loads(data['content_vector'].iloc[0]))

print(title_vector_length, content_vector_length)
```

    1536 1536


### Create table and proxima vector index

Hologres stores data in __tables__ where each object is described by at least one vector. Our table will be called **articles** and each object will be described by both **title** and **content** vectors.

We will start with creating a table and create proxima indexes on both **title** and **content**, and then we will fill it with our precomputed embeddings.


key: _node_content
string_value:
{"id_": "26ec2d2a-93b7-4a84-b0e3-7406012e6487", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e7bb1333-0665-4aae-87f0-682a6c6ac76d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3ebf0fc03bb8cb0cda019036db1abc158d03013b99b40bda7543918043e0a2b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0b552788-bc1d-45ea-bd6d-9376c892d3f0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8136a08c6de5af07a28f349d74d6b6ed726a76d4821ebba3d613d14e9ab86843", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0cbf3664-c28b-458d-8e7a-23c1f743da20", "node_type": "1", "metadata": {}, "hash": "da80087ba7bbac796d5e187be303cea52e7f10533f41f21c528b5e23684826eb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</td>
      <td>[-0.00681863259524107, 0.002171179046854377, 8.</td>
      <td>[-0.0019235472427681088, -0.004023272544145584.</td>
      <td>24998</td>
    </tr>
    <tr>
      <th>24999</th>
      <td>98327</td>
      <td>https://simple.wikipedia.org/wiki/Italian%20Ca.</td>
      <td>Italian Campaign</td>
      <td>Italian Campaign can mean the following:\n\nTh.</td>
      <td>[-0.014151256531476974, -0.008553029969334602,</td>
      <td>[-0.011758845299482346, -0.01346028596162796, .</td>
      <td>24999</td>
    </tr>
  </tbody>
</table>
<p>25000 rows × 7 columns</p>
</div>




```python
title_vector_length = len(json.loads(data['title_vector'].iloc[0]))
content_vector_length = len(json.loads(data['content_vector'].iloc[0]))

print(title_vector_length, content_vector_length)
```

    1536 1536


### Create table and proxima vector index

Hologres stores data in __tables__ where each object is described by at least one vector. Our table will be called **articles** and each object will be described by both **title** and **content** vectors.

We will start with creating a table and create proxima indexes on both **title** and **content**, and then we will fill it with our precomputed embeddings.


```python
cursor.execute('CREATE EXTENSION IF NOT EXISTS proxima;')
create_proxima_table_sql = '''
BEGIN;
DROP TABLE IF EXISTS articles;
CREATE TABLE articles (
    id INT PRIMARY KEY NOT NULL,
    url TEXT,
    title TEXT,
    content TEXT,
    title_vector float4[] check(
        array_ndims(title_vector) = 1 and 
        array_length(title_vector, 1) = 1536
    ), -- define the vectors
    content_vector float4[] check(
        array_ndims(content_vector) = 1 and 
        array_length(content_vector, 1) = 1536
    ),
    vector_id INT
);

-- Create indexes for the vector fields.
call set_table_property(
    'articles',
    'proxima_vectors', 
    '{
        "title_vector":{"algorithm":"Graph","distance_method":"Euclidean","builder_params":{"min_flush_proxima_row_count" : 10}},
        "content_vector":{"algorithm":"Graph","distance_method":"Euclidean","builder_params":{"min_flush_proxima_row_count" : 10}}
    }'
);  

COMMIT;
'''

# Execute the SQL statements (will autocommit)
cursor.execute(create_proxima_table_sql)
```

### Upload data

Now let's upload the data to the Hologres cloud instance using [COPY statement](https://www.alibabacloud.com/help/en/hologres/latest/use-the-copy-statement-to-import-or-export-data). This might take 5-10 minutes according to the network bandwidth.


key: _node_content
string_value:
{"id_": "0cbf3664-c28b-458d-8e7a-23c1f743da20", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e7bb1333-0665-4aae-87f0-682a6c6ac76d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3ebf0fc03bb8cb0cda019036db1abc158d03013b99b40bda7543918043e0a2b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "26ec2d2a-93b7-4a84-b0e3-7406012e6487", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "704bc636bb0675ec790f79e2379c3d3a87bb98585baf0d81d8378bee30f97e93", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "24d5b3b1-b27d-4c49-820d-e9ab9585424b", "node_type": "1", "metadata": {}, "hash": "5fcfaaf6c49a35c048fdc4153f39b4b43acec78e7fe85381fc11b94bc2da053f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11745, "end_char_idx": 14763, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
cursor.execute('CREATE EXTENSION IF NOT EXISTS proxima;')
create_proxima_table_sql = '''
BEGIN;
DROP TABLE IF EXISTS articles;
CREATE TABLE articles (
    id INT PRIMARY KEY NOT NULL,
    url TEXT,
    title TEXT,
    content TEXT,
    title_vector float4[] check(
        array_ndims(title_vector) = 1 and 
        array_length(title_vector, 1) = 1536
    ), -- define the vectors
    content_vector float4[] check(
        array_ndims(content_vector) = 1 and 
        array_length(content_vector, 1) = 1536
    ),
    vector_id INT
);

-- Create indexes for the vector fields.
call set_table_property(
    'articles',
    'proxima_vectors', 
    '{
        "title_vector":{"algorithm":"Graph","distance_method":"Euclidean","builder_params":{"min_flush_proxima_row_count" : 10}},
        "content_vector":{"algorithm":"Graph","distance_method":"Euclidean","builder_params":{"min_flush_proxima_row_count" : 10}}
    }'
);  

COMMIT;
'''

# Execute the SQL statements (will autocommit)
cursor.execute(create_proxima_table_sql)
```

### Upload data

Now let's upload the data to the Hologres cloud instance using [COPY statement](https://www.alibabacloud.com/help/en/hologres/latest/use-the-copy-statement-to-import-or-export-data). This might take 5-10 minutes according to the network bandwidth.


```python
import io

# Path to the unzipped CSV file
csv_file_path = '../../data/vector_database_wikipedia_articles_embedded.csv'

# In SQL, arrays are surrounded by {}, rather than []
def process_file(file_path):
    with open(file_path, 'r') as file:
        for line in file:
            # Replace '[' with '{' and ']' with '}'
            modified_line = line.replace('[', '{').replace(']', '}')
            yield modified_line

# Create a StringIO object to store the modified lines
modified_lines = io.StringIO(''.join(list(process_file(csv_file_path))))

# Create the COPY command for the copy_expert method
copy_command = '''
COPY public.articles (id, url, title, content, title_vector, content_vector, vector_id)
FROM STDIN WITH (FORMAT CSV, HEADER true, DELIMITER ',');
'''

# Execute the COPY command using the copy_expert method
cursor.copy_expert(copy_command, modified_lines)
```

The proxima index will be built in the background. We can do searching during this period but the query will be slow without the vector index. Use this command to wait for finish building the index.


```python
cursor.execute('vacuum articles;')
```


```python
# Check the collection size to make sure all the points have been stored
count_sql = "select count(*) from articles;"
cursor.execute(count_sql)
result = cursor.fetchone()
print(f"Count:{result[0]}")

```

    Count:25000


## Search data

Once the data is uploaded we will start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search. Since the precomputed embeddings were created with `text-embedding-3-small` OpenAI model we also have to use it during search.


key: _node_content
string_value:
{"id_": "24d5b3b1-b27d-4c49-820d-e9ab9585424b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e7bb1333-0665-4aae-87f0-682a6c6ac76d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3ebf0fc03bb8cb0cda019036db1abc158d03013b99b40bda7543918043e0a2b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0cbf3664-c28b-458d-8e7a-23c1f743da20", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "cf16b0e2c9b0bef2b2418a91f4b90bce4a8ba313525ca7183f45bd051e9f6793", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c5b3a3fb-4640-4b05-b0c0-ce63f1630c53", "node_type": "1", "metadata": {}, "hash": "fce9e138c89316a953679eff10b09108029dd886ae9487c5920506bd7caab72a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13053, "end_char_idx": 15525, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import io

# Path to the unzipped CSV file
csv_file_path = '../../data/vector_database_wikipedia_articles_embedded.csv'

# In SQL, arrays are surrounded by {}, rather than []
def process_file(file_path):
    with open(file_path, 'r') as file:
        for line in file:
            # Replace '[' with '{' and ']' with '}'
            modified_line = line.replace('[', '{').replace(']', '}')
            yield modified_line

# Create a StringIO object to store the modified lines
modified_lines = io.StringIO(''.join(list(process_file(csv_file_path))))

# Create the COPY command for the copy_expert method
copy_command = '''
COPY public.articles (id, url, title, content, title_vector, content_vector, vector_id)
FROM STDIN WITH (FORMAT CSV, HEADER true, DELIMITER ',');
'''

# Execute the COPY command using the copy_expert method
cursor.copy_expert(copy_command, modified_lines)
```

The proxima index will be built in the background. We can do searching during this period but the query will be slow without the vector index. Use this command to wait for finish building the index.


```python
cursor.execute('vacuum articles;')
```


```python
# Check the collection size to make sure all the points have been stored
count_sql = "select count(*) from articles;"
cursor.execute(count_sql)
result = cursor.fetchone()
print(f"Count:{result[0]}")

```

    Count:25000


## Search data

Once the data is uploaded we will start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search. Since the precomputed embeddings were created with `text-embedding-3-small` OpenAI model we also have to use it during search.



```python
import openai
def query_knn(query, table_name, vector_name="title_vector", top_k=20):

    # Creates embedding vector from user query
    embedded_query = openai.Embedding.create(
        input=query,
        model="text-embedding-3-small",
    )["data"][0]["embedding"]

    # Convert the embedded_query to PostgreSQL compatible format
    embedded_query_pg = "{" + ",".join(map(str, embedded_query)) + "}"

    # Create SQL query
    query_sql = f"""
    SELECT id, url, title, pm_approx_euclidean_distance({vector_name},'{embedded_query_pg}'::float4[]) AS distance
    FROM {table_name}
    ORDER BY distance
    LIMIT {top_k};
    """
    # Execute the query
    cursor.execute(query_sql)
    results = cursor.fetchall()

    return results
```


key: _node_content
string_value:
{"id_": "c5b3a3fb-4640-4b05-b0c0-ce63f1630c53", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e7bb1333-0665-4aae-87f0-682a6c6ac76d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3ebf0fc03bb8cb0cda019036db1abc158d03013b99b40bda7543918043e0a2b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "24d5b3b1-b27d-4c49-820d-e9ab9585424b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "50b95cd9d07101d70376392814db3ec6481dbea47789abb9c39b02024ee9be1f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "56f771c8-220a-42ae-913a-4acc80545fe0", "node_type": "1", "metadata": {}, "hash": "87ae9784febfe831317d44fdaf3b2e3b385bae5e5cb7b94ebe92c364ede354e0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14149, "end_char_idx": 16421, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
cursor.execute('vacuum articles;')
```


```python
# Check the collection size to make sure all the points have been stored
count_sql = "select count(*) from articles;"
cursor.execute(count_sql)
result = cursor.fetchone()
print(f"Count:{result[0]}")

```

    Count:25000


## Search data

Once the data is uploaded we will start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search. Since the precomputed embeddings were created with `text-embedding-3-small` OpenAI model we also have to use it during search.



```python
import openai
def query_knn(query, table_name, vector_name="title_vector", top_k=20):

    # Creates embedding vector from user query
    embedded_query = openai.Embedding.create(
        input=query,
        model="text-embedding-3-small",
    )["data"][0]["embedding"]

    # Convert the embedded_query to PostgreSQL compatible format
    embedded_query_pg = "{" + ",".join(map(str, embedded_query)) + "}"

    # Create SQL query
    query_sql = f"""
    SELECT id, url, title, pm_approx_euclidean_distance({vector_name},'{embedded_query_pg}'::float4[]) AS distance
    FROM {table_name}
    ORDER BY distance
    LIMIT {top_k};
    """
    # Execute the query
    cursor.execute(query_sql)
    results = cursor.fetchall()

    return results
```


```python
query_results = query_knn("modern art in Europe", "Articles")
for i, result in enumerate(query_results):
    print(f"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})")
```

    1. Museum of Modern Art (Score: 0.501)
    2. Western Europe (Score: 0.485)
    3. Renaissance art (Score: 0.479)
    4. Pop art (Score: 0.472)
    5. Northern Europe (Score: 0.461)
    6. Hellenistic art (Score: 0.458)
    7. Modernist literature (Score: 0.447)
    8. Art film (Score: 0.44)
    9. Central Europe (Score: 0.439)
    10. Art (Score: 0.437)
    11. European (Score: 0.437)
    12. Byzantine art (Score: 0.436)
    13. Postmodernism (Score: 0.435)
    14. Eastern Europe (Score: 0.433)
    15. Cubism (Score: 0.433)
    16. Europe (Score: 0.432)
    17. Impressionism (Score: 0.432)
    18. Bauhaus (Score: 0.431)
    19. Surrealism (Score: 0.429)
    20. Expressionism (Score: 0.429)


key: _node_content
string_value:
{"id_": "56f771c8-220a-42ae-913a-4acc80545fe0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e7bb1333-0665-4aae-87f0-682a6c6ac76d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3ebf0fc03bb8cb0cda019036db1abc158d03013b99b40bda7543918043e0a2b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c5b3a3fb-4640-4b05-b0c0-ce63f1630c53", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/hologres/Getting_started_with_Hologres_and_OpenAI.ipynb", "title": "Using Hologres as a vector database for OpenAI embeddings", "tokens": 4831, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3764d46578c48a12652a3f00210390d5f0082618a945b2c6a52da049ea388d96", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15528, "end_char_idx": 17388, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
query_results = query_knn("modern art in Europe", "Articles")
for i, result in enumerate(query_results):
    print(f"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})")
```

    1. Museum of Modern Art (Score: 0.501)
    2. Western Europe (Score: 0.485)
    3. Renaissance art (Score: 0.479)
    4. Pop art (Score: 0.472)
    5. Northern Europe (Score: 0.461)
    6. Hellenistic art (Score: 0.458)
    7. Modernist literature (Score: 0.447)
    8. Art film (Score: 0.44)
    9. Central Europe (Score: 0.439)
    10. Art (Score: 0.437)
    11. European (Score: 0.437)
    12. Byzantine art (Score: 0.436)
    13. Postmodernism (Score: 0.435)
    14. Eastern Europe (Score: 0.433)
    15. Cubism (Score: 0.433)
    16. Europe (Score: 0.432)
    17. Impressionism (Score: 0.432)
    18. Bauhaus (Score: 0.431)
    19. Surrealism (Score: 0.429)
    20. Expressionism (Score: 0.429)



```python
# This time we'll query using content vector
query_results = query_knn("Famous battles in Scottish history", "Articles", "content_vector")
for i, result in enumerate(query_results):
    print(f"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})")
```

    1. Battle of Bannockburn (Score: 0.489)
    2. Wars of Scottish Independence (Score: 0.474)
    3. 1651 (Score: 0.457)
    4. First War of Scottish Independence (Score: 0.452)
    5. Robert I of Scotland (Score: 0.445)
    6. 841 (Score: 0.441)
    7. 1716 (Score: 0.441)
    8. 1314 (Score: 0.429)
    9. 1263 (Score: 0.428)
    10. William Wallace (Score: 0.426)
    11. Stirling (Score: 0.419)
    12. 1306 (Score: 0.419)
    13. 1746 (Score: 0.418)
    14. 1040s (Score: 0.414)
    15. 1106 (Score: 0.412)
    16. 1304 (Score: 0.411)
    17. David II of Scotland (Score: 0.408)
    18. Braveheart (Score: 0.407)
    19. 1124 (Score: 0.406)
    20. July 27 (Score: 0.405)



```python

```


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb


key: _node_content
string_value:
{"id_": "e9da145e-35ac-4006-882a-a56e6701ddb4", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6d313901-fd23-4dea-86d9-d4baaedbd3fc", "node_type": "1", "metadata": {}, "hash": "da3a6cc971976443e5042af317c47553534dcf064581159176f061bf697d0812", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3150, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Using Chroma for Embeddings Search

This notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.

### What is a Vector Database

A vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.

### Why use a Vector Database

Vector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.


### Demo Flow
The demo flow is:
- **Setup**: Import packages and set any required variables
- **Load data**: Load a dataset and embed it using OpenAI embeddings
- **Chroma**:
    - *Setup*: Here we'll set up the Python client for Chroma. For more details go [here](https://docs.trychroma.com/usage-guide)
    - *Index Data*: We'll create collections with vectors for __titles__ and __content__
    - *Search Data*: We'll run a few searches to confirm it works

Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.

## Setup

Import the required libraries and set the embedding model that we'd like to use.


```python
# Make sure the OpenAI library is installed
%pip install openai

# We'll need to install the Chroma client
%pip install chromadb

# Install wget to pull zip file
%pip install wget

# Install numpy for data manipulation
%pip install numpy
```

    Collecting openai
      Obtaining dependency information for openai from https://files.pythonhosted.org/packages/67/78/7588a047e458cb8075a4089d721d7af5e143ff85a2388d4a28c530be0494/openai-0.27.8-py3-none-any.whl.metadata
      Downloading openai-0.27.8-py3-none-any.whl.metadata (13 kB)
    Collecting requests>=2.20 (from openai)
      Obtaining dependency information for requests>=2.20 from https://files.pythonhosted.org/packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata
      Using cached requests-2.31.0-py3-none-any.whl.metadata (4.


key: _node_content
string_value:
{"id_": "6d313901-fd23-4dea-86d9-d4baaedbd3fc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e9da145e-35ac-4006-882a-a56e6701ddb4", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "712d163f1c2c1cb788a63e3fcd3e1b2fedba0d41cdc9668c9503892a6a6ec121", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "193814f8-4e9b-498a-9a62-7f26d43aaa8a", "node_type": "1", "metadata": {}, "hash": "494e907ac46e0d980cfdd6b423c46a7b1e30acae9fd28fc6e9a1338e69d45b32", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2287, "end_char_idx": 4380, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Make sure the OpenAI library is installed
%pip install openai

# We'll need to install the Chroma client
%pip install chromadb

# Install wget to pull zip file
%pip install wget

# Install numpy for data manipulation
%pip install numpy
```

    Collecting openai
      Obtaining dependency information for openai from https://files.pythonhosted.org/packages/67/78/7588a047e458cb8075a4089d721d7af5e143ff85a2388d4a28c530be0494/openai-0.27.8-py3-none-any.whl.metadata
      Downloading openai-0.27.8-py3-none-any.whl.metadata (13 kB)
    Collecting requests>=2.20 (from openai)
      Obtaining dependency information for requests>=2.20 from https://files.pythonhosted.org/packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl.metadata
      Using cached requests-2.31.0-py3-none-any.whl.metadata (4.6 kB)
    Collecting tqdm (from openai)
      Using cached tqdm-4.65.0-py3-none-any.whl (77 kB)
    Collecting aiohttp (from openai)
      Obtaining dependency information for aiohttp from https://files.pythonhosted.org/packages/fa/9e/49002fde2a97d7df0e162e919c31cf13aa9f184537739743d1239edd0e67/aiohttp-3.8.5-cp310-cp310-macosx_11_0_arm64.whl.metadata
      Downloading aiohttp-3.8.5-cp310-cp310-macosx_11_0_arm64.whl.metadata (7.7 kB)
    Collecting charset-normalizer<4,>=2 (from requests>=2.20->openai)
      Obtaining dependency information for charset-normalizer<4,>=2 from https://files.pythonhosted.org/packages/ec/a7/96835706283d63fefbbbb4f119d52f195af00fc747e67cc54397c56312c8/charset_normalizer-3.2.0-cp310-cp310-macosx_11_0_arm64.whl.metadata
      Using cached charset_normalizer-3.2.0-cp310-cp310-macosx_11_0_arm64.whl.metadata (31 kB)
    Collecting idna<4,>=2.5 (from requests>=2.20->openai)
      Using cached idna-3.4-py3-none-any.whl (61 kB)
    Collecting urllib3<3,>=1.21.1 (from requests>=2.20->openai)
      Obtaining dependency information for urllib3<3,>=1.21.1 from https://files.pythonhosted.org/packages/9b/81/62fd61001fa4b9d0df6e31d47ff49cfa9de4af03adecf339c7bc30656b37/urllib3-2.0.4-py3-none-any.whl.


key: _node_content
string_value:
{"id_": "193814f8-4e9b-498a-9a62-7f26d43aaa8a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6d313901-fd23-4dea-86d9-d4baaedbd3fc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "fe463b61c44a8eb3af0bd04df171fcbf862e4032eaaf99ad7737b516c255ce93", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8fedb0d2-c2da-4e24-95b1-94d2b70bb073", "node_type": "1", "metadata": {}, "hash": "3a85adff5038cd5128ba3a32660df4e898822687a4e911a8aa7b363160bb8c35", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3456, "end_char_idx": 5404, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
8.5-cp310-cp310-macosx_11_0_arm64.whl.metadata
      Downloading aiohttp-3.8.5-cp310-cp310-macosx_11_0_arm64.whl.metadata (7.7 kB)
    Collecting charset-normalizer<4,>=2 (from requests>=2.20->openai)
      Obtaining dependency information for charset-normalizer<4,>=2 from https://files.pythonhosted.org/packages/ec/a7/96835706283d63fefbbbb4f119d52f195af00fc747e67cc54397c56312c8/charset_normalizer-3.2.0-cp310-cp310-macosx_11_0_arm64.whl.metadata
      Using cached charset_normalizer-3.2.0-cp310-cp310-macosx_11_0_arm64.whl.metadata (31 kB)
    Collecting idna<4,>=2.5 (from requests>=2.20->openai)
      Using cached idna-3.4-py3-none-any.whl (61 kB)
    Collecting urllib3<3,>=1.21.1 (from requests>=2.20->openai)
      Obtaining dependency information for urllib3<3,>=1.21.1 from https://files.pythonhosted.org/packages/9b/81/62fd61001fa4b9d0df6e31d47ff49cfa9de4af03adecf339c7bc30656b37/urllib3-2.0.4-py3-none-any.whl.metadata
      Downloading urllib3-2.0.4-py3-none-any.whl.metadata (6.6 kB)
    Collecting certifi>=2017.4.17 (from requests>=2.20->openai)
      Using cached certifi-2023.5.7-py3-none-any.whl (156 kB)
    Collecting attrs>=17.3.0 (from aiohttp->openai)
      Using cached attrs-23.1.0-py3-none-any.whl (61 kB)
    Collecting multidict<7.0,>=4.5 (from aiohttp->openai)
      Using cached multidict-6.0.4-cp310-cp310-macosx_11_0_arm64.whl (29 kB)
    Collecting async-timeout<5.0,>=4.0.0a3 (from aiohttp->openai)
      Using cached async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
    Collecting yarl<2.0,>=1.0 (from aiohttp->openai)
      Using cached yarl-1.9.2-cp310-cp310-macosx_11_0_arm64.whl (62 kB)
    Collecting frozenlist>=1.1.1 (from aiohttp->openai)
      Obtaining dependency information for frozenlist>=1.1.1 from https://files.pythonhosted.org/packages/67/6a/55a49da0fa373ac9aa49ccd5b6393ecc183e2a0904d9449ea3ee1163e0b1/frozenlist-1.4.0-cp310-cp310-macosx_11_0_arm64.whl.metadata
      Downloading frozenlist-1.4.


key: _node_content
string_value:
{"id_": "8fedb0d2-c2da-4e24-95b1-94d2b70bb073", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "193814f8-4e9b-498a-9a62-7f26d43aaa8a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "5bf7250ac75635842d8521cb0812470a1faef05758e6c5f8c6ce97d961b67946", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d2ad719a-d392-441a-976e-30de447fc0a3", "node_type": "1", "metadata": {}, "hash": "3a8293023104a5c65c389a14cc7d2c3875dfc149c27663b687af99e7c89d2e3e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4417, "end_char_idx": 6142, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0.4-py3-none-any.whl.metadata (6.6 kB)
    Collecting certifi>=2017.4.17 (from requests>=2.20->openai)
      Using cached certifi-2023.5.7-py3-none-any.whl (156 kB)
    Collecting attrs>=17.3.0 (from aiohttp->openai)
      Using cached attrs-23.1.0-py3-none-any.whl (61 kB)
    Collecting multidict<7.0,>=4.5 (from aiohttp->openai)
      Using cached multidict-6.0.4-cp310-cp310-macosx_11_0_arm64.whl (29 kB)
    Collecting async-timeout<5.0,>=4.0.0a3 (from aiohttp->openai)
      Using cached async_timeout-4.0.2-py3-none-any.whl (5.8 kB)
    Collecting yarl<2.0,>=1.0 (from aiohttp->openai)
      Using cached yarl-1.9.2-cp310-cp310-macosx_11_0_arm64.whl (62 kB)
    Collecting frozenlist>=1.1.1 (from aiohttp->openai)
      Obtaining dependency information for frozenlist>=1.1.1 from https://files.pythonhosted.org/packages/67/6a/55a49da0fa373ac9aa49ccd5b6393ecc183e2a0904d9449ea3ee1163e0b1/frozenlist-1.4.0-cp310-cp310-macosx_11_0_arm64.whl.metadata
      Downloading frozenlist-1.4.0-cp310-cp310-macosx_11_0_arm64.whl.metadata (5.2 kB)
    Collecting aiosignal>=1.1.2 (from aiohttp->openai)
      Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
    Using cached openai-0.27.8-py3-none-any.whl (73 kB)
    Using cached requests-2.31.0-py3-none-any.whl (62 kB)
    Downloading aiohttp-3.8.5-cp310-cp310-macosx_11_0_arm64.whl (343 kB)
    [2K   [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m343.9/343.9 kB[0m [31m11.4 MB/s[0m eta [36m0:00:00[0m
    [?25hUsing cached charset_normalizer-3.2.0-cp310-cp310-macosx_11_0_arm64.whl (124 kB)
    Downloading frozenlist-1.4.0-cp310-cp310-macosx_11_0_arm64.whl (46 kB)
    [2K   [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m46.0/46.0 kB[0m [31m4.


key: _node_content
string_value:
{"id_": "d2ad719a-d392-441a-976e-30de447fc0a3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8fedb0d2-c2da-4e24-95b1-94d2b70bb073", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "3183224ab57868e1f7b74f168f8714b045de535cfbeaec8894902ecaff40cc78", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "81b46c6a-3e52-4fc5-8909-e4d3a50c9061", "node_type": "1", "metadata": {}, "hash": "cbbb86efe862822f9bafe2078da11a3fb6af7839abee83bb576b461a70874b54", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5324, "end_char_idx": 7067, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
4.0-cp310-cp310-macosx_11_0_arm64.whl.metadata
      Downloading frozenlist-1.4.0-cp310-cp310-macosx_11_0_arm64.whl.metadata (5.2 kB)
    Collecting aiosignal>=1.1.2 (from aiohttp->openai)
      Using cached aiosignal-1.3.1-py3-none-any.whl (7.6 kB)
    Using cached openai-0.27.8-py3-none-any.whl (73 kB)
    Using cached requests-2.31.0-py3-none-any.whl (62 kB)
    Downloading aiohttp-3.8.5-cp310-cp310-macosx_11_0_arm64.whl (343 kB)
    [2K   [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m343.9/343.9 kB[0m [31m11.4 MB/s[0m eta [36m0:00:00[0m
    [?25hUsing cached charset_normalizer-3.2.0-cp310-cp310-macosx_11_0_arm64.whl (124 kB)
    Downloading frozenlist-1.4.0-cp310-cp310-macosx_11_0_arm64.whl (46 kB)
    [2K   [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m46.0/46.0 kB[0m [31m4.4 MB/s[0m eta [36m0:00:00[0m
    [?25hDownloading urllib3-2.0.4-py3-none-any.whl (123 kB)
    [2K   [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m123.9/123.9 kB[0m [31m20.0 MB/s[0m eta [36m0:00:00[0m
    [?25hInstalling collected packages: urllib3, tqdm, multidict, idna, frozenlist, charset-normalizer, certifi, attrs, async-timeout, yarl, requests, aiosignal, aiohttp, openai
    Successfully installed aiohttp-3.8.5 aiosignal-1.3.1 async-timeout-4.0.2 attrs-23.1.0 certifi-2023.5.7 charset-normalizer-3.2.0 frozenlist-1.4.0 idna-3.4 multidict-6.0.4 openai-0.27.8 requests-2.31.0 tqdm-4.65.0 urllib3-2.0.4 yarl-1.9.2
    Note: you may need to restart the kernel to use updated packages.
    Collecting chromadb
      Obtaining dependency information for chromadb from https://files.pythonhosted.org/packages/47/b7/41d975f02818c965cdb8a119cab5a38cfb08e0c1abb18efebe9a373ea97b/chromadb-0.4.2-py3-none-any.


key: _node_content
string_value:
{"id_": "81b46c6a-3e52-4fc5-8909-e4d3a50c9061", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d2ad719a-d392-441a-976e-30de447fc0a3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "1bb8f0b915b789dc47ed17f5aca6414674ff23d44854e36f8d8c4b1c92678faf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a998d9ad-4603-42be-bb58-475d196a10e2", "node_type": "1", "metadata": {}, "hash": "f0bff1d46574a874bc313737bc10eea73393130c9a01b4ed313b415ebcd929ea", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6121, "end_char_idx": 8015, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0/46.0 kB[0m [31m4.4 MB/s[0m eta [36m0:00:00[0m
    [?25hDownloading urllib3-2.0.4-py3-none-any.whl (123 kB)
    [2K   [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m123.9/123.9 kB[0m [31m20.0 MB/s[0m eta [36m0:00:00[0m
    [?25hInstalling collected packages: urllib3, tqdm, multidict, idna, frozenlist, charset-normalizer, certifi, attrs, async-timeout, yarl, requests, aiosignal, aiohttp, openai
    Successfully installed aiohttp-3.8.5 aiosignal-1.3.1 async-timeout-4.0.2 attrs-23.1.0 certifi-2023.5.7 charset-normalizer-3.2.0 frozenlist-1.4.0 idna-3.4 multidict-6.0.4 openai-0.27.8 requests-2.31.0 tqdm-4.65.0 urllib3-2.0.4 yarl-1.9.2
    Note: you may need to restart the kernel to use updated packages.
    Collecting chromadb
      Obtaining dependency information for chromadb from https://files.pythonhosted.org/packages/47/b7/41d975f02818c965cdb8a119cab5a38cfb08e0c1abb18efebe9a373ea97b/chromadb-0.4.2-py3-none-any.whl.metadata
      Downloading chromadb-0.4.2-py3-none-any.whl.metadata (6.9 kB)
    Collecting pandas>=1.3 (from chromadb)
      Obtaining dependency information for pandas>=1.3 from https://files.pythonhosted.org/packages/4a/f6/f620ca62365d83e663a255a41b08d2fc2eaf304e0b8b21bb6d62a7390fe3/pandas-2.0.3-cp310-cp310-macosx_11_0_arm64.whl.metadata
      Using cached pandas-2.0.3-cp310-cp310-macosx_11_0_arm64.whl.metadata (18 kB)
    Requirement already satisfied: requests>=2.28 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from chromadb) (2.31.0)
    Collecting pydantic<2.0,>=1.9 (from chromadb)
      Obtaining dependency information for pydantic<2.0,>=1.9 from https://files.pythonhosted.org/packages/79/3e/6b4d0fb2174beceac9a991ba8e67158b45c35faca9ea4545ae32d47096cd/pydantic-1.10.11-cp310-cp310-macosx_11_0_arm64.whl.metadata
      Using cached pydantic-1.10.11-cp310-cp310-macosx_11_0_arm64.


key: _node_content
string_value:
{"id_": "a998d9ad-4603-42be-bb58-475d196a10e2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "81b46c6a-3e52-4fc5-8909-e4d3a50c9061", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "57367f8bcb998b4bdec07caa2bb404c03a5fe0a707eee56eb12240cfa59a8a4b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c9e06df6-a8bb-4afc-b175-deec95f34554", "node_type": "1", "metadata": {}, "hash": "a747c43aaf3aa919f95211780385a6dba693592494771bcde4005c8b729b5b67", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7050, "end_char_idx": 8929, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
4.2-py3-none-any.whl.metadata
      Downloading chromadb-0.4.2-py3-none-any.whl.metadata (6.9 kB)
    Collecting pandas>=1.3 (from chromadb)
      Obtaining dependency information for pandas>=1.3 from https://files.pythonhosted.org/packages/4a/f6/f620ca62365d83e663a255a41b08d2fc2eaf304e0b8b21bb6d62a7390fe3/pandas-2.0.3-cp310-cp310-macosx_11_0_arm64.whl.metadata
      Using cached pandas-2.0.3-cp310-cp310-macosx_11_0_arm64.whl.metadata (18 kB)
    Requirement already satisfied: requests>=2.28 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from chromadb) (2.31.0)
    Collecting pydantic<2.0,>=1.9 (from chromadb)
      Obtaining dependency information for pydantic<2.0,>=1.9 from https://files.pythonhosted.org/packages/79/3e/6b4d0fb2174beceac9a991ba8e67158b45c35faca9ea4545ae32d47096cd/pydantic-1.10.11-cp310-cp310-macosx_11_0_arm64.whl.metadata
      Using cached pydantic-1.10.11-cp310-cp310-macosx_11_0_arm64.whl.metadata (148 kB)
    Collecting chroma-hnswlib==0.7.1 (from chromadb)
      Obtaining dependency information for chroma-hnswlib==0.7.1 from https://files.pythonhosted.org/packages/a5/d5/54947127f5cb2a1fcef40877fb3e6044495eec0a158ba0956babe4ab2a77/chroma_hnswlib-0.7.1-cp310-cp310-macosx_13_0_arm64.whl.metadata
      Using cached chroma_hnswlib-0.7.1-cp310-cp310-macosx_13_0_arm64.whl.metadata (252 bytes)
    Collecting fastapi<0.100.0,>=0.95.2 (from chromadb)
      Obtaining dependency information for fastapi<0.100.0,>=0.95.2 from https://files.pythonhosted.org/packages/73/eb/03b691afa0b5ffa1e93ed34f97ec1e7855c758efbdcfb16c209af0b0506b/fastapi-0.99.1-py3-none-any.whl.metadata
      Using cached fastapi-0.99.1-py3-none-any.whl.metadata (23 kB)
    Collecting uvicorn[standard]>=0.18.3 (from chromadb)
      Obtaining dependency information for uvicorn[standard]>=0.18.3 from https://files.pythonhosted.


key: _node_content
string_value:
{"id_": "c9e06df6-a8bb-4afc-b175-deec95f34554", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a998d9ad-4603-42be-bb58-475d196a10e2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "21722b67ffa496f9d8e8a8876a692e0cc74e02df264be26c0797895ab765d41e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3da3c845-2918-4287-90b7-79f5335bba1c", "node_type": "1", "metadata": {}, "hash": "63a74415aa0316047bba32682a9b6ff1252e608c967c6a44c612e413f620213f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7903, "end_char_idx": 9906, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
11-cp310-cp310-macosx_11_0_arm64.whl.metadata
      Using cached pydantic-1.10.11-cp310-cp310-macosx_11_0_arm64.whl.metadata (148 kB)
    Collecting chroma-hnswlib==0.7.1 (from chromadb)
      Obtaining dependency information for chroma-hnswlib==0.7.1 from https://files.pythonhosted.org/packages/a5/d5/54947127f5cb2a1fcef40877fb3e6044495eec0a158ba0956babe4ab2a77/chroma_hnswlib-0.7.1-cp310-cp310-macosx_13_0_arm64.whl.metadata
      Using cached chroma_hnswlib-0.7.1-cp310-cp310-macosx_13_0_arm64.whl.metadata (252 bytes)
    Collecting fastapi<0.100.0,>=0.95.2 (from chromadb)
      Obtaining dependency information for fastapi<0.100.0,>=0.95.2 from https://files.pythonhosted.org/packages/73/eb/03b691afa0b5ffa1e93ed34f97ec1e7855c758efbdcfb16c209af0b0506b/fastapi-0.99.1-py3-none-any.whl.metadata
      Using cached fastapi-0.99.1-py3-none-any.whl.metadata (23 kB)
    Collecting uvicorn[standard]>=0.18.3 (from chromadb)
      Obtaining dependency information for uvicorn[standard]>=0.18.3 from https://files.pythonhosted.org/packages/5d/07/b9eac057f7efa56900640a233c1ed63db83568322c6bcbabe98f741d5289/uvicorn-0.23.1-py3-none-any.whl.metadata
      Using cached uvicorn-0.23.1-py3-none-any.whl.metadata (6.2 kB)
    Collecting numpy>=1.21.6 (from chromadb)
      Obtaining dependency information for numpy>=1.21.6 from https://files.pythonhosted.org/packages/1b/cd/9e8313ffd849626c836fffd7881296a74f53a7739bd9ce7a6e22b1fc843b/numpy-1.25.1-cp310-cp310-macosx_11_0_arm64.whl.metadata
      Using cached numpy-1.25.1-cp310-cp310-macosx_11_0_arm64.whl.metadata (5.6 kB)
    Collecting posthog>=2.4.0 (from chromadb)
      Using cached posthog-3.0.1-py2.py3-none-any.whl (37 kB)
    Requirement already satisfied: typing-extensions>=4.5.0 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from chromadb) (4.7.1)
    Collecting pulsar-client>=3.1.0 (from chromadb)
      Obtaining dependency information for pulsar-client>=3.1.0 from https://files.pythonhosted.


key: _node_content
string_value:
{"id_": "3da3c845-2918-4287-90b7-79f5335bba1c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c9e06df6-a8bb-4afc-b175-deec95f34554", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "828bf73fee962d40b49987729ec15243a4626889a2e15e526ca281f0ae72c8f1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "31db0435-625e-47c6-8182-84a019cce0c2", "node_type": "1", "metadata": {}, "hash": "31e5bc06ffe5c69f796c05e65e3e7d768b09ec188e5420d2f9a2d54a84f3f413", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8810, "end_char_idx": 10806, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
3 (from chromadb)
      Obtaining dependency information for uvicorn[standard]>=0.18.3 from https://files.pythonhosted.org/packages/5d/07/b9eac057f7efa56900640a233c1ed63db83568322c6bcbabe98f741d5289/uvicorn-0.23.1-py3-none-any.whl.metadata
      Using cached uvicorn-0.23.1-py3-none-any.whl.metadata (6.2 kB)
    Collecting numpy>=1.21.6 (from chromadb)
      Obtaining dependency information for numpy>=1.21.6 from https://files.pythonhosted.org/packages/1b/cd/9e8313ffd849626c836fffd7881296a74f53a7739bd9ce7a6e22b1fc843b/numpy-1.25.1-cp310-cp310-macosx_11_0_arm64.whl.metadata
      Using cached numpy-1.25.1-cp310-cp310-macosx_11_0_arm64.whl.metadata (5.6 kB)
    Collecting posthog>=2.4.0 (from chromadb)
      Using cached posthog-3.0.1-py2.py3-none-any.whl (37 kB)
    Requirement already satisfied: typing-extensions>=4.5.0 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from chromadb) (4.7.1)
    Collecting pulsar-client>=3.1.0 (from chromadb)
      Obtaining dependency information for pulsar-client>=3.1.0 from https://files.pythonhosted.org/packages/43/85/ab0455008ce3335a1c75a7c500fd8921ab166f34821fa67dc91ae9687a40/pulsar_client-3.2.0-cp310-cp310-macosx_10_15_universal2.whl.metadata
      Using cached pulsar_client-3.2.0-cp310-cp310-macosx_10_15_universal2.whl.metadata (1.0 kB)
    Collecting onnxruntime>=1.14.1 (from chromadb)
      Obtaining dependency information for onnxruntime>=1.14.1 from https://files.pythonhosted.org/packages/cf/06/0c6e355b9ddbebc34d0e21bc5be1e4bd2c124ebd9030525838fa6e65eaa8/onnxruntime-1.15.1-cp310-cp310-macosx_11_0_arm64.whl.metadata
      Using cached onnxruntime-1.15.1-cp310-cp310-macosx_11_0_arm64.whl.metadata (4.0 kB)
    Collecting tokenizers>=0.13.2 (from chromadb)
      Using cached tokenizers-0.13.3-cp310-cp310-macosx_12_0_arm64.whl (3.9 MB)
    Collecting pypika>=0.48.9 (from chromadb)
      Using cached PyPika-0.48.9-py2.py3-none-any.whl
    Requirement already satisfied: tqdm>=4.65.


key: _node_content
string_value:
{"id_": "31db0435-625e-47c6-8182-84a019cce0c2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3da3c845-2918-4287-90b7-79f5335bba1c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "72e29c6be555eab86497a73b53edaf55a16a54e1705bf4df788ea1a8098c5cde", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "48005629-0890-4824-9362-3fc0fc1bed2f", "node_type": "1", "metadata": {}, "hash": "a35813e3cee8b0f51f9ff668bfae389c2851389d3717cdb4e0bcd321fabf5503", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9790, "end_char_idx": 11821, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
1.0 (from chromadb)
      Obtaining dependency information for pulsar-client>=3.1.0 from https://files.pythonhosted.org/packages/43/85/ab0455008ce3335a1c75a7c500fd8921ab166f34821fa67dc91ae9687a40/pulsar_client-3.2.0-cp310-cp310-macosx_10_15_universal2.whl.metadata
      Using cached pulsar_client-3.2.0-cp310-cp310-macosx_10_15_universal2.whl.metadata (1.0 kB)
    Collecting onnxruntime>=1.14.1 (from chromadb)
      Obtaining dependency information for onnxruntime>=1.14.1 from https://files.pythonhosted.org/packages/cf/06/0c6e355b9ddbebc34d0e21bc5be1e4bd2c124ebd9030525838fa6e65eaa8/onnxruntime-1.15.1-cp310-cp310-macosx_11_0_arm64.whl.metadata
      Using cached onnxruntime-1.15.1-cp310-cp310-macosx_11_0_arm64.whl.metadata (4.0 kB)
    Collecting tokenizers>=0.13.2 (from chromadb)
      Using cached tokenizers-0.13.3-cp310-cp310-macosx_12_0_arm64.whl (3.9 MB)
    Collecting pypika>=0.48.9 (from chromadb)
      Using cached PyPika-0.48.9-py2.py3-none-any.whl
    Requirement already satisfied: tqdm>=4.65.0 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from chromadb) (4.65.0)
    Collecting overrides>=7.3.1 (from chromadb)
      Using cached overrides-7.3.1-py3-none-any.whl (17 kB)
    Collecting importlib-resources (from chromadb)
      Obtaining dependency information for importlib-resources from https://files.pythonhosted.org/packages/29/d1/bed03eca30aa05aaf6e0873de091f9385c48705c4a607c2dfe3edbe543e8/importlib_resources-6.0.0-py3-none-any.whl.metadata
      Using cached importlib_resources-6.0.0-py3-none-any.whl.metadata (4.2 kB)
    Collecting starlette<0.28.0,>=0.27.0 (from fastapi<0.100.0,>=0.95.2->chromadb)
      Obtaining dependency information for starlette<0.28.0,>=0.27.0 from https://files.pythonhosted.org/packages/58/f8/e2cca22387965584a409795913b774235752be4176d276714e15e1a58884/starlette-0.27.0-py3-none-any.whl.metadata
      Using cached starlette-0.27.0-py3-none-any.whl.metadata (5.8 kB)
    Collecting coloredlogs (from onnxruntime>=1.14.


key: _node_content
string_value:
{"id_": "48005629-0890-4824-9362-3fc0fc1bed2f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "31db0435-625e-47c6-8182-84a019cce0c2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "031b9c8f34ad1821435ae6571ee8b816be4d5bfdcf82d45c4e8cd5566cc300e6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8dd2943b-f1f9-45df-9836-4d04712993ec", "node_type": "1", "metadata": {}, "hash": "a9771477b207bbed33c5eec2fcbe187b1b39d6c7b39c7b4079f475af7dbab9cc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10734, "end_char_idx": 12870, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
48.9-py2.py3-none-any.whl
    Requirement already satisfied: tqdm>=4.65.0 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from chromadb) (4.65.0)
    Collecting overrides>=7.3.1 (from chromadb)
      Using cached overrides-7.3.1-py3-none-any.whl (17 kB)
    Collecting importlib-resources (from chromadb)
      Obtaining dependency information for importlib-resources from https://files.pythonhosted.org/packages/29/d1/bed03eca30aa05aaf6e0873de091f9385c48705c4a607c2dfe3edbe543e8/importlib_resources-6.0.0-py3-none-any.whl.metadata
      Using cached importlib_resources-6.0.0-py3-none-any.whl.metadata (4.2 kB)
    Collecting starlette<0.28.0,>=0.27.0 (from fastapi<0.100.0,>=0.95.2->chromadb)
      Obtaining dependency information for starlette<0.28.0,>=0.27.0 from https://files.pythonhosted.org/packages/58/f8/e2cca22387965584a409795913b774235752be4176d276714e15e1a58884/starlette-0.27.0-py3-none-any.whl.metadata
      Using cached starlette-0.27.0-py3-none-any.whl.metadata (5.8 kB)
    Collecting coloredlogs (from onnxruntime>=1.14.1->chromadb)
      Using cached coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)
    Collecting flatbuffers (from onnxruntime>=1.14.1->chromadb)
      Obtaining dependency information for flatbuffers from https://files.pythonhosted.org/packages/6f/12/d5c79ee252793ffe845d58a913197bfa02ae9a0b5c9bc3dc4b58d477b9e7/flatbuffers-23.5.26-py2.py3-none-any.whl.metadata
      Using cached flatbuffers-23.5.26-py2.py3-none-any.whl.metadata (850 bytes)
    Requirement already satisfied: packaging in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from onnxruntime>=1.14.1->chromadb) (23.1)
    Collecting protobuf (from onnxruntime>=1.14.1->chromadb)
      Obtaining dependency information for protobuf from https://files.pythonhosted.org/packages/cb/d3/a164038605494d49acc4f9cda1c0bc200b96382c53edd561387263bb181d/protobuf-4.23.4-cp37-abi3-macosx_10_9_universal2.whl.metadata
      Using cached protobuf-4.23.4-cp37-abi3-macosx_10_9_universal2.whl.metadata (540 bytes)
    Collecting sympy (from onnxruntime>=1.14.


key: _node_content
string_value:
{"id_": "8dd2943b-f1f9-45df-9836-4d04712993ec", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "48005629-0890-4824-9362-3fc0fc1bed2f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "b5b542927892ef31a0faebb5247e4e457d78e42331639f76fcdbc0bf66228ed9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "10d57062-da0d-43a9-a4bf-9fa6168e19f3", "node_type": "1", "metadata": {}, "hash": "0c870e984ba4ac3159bd8b49355cba154ae47ec5b1d2e520684c6cae098b0c69", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11748, "end_char_idx": 13910, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
whl.metadata (5.8 kB)
    Collecting coloredlogs (from onnxruntime>=1.14.1->chromadb)
      Using cached coloredlogs-15.0.1-py2.py3-none-any.whl (46 kB)
    Collecting flatbuffers (from onnxruntime>=1.14.1->chromadb)
      Obtaining dependency information for flatbuffers from https://files.pythonhosted.org/packages/6f/12/d5c79ee252793ffe845d58a913197bfa02ae9a0b5c9bc3dc4b58d477b9e7/flatbuffers-23.5.26-py2.py3-none-any.whl.metadata
      Using cached flatbuffers-23.5.26-py2.py3-none-any.whl.metadata (850 bytes)
    Requirement already satisfied: packaging in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from onnxruntime>=1.14.1->chromadb) (23.1)
    Collecting protobuf (from onnxruntime>=1.14.1->chromadb)
      Obtaining dependency information for protobuf from https://files.pythonhosted.org/packages/cb/d3/a164038605494d49acc4f9cda1c0bc200b96382c53edd561387263bb181d/protobuf-4.23.4-cp37-abi3-macosx_10_9_universal2.whl.metadata
      Using cached protobuf-4.23.4-cp37-abi3-macosx_10_9_universal2.whl.metadata (540 bytes)
    Collecting sympy (from onnxruntime>=1.14.1->chromadb)
      Using cached sympy-1.12-py3-none-any.whl (5.7 MB)
    Requirement already satisfied: python-dateutil>=2.8.2 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from pandas>=1.3->chromadb) (2.8.2)
    Collecting pytz>=2020.1 (from pandas>=1.3->chromadb)
      Using cached pytz-2023.3-py2.py3-none-any.whl (502 kB)
    Collecting tzdata>=2022.1 (from pandas>=1.3->chromadb)
      Using cached tzdata-2023.3-py2.py3-none-any.whl (341 kB)
    Requirement already satisfied: six>=1.5 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from posthog>=2.4.0->chromadb) (1.16.0)
    Collecting monotonic>=1.5 (from posthog>=2.4.0->chromadb)
      Using cached monotonic-1.6-py2.py3-none-any.whl (8.2 kB)
    Collecting backoff>=1.10.0 (from posthog>=2.4.0->chromadb)
      Using cached backoff-2.2.1-py3-none-any.whl (15 kB)
    Requirement already satisfied: certifi in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.


key: _node_content
string_value:
{"id_": "10d57062-da0d-43a9-a4bf-9fa6168e19f3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8dd2943b-f1f9-45df-9836-4d04712993ec", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "e96b6263a2c5130a5c9b3a102b6d0dd189f73875534b4b88b2c4cc4ab7898ab7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ea223d31-5ca8-42d1-878e-53367900605e", "node_type": "1", "metadata": {}, "hash": "a89dd86bd6173f8cf6ca5d19f2ea4f78e3482602aae1162d619b5ac1b4ea8642", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12800, "end_char_idx": 14928, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
whl.metadata (540 bytes)
    Collecting sympy (from onnxruntime>=1.14.1->chromadb)
      Using cached sympy-1.12-py3-none-any.whl (5.7 MB)
    Requirement already satisfied: python-dateutil>=2.8.2 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from pandas>=1.3->chromadb) (2.8.2)
    Collecting pytz>=2020.1 (from pandas>=1.3->chromadb)
      Using cached pytz-2023.3-py2.py3-none-any.whl (502 kB)
    Collecting tzdata>=2022.1 (from pandas>=1.3->chromadb)
      Using cached tzdata-2023.3-py2.py3-none-any.whl (341 kB)
    Requirement already satisfied: six>=1.5 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from posthog>=2.4.0->chromadb) (1.16.0)
    Collecting monotonic>=1.5 (from posthog>=2.4.0->chromadb)
      Using cached monotonic-1.6-py2.py3-none-any.whl (8.2 kB)
    Collecting backoff>=1.10.0 (from posthog>=2.4.0->chromadb)
      Using cached backoff-2.2.1-py3-none-any.whl (15 kB)
    Requirement already satisfied: certifi in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from pulsar-client>=3.1.0->chromadb) (2023.5.7)
    Requirement already satisfied: charset-normalizer<4,>=2 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from requests>=2.28->chromadb) (3.2.0)
    Requirement already satisfied: idna<4,>=2.5 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from requests>=2.28->chromadb) (3.4)
    Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from requests>=2.28->chromadb) (2.0.4)
    Collecting click>=7.0 (from uvicorn[standard]>=0.18.3->chromadb)
      Obtaining dependency information for click>=7.0 from https://files.pythonhosted.org/packages/1a/70/e63223f8116931d365993d4a6b7ef653a4d920b41d03de7c59499962821f/click-8.1.6-py3-none-any.whl.metadata
      Using cached click-8.1.6-py3-none-any.whl.metadata (3.0 kB)
    Collecting h11>=0.8 (from uvicorn[standard]>=0.18.


key: _node_content
string_value:
{"id_": "ea223d31-5ca8-42d1-878e-53367900605e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "10d57062-da0d-43a9-a4bf-9fa6168e19f3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "67e92f787292ab99157f792b7893d834bbe64a8ec5fb6d0b9fd7e11b3a62de28", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "959c0e8a-41f7-4068-976c-e66568feb767", "node_type": "1", "metadata": {}, "hash": "4d4960667d4c09bc9bad056f6c3e5c657eb02277118bfed7992df10cfc65cdec", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13910, "end_char_idx": 15912, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
10/site-packages (from pulsar-client>=3.1.0->chromadb) (2023.5.7)
    Requirement already satisfied: charset-normalizer<4,>=2 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from requests>=2.28->chromadb) (3.2.0)
    Requirement already satisfied: idna<4,>=2.5 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from requests>=2.28->chromadb) (3.4)
    Requirement already satisfied: urllib3<3,>=1.21.1 in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (from requests>=2.28->chromadb) (2.0.4)
    Collecting click>=7.0 (from uvicorn[standard]>=0.18.3->chromadb)
      Obtaining dependency information for click>=7.0 from https://files.pythonhosted.org/packages/1a/70/e63223f8116931d365993d4a6b7ef653a4d920b41d03de7c59499962821f/click-8.1.6-py3-none-any.whl.metadata
      Using cached click-8.1.6-py3-none-any.whl.metadata (3.0 kB)
    Collecting h11>=0.8 (from uvicorn[standard]>=0.18.3->chromadb)
      Using cached h11-0.14.0-py3-none-any.whl (58 kB)
    Collecting httptools>=0.5.0 (from uvicorn[standard]>=0.18.3->chromadb)
      Obtaining dependency information for httptools>=0.5.0 from https://files.pythonhosted.org/packages/8f/71/d535e9f6967958d21b8fe1baeb7efb6304b86e8fcff44d0bda8690e0aec9/httptools-0.6.0-cp310-cp310-macosx_10_9_universal2.whl.metadata
      Using cached httptools-0.6.0-cp310-cp310-macosx_10_9_universal2.whl.metadata (3.6 kB)
    Collecting python-dotenv>=0.13 (from uvicorn[standard]>=0.18.3->chromadb)
      Using cached python_dotenv-1.0.0-py3-none-any.whl (19 kB)
    Collecting pyyaml>=5.1 (from uvicorn[standard]>=0.18.3->chromadb)
      Obtaining dependency information for pyyaml>=5.1 from https://files.pythonhosted.org/packages/5b/07/10033a403b23405a8fc48975444463d3d10a5c2736b7eb2550b07b367429/PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl.metadata
      Using cached PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl.metadata (2.


key: _node_content
string_value:
{"id_": "959c0e8a-41f7-4068-976c-e66568feb767", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ea223d31-5ca8-42d1-878e-53367900605e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8c5a59adb1728a915245929e31a77a048e968586ac74d3fc8abca2cf58e2e21a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9a8e842c-65d3-44d6-b954-1528214ed44d", "node_type": "1", "metadata": {}, "hash": "d3d57c5eba9d2e0f23e4154c915b76b48b8d72298d685642b4ca716a0f19d8d4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14928, "end_char_idx": 16767, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
3->chromadb)
      Using cached h11-0.14.0-py3-none-any.whl (58 kB)
    Collecting httptools>=0.5.0 (from uvicorn[standard]>=0.18.3->chromadb)
      Obtaining dependency information for httptools>=0.5.0 from https://files.pythonhosted.org/packages/8f/71/d535e9f6967958d21b8fe1baeb7efb6304b86e8fcff44d0bda8690e0aec9/httptools-0.6.0-cp310-cp310-macosx_10_9_universal2.whl.metadata
      Using cached httptools-0.6.0-cp310-cp310-macosx_10_9_universal2.whl.metadata (3.6 kB)
    Collecting python-dotenv>=0.13 (from uvicorn[standard]>=0.18.3->chromadb)
      Using cached python_dotenv-1.0.0-py3-none-any.whl (19 kB)
    Collecting pyyaml>=5.1 (from uvicorn[standard]>=0.18.3->chromadb)
      Obtaining dependency information for pyyaml>=5.1 from https://files.pythonhosted.org/packages/5b/07/10033a403b23405a8fc48975444463d3d10a5c2736b7eb2550b07b367429/PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl.metadata
      Using cached PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl.metadata (2.1 kB)
    Collecting uvloop!=0.15.0,!=0.15.1,>=0.14.0 (from uvicorn[standard]>=0.18.3->chromadb)
      Using cached uvloop-0.17.0-cp310-cp310-macosx_10_9_universal2.whl (2.1 MB)
    Collecting watchfiles>=0.13 (from uvicorn[standard]>=0.18.3->chromadb)
      Using cached watchfiles-0.19.0-cp37-abi3-macosx_11_0_arm64.whl (388 kB)
    Collecting websockets>=10.4 (from uvicorn[standard]>=0.18.3->chromadb)
      Using cached websockets-11.0.3-cp310-cp310-macosx_11_0_arm64.whl (121 kB)
    Collecting anyio<5,>=3.4.0 (from starlette<0.28.0,>=0.27.0->fastapi<0.100.0,>=0.95.2->chromadb)
      Obtaining dependency information for anyio<5,>=3.4.0 from https://files.pythonhosted.org/packages/19/24/44299477fe7dcc9cb58d0a57d5a7588d6af2ff403fdd2d47a246c91a3246/anyio-3.7.1-py3-none-any.whl.metadata
      Using cached anyio-3.7.1-py3-none-any.whl.metadata (4.


key: _node_content
string_value:
{"id_": "9a8e842c-65d3-44d6-b954-1528214ed44d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "959c0e8a-41f7-4068-976c-e66568feb767", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "2a61254633a16831d1fd46a8cdd7d21bc1749a661e71d44998c53b35463ae29c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "da137ef5-b66b-4153-b3f2-6ade987cebfe", "node_type": "1", "metadata": {}, "hash": "532a6220d13418b7ed177e24bc801f08a6a339411958723f50510aeee323d0ef", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15821, "end_char_idx": 17678, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
whl.metadata
      Using cached PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl.metadata (2.1 kB)
    Collecting uvloop!=0.15.0,!=0.15.1,>=0.14.0 (from uvicorn[standard]>=0.18.3->chromadb)
      Using cached uvloop-0.17.0-cp310-cp310-macosx_10_9_universal2.whl (2.1 MB)
    Collecting watchfiles>=0.13 (from uvicorn[standard]>=0.18.3->chromadb)
      Using cached watchfiles-0.19.0-cp37-abi3-macosx_11_0_arm64.whl (388 kB)
    Collecting websockets>=10.4 (from uvicorn[standard]>=0.18.3->chromadb)
      Using cached websockets-11.0.3-cp310-cp310-macosx_11_0_arm64.whl (121 kB)
    Collecting anyio<5,>=3.4.0 (from starlette<0.28.0,>=0.27.0->fastapi<0.100.0,>=0.95.2->chromadb)
      Obtaining dependency information for anyio<5,>=3.4.0 from https://files.pythonhosted.org/packages/19/24/44299477fe7dcc9cb58d0a57d5a7588d6af2ff403fdd2d47a246c91a3246/anyio-3.7.1-py3-none-any.whl.metadata
      Using cached anyio-3.7.1-py3-none-any.whl.metadata (4.7 kB)
    Collecting humanfriendly>=9.1 (from coloredlogs->onnxruntime>=1.14.1->chromadb)
      Using cached humanfriendly-10.0-py2.py3-none-any.whl (86 kB)
    Collecting mpmath>=0.19 (from sympy->onnxruntime>=1.14.1->chromadb)
      Using cached mpmath-1.3.0-py3-none-any.whl (536 kB)
    Collecting sniffio>=1.1 (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi<0.100.0,>=0.95.2->chromadb)
      Using cached sniffio-1.3.0-py3-none-any.whl (10 kB)
    Collecting exceptiongroup (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi<0.100.0,>=0.95.2->chromadb)
      Obtaining dependency information for exceptiongroup from https://files.pythonhosted.org/packages/fe/17/f43b7c9ccf399d72038042ee72785c305f6c6fdc6231942f8ab99d995742/exceptiongroup-1.1.2-py3-none-any.whl.metadata
      Using cached exceptiongroup-1.1.2-py3-none-any.whl.metadata (6.1 kB)
    Downloading chromadb-0.4.2-py3-none-any.


key: _node_content
string_value:
{"id_": "da137ef5-b66b-4153-b3f2-6ade987cebfe", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9a8e842c-65d3-44d6-b954-1528214ed44d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "43bf7045ae076615ddd241a374727713a0d71da849a9d6e669dad81a3da3432e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "64df5e66-e48a-4b53-9827-7f1946850904", "node_type": "1", "metadata": {}, "hash": "fc48370691c2af01a41535cb3dc824f9ffb04d104cbfc3a7c4260dc312e6f4ca", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16677, "end_char_idx": 18461, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
7.1-py3-none-any.whl.metadata
      Using cached anyio-3.7.1-py3-none-any.whl.metadata (4.7 kB)
    Collecting humanfriendly>=9.1 (from coloredlogs->onnxruntime>=1.14.1->chromadb)
      Using cached humanfriendly-10.0-py2.py3-none-any.whl (86 kB)
    Collecting mpmath>=0.19 (from sympy->onnxruntime>=1.14.1->chromadb)
      Using cached mpmath-1.3.0-py3-none-any.whl (536 kB)
    Collecting sniffio>=1.1 (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi<0.100.0,>=0.95.2->chromadb)
      Using cached sniffio-1.3.0-py3-none-any.whl (10 kB)
    Collecting exceptiongroup (from anyio<5,>=3.4.0->starlette<0.28.0,>=0.27.0->fastapi<0.100.0,>=0.95.2->chromadb)
      Obtaining dependency information for exceptiongroup from https://files.pythonhosted.org/packages/fe/17/f43b7c9ccf399d72038042ee72785c305f6c6fdc6231942f8ab99d995742/exceptiongroup-1.1.2-py3-none-any.whl.metadata
      Using cached exceptiongroup-1.1.2-py3-none-any.whl.metadata (6.1 kB)
    Downloading chromadb-0.4.2-py3-none-any.whl (399 kB)
    [2K   [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m399.3/399.3 kB[0m [31m12.8 MB/s[0m eta [36m0:00:00[0m
    [?25hUsing cached chroma_hnswlib-0.7.1-cp310-cp310-macosx_13_0_arm64.whl (195 kB)
    Using cached fastapi-0.99.1-py3-none-any.whl (58 kB)
    Using cached numpy-1.25.1-cp310-cp310-macosx_11_0_arm64.whl (14.0 MB)
    Using cached onnxruntime-1.15.1-cp310-cp310-macosx_11_0_arm64.whl (6.1 MB)
    Using cached pandas-2.0.3-cp310-cp310-macosx_11_0_arm64.whl (10.8 MB)
    Using cached pulsar_client-3.2.0-cp310-cp310-macosx_10_15_universal2.whl (10.8 MB)
    Using cached pydantic-1.10.11-cp310-cp310-macosx_11_0_arm64.whl (2.5 MB)
    Using cached importlib_resources-6.0.0-py3-none-any.whl (31 kB)
    Using cached click-8.1.6-py3-none-any.


key: _node_content
string_value:
{"id_": "64df5e66-e48a-4b53-9827-7f1946850904", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "da137ef5-b66b-4153-b3f2-6ade987cebfe", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "a6c631432a4fd44f8849d95a92bac843ee984095a43aa88bf26a65aee7b5a2c9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "420167d3-b2c2-4833-b7be-aed2680dde3c", "node_type": "1", "metadata": {}, "hash": "be62f03cdee8aa6b42d698d98c708d7dcfddc1eb2c2ce61ffa020c5c87f7cea7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17595, "end_char_idx": 19469, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
1.2-py3-none-any.whl.metadata (6.1 kB)
    Downloading chromadb-0.4.2-py3-none-any.whl (399 kB)
    [2K   [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m399.3/399.3 kB[0m [31m12.8 MB/s[0m eta [36m0:00:00[0m
    [?25hUsing cached chroma_hnswlib-0.7.1-cp310-cp310-macosx_13_0_arm64.whl (195 kB)
    Using cached fastapi-0.99.1-py3-none-any.whl (58 kB)
    Using cached numpy-1.25.1-cp310-cp310-macosx_11_0_arm64.whl (14.0 MB)
    Using cached onnxruntime-1.15.1-cp310-cp310-macosx_11_0_arm64.whl (6.1 MB)
    Using cached pandas-2.0.3-cp310-cp310-macosx_11_0_arm64.whl (10.8 MB)
    Using cached pulsar_client-3.2.0-cp310-cp310-macosx_10_15_universal2.whl (10.8 MB)
    Using cached pydantic-1.10.11-cp310-cp310-macosx_11_0_arm64.whl (2.5 MB)
    Using cached importlib_resources-6.0.0-py3-none-any.whl (31 kB)
    Using cached click-8.1.6-py3-none-any.whl (97 kB)
    Using cached httptools-0.6.0-cp310-cp310-macosx_10_9_universal2.whl (237 kB)
    Using cached PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl (169 kB)
    Using cached starlette-0.27.0-py3-none-any.whl (66 kB)
    Using cached flatbuffers-23.5.26-py2.py3-none-any.whl (26 kB)
    Using cached protobuf-4.23.4-cp37-abi3-macosx_10_9_universal2.whl (400 kB)
    Using cached uvicorn-0.23.1-py3-none-any.whl (59 kB)
    Using cached anyio-3.7.1-py3-none-any.whl (80 kB)
    Using cached exceptiongroup-1.1.2-py3-none-any.whl (14 kB)
    Installing collected packages: tokenizers, pytz, pypika, mpmath, monotonic, flatbuffers, websockets, uvloop, tzdata, sympy, sniffio, pyyaml, python-dotenv, pydantic, pulsar-client, protobuf, overrides, numpy, importlib-resources, humanfriendly, httptools, h11, exceptiongroup, click, backoff, uvicorn, posthog, pandas, coloredlogs, chroma-hnswlib, anyio, watchfiles, starlette, onnxruntime, fastapi, chromadb
    Successfully installed anyio-3.7.1 backoff-2.2.


key: _node_content
string_value:
{"id_": "420167d3-b2c2-4833-b7be-aed2680dde3c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "64df5e66-e48a-4b53-9827-7f1946850904", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "eefa500abf2c84b6c61c909f2583dd50554d96b4d7ab0f955acf8f8100976f97", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4d12b45a-a731-4e31-b6eb-069535d21509", "node_type": "1", "metadata": {}, "hash": "0648dffc3582738fbfdcfe4d638c81845d49076868d83d09ff3cb1ab80c6b817", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18390, "end_char_idx": 20088, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0.0-py3-none-any.whl (31 kB)
    Using cached click-8.1.6-py3-none-any.whl (97 kB)
    Using cached httptools-0.6.0-cp310-cp310-macosx_10_9_universal2.whl (237 kB)
    Using cached PyYAML-6.0.1-cp310-cp310-macosx_11_0_arm64.whl (169 kB)
    Using cached starlette-0.27.0-py3-none-any.whl (66 kB)
    Using cached flatbuffers-23.5.26-py2.py3-none-any.whl (26 kB)
    Using cached protobuf-4.23.4-cp37-abi3-macosx_10_9_universal2.whl (400 kB)
    Using cached uvicorn-0.23.1-py3-none-any.whl (59 kB)
    Using cached anyio-3.7.1-py3-none-any.whl (80 kB)
    Using cached exceptiongroup-1.1.2-py3-none-any.whl (14 kB)
    Installing collected packages: tokenizers, pytz, pypika, mpmath, monotonic, flatbuffers, websockets, uvloop, tzdata, sympy, sniffio, pyyaml, python-dotenv, pydantic, pulsar-client, protobuf, overrides, numpy, importlib-resources, humanfriendly, httptools, h11, exceptiongroup, click, backoff, uvicorn, posthog, pandas, coloredlogs, chroma-hnswlib, anyio, watchfiles, starlette, onnxruntime, fastapi, chromadb
    Successfully installed anyio-3.7.1 backoff-2.2.1 chroma-hnswlib-0.7.1 chromadb-0.4.2 click-8.1.6 coloredlogs-15.0.1 exceptiongroup-1.1.2 fastapi-0.99.1 flatbuffers-23.5.26 h11-0.14.0 httptools-0.6.0 humanfriendly-10.0 importlib-resources-6.0.0 monotonic-1.6 mpmath-1.3.0 numpy-1.25.1 onnxruntime-1.15.1 overrides-7.3.1 pandas-2.0.3 posthog-3.0.1 protobuf-4.23.4 pulsar-client-3.2.0 pydantic-1.10.11 pypika-0.48.9 python-dotenv-1.0.0 pytz-2023.3 pyyaml-6.0.1 sniffio-1.3.0 starlette-0.27.0 sympy-1.12 tokenizers-0.13.3 tzdata-2023.3 uvicorn-0.23.1 uvloop-0.17.0 watchfiles-0.19.0 websockets-11.0.3
    Note: you may need to restart the kernel to use updated packages.


key: _node_content
string_value:
{"id_": "4d12b45a-a731-4e31-b6eb-069535d21509", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "420167d3-b2c2-4833-b7be-aed2680dde3c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "6e16a96eea041d876d6b2c6736e6c7515f0109628d44896f40ef1ee99470ba2d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8df577df-ed92-49a0-9b78-b9cc041566ca", "node_type": "1", "metadata": {}, "hash": "bb97a35ff3877fb498af5413fb94a64d443cfb7e6fa92291c630aa3306f2ccca", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19110, "end_char_idx": 20985, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
uvloop, tzdata, sympy, sniffio, pyyaml, python-dotenv, pydantic, pulsar-client, protobuf, overrides, numpy, importlib-resources, humanfriendly, httptools, h11, exceptiongroup, click, backoff, uvicorn, posthog, pandas, coloredlogs, chroma-hnswlib, anyio, watchfiles, starlette, onnxruntime, fastapi, chromadb
    Successfully installed anyio-3.7.1 backoff-2.2.1 chroma-hnswlib-0.7.1 chromadb-0.4.2 click-8.1.6 coloredlogs-15.0.1 exceptiongroup-1.1.2 fastapi-0.99.1 flatbuffers-23.5.26 h11-0.14.0 httptools-0.6.0 humanfriendly-10.0 importlib-resources-6.0.0 monotonic-1.6 mpmath-1.3.0 numpy-1.25.1 onnxruntime-1.15.1 overrides-7.3.1 pandas-2.0.3 posthog-3.0.1 protobuf-4.23.4 pulsar-client-3.2.0 pydantic-1.10.11 pypika-0.48.9 python-dotenv-1.0.0 pytz-2023.3 pyyaml-6.0.1 sniffio-1.3.0 starlette-0.27.0 sympy-1.12 tokenizers-0.13.3 tzdata-2023.3 uvicorn-0.23.1 uvloop-0.17.0 watchfiles-0.19.0 websockets-11.0.3
    Note: you may need to restart the kernel to use updated packages.
    Collecting wget
      Using cached wget-3.2.zip (10 kB)
      Preparing metadata (setup.py) ... [?25ldone
    [?25hBuilding wheels for collected packages: wget
      Building wheel for wget (setup.py) ... [?25ldone
    [?25h  Created wheel for wget: filename=wget-3.2-py3-none-any.whl size=9657 sha256=b2d83c5fcdeab398d0a4e9808a470bbf725fffea4a6130e731c6097b9561005b
      Stored in directory: /Users/antontroynikov/Library/Caches/pip/wheels/8b/f1/7f/5c94f0a7a505ca1c81cd1d9208ae2064675d97582078e6c769
    Successfully built wget
    Installing collected packages: wget
    Successfully installed wget-3.2
    Note: you may need to restart the kernel to use updated packages.
    Requirement already satisfied: numpy in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (1.25.1)
    Note: you may need to restart the kernel to use updated packages.


key: _node_content
string_value:
{"id_": "8df577df-ed92-49a0-9b78-b9cc041566ca", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4d12b45a-a731-4e31-b6eb-069535d21509", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "a694c829048ae6db02e60c5b4e5721efeef0ec63277083f57de3fb15e770e9f3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "95d6e6c3-263f-4c9b-8998-f2fb4d1f46fb", "node_type": "1", "metadata": {}, "hash": "cc5552ba2aa38db470f007100d75c8ebf3eaacbb31b332e54b1079d48a6a9459", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19851, "end_char_idx": 22189, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0.0 pytz-2023.3 pyyaml-6.0.1 sniffio-1.3.0 starlette-0.27.0 sympy-1.12 tokenizers-0.13.3 tzdata-2023.3 uvicorn-0.23.1 uvloop-0.17.0 watchfiles-0.19.0 websockets-11.0.3
    Note: you may need to restart the kernel to use updated packages.
    Collecting wget
      Using cached wget-3.2.zip (10 kB)
      Preparing metadata (setup.py) ... [?25ldone
    [?25hBuilding wheels for collected packages: wget
      Building wheel for wget (setup.py) ... [?25ldone
    [?25h  Created wheel for wget: filename=wget-3.2-py3-none-any.whl size=9657 sha256=b2d83c5fcdeab398d0a4e9808a470bbf725fffea4a6130e731c6097b9561005b
      Stored in directory: /Users/antontroynikov/Library/Caches/pip/wheels/8b/f1/7f/5c94f0a7a505ca1c81cd1d9208ae2064675d97582078e6c769
    Successfully built wget
    Installing collected packages: wget
    Successfully installed wget-3.2
    Note: you may need to restart the kernel to use updated packages.
    Requirement already satisfied: numpy in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (1.25.1)
    Note: you may need to restart the kernel to use updated packages.



```python
import openai
import pandas as pd
import os
import wget
from ast import literal_eval

# Chroma's client library for Python
import chromadb

# I've set this to our new embeddings model, this can be changed to the embedding model of your choice
EMBEDDING_MODEL = "text-embedding-3-small"

# Ignore unclosed SSL socket warnings - optional in case you get these errors
import warnings

warnings.filterwarnings(action="ignore", message="unclosed", category=ResourceWarning)
warnings.filterwarnings("ignore", category=DeprecationWarning) 
```

## Load data

In this section we'll load embedded data that we've prepared previous to this session.


```python
embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```




    'vector_database_wikipedia_articles_embedded.zip'




```python
import zipfile
with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip","r") as zip_ref:
    zip_ref.extractall("../data")
```


```python
article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')
```


```python
article_df.head()
```


key: _node_content
string_value:
{"id_": "95d6e6c3-263f-4c9b-8998-f2fb4d1f46fb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8df577df-ed92-49a0-9b78-b9cc041566ca", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "bf42f226b94a5c82b0ea8c4149292011f2eff6474192952a02215daa5b86db73", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "28c11293-e062-4672-81bf-e7ff0f4ec01e", "node_type": "1", "metadata": {}, "hash": "55d75d734c6c465942a4bbe99c9e2550ca1dde90d5af183d906f3fc23ae149c3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20777, "end_char_idx": 23205, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Requirement already satisfied: numpy in /Users/antontroynikov/miniforge3/envs/chroma-openai-cookbook/lib/python3.10/site-packages (1.25.1)
    Note: you may need to restart the kernel to use updated packages.



```python
import openai
import pandas as pd
import os
import wget
from ast import literal_eval

# Chroma's client library for Python
import chromadb

# I've set this to our new embeddings model, this can be changed to the embedding model of your choice
EMBEDDING_MODEL = "text-embedding-3-small"

# Ignore unclosed SSL socket warnings - optional in case you get these errors
import warnings

warnings.filterwarnings(action="ignore", message="unclosed", category=ResourceWarning)
warnings.filterwarnings("ignore", category=DeprecationWarning) 
```

## Load data

In this section we'll load embedded data that we've prepared previous to this session.


```python
embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```




    'vector_database_wikipedia_articles_embedded.zip'




```python
import zipfile
with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip","r") as zip_ref:
    zip_ref.extractall("../data")
```


```python
article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')
```


```python
article_df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>url</th>
      <th>title</th>
      <th>text</th>
      <th>title_vector</th>
      <th>content_vector</th>
      <th>vector_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>https://simple.wikipedia.org/wiki/April</td>
      <td>April</td>
      <td>April is the fourth month of the year in the J...</td>
      <td>[0.001009464613161981, -0.020700545981526375, ...</td>
      <td>[-0.011253940872848034, -0.013491976074874401,...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>https://simple.wikipedia.org/wiki/August</td>
      <td>August</td>
      <td>August (Aug.)


key: _node_content
string_value:
{"id_": "28c11293-e062-4672-81bf-e7ff0f4ec01e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "95d6e6c3-263f-4c9b-8998-f2fb4d1f46fb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "5ab0f70936dfab0b4238ff107eed102792d23e0750a4ec7fc444661a843d47b0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5126e413-ba75-4122-b94d-00649c37653f", "node_type": "1", "metadata": {}, "hash": "0eb9f1fbcad7c87e0ae1cc80d55f22c888212d66f0901dbcd88725f848899c3c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 22058, "end_char_idx": 24236, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')
```


```python
article_df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>url</th>
      <th>title</th>
      <th>text</th>
      <th>title_vector</th>
      <th>content_vector</th>
      <th>vector_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>https://simple.wikipedia.org/wiki/April</td>
      <td>April</td>
      <td>April is the fourth month of the year in the J...</td>
      <td>[0.001009464613161981, -0.020700545981526375, ...</td>
      <td>[-0.011253940872848034, -0.013491976074874401,...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>https://simple.wikipedia.org/wiki/August</td>
      <td>August</td>
      <td>August (Aug.) is the eighth month of the year ...</td>
      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>
      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>6</td>
      <td>https://simple.wikipedia.org/wiki/Art</td>
      <td>Art</td>
      <td>Art is a creative activity that expresses imag...</td>
      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>
      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>
      <td>2</td>
    </tr>
    <tr>
      <th>3</th>
      <td>8</td>
      <td>https://simple.wikipedia.org/wiki/A</td>
      <td>A</td>
      <td>A or a is the first letter of the English alph...</td>
      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>
      <td>[0.024894846603274345, -0.022186409682035446, ...</td>
      <td>3</td>
    </tr>
    <tr>
      <th>4</th>
      <td>9</td>
      <td>https://simple.wikipedia.org/wiki/Air</td>
      <td>Air</td>
      <td>Air refers to the Earth's atmosphere.


key: _node_content
string_value:
{"id_": "5126e413-ba75-4122-b94d-00649c37653f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "28c11293-e062-4672-81bf-e7ff0f4ec01e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "5e7cf39b2939fc387f82f1b1edcd54492de9e9c10b9794bd1c595ccf56724885", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7f69b41e-00f9-44f3-b75e-45497d449742", "node_type": "1", "metadata": {}, "hash": "e31dbdf83f0b175955f34de70464687d224d388059d362c5fb12d62947de6bde", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23206, "end_char_idx": 25404, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
is the eighth month of the year ...</td>
      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>
      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>6</td>
      <td>https://simple.wikipedia.org/wiki/Art</td>
      <td>Art</td>
      <td>Art is a creative activity that expresses imag...</td>
      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>
      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>
      <td>2</td>
    </tr>
    <tr>
      <th>3</th>
      <td>8</td>
      <td>https://simple.wikipedia.org/wiki/A</td>
      <td>A</td>
      <td>A or a is the first letter of the English alph...</td>
      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>
      <td>[0.024894846603274345, -0.022186409682035446, ...</td>
      <td>3</td>
    </tr>
    <tr>
      <th>4</th>
      <td>9</td>
      <td>https://simple.wikipedia.org/wiki/Air</td>
      <td>Air</td>
      <td>Air refers to the Earth's atmosphere. Air is a...</td>
      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>
      <td>[0.021524671465158463, 0.018522677943110466, -...</td>
      <td>4</td>
    </tr>
  </tbody>
</table>
</div>




```python
# Read vectors from strings back into a list
article_df['title_vector'] = article_df.title_vector.apply(literal_eval)
article_df['content_vector'] = article_df.content_vector.apply(literal_eval)

# Set vector_id to be a string
article_df['vector_id'] = article_df['vector_id'].apply(str)
```


```python
article_df.info(show_counts=True)
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 25000 entries, 0 to 24999
    Data columns (total 7 columns):
     #   Column          Non-Null Count  Dtype 
    ---  ------          --------------  ----- 
     0   id              25000 non-null  int64 
     1   url             25000 non-null  object
     2   title           25000 non-null  object
     3   text            25000 non-null  object
     4   title_vector    25000 non-null  object
     5   content_vector  25000 non-null  object
     6   vector_id       25000 non-null  object
    dtypes: int64(1), object(6)
    memory usage: 1.3+ MB


key: _node_content
string_value:
{"id_": "7f69b41e-00f9-44f3-b75e-45497d449742", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5126e413-ba75-4122-b94d-00649c37653f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "2ece711683c6310cd604958b8cd5be14dbba53c8c66e3636993a862eb40a39da", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f21f8af9-0dcb-42a2-bc9b-5c633663909f", "node_type": "1", "metadata": {}, "hash": "6915acaf8b4f99ccbaf24180f03eb4a16e0105699601e93c153d4568a5fff1dd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24237, "end_char_idx": 27234, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Air is a...</td>
      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>
      <td>[0.021524671465158463, 0.018522677943110466, -...</td>
      <td>4</td>
    </tr>
  </tbody>
</table>
</div>




```python
# Read vectors from strings back into a list
article_df['title_vector'] = article_df.title_vector.apply(literal_eval)
article_df['content_vector'] = article_df.content_vector.apply(literal_eval)

# Set vector_id to be a string
article_df['vector_id'] = article_df['vector_id'].apply(str)
```


```python
article_df.info(show_counts=True)
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 25000 entries, 0 to 24999
    Data columns (total 7 columns):
     #   Column          Non-Null Count  Dtype 
    ---  ------          --------------  ----- 
     0   id              25000 non-null  int64 
     1   url             25000 non-null  object
     2   title           25000 non-null  object
     3   text            25000 non-null  object
     4   title_vector    25000 non-null  object
     5   content_vector  25000 non-null  object
     6   vector_id       25000 non-null  object
    dtypes: int64(1), object(6)
    memory usage: 1.3+ MB


# Chroma

We'll index these embedded documents in a vector database and search them. The first option we'll look at is **Chroma**, an easy to use open-source self-hosted in-memory vector database, designed for working with embeddings together with LLMs. 

In this section, we will:
- Instantiate the Chroma client
- Create collections for each class of embedding 
- Query each collection 

### Instantiate the Chroma client

Create the Chroma client. By default, Chroma is ephemeral and runs in memory. 
However, you can easily set up a persistent configuration which writes to disk.


```python
chroma_client = chromadb.EphemeralClient() # Equivalent to chromadb.Client(), ephemeral.
# Uncomment for persistent client
# chroma_client = chromadb.PersistentClient()
```

### Create collections

Chroma collections allow you to store and filter with arbitrary metadata, making it easy to query subsets of the embedded data. 

Chroma is already integrated with OpenAI's embedding functions. The best way to use them is on construction of a collection, as follows.
Alternatively, you can 'bring your own embeddings'. More information can be found [here](https://docs.trychroma.com/embeddings)


```python
from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction

# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.

# Note. alternatively you can set a temporary env variable like this:
# os.environ["OPENAI_API_KEY"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

if os.getenv("OPENAI_API_KEY") is not None:
    openai.api_key = os.getenv("OPENAI_API_KEY")
    print ("OPENAI_API_KEY is ready")
else:
    print ("OPENAI_API_KEY environment variable not found")


key: _node_content
string_value:
{"id_": "f21f8af9-0dcb-42a2-bc9b-5c633663909f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7f69b41e-00f9-44f3-b75e-45497d449742", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "d856fba364552c3d638a374629a47bba05767b95bc0837ae152492f23757fee6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "727ac10d-6681-432e-9201-bfab17b234a3", "node_type": "1", "metadata": {}, "hash": "4cab9f149edcde8976c9c43075473e4fd728f89cf4451a534e42faddc7ac53e0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25407, "end_char_idx": 28948, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Chroma

We'll index these embedded documents in a vector database and search them. The first option we'll look at is **Chroma**, an easy to use open-source self-hosted in-memory vector database, designed for working with embeddings together with LLMs. 

In this section, we will:
- Instantiate the Chroma client
- Create collections for each class of embedding 
- Query each collection 

### Instantiate the Chroma client

Create the Chroma client. By default, Chroma is ephemeral and runs in memory. 
However, you can easily set up a persistent configuration which writes to disk.


```python
chroma_client = chromadb.EphemeralClient() # Equivalent to chromadb.Client(), ephemeral.
# Uncomment for persistent client
# chroma_client = chromadb.PersistentClient()
```

### Create collections

Chroma collections allow you to store and filter with arbitrary metadata, making it easy to query subsets of the embedded data. 

Chroma is already integrated with OpenAI's embedding functions. The best way to use them is on construction of a collection, as follows.
Alternatively, you can 'bring your own embeddings'. More information can be found [here](https://docs.trychroma.com/embeddings)


```python
from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction

# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.

# Note. alternatively you can set a temporary env variable like this:
# os.environ["OPENAI_API_KEY"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

if os.getenv("OPENAI_API_KEY") is not None:
    openai.api_key = os.getenv("OPENAI_API_KEY")
    print ("OPENAI_API_KEY is ready")
else:
    print ("OPENAI_API_KEY environment variable not found")


embedding_function = OpenAIEmbeddingFunction(api_key=os.environ.get('OPENAI_API_KEY'), model_name=EMBEDDING_MODEL)

wikipedia_content_collection = chroma_client.create_collection(name='wikipedia_content', embedding_function=embedding_function)
wikipedia_title_collection = chroma_client.create_collection(name='wikipedia_titles', embedding_function=embedding_function)
```

    OPENAI_API_KEY is ready


### Populate the collections

Chroma collections allow you to populate, and filter on, whatever metadata you like. Chroma can also store the text alongside the vectors, and return everything in a single `query` call, when this is more convenient. 

For this use-case, we'll just store the embeddings and IDs, and use these to index the original dataframe. 


```python
# Add the content vectors
wikipedia_content_collection.add(
    ids=article_df.vector_id.tolist(),
    embeddings=article_df.content_vector.tolist(),
)

# Add the title vectors
wikipedia_title_collection.add(
    ids=article_df.vector_id.tolist(),
    embeddings=article_df.title_vector.tolist(),
)
```

### Search the collections

Chroma handles embedding queries for you if an embedding function is set, like in this example.


```python
def query_collection(collection, query, max_results, dataframe):
    results = collection.query(query_texts=query, n_results=max_results, include=['distances']) 
    df = pd.DataFrame({
                'id':results['ids'][0], 
                'score':results['distances'][0],
                'title': dataframe[dataframe.vector_id.isin(results['ids'][0])]['title'],
                'content': dataframe[dataframe.vector_id.isin(results['ids'][0])]['text'],
                })
    
    return df
```


key: _node_content
string_value:
{"id_": "727ac10d-6681-432e-9201-bfab17b234a3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f21f8af9-0dcb-42a2-bc9b-5c633663909f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "a5064641359a1e2fbf00e5df2f2ed51559f34313de1413b20581dc9b1f00528b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e246b2bf-b85e-4dd7-b6e5-91a29faba2ff", "node_type": "1", "metadata": {}, "hash": "3b60c49431a6444e1de36ffa84f1f5ac907c937da763d81595ffcda70ae57123", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27237, "end_char_idx": 29153, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
embedding_function = OpenAIEmbeddingFunction(api_key=os.environ.get('OPENAI_API_KEY'), model_name=EMBEDDING_MODEL)

wikipedia_content_collection = chroma_client.create_collection(name='wikipedia_content', embedding_function=embedding_function)
wikipedia_title_collection = chroma_client.create_collection(name='wikipedia_titles', embedding_function=embedding_function)
```

    OPENAI_API_KEY is ready


### Populate the collections

Chroma collections allow you to populate, and filter on, whatever metadata you like. Chroma can also store the text alongside the vectors, and return everything in a single `query` call, when this is more convenient. 

For this use-case, we'll just store the embeddings and IDs, and use these to index the original dataframe. 


```python
# Add the content vectors
wikipedia_content_collection.add(
    ids=article_df.vector_id.tolist(),
    embeddings=article_df.content_vector.tolist(),
)

# Add the title vectors
wikipedia_title_collection.add(
    ids=article_df.vector_id.tolist(),
    embeddings=article_df.title_vector.tolist(),
)
```

### Search the collections

Chroma handles embedding queries for you if an embedding function is set, like in this example.


```python
def query_collection(collection, query, max_results, dataframe):
    results = collection.query(query_texts=query, n_results=max_results, include=['distances']) 
    df = pd.DataFrame({
                'id':results['ids'][0], 
                'score':results['distances'][0],
                'title': dataframe[dataframe.vector_id.isin(results['ids'][0])]['title'],
                'content': dataframe[dataframe.vector_id.isin(results['ids'][0])]['text'],
                })
    
    return df
```


```python
title_query_result = query_collection(
    collection=wikipedia_title_collection,
    query="modern art in Europe",
    max_results=10,
    dataframe=article_df
)
title_query_result.head()
```


key: _node_content
string_value:
{"id_": "e246b2bf-b85e-4dd7-b6e5-91a29faba2ff", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "727ac10d-6681-432e-9201-bfab17b234a3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f707ac8cbb10a5cc5e3139209e55fd723b86a39e6894247a51d6b44bef087599", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "07521441-9c66-40e5-8674-f72e8831710e", "node_type": "1", "metadata": {}, "hash": "8261dc1b40eb4d98829a6e56f7c6e2ebddb0e2ea3c57980015bc72ad04bd8391", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27641, "end_char_idx": 30512, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Populate the collections

Chroma collections allow you to populate, and filter on, whatever metadata you like. Chroma can also store the text alongside the vectors, and return everything in a single `query` call, when this is more convenient. 

For this use-case, we'll just store the embeddings and IDs, and use these to index the original dataframe. 


```python
# Add the content vectors
wikipedia_content_collection.add(
    ids=article_df.vector_id.tolist(),
    embeddings=article_df.content_vector.tolist(),
)

# Add the title vectors
wikipedia_title_collection.add(
    ids=article_df.vector_id.tolist(),
    embeddings=article_df.title_vector.tolist(),
)
```

### Search the collections

Chroma handles embedding queries for you if an embedding function is set, like in this example.


```python
def query_collection(collection, query, max_results, dataframe):
    results = collection.query(query_texts=query, n_results=max_results, include=['distances']) 
    df = pd.DataFrame({
                'id':results['ids'][0], 
                'score':results['distances'][0],
                'title': dataframe[dataframe.vector_id.isin(results['ids'][0])]['title'],
                'content': dataframe[dataframe.vector_id.isin(results['ids'][0])]['text'],
                })
    
    return df
```


```python
title_query_result = query_collection(
    collection=wikipedia_title_collection,
    query="modern art in Europe",
    max_results=10,
    dataframe=article_df
)
title_query_result.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>score</th>
      <th>title</th>
      <th>content</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>2</th>
      <td>23266</td>
      <td>0.249646</td>
      <td>Art</td>
      <td>Art is a creative activity that expresses imag...</td>
    </tr>
    <tr>
      <th>11777</th>
      <td>15436</td>
      <td>0.271688</td>
      <td>Hellenistic art</td>
      <td>The art of the Hellenistic time (from 400 B.C....</td>
    </tr>
    <tr>
      <th>12178</th>
      <td>23265</td>
      <td>0.279306</td>
      <td>Byzantine art</td>
      <td>Byzantine art is a form of Christian Greek art...</td>
    </tr>
    <tr>
      <th>13215</th>
      <td>11777</td>
      <td>0.294415</td>
      <td>Art film</td>
      <td>Art films are a type of movie that is very dif...</td>
    </tr>
    <tr>
      <th>15436</th>
      <td>22108</td>
      <td>0.305937</td>
      <td>Renaissance art</td>
      <td>Many of the most famous and best-loved works o...</td>
    </tr>
  </tbody>
</table>
</div>


key: _node_content
string_value:
{"id_": "07521441-9c66-40e5-8674-f72e8831710e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5fc992a7-eca1-450f-810a-51caf93110e8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e246b2bf-b85e-4dd7-b6e5-91a29faba2ff", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "15f1d4fcb09d1998d0efb4b49cbb3c4072d5dd45a01649a0304ada84c6a7d110", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30517, "end_char_idx": 32346, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
content_query_result = query_collection(
    collection=wikipedia_content_collection,
    query="Famous battles in Scottish history",
    max_results=10,
    dataframe=article_df
)
content_query_result.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>score</th>
      <th>title</th>
      <th>content</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>2923</th>
      <td>13135</td>
      <td>0.261328</td>
      <td>1651</td>
      <td>\n\nEvents \n January 1 – Charles II crowned K...</td>
    </tr>
    <tr>
      <th>3694</th>
      <td>13571</td>
      <td>0.277058</td>
      <td>Stirling</td>
      <td>Stirling () is a city in the middle of Scotlan...</td>
    </tr>
    <tr>
      <th>6248</th>
      <td>2923</td>
      <td>0.294823</td>
      <td>841</td>
      <td>\n\nEvents \n June 25: Battle of Fontenay – Lo...</td>
    </tr>
    <tr>
      <th>6297</th>
      <td>13568</td>
      <td>0.300756</td>
      <td>1746</td>
      <td>\n\nEvents \n January 8 – Bonnie Prince Charli...</td>
    </tr>
    <tr>
      <th>11702</th>
      <td>11708</td>
      <td>0.307572</td>
      <td>William Wallace</td>
      <td>William Wallace was a Scottish knight who foug...</td>
    </tr>
  </tbody>
</table>
</div>



Now that you've got a basic embeddings search running, you can [hop over to the Chroma docs](https://docs.trychroma.com/usage-guide#using-where-filters) to learn more about how to add filters to your query, update/delete data in your collections, and deploy Chroma.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb


key: _node_content
string_value:
{"id_": "81f60d20-4843-497c-ab02-aca94efef3a0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fa541af6-1f04-44f8-8d53-438b9e8fd1d6", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c85a37b9-e54d-4bfd-a9f2-2a9a53b614cd", "node_type": "1", "metadata": {}, "hash": "2a683a846ba84360e06402d30d14238dd58a09a52d66cff465ff09bba1bf294b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3231, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Robust Question Answering with Chroma and OpenAI 

This notebook guides you step-by-step through answering questions about a collection of data, using [Chroma](https://trychroma.com), an open-source embeddings database, along with OpenAI's [text embeddings](https://platform.openai.com/docs/guides/embeddings/use-cases) and [chat completion](https://platform.openai.com/docs/guides/chat) API's. 

Additionally, this notebook demonstrates some of the tradeoffs in making a question answering system more robust. As we shall see, *simple querying doesn't always create the best results*! 

## Question Answering with LLMs

Large language models (LLMs) like OpenAI's ChatGPT can be used to answer questions about data that the model may not have been trained on, or have access to. For example;

- Personal data like e-mails and notes
- Highly specialized data like archival or legal documents
- Newly created data like recent news stories

In order to overcome this limitation, we can use a data store which is amenable to querying in natural language, just like the LLM itself. An embeddings store like Chroma represents documents as [embeddings](https://openai.com/blog/introducing-text-and-code-embeddings), alongside the documents themselves. 

By embedding a text query, Chroma can find relevant documents, which we can then pass to the LLM to answer our question. We'll show detailed examples and variants of this approach. 

# Setup and preliminaries

First we make sure the python dependencies we need are installed. 


```python
%pip install -qU openai chromadb pandas
```

    Note: you may need to restart the kernel to use updated packages.


We use OpenAI's API's throughout this notebook. You can get an API key from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys)

You can add your API key as an environment variable by executing the command `export OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx` in a terminal. Note that you will need to reload the notebook if the environment variable wasn't set yet. Alternatively, you can set it in the notebook, see below. 


```python
import os

# Uncomment the following line to set the environment variable in the notebook
# os.environ["OPENAI_API_KEY"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

if os.getenv("OPENAI_API_KEY") is not None:
    print("OPENAI_API_KEY is ready")
    import openai
    openai.api_key = os.getenv("OPENAI_API_KEY")
else:
    print("OPENAI_API_KEY environment variable not found")
```

    OPENAI_API_KEY is ready


# Dataset

Throughout this notebook, we use the [SciFact dataset](https://github.com/allenai/scifact). This is a curated dataset of expert annotated scientific claims, with an accompanying text corpus of paper titles and abstracts. Each claim may be supported, contradicted, or not have enough evidence either way, according to the documents in the corpus. 

Having the corpus available as ground-truth allows us to investigate how well the following approaches to LLM question answering perform. 


```python
# Load the claim dataset
import pandas as pd

data_path = '../../data'

claim_df = pd.read_json(f'{data_path}/scifact_claims.jsonl', lines=True)
claim_df.head()
```


key: _node_content
string_value:
{"id_": "c85a37b9-e54d-4bfd-a9f2-2a9a53b614cd", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fa541af6-1f04-44f8-8d53-438b9e8fd1d6", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "81f60d20-4843-497c-ab02-aca94efef3a0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e5dad5a280180581156d07349f86803495a87fe18d2cc48935707f2548b0f9ca", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0abce852-ad77-484f-980f-94c3ef30fbd3", "node_type": "1", "metadata": {}, "hash": "266c03c3164aa015d5aee257ec93c88a5e2583aa158dfe4fe1195110e1268966", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1528, "end_char_idx": 4626, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
%pip install -qU openai chromadb pandas
```

    Note: you may need to restart the kernel to use updated packages.


We use OpenAI's API's throughout this notebook. You can get an API key from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys)

You can add your API key as an environment variable by executing the command `export OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx` in a terminal. Note that you will need to reload the notebook if the environment variable wasn't set yet. Alternatively, you can set it in the notebook, see below. 


```python
import os

# Uncomment the following line to set the environment variable in the notebook
# os.environ["OPENAI_API_KEY"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

if os.getenv("OPENAI_API_KEY") is not None:
    print("OPENAI_API_KEY is ready")
    import openai
    openai.api_key = os.getenv("OPENAI_API_KEY")
else:
    print("OPENAI_API_KEY environment variable not found")
```

    OPENAI_API_KEY is ready


# Dataset

Throughout this notebook, we use the [SciFact dataset](https://github.com/allenai/scifact). This is a curated dataset of expert annotated scientific claims, with an accompanying text corpus of paper titles and abstracts. Each claim may be supported, contradicted, or not have enough evidence either way, according to the documents in the corpus. 

Having the corpus available as ground-truth allows us to investigate how well the following approaches to LLM question answering perform. 


```python
# Load the claim dataset
import pandas as pd

data_path = '../../data'

claim_df = pd.read_json(f'{data_path}/scifact_claims.jsonl', lines=True)
claim_df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>claim</th>
      <th>evidence</th>
      <th>cited_doc_ids</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>0-dimensional biomaterials show inductive prop...</td>
      <td>{}</td>
      <td>[31715818]</td>
    </tr>
    <tr>
      <th>1</th>
      <td>3</td>
      <td>1,000 genomes project enables mapping of genet...</td>
      <td>{'14717500': [{'sentences': [2, 5], 'label': '...</td>
      <td>[14717500]</td>
    </tr>
    <tr>
      <th>2</th>
      <td>5</td>
      <td>1/2000 in UK have abnormal PrP positivity.</td>
      <td>{'13734012': [{'sentences': [4], 'label': 'SUP...</td>
      <td>[13734012]</td>
    </tr>
    <tr>
      <th>3</th>
      <td>13</td>
      <td>5% of perinatal mortality is due to low birth ...</td>
      <td>{}</td>
      <td>[1606628]</td>
    </tr>
    <tr>
      <th>4</th>
      <td>36</td>
      <td>A deficiency of vitamin B12 increases blood le...</td>
      <td>{}</td>
      <td>[5152028, 11705328]</td>
    </tr>
  </tbody>
</table>
</div>


key: _node_content
string_value:
{"id_": "0abce852-ad77-484f-980f-94c3ef30fbd3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fa541af6-1f04-44f8-8d53-438b9e8fd1d6", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c85a37b9-e54d-4bfd-a9f2-2a9a53b614cd", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "68c24ecd4ade173279c9351091e77c19fb71d14bb1aae504c4aac90a6a212f07", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2d2d0d96-ad8b-46a1-a7f7-5c0556427402", "node_type": "1", "metadata": {}, "hash": "355fbc1e7a71331fb5247d3623d9b8fb0538274ed8dba5cc363dcba2fd1c0cbf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4630, "end_char_idx": 7216, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Just asking the model

GPT-3.5 was trained on a large amount of scientific information. As a baseline, we'd like to understand what the model already knows without any further context. This will allow us to calibrate overall performance. 

We construct an appropriate prompt, with some example facts, then query the model with each claim in the dataset. We ask the model to assess a claim as 'True', 'False', or 'NEE' if there is not enough evidence one way or the other. 


```python
def build_prompt(claim):
    return [
        {"role": "system", "content": "I will ask you to assess a scientific claim. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence."},
        {"role": "user", "content": f"""        
Example:

Claim:
0-dimensional biomaterials show inductive properties.

Assessment:
False

Claim:
1/2000 in UK have abnormal PrP positivity.

Assessment:
True

Claim:
Aspirin inhibits the production of PGE2.

Assessment:
False

End of examples. Assess the following claim:

Claim:
{claim}

Assessment:
"""}
    ]


def assess_claims(claims):
    responses = []
    # Query the OpenAI API
    for claim in claims:
        response = openai.ChatCompletion.create(
            model='gpt-3.5-turbo',
            messages=build_prompt(claim),
            max_tokens=3,
        )
        # Strip any punctuation or whitespace from the response
        responses.append(response.choices[0].message.content.strip('., '))

    return responses
```

We sample 100 claims from the dataset


```python
# Let's take a look at 100 claims
samples = claim_df.sample(50)

claims = samples['claim'].tolist() 

```

We evaluate the ground-truth according to the dataset. From the dataset description, each claim is either supported or contradicted by the evidence, or else there isn't enough evidence either way. 


```python
def get_groundtruth(evidence):
    groundtruth = []
    for e in evidence:
        # Evidence is empty 
        if len(e) == 0:
            groundtruth.append('NEE')
        else:
            # In this dataset, all evidence for a given claim is consistent, either SUPPORT or CONTRADICT
            if list(e.values())[0][0]['label'] == 'SUPPORT':
                groundtruth.append('True')
            else:
                groundtruth.append('False')
    return groundtruth
```


```python
evidence = samples['evidence'].tolist()
groundtruth = get_groundtruth(evidence)
```

We also output the confusion matrix, comparing the model's assessments with the ground truth, in an easy to read table.


key: _node_content
string_value:
{"id_": "2d2d0d96-ad8b-46a1-a7f7-5c0556427402", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fa541af6-1f04-44f8-8d53-438b9e8fd1d6", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0abce852-ad77-484f-980f-94c3ef30fbd3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1353bc944e06942d6f92ed95dc0e4a2aa4d95ff79e82e6bcf6c69da7b0ff93fb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "830427fb-a32b-4b4f-80fe-ce967bf99b99", "node_type": "1", "metadata": {}, "hash": "2585e3d072f75550210dd85dba3e086b3331ed428f25dc567392fffc97703a33", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5728, "end_char_idx": 8574, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
def assess_claims(claims):
    responses = []
    # Query the OpenAI API
    for claim in claims:
        response = openai.ChatCompletion.create(
            model='gpt-3.5-turbo',
            messages=build_prompt(claim),
            max_tokens=3,
        )
        # Strip any punctuation or whitespace from the response
        responses.append(response.choices[0].message.content.strip('., '))

    return responses
```

We sample 100 claims from the dataset


```python
# Let's take a look at 100 claims
samples = claim_df.sample(50)

claims = samples['claim'].tolist() 

```

We evaluate the ground-truth according to the dataset. From the dataset description, each claim is either supported or contradicted by the evidence, or else there isn't enough evidence either way. 


```python
def get_groundtruth(evidence):
    groundtruth = []
    for e in evidence:
        # Evidence is empty 
        if len(e) == 0:
            groundtruth.append('NEE')
        else:
            # In this dataset, all evidence for a given claim is consistent, either SUPPORT or CONTRADICT
            if list(e.values())[0][0]['label'] == 'SUPPORT':
                groundtruth.append('True')
            else:
                groundtruth.append('False')
    return groundtruth
```


```python
evidence = samples['evidence'].tolist()
groundtruth = get_groundtruth(evidence)
```

We also output the confusion matrix, comparing the model's assessments with the ground truth, in an easy to read table. 


```python
def confusion_matrix(inferred, groundtruth):
    assert len(inferred) == len(groundtruth)
    confusion = {
        'True': {'True': 0, 'False': 0, 'NEE': 0},
        'False': {'True': 0, 'False': 0, 'NEE': 0},
        'NEE': {'True': 0, 'False': 0, 'NEE': 0},
    }
    for i, g in zip(inferred, groundtruth):
        confusion[i][g] += 1

    # Pretty print the confusion matrix
    print('\tGroundtruth')
    print('\tTrue\tFalse\tNEE')
    for i in confusion:
        print(i, end='\t')
        for g in confusion[i]:
            print(confusion[i][g], end='\t')
        print()

    return confusion
```

We ask the model to directly assess the claims, without additional context. 


```python
gpt_inferred = assess_claims(claims)
confusion_matrix(gpt_inferred, groundtruth)
```

    	Groundtruth
    	True	False	NEE
    True	15	5	14	
    False	0	2	1	
    NEE	3	3	7	





    {'True': {'True': 15, 'False': 5, 'NEE': 14},
     'False': {'True': 0, 'False': 2, 'NEE': 1},
     'NEE': {'True': 3, 'False': 3, 'NEE': 7}}



## Results

From these results we see that the LLM is strongly biased to assess claims as true, even when they are false, and also tends to assess false claims as not having enough evidence. Note that 'not enough evidence' is with respect to the model's assessment of the claim in a vacuum, without additional context.


key: _node_content
string_value:
{"id_": "830427fb-a32b-4b4f-80fe-ce967bf99b99", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fa541af6-1f04-44f8-8d53-438b9e8fd1d6", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2d2d0d96-ad8b-46a1-a7f7-5c0556427402", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "65855cb9c8543815167a0c2902bf9fc5a5f8d6bf925dfee8644a570afe3d979b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "916a73e2-7c1e-41c2-946e-112e8c6c3075", "node_type": "1", "metadata": {}, "hash": "dbcd26f0518ab002681e4ec6edc0956a9caf6d5ea8419af30b57c73648576cf6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7919, "end_char_idx": 8951, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
gpt_inferred = assess_claims(claims)
confusion_matrix(gpt_inferred, groundtruth)
```

    	Groundtruth
    	True	False	NEE
    True	15	5	14	
    False	0	2	1	
    NEE	3	3	7	





    {'True': {'True': 15, 'False': 5, 'NEE': 14},
     'False': {'True': 0, 'False': 2, 'NEE': 1},
     'NEE': {'True': 3, 'False': 3, 'NEE': 7}}



## Results

From these results we see that the LLM is strongly biased to assess claims as true, even when they are false, and also tends to assess false claims as not having enough evidence. Note that 'not enough evidence' is with respect to the model's assessment of the claim in a vacuum, without additional context.


# Adding context 

We now add the additional context available from the corpus of paper titles and abstracts. This section shows how to load a text corpus into Chroma, using OpenAI text embeddings. 

First, we load the text corpus. 


```python
# Load the corpus into a dataframe
corpus_df = pd.read_json(f'{data_path}/scifact_corpus.jsonl', lines=True)
corpus_df.head()
```


key: _node_content
string_value:
{"id_": "916a73e2-7c1e-41c2-946e-112e8c6c3075", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fa541af6-1f04-44f8-8d53-438b9e8fd1d6", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "830427fb-a32b-4b4f-80fe-ce967bf99b99", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e2705f8b9280fe72f564b5247c116abc6f1866ebcd2e6ea120aec0d1c98e5e3a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3e69568d-76af-40ab-9174-690d684748af", "node_type": "1", "metadata": {}, "hash": "76197a26079d0781a448b68ca74f4b47cd67db1631ef0f0f6dc0a516ac4e0d45", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7919, "end_char_idx": 10477, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
gpt_inferred = assess_claims(claims)
confusion_matrix(gpt_inferred, groundtruth)
```

    	Groundtruth
    	True	False	NEE
    True	15	5	14	
    False	0	2	1	
    NEE	3	3	7	





    {'True': {'True': 15, 'False': 5, 'NEE': 14},
     'False': {'True': 0, 'False': 2, 'NEE': 1},
     'NEE': {'True': 3, 'False': 3, 'NEE': 7}}



## Results

From these results we see that the LLM is strongly biased to assess claims as true, even when they are false, and also tends to assess false claims as not having enough evidence. Note that 'not enough evidence' is with respect to the model's assessment of the claim in a vacuum, without additional context.


# Adding context 

We now add the additional context available from the corpus of paper titles and abstracts. This section shows how to load a text corpus into Chroma, using OpenAI text embeddings. 

First, we load the text corpus. 


```python
# Load the corpus into a dataframe
corpus_df = pd.read_json(f'{data_path}/scifact_corpus.jsonl', lines=True)
corpus_df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>doc_id</th>
      <th>title</th>
      <th>abstract</th>
      <th>structured</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>4983</td>
      <td>Microstructural development of human newborn c...</td>
      <td>[Alterations of the architecture of cerebral w...</td>
      <td>False</td>
    </tr>
    <tr>
      <th>1</th>
      <td>5836</td>
      <td>Induction of myelodysplasia by myeloid-derived...</td>
      <td>[Myelodysplastic syndromes (MDS) are age-depen...</td>
      <td>False</td>
    </tr>
    <tr>
      <th>2</th>
      <td>7912</td>
      <td>BC1 RNA, the transcript from a master gene for...</td>
      <td>[ID elements are short interspersed elements (...</td>
      <td>False</td>
    </tr>
    <tr>
      <th>3</th>
      <td>18670</td>
      <td>The DNA Methylome of Human Peripheral Blood Mo...</td>
      <td>[DNA methylation plays an important role in bi...</td>
      <td>False</td>
    </tr>
    <tr>
      <th>4</th>
      <td>19238</td>
      <td>The human myelin basic protein gene is include...</td>
      <td>[Two human Golli (for gene expressed in the ol...</td>
      <td>False</td>
    </tr>
  </tbody>
</table>
</div>


key: _node_content
string_value:
{"id_": "3e69568d-76af-40ab-9174-690d684748af", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fa541af6-1f04-44f8-8d53-438b9e8fd1d6", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "916a73e2-7c1e-41c2-946e-112e8c6c3075", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "70b42dac93d846ba1ddf24ce895f1bb40d71e03fc8f46791f42dfca237e69716", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "136135ab-7b28-4c16-b257-2b55fca19836", "node_type": "1", "metadata": {}, "hash": "4141e1ba657640c16d1e19319fd2889088058dddfcd42bf528197d0f5098e009", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10481, "end_char_idx": 13640, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Loading the corpus into Chroma

The next step is to load the corpus into Chroma. Given an embedding function, Chroma will automatically handle embedding each document, and will store it alongside its text and metadata, making it simple to query.

We instantiate a (ephemeral) Chroma client, and create a collection for the SciFact title and abstract corpus. 
Chroma can also be instantiated in a persisted configuration; learn more at the [Chroma docs](https://docs.trychroma.com/usage-guide?lang=py). 


```python
import chromadb
from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction

# We initialize an embedding function, and provide it to the collection.
embedding_function = OpenAIEmbeddingFunction(api_key=os.getenv("OPENAI_API_KEY"))

chroma_client = chromadb.Client() # Ephemeral by default
scifact_corpus_collection = chroma_client.create_collection(name='scifact_corpus', embedding_function=embedding_function)
```

    Running Chroma using direct local API.
    Using DuckDB in-memory for database. Data will be transient.


Next we load the corpus into Chroma. Because this data loading is memory intensive, we recommend using a batched loading scheme in batches of 50-1000. For this example it should take just over one minute for the entire corpus. It's being embedded in the background, automatically, using the `embedding_function` we specified earlier.


```python
batch_size = 100

for i in range(0, len(corpus_df), batch_size):
    batch_df = corpus_df[i:i+batch_size]
    scifact_corpus_collection.add(
        ids=batch_df['doc_id'].apply(lambda x: str(x)).tolist(), # Chroma takes string IDs.
        documents=(batch_df['title'] + '. ' + batch_df['abstract'].apply(lambda x: ' '.join(x))).to_list(), # We concatenate the title and abstract.
        metadatas=[{"structured": structured} for structured in batch_df['structured'].to_list()] # We also store the metadata, though we don't use it in this example.
    )
```

## Retrieving context

Next we retrieve documents from the corpus which may be relevant to each claim in our sample. We want to provide these as context to the LLM for evaluating the claims. We retrieve the 3 most relevant documents for each claim, according to the embedding distance. 


```python
claim_query_result = scifact_corpus_collection.query(query_texts=claims, include=['documents', 'distances'], n_results=3)
```

We create a new prompt, this time taking into account the additional context we retrieve from the corpus. 


```python
def build_prompt_with_context(claim, context):
    return [{'role': 'system', 'content': "I will ask you to assess whether a particular scientific claim, based on evidence provided. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence."}, 
            {'role': 'user', 'content': f""""
The evidence is the following:

{' '.join(context)}

Assess the following claim on the basis of the evidence. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence. Do not output any other text. 

Claim:
{claim}

Assessment:
"""}]


key: _node_content
string_value:
{"id_": "136135ab-7b28-4c16-b257-2b55fca19836", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fa541af6-1f04-44f8-8d53-438b9e8fd1d6", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3e69568d-76af-40ab-9174-690d684748af", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4c5a5cca7fac41d07e1d3db95a8568244b9a239bf3b430fd4ebb19a8dfa5a1cc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5e5a2fd2-4b27-4211-a316-6f707cb87660", "node_type": "1", "metadata": {}, "hash": "0628ac2ec49633ccfa9cdc83edaaf9ad092c43126c21199ecdd621dbcda51b1d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12733, "end_char_idx": 15739, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
claim_query_result = scifact_corpus_collection.query(query_texts=claims, include=['documents', 'distances'], n_results=3)
```

We create a new prompt, this time taking into account the additional context we retrieve from the corpus. 


```python
def build_prompt_with_context(claim, context):
    return [{'role': 'system', 'content': "I will ask you to assess whether a particular scientific claim, based on evidence provided. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence."}, 
            {'role': 'user', 'content': f""""
The evidence is the following:

{' '.join(context)}

Assess the following claim on the basis of the evidence. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence. Do not output any other text. 

Claim:
{claim}

Assessment:
"""}]


def assess_claims_with_context(claims, contexts):
    responses = []
    # Query the OpenAI API
    for claim, context in zip(claims, contexts):
        # If no evidence is provided, return NEE
        if len(context) == 0:
            responses.append('NEE')
            continue
        response = openai.ChatCompletion.create(
            model='gpt-3.5-turbo',
            messages=build_prompt_with_context(claim=claim, context=context),
            max_tokens=3,
        )
        # Strip any punctuation or whitespace from the response
        responses.append(response.choices[0].message.content.strip('., '))

    return responses
```

Then ask the model to evaluate the claims with the retrieved context. 


```python
gpt_with_context_evaluation = assess_claims_with_context(claims, claim_query_result['documents'])
confusion_matrix(gpt_with_context_evaluation, groundtruth)
```

    	Groundtruth
    	True	False	NEE
    True	16	2	8	
    False	1	6	5	
    NEE	1	2	9	





    {'True': {'True': 16, 'False': 2, 'NEE': 8},
     'False': {'True': 1, 'False': 6, 'NEE': 5},
     'NEE': {'True': 1, 'False': 2, 'NEE': 9}}



## Results

We see that the model is a lot less likely to evaluate a False claim as true (2 instances VS 5 previously), but that claims without enough evidence are still often assessed as True or False.

Taking a look at the retrieved documents, we see that they are sometimes not relevant to the claim - this causes the model to be confused by the extra information, and it may decide that sufficient evidence is present, even when the information is irrelevant. This happens because we always ask for the 3 'most' relevant documents, but these might not be relevant at all beyond a certain point.  

## Filtering context on relevance

Along with the documents themselves, Chroma returns a distance score. We can try thresholding on distance, so that fewer irrelevant documents make it into the context we provide the model. 

If, after filtering on the threshold, no context documents remain, we bypass the model and simply return that there is not enough evidence.


key: _node_content
string_value:
{"id_": "5e5a2fd2-4b27-4211-a316-6f707cb87660", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fa541af6-1f04-44f8-8d53-438b9e8fd1d6", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "136135ab-7b28-4c16-b257-2b55fca19836", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a2034b8a20f8446978940ff03e81095fa0b76e455ed1e0671ad3cbf96ccbc2f1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "43e766bb-2805-4caa-9043-88817f100767", "node_type": "1", "metadata": {}, "hash": "0debb42270b184f1c2c79545a873540128782d62b2012be14f939979e8a2e7a5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14361, "end_char_idx": 16808, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
gpt_with_context_evaluation = assess_claims_with_context(claims, claim_query_result['documents'])
confusion_matrix(gpt_with_context_evaluation, groundtruth)
```

    	Groundtruth
    	True	False	NEE
    True	16	2	8	
    False	1	6	5	
    NEE	1	2	9	





    {'True': {'True': 16, 'False': 2, 'NEE': 8},
     'False': {'True': 1, 'False': 6, 'NEE': 5},
     'NEE': {'True': 1, 'False': 2, 'NEE': 9}}



## Results

We see that the model is a lot less likely to evaluate a False claim as true (2 instances VS 5 previously), but that claims without enough evidence are still often assessed as True or False.

Taking a look at the retrieved documents, we see that they are sometimes not relevant to the claim - this causes the model to be confused by the extra information, and it may decide that sufficient evidence is present, even when the information is irrelevant. This happens because we always ask for the 3 'most' relevant documents, but these might not be relevant at all beyond a certain point.  

## Filtering context on relevance

Along with the documents themselves, Chroma returns a distance score. We can try thresholding on distance, so that fewer irrelevant documents make it into the context we provide the model. 

If, after filtering on the threshold, no context documents remain, we bypass the model and simply return that there is not enough evidence. 


```python
def filter_query_result(query_result, distance_threshold=0.25):
# For each query result, retain only the documents whose distance is below the threshold
    for ids, docs, distances in zip(query_result['ids'], query_result['documents'], query_result['distances']):
        for i in range(len(ids)-1, -1, -1):
            if distances[i] > distance_threshold:
                ids.pop(i)
                docs.pop(i)
                distances.pop(i)
    return query_result

```


```python
filtered_claim_query_result = filter_query_result(claim_query_result)
```

Now we assess the claims using this cleaner context. 


```python
gpt_with_filtered_context_evaluation = assess_claims_with_context(claims, filtered_claim_query_result['documents'])
confusion_matrix(gpt_with_filtered_context_evaluation, groundtruth)
```

    	Groundtruth
    	True	False	NEE
    True	10	2	1	
    False	0	2	1	
    NEE	8	6	20	





    {'True': {'True': 10, 'False': 2, 'NEE': 1},
     'False': {'True': 0, 'False': 2, 'NEE': 1},
     'NEE': {'True': 8, 'False': 6, 'NEE': 20}}


key: _node_content
string_value:
{"id_": "43e766bb-2805-4caa-9043-88817f100767", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fa541af6-1f04-44f8-8d53-438b9e8fd1d6", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5e5a2fd2-4b27-4211-a316-6f707cb87660", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4cf8d2f39e5c8c2728a6728861bd1206e9ac12b6b3652c3e533cad19a75c30c1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "49ebb8ad-4eac-4311-b927-842c9f3ac5ac", "node_type": "1", "metadata": {}, "hash": "725e8f3d5074fb077f7affdec9d8e9d486e4728c381e176262d779f22aaa5afd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15743, "end_char_idx": 18984, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def filter_query_result(query_result, distance_threshold=0.25):
# For each query result, retain only the documents whose distance is below the threshold
    for ids, docs, distances in zip(query_result['ids'], query_result['documents'], query_result['distances']):
        for i in range(len(ids)-1, -1, -1):
            if distances[i] > distance_threshold:
                ids.pop(i)
                docs.pop(i)
                distances.pop(i)
    return query_result

```


```python
filtered_claim_query_result = filter_query_result(claim_query_result)
```

Now we assess the claims using this cleaner context. 


```python
gpt_with_filtered_context_evaluation = assess_claims_with_context(claims, filtered_claim_query_result['documents'])
confusion_matrix(gpt_with_filtered_context_evaluation, groundtruth)
```

    	Groundtruth
    	True	False	NEE
    True	10	2	1	
    False	0	2	1	
    NEE	8	6	20	





    {'True': {'True': 10, 'False': 2, 'NEE': 1},
     'False': {'True': 0, 'False': 2, 'NEE': 1},
     'NEE': {'True': 8, 'False': 6, 'NEE': 20}}



## Results

The model now assesses many fewer claims as True or False when there is not enough evidence present. However, it now biases away from certainty. Most claims are now assessed as having not enough evidence, because a large fraction of them are filtered out by the distance threshold. It's possible to tune the distance threshold to find the optimal operating point, but this can be difficult, and is dataset and embedding model dependent. 

# Hypothetical Document Embeddings: Using hallucinations productively

We want to be able to retrieve relevant documents, without retrieving less relevant ones which might confuse the model. One way to accomplish this is to improve the retrieval query. 

Until now, we have queried the dataset using _claims_ which are single sentence statements, while the corpus contains _abstracts_ describing a scientific paper. Intuitively, while these might be related, there are significant differences in their structure and meaning. These differences are encoded by the embedding model, and so influence the distances between the query and the most relevant results. 

We can overcome this by leveraging the power of LLMs to generate relevant text. While the facts might be hallucinated, the content and structure of the documents the models generate is more similar to the documents in our corpus, than the queries are. This could lead to better queries and hence better results. 

This approach is called [Hypothetical Document Embeddings (HyDE)](https://arxiv.org/abs/2212.10496), and has been shown to be quite good at the retrieval task. It should help us bring more relevant information into the context, without polluting it.  

TL;DR:
- you get much better matches when you embed whole abstracts rather than single sentences
- but claims are usually single sentences
- So HyDE shows that using GPT3 to expand claims into hallucinated abstracts and then searching based on those abstracts works (claims -> abstracts -> results) better than searching directly (claims -> results)

First, we use in-context examples to prompt the model to generate documents similar to what's in the corpus, for each claim we want to assess.


key: _node_content
string_value:
{"id_": "49ebb8ad-4eac-4311-b927-842c9f3ac5ac", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fa541af6-1f04-44f8-8d53-438b9e8fd1d6", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "43e766bb-2805-4caa-9043-88817f100767", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "16d3c426bc313b207cfe8f5e8df5dc47b41e085a89aa8ae9c8e5e4e3edc801e9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "82c74463-c40c-415d-a948-45e6283df9ea", "node_type": "1", "metadata": {}, "hash": "3ce52a35ffd3ed9ac882558ad576d3512cb63f01ed926b3e9164054a92403d5f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18988, "end_char_idx": 22163, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def build_hallucination_prompt(claim):
    return [{'role': 'system', 'content': """I will ask you to write an abstract for a scientific paper which supports or refutes a given claim. It should be written in scientific language, include a title. Output only one abstract, then stop.
    
    An Example:

    Claim:
    A high microerythrocyte count raises vulnerability to severe anemia in homozygous alpha (+)- thalassemia trait subjects.

    Abstract:
    BACKGROUND The heritable haemoglobinopathy alpha(+)-thalassaemia is caused by the reduced synthesis of alpha-globin chains that form part of normal adult haemoglobin (Hb). Individuals homozygous for alpha(+)-thalassaemia have microcytosis and an increased erythrocyte count. Alpha(+)-thalassaemia homozygosity confers considerable protection against severe malaria, including severe malarial anaemia (SMA) (Hb concentration < 50 g/l), but does not influence parasite count. We tested the hypothesis that the erythrocyte indices associated with alpha(+)-thalassaemia homozygosity provide a haematological benefit during acute malaria.   
    METHODS AND FINDINGS Data from children living on the north coast of Papua New Guinea who had participated in a case-control study of the protection afforded by alpha(+)-thalassaemia against severe malaria were reanalysed to assess the genotype-specific reduction in erythrocyte count and Hb levels associated with acute malarial disease. We observed a reduction in median erythrocyte count of approximately 1.5 x 10(12)/l in all children with acute falciparum malaria relative to values in community children (p < 0.001). We developed a simple mathematical model of the linear relationship between Hb concentration and erythrocyte count. This model predicted that children homozygous for alpha(+)-thalassaemia lose less Hb than children of normal genotype for a reduction in erythrocyte count of >1.1 x 10(12)/l as a result of the reduced mean cell Hb in homozygous alpha(+)-thalassaemia. In addition, children homozygous for alpha(+)-thalassaemia require a 10% greater reduction in erythrocyte count than children of normal genotype (p = 0.02) for Hb concentration to fall to 50 g/l, the cutoff for SMA. We estimated that the haematological profile in children homozygous for alpha(+)-thalassaemia reduces the risk of SMA during acute malaria compared to children of normal genotype (relative risk 0.52; 95% confidence interval [CI] 0.24-1.12, p = 0.09).   
    CONCLUSIONS The increased erythrocyte count and microcytosis in children homozygous for alpha(+)-thalassaemia may contribute substantially to their protection against SMA. A lower concentration of Hb per erythrocyte and a larger population of erythrocytes may be a biologically advantageous strategy against the significant reduction in erythrocyte count that occurs during acute infection with the malaria parasite Plasmodium falciparum. This haematological profile may reduce the risk of anaemia by other Plasmodium species, as well as other causes of anaemia. Other host polymorphisms that induce an increased erythrocyte count and microcytosis may confer a similar advantage.

    End of example.


key: _node_content
string_value:
{"id_": "82c74463-c40c-415d-a948-45e6283df9ea", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fa541af6-1f04-44f8-8d53-438b9e8fd1d6", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "49ebb8ad-4eac-4311-b927-842c9f3ac5ac", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c725155626775468ccff264bfb16d4b7caaee7ebebe6cc2f91f148bf87509b1d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "816f56ed-e661-4244-9b19-c9a50e234614", "node_type": "1", "metadata": {}, "hash": "89ee72a4c1a935544a9770be131bcf7707e09d9833f115e340c7d1adc505a3f5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20622, "end_char_idx": 23668, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We developed a simple mathematical model of the linear relationship between Hb concentration and erythrocyte count. This model predicted that children homozygous for alpha(+)-thalassaemia lose less Hb than children of normal genotype for a reduction in erythrocyte count of >1.1 x 10(12)/l as a result of the reduced mean cell Hb in homozygous alpha(+)-thalassaemia. In addition, children homozygous for alpha(+)-thalassaemia require a 10% greater reduction in erythrocyte count than children of normal genotype (p = 0.02) for Hb concentration to fall to 50 g/l, the cutoff for SMA. We estimated that the haematological profile in children homozygous for alpha(+)-thalassaemia reduces the risk of SMA during acute malaria compared to children of normal genotype (relative risk 0.52; 95% confidence interval [CI] 0.24-1.12, p = 0.09).   
    CONCLUSIONS The increased erythrocyte count and microcytosis in children homozygous for alpha(+)-thalassaemia may contribute substantially to their protection against SMA. A lower concentration of Hb per erythrocyte and a larger population of erythrocytes may be a biologically advantageous strategy against the significant reduction in erythrocyte count that occurs during acute infection with the malaria parasite Plasmodium falciparum. This haematological profile may reduce the risk of anaemia by other Plasmodium species, as well as other causes of anaemia. Other host polymorphisms that induce an increased erythrocyte count and microcytosis may confer a similar advantage.

    End of example. 
    
    """}, {'role': 'user', 'content': f""""
    Perform the task for the following claim.

    Claim:
    {claim}

    Abstract:
    """}]


def hallucinate_evidence(claims):
    # Query the OpenAI API
    responses = []
    # Query the OpenAI API
    for claim in claims:
        response = openai.ChatCompletion.create(
            model='gpt-3.5-turbo',
            messages=build_hallucination_prompt(claim),
        )
        responses.append(response.choices[0].message.content)
    return responses
```

We hallucinate a document for each claim.

*NB: This can take a while, about 30m for 100 claims*. You can reduce the number of claims we want to assess to get results more quickly. 


```python
hallucinated_evidence = hallucinate_evidence(claims)
```

We use the hallucinated documents as queries into the corpus, and filter the results using the same distance threshold. 


```python
hallucinated_query_result = scifact_corpus_collection.query(query_texts=hallucinated_evidence, include=['documents', 'distances'], n_results=3)
filtered_hallucinated_query_result = filter_query_result(hallucinated_query_result)
```

We then ask the model to assess the claims, using the new context. 


```python
gpt_with_hallucinated_context_evaluation = assess_claims_with_context(claims, filtered_hallucinated_query_result['documents'])
confusion_matrix(gpt_with_hallucinated_context_evaluation, groundtruth)
```

    	Groundtruth
    	True	False	NEE
    True	15	2	5	
    False	1	5	4	
    NEE	2	3	13


key: _node_content
string_value:
{"id_": "816f56ed-e661-4244-9b19-c9a50e234614", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fa541af6-1f04-44f8-8d53-438b9e8fd1d6", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "82c74463-c40c-415d-a948-45e6283df9ea", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b2a33432a960093a8a6bfd74a74c1ea6ffb5ad60bb14553270d5bc28d560f2ea", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 22148, "end_char_idx": 24502, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
End of example. 
    
    """}, {'role': 'user', 'content': f""""
    Perform the task for the following claim.

    Claim:
    {claim}

    Abstract:
    """}]


def hallucinate_evidence(claims):
    # Query the OpenAI API
    responses = []
    # Query the OpenAI API
    for claim in claims:
        response = openai.ChatCompletion.create(
            model='gpt-3.5-turbo',
            messages=build_hallucination_prompt(claim),
        )
        responses.append(response.choices[0].message.content)
    return responses
```

We hallucinate a document for each claim.

*NB: This can take a while, about 30m for 100 claims*. You can reduce the number of claims we want to assess to get results more quickly. 


```python
hallucinated_evidence = hallucinate_evidence(claims)
```

We use the hallucinated documents as queries into the corpus, and filter the results using the same distance threshold. 


```python
hallucinated_query_result = scifact_corpus_collection.query(query_texts=hallucinated_evidence, include=['documents', 'distances'], n_results=3)
filtered_hallucinated_query_result = filter_query_result(hallucinated_query_result)
```

We then ask the model to assess the claims, using the new context. 


```python
gpt_with_hallucinated_context_evaluation = assess_claims_with_context(claims, filtered_hallucinated_query_result['documents'])
confusion_matrix(gpt_with_hallucinated_context_evaluation, groundtruth)
```

    	Groundtruth
    	True	False	NEE
    True	15	2	5	
    False	1	5	4	
    NEE	2	3	13	





    {'True': {'True': 15, 'False': 2, 'NEE': 5},
     'False': {'True': 1, 'False': 5, 'NEE': 4},
     'NEE': {'True': 2, 'False': 3, 'NEE': 13}}



## Results

Combining HyDE with a simple distance threshold leads to a significant improvement. The model no longer biases assessing claims as True, nor toward their not being enough evidence. It also correctly assesses when there isn't enough evidence more often.

# Conclusion

Equipping LLMs with a context based on a corpus of documents is a powerful technique for bringing the general reasoning and natural language interactions of LLMs to your own data. However, it's important to know that naive query and retrieval may not produce the best possible results! Ultimately understanding the data will help get the most out of the retrieval based question-answering approach.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb


key: _node_content
string_value:
{"id_": "3bcb320d-ecc2-4a8f-a3a0-48dd8a5b934f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bb8e566e-f3ea-4493-a947-9bbf426f140e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f27b9967-7bbb-42ee-8e46-909dfa99a2c0", "node_type": "1", "metadata": {}, "hash": "088acc09ac6d00e2f69d60f7eaec540a731a22f958b8c7d25f609591117eb096", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3626, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Using Redis for Embeddings Search

This notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.

### What is a Vector Database

A vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.

### Why use a Vector Database

Vector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.


### Demo Flow
The demo flow is:
- **Setup**: Import packages and set any required variables
- **Load data**: Load a dataset and embed it using OpenAI embeddings
- **Redis**
    - *Setup*: Set up the Redis-Py client. For more details go [here](https://github.com/redis/redis-py)
    - *Index Data*: Create the search index for vector search and hybrid search (vector + full-text search) on all available fields.
    - *Search Data*: Run a few example queries with various goals in mind.

Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.

## Setup

Import the required libraries and set the embedding model that we'd like to use.


```python
# We'll need to install the Redis client
!pip install redis

#Install wget to pull zip file
!pip install wget
```


```python
import openai

from typing import List, Iterator
import pandas as pd
import numpy as np
import os
import wget
from ast import literal_eval

# Redis client library for Python
import redis

# I've set this to our new embeddings model, this can be changed to the embedding model of your choice
EMBEDDING_MODEL = "text-embedding-3-small"

# Ignore unclosed SSL socket warnings - optional in case you get these errors
import warnings

warnings.filterwarnings(action="ignore", message="unclosed", category=ResourceWarning)
warnings.filterwarnings("ignore", category=DeprecationWarning) 
```

## Load data

In this section we'll load embedded data that we've prepared previous to this session.


```python
embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```


```python
import zipfile
with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip","r") as zip_ref:
    zip_ref.extractall("../data")
```


```python
article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')
```


```python
article_df.head()
```


key: _node_content
string_value:
{"id_": "f27b9967-7bbb-42ee-8e46-909dfa99a2c0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bb8e566e-f3ea-4493-a947-9bbf426f140e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3bcb320d-ecc2-4a8f-a3a0-48dd8a5b934f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "6cb0e5791b2675568000a729aced58af9bb81fb4436908ee7d55e1b5282e3bf8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a9aa3147-ccb1-45e9-a825-4e0befd63cac", "node_type": "1", "metadata": {}, "hash": "55d75d734c6c465942a4bbe99c9e2550ca1dde90d5af183d906f3fc23ae149c3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2312, "end_char_idx": 4642, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# We'll need to install the Redis client
!pip install redis

#Install wget to pull zip file
!pip install wget
```


```python
import openai

from typing import List, Iterator
import pandas as pd
import numpy as np
import os
import wget
from ast import literal_eval

# Redis client library for Python
import redis

# I've set this to our new embeddings model, this can be changed to the embedding model of your choice
EMBEDDING_MODEL = "text-embedding-3-small"

# Ignore unclosed SSL socket warnings - optional in case you get these errors
import warnings

warnings.filterwarnings(action="ignore", message="unclosed", category=ResourceWarning)
warnings.filterwarnings("ignore", category=DeprecationWarning) 
```

## Load data

In this section we'll load embedded data that we've prepared previous to this session.


```python
embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```


```python
import zipfile
with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip","r") as zip_ref:
    zip_ref.extractall("../data")
```


```python
article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')
```


```python
article_df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>url</th>
      <th>title</th>
      <th>text</th>
      <th>title_vector</th>
      <th>content_vector</th>
      <th>vector_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>https://simple.wikipedia.org/wiki/April</td>
      <td>April</td>
      <td>April is the fourth month of the year in the J...</td>
      <td>[0.001009464613161981, -0.020700545981526375, ...</td>
      <td>[-0.011253940872848034, -0.013491976074874401,...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>https://simple.wikipedia.org/wiki/August</td>
      <td>August</td>
      <td>August (Aug.)


key: _node_content
string_value:
{"id_": "a9aa3147-ccb1-45e9-a825-4e0befd63cac", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bb8e566e-f3ea-4493-a947-9bbf426f140e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f27b9967-7bbb-42ee-8e46-909dfa99a2c0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a689ea010c1bc5251888130f4573694442c5ee0d666e0b4fb260dcaf8e91fe5e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7ec25935-b096-4410-af4d-014dafd638dc", "node_type": "1", "metadata": {}, "hash": "18e7b60db5538910c0b015b09df347c6a085ba812f72bb29fa3eeccb225ebb90", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3495, "end_char_idx": 5673, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: _node_content
string_value:
{"id_": "7ec25935-b096-4410-af4d-014dafd638dc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bb8e566e-f3ea-4493-a947-9bbf426f140e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a9aa3147-ccb1-45e9-a825-4e0befd63cac", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ba98c8b990e273960d99a1a8b9c6fac5aa624908bdcb9d71eb7037bfc7172e54", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4a9f5ad3-51c1-4625-a793-16efba449e1a", "node_type": "1", "metadata": {}, "hash": "647a53a8136e9b26b4b4b4cefb65311c24b5a2c31beb16c74dc5582c5bf676be", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4643, "end_char_idx": 6972, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
is the eighth month of the year ...</td>
      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>
      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>6</td>
      <td>https://simple.wikipedia.org/wiki/Art</td>
      <td>Art</td>
      <td>Art is a creative activity that expresses imag...</td>
      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>
      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>
      <td>2</td>
    </tr>
    <tr>
      <th>3</th>
      <td>8</td>
      <td>https://simple.wikipedia.org/wiki/A</td>
      <td>A</td>
      <td>A or a is the first letter of the English alph...</td>
      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>
      <td>[0.024894846603274345, -0.022186409682035446, ...</td>
      <td>3</td>
    </tr>
    <tr>
      <th>4</th>
      <td>9</td>
      <td>https://simple.wikipedia.org/wiki/Air</td>
      <td>Air</td>
      <td>Air refers to the Earth's atmosphere. Air is a...</td>
      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>
      <td>[0.021524671465158463, 0.018522677943110466, -...</td>
      <td>4</td>
    </tr>
  </tbody>
</table>
</div>




```python
# Read vectors from strings back into a list
article_df['title_vector'] = article_df.title_vector.apply(literal_eval)
article_df['content_vector'] = article_df.content_vector.apply(literal_eval)

# Set vector_id to be a string
article_df['vector_id'] = article_df['vector_id'].apply(str)
```


```python
article_df.info(show_counts=True)
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 25000 entries, 0 to 24999
    Data columns (total 7 columns):
     #   Column          Non-Null Count  Dtype 
    ---  ------          --------------  ----- 
     0   id              25000 non-null  int64 
     1   url             25000 non-null  object
     2   title           25000 non-null  object
     3   text            25000 non-null  object
     4   title_vector    25000 non-null  object
     5   content_vector  25000 non-null  object
     6   vector_id       25000 non-null  object
    dtypes: int64(1), object(6)
    memory usage: 1.3+ MB


# Redis

The next vector database covered in this tutorial is **[Redis](https://redis.io)**. You most likely already know Redis.


key: _node_content
string_value:
{"id_": "4a9f5ad3-51c1-4625-a793-16efba449e1a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bb8e566e-f3ea-4493-a947-9bbf426f140e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7ec25935-b096-4410-af4d-014dafd638dc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c28052e04f52063da569dfeb941e12b763aed2cd054bf995ed1bacca8fbfbb15", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "93490e13-bb64-4019-af66-9e3e3c5afa46", "node_type": "1", "metadata": {}, "hash": "06777b422b8a962b9e3d3037f35aa5facb415c0cb6155bf970990a0f6ca82fbf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5674, "end_char_idx": 8400, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Air is a...</td>
      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>
      <td>[0.021524671465158463, 0.018522677943110466, -...</td>
      <td>4</td>
    </tr>
  </tbody>
</table>
</div>




```python
# Read vectors from strings back into a list
article_df['title_vector'] = article_df.title_vector.apply(literal_eval)
article_df['content_vector'] = article_df.content_vector.apply(literal_eval)

# Set vector_id to be a string
article_df['vector_id'] = article_df['vector_id'].apply(str)
```


```python
article_df.info(show_counts=True)
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 25000 entries, 0 to 24999
    Data columns (total 7 columns):
     #   Column          Non-Null Count  Dtype 
    ---  ------          --------------  ----- 
     0   id              25000 non-null  int64 
     1   url             25000 non-null  object
     2   title           25000 non-null  object
     3   text            25000 non-null  object
     4   title_vector    25000 non-null  object
     5   content_vector  25000 non-null  object
     6   vector_id       25000 non-null  object
    dtypes: int64(1), object(6)
    memory usage: 1.3+ MB


# Redis

The next vector database covered in this tutorial is **[Redis](https://redis.io)**. You most likely already know Redis. What you might not be aware of is the [RediSearch module](https://github.com/RediSearch/RediSearch). Enterprises have been using Redis with the RediSearch module for years now across all major cloud providers, Redis Cloud, and on premise. Recently, the Redis team added vector storage and search capability to this module in addition to the features RediSearch already had.

Given the large ecosystem around Redis, there are most likely client libraries in the language you need. You can use any standard Redis client library to run RediSearch commands, but it's easiest to use a library that wraps the RediSearch API. Below are a few examples, but you can find more client libraries [here](https://redis.io/resources/clients/).

| Project | Language | License | Author | Stars |
|----------|---------|--------|---------|-------|
| [jedis][jedis-url] | Java | MIT | [Redis][redis-url] | ![Stars][jedis-stars] |
| [redis-py][redis-py-url] | Python | MIT | [Redis][redis-url] | ![Stars][redis-py-stars] |
| [node-redis][node-redis-url] | Node.js | MIT | [Redis][redis-url] | ![Stars][node-redis-stars] |
| [nredisstack][nredisstack-url] | .NET | MIT | [Redis][redis-url] | ![Stars][nredisstack-stars] |
| [redisearch-go][redisearch-go-url] | Go | BSD | [Redis][redisearch-go-author] | [![redisearch-go-stars]][redisearch-go-url] |
| [redisearch-api-rs][redisearch-api-rs-url] | Rust | BSD | [Redis][redisearch-api-rs-author] | [!


key: _node_content
string_value:
{"id_": "93490e13-bb64-4019-af66-9e3e3c5afa46", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bb8e566e-f3ea-4493-a947-9bbf426f140e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4a9f5ad3-51c1-4625-a793-16efba449e1a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1ca1cfe187425171a779b3643ef7c9357ba9a22e6d91c49e34c0351af54af32d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4c5de6df-4b4e-44b5-9993-514729ea7267", "node_type": "1", "metadata": {}, "hash": "845a012f52e37a98b828c0f17f31fdfd1a68f9a6186ab60e3678e5e6f272f198", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6937, "end_char_idx": 10175, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You most likely already know Redis. What you might not be aware of is the [RediSearch module](https://github.com/RediSearch/RediSearch). Enterprises have been using Redis with the RediSearch module for years now across all major cloud providers, Redis Cloud, and on premise. Recently, the Redis team added vector storage and search capability to this module in addition to the features RediSearch already had.

Given the large ecosystem around Redis, there are most likely client libraries in the language you need. You can use any standard Redis client library to run RediSearch commands, but it's easiest to use a library that wraps the RediSearch API. Below are a few examples, but you can find more client libraries [here](https://redis.io/resources/clients/).

| Project | Language | License | Author | Stars |
|----------|---------|--------|---------|-------|
| [jedis][jedis-url] | Java | MIT | [Redis][redis-url] | ![Stars][jedis-stars] |
| [redis-py][redis-py-url] | Python | MIT | [Redis][redis-url] | ![Stars][redis-py-stars] |
| [node-redis][node-redis-url] | Node.js | MIT | [Redis][redis-url] | ![Stars][node-redis-stars] |
| [nredisstack][nredisstack-url] | .NET | MIT | [Redis][redis-url] | ![Stars][nredisstack-stars] |
| [redisearch-go][redisearch-go-url] | Go | BSD | [Redis][redisearch-go-author] | [![redisearch-go-stars]][redisearch-go-url] |
| [redisearch-api-rs][redisearch-api-rs-url] | Rust | BSD | [Redis][redisearch-api-rs-author] | [![redisearch-api-rs-stars]][redisearch-api-rs-url] |

[redis-url]: https://redis.com

[redis-py-url]: https://github.com/redis/redis-py
[redis-py-stars]: https://img.shields.io/github/stars/redis/redis-py.svg?style=social&amp;label=Star&amp;maxAge=2592000
[redis-py-package]: https://pypi.python.org/pypi/redis

[jedis-url]: https://github.com/redis/jedis
[jedis-stars]: https://img.shields.io/github/stars/redis/jedis.svg?style=social&amp;label=Star&amp;maxAge=2592000
[Jedis-package]: https://search.maven.org/artifact/redis.clients/jedis

[nredisstack-url]: https://github.com/redis/nredisstack
[nredisstack-stars]: https://img.shields.io/github/stars/redis/nredisstack.svg?style=social&amp;label=Star&amp;maxAge=2592000
[nredisstack-package]: https://www.nuget.org/packages/nredisstack/

[node-redis-url]: https://github.com/redis/node-redis
[node-redis-stars]: https://img.shields.io/github/stars/redis/node-redis.svg?style=social&amp;label=Star&amp;maxAge=2592000
[node-redis-package]: https://www.npmjs.com/package/redis

[redis-om-python-url]: https://github.com/redis/redis-om-python
[redis-om-python-author]: https://redis.com
[redis-om-python-stars]: https://img.shields.io/github/stars/redis/redis-om-python.svg?style=social&amp;label=Star&amp;maxAge=2592000

[redisearch-go-url]: https://github.com/RediSearch/redisearch-go
[redisearch-go-author]: https://redis.com
[redisearch-go-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-go.svg?style=social&amp;label=Star&amp;maxAge=2592000

[redisearch-api-rs-url]: https://github.com/RediSearch/redisearch-api-rs
[redisearch-api-rs-author]: https://redis.com
[redisearch-api-rs-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-api-rs.svg?style=social&amp;label=Star&amp;maxAge=2592000


key: _node_content
string_value:
{"id_": "4c5de6df-4b4e-44b5-9993-514729ea7267", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bb8e566e-f3ea-4493-a947-9bbf426f140e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "93490e13-bb64-4019-af66-9e3e3c5afa46", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b39f91b81ebb87ad5a5b4803bbe38240f481907ca6a60356b248d670b3652a4a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b0922abe-7468-4878-8eca-b7d42c7bd569", "node_type": "1", "metadata": {}, "hash": "4b9ab54977f54ba8b00795c25cff3079b45023950e8f5130d45a560a1d613543", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10178, "end_char_idx": 13003, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In the below cells, we will walk you through using Redis as a vector database. Since many of you are likely already used to the Redis API, this should be familiar to most.

## Setup

There are many ways to deploy Redis with RediSearch. The easiest way to get started is to use Docker, but there are are many potential options for deployment. For other deployment options, see the [redis directory](./redis) in this repo.

For this tutorial, we will use Redis Stack on Docker.

Start a version of Redis with RediSearch (Redis Stack) by running the following docker command

```bash
$ cd redis
$ docker compose up -d
```
This also includes the [RedisInsight](https://redis.com/redis-enterprise/redis-insight/) GUI for managing your Redis database which you can view at [http://localhost:8001](http://localhost:8001) once you start the docker container.

You're all set up and ready to go! Next, we import and create our client for communicating with the Redis database we just created.


```python
import redis
from redis.commands.search.indexDefinition import (
    IndexDefinition,
    IndexType
)
from redis.commands.search.query import Query
from redis.commands.search.field import (
    TextField,
    VectorField
)

REDIS_HOST =  "localhost"
REDIS_PORT = 6379
REDIS_PASSWORD = "" # default for passwordless Redis

# Connect to Redis
redis_client = redis.Redis(
    host=REDIS_HOST,
    port=REDIS_PORT,
    password=REDIS_PASSWORD
)
redis_client.ping()
```




    True



## Creating a Search Index

The below cells will show how to specify and create a search index in Redis. We will

1. Set some constants for defining our index like the distance metric and the index name
2. Define the index schema with RediSearch fields
3. Create the index



```python
# Constants
VECTOR_DIM = len(article_df['title_vector'][0]) # length of the vectors
VECTOR_NUMBER = len(article_df)                 # initial number of vectors
INDEX_NAME = "embeddings-index"                 # name of the search index
PREFIX = "doc"                                  # prefix for the document keys
DISTANCE_METRIC = "COSINE"                      # distance metric for the vectors (ex. COSINE, IP, L2)
```


```python
# Define RediSearch fields for each of the columns in the dataset
title = TextField(name="title")
url = TextField(name="url")
text = TextField(name="text")
title_embedding = VectorField("title_vector",
    "FLAT", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER,
    }
)
text_embedding = VectorField("content_vector",
    "FLAT", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER,
    }
)
fields = [title, url, text, title_embedding, text_embedding]
```


key: _node_content
string_value:
{"id_": "b0922abe-7468-4878-8eca-b7d42c7bd569", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bb8e566e-f3ea-4493-a947-9bbf426f140e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4c5de6df-4b4e-44b5-9993-514729ea7267", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d43be6244787ff80f97e206d0363aebc1629a61e741af83d693c19e759c9d6ac", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8577fa02-8ed9-4be1-a4c2-d9ac48709dd6", "node_type": "1", "metadata": {}, "hash": "39ab50975f0a16a65a7b2763f6d87b336d84a7e1a0abc07e558b57eddd8b6425", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11647, "end_char_idx": 14591, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
True



## Creating a Search Index

The below cells will show how to specify and create a search index in Redis. We will

1. Set some constants for defining our index like the distance metric and the index name
2. Define the index schema with RediSearch fields
3. Create the index



```python
# Constants
VECTOR_DIM = len(article_df['title_vector'][0]) # length of the vectors
VECTOR_NUMBER = len(article_df)                 # initial number of vectors
INDEX_NAME = "embeddings-index"                 # name of the search index
PREFIX = "doc"                                  # prefix for the document keys
DISTANCE_METRIC = "COSINE"                      # distance metric for the vectors (ex. COSINE, IP, L2)
```


```python
# Define RediSearch fields for each of the columns in the dataset
title = TextField(name="title")
url = TextField(name="url")
text = TextField(name="text")
title_embedding = VectorField("title_vector",
    "FLAT", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER,
    }
)
text_embedding = VectorField("content_vector",
    "FLAT", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER,
    }
)
fields = [title, url, text, title_embedding, text_embedding]
```


```python
# Check if index exists
try:
    redis_client.ft(INDEX_NAME).info()
    print("Index already exists")
except:
    # Create RediSearch Index
    redis_client.ft(INDEX_NAME).create_index(
        fields = fields,
        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)
    )
```

## Load Documents into the Index

Now that we have a search index, we can load documents into it. We will use the same documents we used in the previous examples. In Redis, either the Hash or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The below cells will show how to load documents into the index.


```python
def index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):
    records = documents.to_dict("records")
    for doc in records:
        key = f"{prefix}:{str(doc['id'])}"

        # create byte vectors for title and content
        title_embedding = np.array(doc["title_vector"], dtype=np.float32).tobytes()
        content_embedding = np.array(doc["content_vector"], dtype=np.float32).tobytes()

        # replace list of floats with byte vectors
        doc["title_vector"] = title_embedding
        doc["content_vector"] = content_embedding

        client.hset(key, mapping = doc)
```


```python
index_documents(redis_client, PREFIX, article_df)
print(f"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}")
```

    Loaded 25000 documents in Redis search index with name: embeddings-index


key: _node_content
string_value:
{"id_": "8577fa02-8ed9-4be1-a4c2-d9ac48709dd6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bb8e566e-f3ea-4493-a947-9bbf426f140e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b0922abe-7468-4878-8eca-b7d42c7bd569", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "dc23a3b876b5ee4ec3874bfa359efa6e05ce21be0c374f5d89fde4da54009918", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cea5f8a5-4366-4fef-bff0-fcfb9e27a1d3", "node_type": "1", "metadata": {}, "hash": "14c41fb6388a9b347e8d726c9e619ceb3216ba48c7a37f4be7844bc2417bedb4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13006, "end_char_idx": 15568, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Check if index exists
try:
    redis_client.ft(INDEX_NAME).info()
    print("Index already exists")
except:
    # Create RediSearch Index
    redis_client.ft(INDEX_NAME).create_index(
        fields = fields,
        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)
    )
```

## Load Documents into the Index

Now that we have a search index, we can load documents into it. We will use the same documents we used in the previous examples. In Redis, either the Hash or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The below cells will show how to load documents into the index.


```python
def index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):
    records = documents.to_dict("records")
    for doc in records:
        key = f"{prefix}:{str(doc['id'])}"

        # create byte vectors for title and content
        title_embedding = np.array(doc["title_vector"], dtype=np.float32).tobytes()
        content_embedding = np.array(doc["content_vector"], dtype=np.float32).tobytes()

        # replace list of floats with byte vectors
        doc["title_vector"] = title_embedding
        doc["content_vector"] = content_embedding

        client.hset(key, mapping = doc)
```


```python
index_documents(redis_client, PREFIX, article_df)
print(f"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}")
```

    Loaded 25000 documents in Redis search index with name: embeddings-index


## Running Search Queries

Now that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database. Each example will demonstrate specific features to keep in mind when developing your search application with Redis.

1. **Return Fields**: You can specify which fields you want to return in the search results. This is useful if you only want to return a subset of the fields in your documents and doesn't require a separate call to retrieve documents. In the below example, we will only return the `title` field in the search results.
2. **Hybrid Search**: You can combine vector search with any of the other RediSearch fields for hybrid search such as full text search, tag, geo, and numeric. In the below example, we will combine vector search with full text search.


key: _node_content
string_value:
{"id_": "cea5f8a5-4366-4fef-bff0-fcfb9e27a1d3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bb8e566e-f3ea-4493-a947-9bbf426f140e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8577fa02-8ed9-4be1-a4c2-d9ac48709dd6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7ed0b7e400e1aecc8f3b033407fd9a2ae028bb8bab93f18befcf9df90ba71122", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "640ca9f5-d17c-4b0f-a71a-f8c2826e8a36", "node_type": "1", "metadata": {}, "hash": "5765ce945710b5e3b8c7b151bd19436eaaff407dedd1df6a8bf9fdfd431e2c4a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13720, "end_char_idx": 16762, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):
    records = documents.to_dict("records")
    for doc in records:
        key = f"{prefix}:{str(doc['id'])}"

        # create byte vectors for title and content
        title_embedding = np.array(doc["title_vector"], dtype=np.float32).tobytes()
        content_embedding = np.array(doc["content_vector"], dtype=np.float32).tobytes()

        # replace list of floats with byte vectors
        doc["title_vector"] = title_embedding
        doc["content_vector"] = content_embedding

        client.hset(key, mapping = doc)
```


```python
index_documents(redis_client, PREFIX, article_df)
print(f"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}")
```

    Loaded 25000 documents in Redis search index with name: embeddings-index


## Running Search Queries

Now that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database. Each example will demonstrate specific features to keep in mind when developing your search application with Redis.

1. **Return Fields**: You can specify which fields you want to return in the search results. This is useful if you only want to return a subset of the fields in your documents and doesn't require a separate call to retrieve documents. In the below example, we will only return the `title` field in the search results.
2. **Hybrid Search**: You can combine vector search with any of the other RediSearch fields for hybrid search such as full text search, tag, geo, and numeric. In the below example, we will combine vector search with full text search.



```python
def search_redis(
    redis_client: redis.Redis,
    user_query: str,
    index_name: str = "embeddings-index",
    vector_field: str = "title_vector",
    return_fields: list = ["title", "url", "text", "vector_score"],
    hybrid_fields = "*",
    k: int = 20,
) -> List[dict]:

    # Creates embedding vector from user query
    embedded_query = openai.Embedding.create(input=user_query,
                                            model=EMBEDDING_MODEL,
                                            )["data"][0]['embedding']

    # Prepare the Query
    base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'
    query = (
        Query(base_query)
         .return_fields(*return_fields)
         .sort_by("vector_score")
         .paging(0, k)
         .dialect(2)
    )
    params_dict = {"vector": np.array(embedded_query).astype(dtype=np.float32).tobytes()}

    # perform vector search
    results = redis_client.ft(index_name).search(query, params_dict)
    for i, article in enumerate(results.docs):
        score = 1 - float(article.vector_score)
        print(f"{i}. {article.title} (Score: {round(score ,3) })")
    return results.docs
```


key: _node_content
string_value:
{"id_": "640ca9f5-d17c-4b0f-a71a-f8c2826e8a36", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bb8e566e-f3ea-4493-a947-9bbf426f140e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cea5f8a5-4366-4fef-bff0-fcfb9e27a1d3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e09f9719e2068d70221001567bc4fa69fb93f8b47e0b190d1879427f99fdb0e6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b74e7a6c-5102-49e5-a25b-50dd5cb719b2", "node_type": "1", "metadata": {}, "hash": "b87ee3e2b78c1e1e0d083d238aa850d7def6c3e19e6b872ed42bcf58c8d1f8d0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15572, "end_char_idx": 18252, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def search_redis(
    redis_client: redis.Redis,
    user_query: str,
    index_name: str = "embeddings-index",
    vector_field: str = "title_vector",
    return_fields: list = ["title", "url", "text", "vector_score"],
    hybrid_fields = "*",
    k: int = 20,
) -> List[dict]:

    # Creates embedding vector from user query
    embedded_query = openai.Embedding.create(input=user_query,
                                            model=EMBEDDING_MODEL,
                                            )["data"][0]['embedding']

    # Prepare the Query
    base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'
    query = (
        Query(base_query)
         .return_fields(*return_fields)
         .sort_by("vector_score")
         .paging(0, k)
         .dialect(2)
    )
    params_dict = {"vector": np.array(embedded_query).astype(dtype=np.float32).tobytes()}

    # perform vector search
    results = redis_client.ft(index_name).search(query, params_dict)
    for i, article in enumerate(results.docs):
        score = 1 - float(article.vector_score)
        print(f"{i}. {article.title} (Score: {round(score ,3) })")
    return results.docs
```


```python
# For using OpenAI to generate query embedding
openai.api_key = os.getenv("OPENAI_API_KEY", "sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")
results = search_redis(redis_client, 'modern art in Europe', k=10)
```

    0. Museum of Modern Art (Score: 0.875)
    1. Western Europe (Score: 0.867)
    2. Renaissance art (Score: 0.864)
    3. Pop art (Score: 0.86)
    4. Northern Europe (Score: 0.855)
    5. Hellenistic art (Score: 0.853)
    6. Modernist literature (Score: 0.847)
    7. Art film (Score: 0.843)
    8. Central Europe (Score: 0.843)
    9. European (Score: 0.841)



```python
results = search_redis(redis_client, 'Famous battles in Scottish history', vector_field='content_vector', k=10)
```

    0. Battle of Bannockburn (Score: 0.869)
    1. Wars of Scottish Independence (Score: 0.861)
    2. 1651 (Score: 0.853)
    3. First War of Scottish Independence (Score: 0.85)
    4. Robert I of Scotland (Score: 0.846)
    5. 841 (Score: 0.844)
    6. 1716 (Score: 0.844)
    7. 1314 (Score: 0.837)
    8. 1263 (Score: 0.836)
    9. William Wallace (Score: 0.835)


## Hybrid Queries with Redis

The previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the below example, we will combine vector search with full text search.


```python
def create_hybrid_field(field_name: str, value: str) -> str:
    return f'@{field_name}:"{value}"'
```


key: _node_content
string_value:
{"id_": "b74e7a6c-5102-49e5-a25b-50dd5cb719b2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bb8e566e-f3ea-4493-a947-9bbf426f140e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "640ca9f5-d17c-4b0f-a71a-f8c2826e8a36", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0502f54a747773e52cc858c77c9c8a4fcec263117b8cfac90516a862cf82df60", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1d7b5a4b-51b8-41a7-b0cf-4ab98e4ab79d", "node_type": "1", "metadata": {}, "hash": "6cb263833ce2db73fc6ea420ea725838d71a38887aa9f8c5fbb0f7d90ed6e5ce", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17354, "end_char_idx": 20462, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
results = search_redis(redis_client, 'Famous battles in Scottish history', vector_field='content_vector', k=10)
```

    0. Battle of Bannockburn (Score: 0.869)
    1. Wars of Scottish Independence (Score: 0.861)
    2. 1651 (Score: 0.853)
    3. First War of Scottish Independence (Score: 0.85)
    4. Robert I of Scotland (Score: 0.846)
    5. 841 (Score: 0.844)
    6. 1716 (Score: 0.844)
    7. 1314 (Score: 0.837)
    8. 1263 (Score: 0.836)
    9. William Wallace (Score: 0.835)


## Hybrid Queries with Redis

The previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the below example, we will combine vector search with full text search.


```python
def create_hybrid_field(field_name: str, value: str) -> str:
    return f'@{field_name}:"{value}"'
```


```python
# search the content vector for articles about famous battles in Scottish history and only include results with Scottish in the title
results = search_redis(redis_client,
                       "Famous battles in Scottish history",
                       vector_field="title_vector",
                       k=5,
                       hybrid_fields=create_hybrid_field("title", "Scottish")
                       )
```

    0. First War of Scottish Independence (Score: 0.892)
    1. Wars of Scottish Independence (Score: 0.889)
    2. Second War of Scottish Independence (Score: 0.879)
    3. List of Scottish monarchs (Score: 0.873)
    4. Scottish Borders (Score: 0.863)



```python
# run a hybrid query for articles about Art in the title vector and only include results with the phrase "Leonardo da Vinci" in the text
results = search_redis(redis_client,
                       "Art",
                       vector_field="title_vector",
                       k=5,
                       hybrid_fields=create_hybrid_field("text", "Leonardo da Vinci")
                       )

# find specific mention of Leonardo da Vinci in the text that our full-text-search query returned
mention = [sentence for sentence in results[0].text.split("\n") if "Leonardo da Vinci" in sentence][0]
mention
```

    0. Art (Score: 1.0)
    1. Paint (Score: 0.896)
    2. Renaissance art (Score: 0.88)
    3. Painting (Score: 0.874)
    4. Renaissance (Score: 0.846)





    'In Europe, after the Middle Ages, there was a "Renaissance" which means "rebirth". People rediscovered science and artists were allowed to paint subjects other than religious subjects. People like Michelangelo and Leonardo da Vinci still painted religious pictures, but they also now could paint mythological pictures too. These artists also invented perspective where things in the distance look smaller in the picture. This was new because in the Middle Ages people would paint all the figures close up and just overlapping each other. These artists used nudity regularly in their art.'



For more example with Redis as a vector database, see the README and examples within the ``vector_databases/redis`` directory of this repository


key: _node_content
string_value:
{"id_": "1d7b5a4b-51b8-41a7-b0cf-4ab98e4ab79d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bb8e566e-f3ea-4493-a947-9bbf426f140e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b74e7a6c-5102-49e5-a25b-50dd5cb719b2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1455475e46fc448513bd9d1ee7df7b5daaf42774e6fae9058747d1ec6747a942", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18942, "end_char_idx": 20479, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# run a hybrid query for articles about Art in the title vector and only include results with the phrase "Leonardo da Vinci" in the text
results = search_redis(redis_client,
                       "Art",
                       vector_field="title_vector",
                       k=5,
                       hybrid_fields=create_hybrid_field("text", "Leonardo da Vinci")
                       )

# find specific mention of Leonardo da Vinci in the text that our full-text-search query returned
mention = [sentence for sentence in results[0].text.split("\n") if "Leonardo da Vinci" in sentence][0]
mention
```

    0. Art (Score: 1.0)
    1. Paint (Score: 0.896)
    2. Renaissance art (Score: 0.88)
    3. Painting (Score: 0.874)
    4. Renaissance (Score: 0.846)





    'In Europe, after the Middle Ages, there was a "Renaissance" which means "rebirth". People rediscovered science and artists were allowed to paint subjects other than religious subjects. People like Michelangelo and Leonardo da Vinci still painted religious pictures, but they also now could paint mythological pictures too. These artists also invented perspective where things in the distance look smaller in the picture. This was new because in the Middle Ages people would paint all the figures close up and just overlapping each other. These artists used nudity regularly in their art.'



For more example with Redis as a vector database, see the README and examples within the ``vector_databases/redis`` directory of this repository


```python

```


key: _node_content
string_value:
{"id_": "ba3f9aba-1ddf-4f2a-8d19-21d0dbf26a79", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "68b5f57c-481f-4216-892b-db9473392cd1", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b08d2ce2199cb0ef05b32554508f11d39da0689c732dce096a9cbf3dd901eca0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e5cbc5d6-b656-4053-9adf-0232aae3d60f", "node_type": "1", "metadata": {}, "hash": "60252f48545fffa75c59a4eb516fd84fc4c1ac8dcd6a08004bbf9cef8f9bfd57", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2501, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Redis

### What is Redis?

Most developers from a web services background are probably familiar with Redis. At it's core, Redis is an open-source key-value store that can be used as a cache, message broker, and database. Developers choice Redis because it is fast, has a large ecosystem of client libraries, and has been deployed by major enterprises for years.

In addition to the traditional uses of Redis. Redis also provides [Redis Modules](https://redis.io/modules) which are a way to extend Redis with new capabilities, commands and data types. Example modules include [RedisJSON](https://redis.io/docs/stack/json/), [RedisTimeSeries](https://redis.io/docs/stack/timeseries/), [RedisBloom](https://redis.io/docs/stack/bloom/) and [RediSearch](https://redis.io/docs/stack/search/).


### Deployment options

There are a number of ways to deploy Redis. For local development, the quickest method is to use the [Redis Stack docker container](https://hub.docker.com/r/redis/redis-stack) which we will use here. Redis Stack contains a number of Redis modules that can be used together to create a fast, multi-model data store and query engine.

For production use cases, The easiest way to get started is to use the [Redis Cloud](https://redislabs.com/redis-enterprise-cloud/overview/) service. Redis Cloud is a fully managed Redis service. You can also deploy Redis on your own infrastructure using [Redis Enterprise](https://redislabs.com/redis-enterprise/overview/). Redis Enterprise is a fully managed Redis service that can be deployed in kubernetes, on-premises or in the cloud.

Additionally, every major cloud provider ([AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-e6y7ork67pjwg?sr=0-2&ref_=beagle&applicationId=AWSMPContessa), [Google Marketplace](https://console.cloud.google.com/marketplace/details/redislabs-public/redis-enterprise?pli=1), or [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/garantiadata.redis_enterprise_1sp_public_preview?tab=Overview)) offers Redis Enterprise in a marketplace offering.


### What is RediSearch?

RediSearch is a [Redis module](https://redis.io/modules) that provides querying, secondary indexing, full-text search and vector search for Redis. To use RediSearch, you first declare indexes on your Redis data. You can then use the RediSearch clients to query that data. For more information on the feature set of RediSearch, see the [RediSearch documentation](https://redis.io/docs/stack/search/).


key: _node_content
string_value:
{"id_": "e5cbc5d6-b656-4053-9adf-0232aae3d60f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "68b5f57c-481f-4216-892b-db9473392cd1", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b08d2ce2199cb0ef05b32554508f11d39da0689c732dce096a9cbf3dd901eca0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ba3f9aba-1ddf-4f2a-8d19-21d0dbf26a79", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "80a34cf61b078186617c84d636e9a15d7ce53212072a42d702fa9297589564f7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4b7ab0cc-b40d-4ae1-ba53-86e257dd50dc", "node_type": "1", "metadata": {}, "hash": "cb7257b683082a9e2a155f60bb6de86a717353be30ded39f860e2f3d5be40b32", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 791, "end_char_idx": 3909, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Deployment options

There are a number of ways to deploy Redis. For local development, the quickest method is to use the [Redis Stack docker container](https://hub.docker.com/r/redis/redis-stack) which we will use here. Redis Stack contains a number of Redis modules that can be used together to create a fast, multi-model data store and query engine.

For production use cases, The easiest way to get started is to use the [Redis Cloud](https://redislabs.com/redis-enterprise-cloud/overview/) service. Redis Cloud is a fully managed Redis service. You can also deploy Redis on your own infrastructure using [Redis Enterprise](https://redislabs.com/redis-enterprise/overview/). Redis Enterprise is a fully managed Redis service that can be deployed in kubernetes, on-premises or in the cloud.

Additionally, every major cloud provider ([AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-e6y7ork67pjwg?sr=0-2&ref_=beagle&applicationId=AWSMPContessa), [Google Marketplace](https://console.cloud.google.com/marketplace/details/redislabs-public/redis-enterprise?pli=1), or [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/garantiadata.redis_enterprise_1sp_public_preview?tab=Overview)) offers Redis Enterprise in a marketplace offering.


### What is RediSearch?

RediSearch is a [Redis module](https://redis.io/modules) that provides querying, secondary indexing, full-text search and vector search for Redis. To use RediSearch, you first declare indexes on your Redis data. You can then use the RediSearch clients to query that data. For more information on the feature set of RediSearch, see the [RediSearch documentation](https://redis.io/docs/stack/search/).


### Features

RediSearch uses compressed, inverted indexes for fast indexing with a low memory footprint. RediSearch indexes enhance Redis by providing exact-phrase matching, fuzzy search, and numeric filtering, among many other features. Such as:

* Full-Text indexing of multiple fields in Redis hashes
* Incremental indexing without performance loss
* Vector similarity search
* Document ranking (using [tf-idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf), with optional user-provided weights)
* Field weighting
* Complex boolean queries with AND, OR, and NOT operators
* Prefix matching, fuzzy matching, and exact-phrase queries
* Support for [double-metaphone phonetic matching](https://redis.io/docs/stack/search/reference/phonetic_matching/)
* Auto-complete suggestions (with fuzzy prefix suggestions)
* Stemming-based query expansion in [many languages](https://redis.io/docs/stack/search/reference/stemming/) (using [Snowball](http://snowballstem.org/))
* Support for Chinese-language tokenization and querying (using [Friso](https://github.com/lionsoul2014/friso))
* Numeric filters and ranges
* Geospatial searches using [Redis geospatial indexing](/commands/georadius)
* A powerful aggregations engine
* Supports for all utf-8 encoded text
* Retrieve full documents, selected fields, or only the document IDs
* Sorting results (for example, by creation date)
* JSON support through RedisJSON


key: _node_content
string_value:
{"id_": "4b7ab0cc-b40d-4ae1-ba53-86e257dd50dc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "68b5f57c-481f-4216-892b-db9473392cd1", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b08d2ce2199cb0ef05b32554508f11d39da0689c732dce096a9cbf3dd901eca0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e5cbc5d6-b656-4053-9adf-0232aae3d60f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "bd34788ffe83d6511cd0a9ff2503d8fab821af0fb6a3c6c53271ad57793cbc8d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2e7664f1-cf9c-4b97-b8cd-a81d134cb34d", "node_type": "1", "metadata": {}, "hash": "dc5a05be55dbfc5b2b58c5484e6582f2da7e8cd8fdf126f134a2b850ab78a20c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2504, "end_char_idx": 6474, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Features

RediSearch uses compressed, inverted indexes for fast indexing with a low memory footprint. RediSearch indexes enhance Redis by providing exact-phrase matching, fuzzy search, and numeric filtering, among many other features. Such as:

* Full-Text indexing of multiple fields in Redis hashes
* Incremental indexing without performance loss
* Vector similarity search
* Document ranking (using [tf-idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf), with optional user-provided weights)
* Field weighting
* Complex boolean queries with AND, OR, and NOT operators
* Prefix matching, fuzzy matching, and exact-phrase queries
* Support for [double-metaphone phonetic matching](https://redis.io/docs/stack/search/reference/phonetic_matching/)
* Auto-complete suggestions (with fuzzy prefix suggestions)
* Stemming-based query expansion in [many languages](https://redis.io/docs/stack/search/reference/stemming/) (using [Snowball](http://snowballstem.org/))
* Support for Chinese-language tokenization and querying (using [Friso](https://github.com/lionsoul2014/friso))
* Numeric filters and ranges
* Geospatial searches using [Redis geospatial indexing](/commands/georadius)
* A powerful aggregations engine
* Supports for all utf-8 encoded text
* Retrieve full documents, selected fields, or only the document IDs
* Sorting results (for example, by creation date)
* JSON support through RedisJSON


### Clients

Given the large ecosystem around Redis, there are most likely client libraries in the language you need. You can use any standard Redis client library to run RediSearch commands, but it's easiest to use a library that wraps the RediSearch API. Below are a few examples, but you can find more client libraries [here](https://redis.io/resources/clients/).

| Project | Language | License | Author | Stars |
|----------|---------|--------|---------|-------|
| [jedis][jedis-url] | Java | MIT | [Redis][redis-url] | ![Stars][jedis-stars] |
| [redis-py][redis-py-url] | Python | MIT | [Redis][redis-url] | ![Stars][redis-py-stars] |
| [node-redis][node-redis-url] | Node.js | MIT | [Redis][redis-url] | ![Stars][node-redis-stars] |
| [nredisstack][nredisstack-url] | .NET | MIT | [Redis][redis-url] | ![Stars][nredisstack-stars] |

[redis-url]: https://redis.com

[redis-py-url]: https://github.com/redis/redis-py
[redis-py-stars]: https://img.shields.io/github/stars/redis/redis-py.svg?style=social&amp;label=Star&amp;maxAge=2592000
[redis-py-package]: https://pypi.python.org/pypi/redis

[jedis-url]: https://github.com/redis/jedis
[jedis-stars]: https://img.shields.io/github/stars/redis/jedis.svg?style=social&amp;label=Star&amp;maxAge=2592000
[Jedis-package]: https://search.maven.org/artifact/redis.clients/jedis

[nredisstack-url]: https://github.com/redis/nredisstack
[nredisstack-stars]: https://img.shields.io/github/stars/redis/nredisstack.svg?style=social&amp;label=Star&amp;maxAge=2592000
[nredisstack-package]: https://www.nuget.org/packages/nredisstack/

[node-redis-url]: https://github.com/redis/node-redis
[node-redis-stars]: https://img.shields.io/github/stars/redis/node-redis.svg?style=social&amp;label=Star&amp;maxAge=2592000
[node-redis-package]: https://www.npmjs.com/package/redis

[redis-om-python-url]: https://github.com/redis/redis-om-python
[redis-om-python-author]: https://redis.com
[redis-om-python-stars]: https://img.shields.io/github/stars/redis/redis-om-python.svg?style=social&amp;label=Star&amp;maxAge=2592000

[redisearch-go-url]: https://github.com/RediSearch/redisearch-go
[redisearch-go-author]: https://redis.com
[redisearch-go-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-go.svg?style=social&amp;label=Star&amp;maxAge=2592000

[redisearch-api-rs-url]: https://github.com/RediSearch/redisearch-api-rs
[redisearch-api-rs-author]: https://redis.com
[redisearch-api-rs-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-api-rs.svg?style=social&amp;label=Star&amp;maxAge=2592000


key: _node_content
string_value:
{"id_": "2e7664f1-cf9c-4b97-b8cd-a81d134cb34d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "68b5f57c-481f-4216-892b-db9473392cd1", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b08d2ce2199cb0ef05b32554508f11d39da0689c732dce096a9cbf3dd901eca0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4b7ab0cc-b40d-4ae1-ba53-86e257dd50dc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "9b21f1fabb3bcd7cfc5b7c63fb1a4bee3ce73426fe0a6272cc4f44e2fa77c13e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6477, "end_char_idx": 9609, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Deployment Options

There are many ways to deploy Redis with RediSearch. The easiest way to get started is to use Docker, but there are are many potential options for deployment such as

- [Redis Cloud](https://redis.com/redis-enterprise-cloud/overview/)
- Cloud marketplaces: [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-e6y7ork67pjwg?sr=0-2&ref_=beagle&applicationId=AWSMPContessa), [Google Marketplace](https://console.cloud.google.com/marketplace/details/redislabs-public/redis-enterprise?pli=1), or [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/garantiadata.redis_enterprise_1sp_public_preview?tab=Overview)
- On-premise: [Redis Enterprise Software](https://redis.com/redis-enterprise-software/overview/)
- Kubernetes: [Redis Enterprise Software on Kubernetes](https://docs.redis.com/latest/kubernetes/)
- [Docker (RediSearch)](https://hub.docker.com/r/redislabs/redisearch)
- [Docker (Redis Stack)](https://hub.docker.com/r/redis/redis-stack)


### Cluster support

RediSearch has a distributed cluster version that scales to billions of documents across hundreds of servers. At the moment, distributed RediSearch is available as part of [Redis Enterprise Cloud](https://redis.com/redis-enterprise-cloud/overview/) and [Redis Enterprise Software](https://redis.com/redis-enterprise-software/overview/).

See [RediSearch on Redis Enterprise](https://redis.com/modules/redisearch/) for more information.

### Examples

- [Product Search](https://github.com/RedisVentures/redis-product-search) - eCommerce product search (with image and text)
- [Product Recommendations with DocArray / Jina](https://github.com/jina-ai/product-recommendation-redis-docarray) - Content-based product recommendations example with Redis and DocArray.
- [Redis VSS in RecSys](https://github.com/RedisVentures/Redis-Recsys) - 3 end-to-end Redis & NVIDIA Merlin Recommendation System Architectures.
- [Azure OpenAI Embeddings Q&A](https://github.com/ruoccofabrizio/azure-open-ai-embeddings-qna) - OpenAI and Redis as a Q&A service on Azure.
- [ArXiv Paper Search](https://github.com/RedisVentures/redis-arXiv-search) - Semantic search over arXiv scholarly papers


### More Resources

For more information on how to use Redis as a vector database, check out the following resources:

- [Redis Vector Similarity Docs](https://redis.io/docs/stack/search/reference/vectors/) - Redis official docs for Vector Search.
- [Redis-py Search Docs](https://redis.readthedocs.io/en/latest/redismodules.html#redisearch-commands) - Redis-py client library docs for RediSearch.
- [Vector Similarity Search: From Basics to Production](https://mlops.community/vector-similarity-search-from-basics-to-production/) - Introductory blog post to VSS and Redis as a VectorDB.
- [AI-Powered Document Search](https://datasciencedojo.com/blog/ai-powered-document-search/) - Blog post covering AI Powered Document Search Use Cases & Architectures.
- [Vector Database Benchmarks](https://jina.ai/news/benchmark-vector-search-databases-with-one-million-data/) - Jina AI VectorDB benchmarks comparing Redis against others.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb


key: _node_content
string_value:
{"id_": "039f63a7-e7c2-4994-9873-5e6674c9f478", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "51426527-c022-42e4-8c98-3975aae6d412", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "703f38bf-0392-47ff-a008-08f51c1fcc31", "node_type": "1", "metadata": {}, "hash": "d570b3617e31f41433e5b1d8fd413e18a61b6b06e39832d89f4fc2cfe4ddd3fe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3027, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Using Redis as a Vector Database with OpenAI

This notebook provides an introduction to using Redis as a vector database with OpenAI embeddings. Redis is a scalable, real-time database that can be used as a vector database when using the [RediSearch Module](https://oss.redislabs.com/redisearch/). The RediSearch module allows you to index and search for vectors in Redis. This notebook will show you how to use the RediSearch module to index and search for vectors created by using the OpenAI API and stored in Redis.

### What is Redis?

Most developers from a web services background are probably familiar with Redis. At it's core, Redis is an open-source key-value store that can be used as a cache, message broker, and database. Developers choice Redis because it is fast, has a large ecosystem of client libraries, and has been deployed by major enterprises for years.

In addition to the traditional uses of Redis. Redis also provides [Redis Modules](https://redis.io/modules) which are a way to extend Redis with new data types and commands. Example modules include [RedisJSON](https://redis.io/docs/stack/json/), [RedisTimeSeries](https://redis.io/docs/stack/timeseries/), [RedisBloom](https://redis.io/docs/stack/bloom/) and [RediSearch](https://redis.io/docs/stack/search/).

### What is RediSearch?

RediSearch is a [Redis module](https://redis.io/modules) that provides querying, secondary indexing, full-text search and vector search for Redis. To use RediSearch, you first declare indexes on your Redis data. You can then use the RediSearch clients to query that data. For more information on the feature set of RediSearch, see the [README](./README.md) or the [RediSearch documentation](https://redis.io/docs/stack/search/).

### Deployment options

There are a number of ways to deploy Redis. For local development, the quickest method is to use the [Redis Stack docker container](https://hub.docker.com/r/redis/redis-stack) which we will use here. Redis Stack contains a number of Redis modules that can be used together to create a fast, multi-model data store and query engine.

For production use cases, The easiest way to get started is to use the [Redis Cloud](https://redislabs.com/redis-enterprise-cloud/overview/) service. Redis Cloud is a fully managed Redis service. You can also deploy Redis on your own infrastructure using [Redis Enterprise](https://redislabs.com/redis-enterprise/overview/). Redis Enterprise is a fully managed Redis service that can be deployed in kubernetes, on-premises or in the cloud.

Additionally, every major cloud provider ([AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-e6y7ork67pjwg?sr=0-2&ref_=beagle&applicationId=AWSMPContessa), [Google Marketplace](https://console.cloud.google.com/marketplace/details/redislabs-public/redis-enterprise?pli=1), or [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/garantiadata.redis_enterprise_1sp_public_preview?tab=Overview)) offers Redis Enterprise in a marketplace offering.


key: _node_content
string_value:
{"id_": "703f38bf-0392-47ff-a008-08f51c1fcc31", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "51426527-c022-42e4-8c98-3975aae6d412", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "039f63a7-e7c2-4994-9873-5e6674c9f478", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f12a9bd3e27a18166f0027474e77a1d5d03b7a5c1c41d4a3e85aefd6888a96b3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "01026a6e-f3c8-47ee-ab6c-127508e6757a", "node_type": "1", "metadata": {}, "hash": "2a79989f01b54851ba761c4f72e140fa361cc4be1f7d2f59f6bff5623f6332ec", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3031, "end_char_idx": 5765, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Prerequisites

Before we start this project, we need to set up the following:

* start a Redis database with RediSearch (redis-stack)
* install libraries
    * [Redis-py](https://github.com/redis/redis-py)
* get your [OpenAI API key](https://beta.openai.com/account/api-keys)

===========================================================

### Start Redis

To keep this example simple, we will use the Redis Stack docker container which we can start as follows

```bash
$ docker-compose up -d
```

This also includes the [RedisInsight](https://redis.com/redis-enterprise/redis-insight/) GUI for managing your Redis database which you can view at [http://localhost:8001](http://localhost:8001) once you start the docker container.

You're all set up and ready to go! Next, we import and create our client for communicating with the Redis database we just created.

## Install Requirements

Redis-Py is the python client for communicating with Redis. We will use this to communicate with our Redis-stack database. 


```python
! pip install redis wget pandas openai
```

===========================================================
## Prepare your OpenAI API key

The `OpenAI API key` is used for vectorization of query data.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it to your environment variables as `OPENAI_API_KEY` by using following command:


```python
! export OPENAI_API_KEY="your API key"
```


```python
# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.
import os
import openai

# Note. alternatively you can set a temporary env variable like this:
# os.environ["OPENAI_API_KEY"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

if os.getenv("OPENAI_API_KEY") is not None:
    openai.api_key = os.getenv("OPENAI_API_KEY")
    print ("OPENAI_API_KEY is ready")
else:
    print ("OPENAI_API_KEY environment variable not found")
```

    OPENAI_API_KEY is ready


## Load data

In this section we'll load embedded data that has already been converted into vectors. We'll use this data to create an index in Redis and then search for similar vectors.


```python
import sys
import numpy as np
import pandas as pd
from typing import List

# use helper function in nbutils.py to download and read the data
# this should take from 5-10 min to run
if os.getcwd() not in sys.path:
    sys.path.append(os.getcwd())
import nbutils

nbutils.download_wikipedia_data()
data = nbutils.read_wikipedia_data()

data.head()
```

    File Downloaded


key: _node_content
string_value:
{"id_": "01026a6e-f3c8-47ee-ab6c-127508e6757a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "51426527-c022-42e4-8c98-3975aae6d412", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "703f38bf-0392-47ff-a008-08f51c1fcc31", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "9f09fccf3f16dd31a15ad373413be01a4367092852784baecb0265e294d78a38", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9305de9f-3673-4b08-a4d9-ecd677eef5b4", "node_type": "1", "metadata": {}, "hash": "8cd0a9c9c8bed21afa4c25e5abe679fb75283551f60ad360179fa48e2e466a3e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4518, "end_char_idx": 6782, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
! export OPENAI_API_KEY="your API key"
```


```python
# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.
import os
import openai

# Note. alternatively you can set a temporary env variable like this:
# os.environ["OPENAI_API_KEY"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

if os.getenv("OPENAI_API_KEY") is not None:
    openai.api_key = os.getenv("OPENAI_API_KEY")
    print ("OPENAI_API_KEY is ready")
else:
    print ("OPENAI_API_KEY environment variable not found")
```

    OPENAI_API_KEY is ready


## Load data

In this section we'll load embedded data that has already been converted into vectors. We'll use this data to create an index in Redis and then search for similar vectors.


```python
import sys
import numpy as np
import pandas as pd
from typing import List

# use helper function in nbutils.py to download and read the data
# this should take from 5-10 min to run
if os.getcwd() not in sys.path:
    sys.path.append(os.getcwd())
import nbutils

nbutils.download_wikipedia_data()
data = nbutils.read_wikipedia_data()

data.head()
```

    File Downloaded





<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>url</th>
      <th>title</th>
      <th>text</th>
      <th>title_vector</th>
      <th>content_vector</th>
      <th>vector_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>https://simple.wikipedia.org/wiki/April</td>
      <td>April</td>
      <td>April is the fourth month of the year in the J...</td>
      <td>[0.001009464613161981, -0.020700545981526375, ...</td>
      <td>[-0.011253940872848034, -0.013491976074874401,...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>https://simple.wikipedia.org/wiki/August</td>
      <td>August</td>
      <td>August (Aug.)


key: _node_content
string_value:
{"id_": "9305de9f-3673-4b08-a4d9-ecd677eef5b4", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "51426527-c022-42e4-8c98-3975aae6d412", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "01026a6e-f3c8-47ee-ab6c-127508e6757a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "04631ed923641cbd6b693a87c9df5d6bd071760be3fd82d5cbe7553c01255924", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dcbe177f-6345-4b71-bd7b-57516fcb421c", "node_type": "1", "metadata": {}, "hash": "a38d440fd46e3d8ec15186b41e85bf1941379cf88d54866f98815a4eabb1d814", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5771, "end_char_idx": 7813, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>url</th>
      <th>title</th>
      <th>text</th>
      <th>title_vector</th>
      <th>content_vector</th>
      <th>vector_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>https://simple.wikipedia.org/wiki/April</td>
      <td>April</td>
      <td>April is the fourth month of the year in the J...</td>
      <td>[0.001009464613161981, -0.020700545981526375, ...</td>
      <td>[-0.011253940872848034, -0.013491976074874401,...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>https://simple.wikipedia.org/wiki/August</td>
      <td>August</td>
      <td>August (Aug.) is the eighth month of the year ...</td>
      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>
      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>6</td>
      <td>https://simple.wikipedia.org/wiki/Art</td>
      <td>Art</td>
      <td>Art is a creative activity that expresses imag...</td>
      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>
      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>
      <td>2</td>
    </tr>
    <tr>
      <th>3</th>
      <td>8</td>
      <td>https://simple.wikipedia.org/wiki/A</td>
      <td>A</td>
      <td>A or a is the first letter of the English alph...</td>
      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>
      <td>[0.024894846603274345, -0.022186409682035446, ...</td>
      <td>3</td>
    </tr>
    <tr>
      <th>4</th>
      <td>9</td>
      <td>https://simple.wikipedia.org/wiki/Air</td>
      <td>Air</td>
      <td>Air refers to the Earth's atmosphere.


key: _node_content
string_value:
{"id_": "dcbe177f-6345-4b71-bd7b-57516fcb421c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "51426527-c022-42e4-8c98-3975aae6d412", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9305de9f-3673-4b08-a4d9-ecd677eef5b4", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d90c715b264776998131e75bb9ab0ca7a5ba3cccd1fef786e7d83411ca4327d6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "54a6076c-e8f1-4708-ab51-6cf411699abe", "node_type": "1", "metadata": {}, "hash": "a608461bed49331890805a388d9965aafc86c90bcef890b0aa3e983daf0e5a7a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6783, "end_char_idx": 8997, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
is the eighth month of the year ...</td>
      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>
      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>6</td>
      <td>https://simple.wikipedia.org/wiki/Art</td>
      <td>Art</td>
      <td>Art is a creative activity that expresses imag...</td>
      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>
      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>
      <td>2</td>
    </tr>
    <tr>
      <th>3</th>
      <td>8</td>
      <td>https://simple.wikipedia.org/wiki/A</td>
      <td>A</td>
      <td>A or a is the first letter of the English alph...</td>
      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>
      <td>[0.024894846603274345, -0.022186409682035446, ...</td>
      <td>3</td>
    </tr>
    <tr>
      <th>4</th>
      <td>9</td>
      <td>https://simple.wikipedia.org/wiki/Air</td>
      <td>Air</td>
      <td>Air refers to the Earth's atmosphere. Air is a...</td>
      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>
      <td>[0.021524671465158463, 0.018522677943110466, -...</td>
      <td>4</td>
    </tr>
  </tbody>
</table>
</div>



## Connect to Redis

Now that we have our Redis database running, we can connect to it using the Redis-py client. We will use the default host and port for the Redis database which is `localhost:6379`.




```python
import redis
from redis.commands.search.indexDefinition import (
    IndexDefinition,
    IndexType
)
from redis.commands.search.query import Query
from redis.commands.search.field import (
    TextField,
    VectorField
)

REDIS_HOST =  "localhost"
REDIS_PORT = 6379
REDIS_PASSWORD = "" # default for passwordless Redis

# Connect to Redis
redis_client = redis.Redis(
    host=REDIS_HOST,
    port=REDIS_PORT,
    password=REDIS_PASSWORD
)
redis_client.ping()
```




    True



## Creating a Search Index in Redis

The below cells will show how to specify and create a search index in Redis. We will:

1. Set some constants for defining our index like the distance metric and the index name
2. Define the index schema with RediSearch fields
3. Create the index


key: _node_content
string_value:
{"id_": "54a6076c-e8f1-4708-ab51-6cf411699abe", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "51426527-c022-42e4-8c98-3975aae6d412", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dcbe177f-6345-4b71-bd7b-57516fcb421c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "57e2fb0a9a329b51ab6508b9ccae3e2d0b863a3afb9dafffdb8c8f2ff91376e5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2e67afee-d882-47d9-9489-fec6fd43d092", "node_type": "1", "metadata": {}, "hash": "2aef452339ec5229e17ec0e24316586af693499bb31ac7542bce86a3639e450d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7814, "end_char_idx": 10752, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Air is a...</td>
      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>
      <td>[0.021524671465158463, 0.018522677943110466, -...</td>
      <td>4</td>
    </tr>
  </tbody>
</table>
</div>



## Connect to Redis

Now that we have our Redis database running, we can connect to it using the Redis-py client. We will use the default host and port for the Redis database which is `localhost:6379`.




```python
import redis
from redis.commands.search.indexDefinition import (
    IndexDefinition,
    IndexType
)
from redis.commands.search.query import Query
from redis.commands.search.field import (
    TextField,
    VectorField
)

REDIS_HOST =  "localhost"
REDIS_PORT = 6379
REDIS_PASSWORD = "" # default for passwordless Redis

# Connect to Redis
redis_client = redis.Redis(
    host=REDIS_HOST,
    port=REDIS_PORT,
    password=REDIS_PASSWORD
)
redis_client.ping()
```




    True



## Creating a Search Index in Redis

The below cells will show how to specify and create a search index in Redis. We will:

1. Set some constants for defining our index like the distance metric and the index name
2. Define the index schema with RediSearch fields
3. Create the index


```python
# Constants
VECTOR_DIM = len(data['title_vector'][0]) # length of the vectors
VECTOR_NUMBER = len(data)                 # initial number of vectors
INDEX_NAME = "embeddings-index"           # name of the search index
PREFIX = "doc"                            # prefix for the document keys
DISTANCE_METRIC = "COSINE"                # distance metric for the vectors (ex. COSINE, IP, L2)
```


```python
# Define RediSearch fields for each of the columns in the dataset
title = TextField(name="title")
url = TextField(name="url")
text = TextField(name="text")
title_embedding = VectorField("title_vector",
    "FLAT", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER,
    }
)
text_embedding = VectorField("content_vector",
    "FLAT", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER,
    }
)
fields = [title, url, text, title_embedding, text_embedding]
```


```python
# Check if index exists
try:
    redis_client.ft(INDEX_NAME).info()
    print("Index already exists")
except:
    # Create RediSearch Index
    redis_client.ft(INDEX_NAME).create_index(
        fields = fields,
        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)
)
```

## Load Documents into the Index

Now that we have a search index, we can load documents into it. We will use the same documents we used in the previous examples. In Redis, either the HASH or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The below cells will show how to load documents into the index.


key: _node_content
string_value:
{"id_": "2e67afee-d882-47d9-9489-fec6fd43d092", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "51426527-c022-42e4-8c98-3975aae6d412", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "54a6076c-e8f1-4708-ab51-6cf411699abe", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e1132ed715785cd6b2613ba802c4e18e1417baabdde0c713f7f0e56e917b3ce2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2a227047-ff10-4554-8435-4bb38d4d43a9", "node_type": "1", "metadata": {}, "hash": "4a0d2119b71c60a4b5306fee9f383594b7e682068d2543563c32df6bf2bd7628", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9403, "end_char_idx": 11963, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Define RediSearch fields for each of the columns in the dataset
title = TextField(name="title")
url = TextField(name="url")
text = TextField(name="text")
title_embedding = VectorField("title_vector",
    "FLAT", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER,
    }
)
text_embedding = VectorField("content_vector",
    "FLAT", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER,
    }
)
fields = [title, url, text, title_embedding, text_embedding]
```


```python
# Check if index exists
try:
    redis_client.ft(INDEX_NAME).info()
    print("Index already exists")
except:
    # Create RediSearch Index
    redis_client.ft(INDEX_NAME).create_index(
        fields = fields,
        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)
)
```

## Load Documents into the Index

Now that we have a search index, we can load documents into it. We will use the same documents we used in the previous examples. In Redis, either the HASH or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The below cells will show how to load documents into the index.


```python
def index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):
    records = documents.to_dict("records")
    for doc in records:
        key = f"{prefix}:{str(doc['id'])}"

        # create byte vectors for title and content
        title_embedding = np.array(doc["title_vector"], dtype=np.float32).tobytes()
        content_embedding = np.array(doc["content_vector"], dtype=np.float32).tobytes()

        # replace list of floats with byte vectors
        doc["title_vector"] = title_embedding
        doc["content_vector"] = content_embedding

        client.hset(key, mapping = doc)
```


```python
index_documents(redis_client, PREFIX, data)
print(f"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}")
```

    Loaded 25000 documents in Redis search index with name: embeddings-index


## Simple Vector Search Queries with OpenAI Query Embeddings

Now that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database.


key: _node_content
string_value:
{"id_": "2a227047-ff10-4554-8435-4bb38d4d43a9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "51426527-c022-42e4-8c98-3975aae6d412", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2e67afee-d882-47d9-9489-fec6fd43d092", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0d347f4355c6b4eedb1ffc51149d5c237b5cc4d80796b7aa1925d1ee41b7dc1d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f73eb3fc-842c-4e5a-8c11-70e8469a4c43", "node_type": "1", "metadata": {}, "hash": "0bb1516120e00d61d536bb174f6c12726c16166080eef2402122c96f1832e3ab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10755, "end_char_idx": 13727, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):
    records = documents.to_dict("records")
    for doc in records:
        key = f"{prefix}:{str(doc['id'])}"

        # create byte vectors for title and content
        title_embedding = np.array(doc["title_vector"], dtype=np.float32).tobytes()
        content_embedding = np.array(doc["content_vector"], dtype=np.float32).tobytes()

        # replace list of floats with byte vectors
        doc["title_vector"] = title_embedding
        doc["content_vector"] = content_embedding

        client.hset(key, mapping = doc)
```


```python
index_documents(redis_client, PREFIX, data)
print(f"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}")
```

    Loaded 25000 documents in Redis search index with name: embeddings-index


## Simple Vector Search Queries with OpenAI Query Embeddings

Now that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database.


```python
def search_redis(
    redis_client: redis.Redis,
    user_query: str,
    index_name: str = "embeddings-index",
    vector_field: str = "title_vector",
    return_fields: list = ["title", "url", "text", "vector_score"],
    hybrid_fields = "*",
    k: int = 20,
    print_results: bool = True,
) -> List[dict]:

    # Creates embedding vector from user query
    embedded_query = openai.Embedding.create(input=user_query,
                                            model="text-embedding-3-small",
                                            )["data"][0]['embedding']

    # Prepare the Query
    base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'
    query = (
        Query(base_query)
         .return_fields(*return_fields)
         .sort_by("vector_score")
         .paging(0, k)
         .dialect(2)
    )
    params_dict = {"vector": np.array(embedded_query).astype(dtype=np.float32).tobytes()}

    # perform vector search
    results = redis_client.ft(index_name).search(query, params_dict)
    if print_results:
        for i, article in enumerate(results.docs):
            score = 1 - float(article.vector_score)
            print(f"{i}. {article.title} (Score: {round(score ,3) })")
    return results.docs
```


```python
# For using OpenAI to generate query embedding
results = search_redis(redis_client, 'modern art in Europe', k=10)
```

    0. Museum of Modern Art (Score: 0.875)
    1. Western Europe (Score: 0.868)
    2. Renaissance art (Score: 0.864)
    3. Pop art (Score: 0.86)
    4. Northern Europe (Score: 0.855)
    5. Hellenistic art (Score: 0.853)
    6. Modernist literature (Score: 0.847)
    7. Art film (Score: 0.843)
    8. Central Europe (Score: 0.843)
    9. European (Score: 0.841)


key: _node_content
string_value:
{"id_": "f73eb3fc-842c-4e5a-8c11-70e8469a4c43", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "51426527-c022-42e4-8c98-3975aae6d412", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2a227047-ff10-4554-8435-4bb38d4d43a9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "6b75a9d29ddd5a7b652a6f859b49e020105eeb98194063047412457a444b936d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b435e1ec-66c3-4142-9eac-670a1e037161", "node_type": "1", "metadata": {}, "hash": "559d12af48b71ced4c4154ceb046b72a950acd30db6df770a4129b5e21b40cc3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13234, "end_char_idx": 15300, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# For using OpenAI to generate query embedding
results = search_redis(redis_client, 'modern art in Europe', k=10)
```

    0. Museum of Modern Art (Score: 0.875)
    1. Western Europe (Score: 0.868)
    2. Renaissance art (Score: 0.864)
    3. Pop art (Score: 0.86)
    4. Northern Europe (Score: 0.855)
    5. Hellenistic art (Score: 0.853)
    6. Modernist literature (Score: 0.847)
    7. Art film (Score: 0.843)
    8. Central Europe (Score: 0.843)
    9. European (Score: 0.841)



```python
results = search_redis(redis_client, 'Famous battles in Scottish history', vector_field='content_vector', k=10)
```

    0. Battle of Bannockburn (Score: 0.869)
    1. Wars of Scottish Independence (Score: 0.861)
    2. 1651 (Score: 0.853)
    3. First War of Scottish Independence (Score: 0.85)
    4. Robert I of Scotland (Score: 0.846)
    5. 841 (Score: 0.844)
    6. 1716 (Score: 0.844)
    7. 1314 (Score: 0.837)
    8. 1263 (Score: 0.836)
    9. William Wallace (Score: 0.835)


## Hybrid Queries with Redis

The previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the below example, we will combine vector search with full text search.


```python
def create_hybrid_field(field_name: str, value: str) -> str:
    return f'@{field_name}:"{value}"'

# search the content vector for articles about famous battles in Scottish history and only include results with Scottish in the title
results = search_redis(redis_client,
                       "Famous battles in Scottish history",
                       vector_field="title_vector",
                       k=5,
                       hybrid_fields=create_hybrid_field("title", "Scottish")
                       )
```

    0. First War of Scottish Independence (Score: 0.892)
    1. Wars of Scottish Independence (Score: 0.889)
    2. Second War of Scottish Independence (Score: 0.879)
    3. List of Scottish monarchs (Score: 0.873)
    4. Scottish Borders (Score: 0.863)


key: _node_content
string_value:
{"id_": "b435e1ec-66c3-4142-9eac-670a1e037161", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "51426527-c022-42e4-8c98-3975aae6d412", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f73eb3fc-842c-4e5a-8c11-70e8469a4c43", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "5707566de7a33eb1fdb41d81cafaaf0426027e4ac6f0d64985a0d55f326d5dca", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0c9815d8-fc5d-4cb8-a3d9-c0a8893338a4", "node_type": "1", "metadata": {}, "hash": "3f934ea4c249ac489b58c50e639a9dec48926fc9d23d9da862fa9e7d9a05eab7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14227, "end_char_idx": 17343, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Hybrid Queries with Redis

The previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the below example, we will combine vector search with full text search.


```python
def create_hybrid_field(field_name: str, value: str) -> str:
    return f'@{field_name}:"{value}"'

# search the content vector for articles about famous battles in Scottish history and only include results with Scottish in the title
results = search_redis(redis_client,
                       "Famous battles in Scottish history",
                       vector_field="title_vector",
                       k=5,
                       hybrid_fields=create_hybrid_field("title", "Scottish")
                       )
```

    0. First War of Scottish Independence (Score: 0.892)
    1. Wars of Scottish Independence (Score: 0.889)
    2. Second War of Scottish Independence (Score: 0.879)
    3. List of Scottish monarchs (Score: 0.873)
    4. Scottish Borders (Score: 0.863)



```python
# run a hybrid query for articles about Art in the title vector and only include results with the phrase "Leonardo da Vinci" in the text
results = search_redis(redis_client,
                       "Art",
                       vector_field="title_vector",
                       k=5,
                       hybrid_fields=create_hybrid_field("text", "Leonardo da Vinci")
                       )

# find specific mention of Leonardo da Vinci in the text that our full-text-search query returned
mention = [sentence for sentence in results[0].text.split("\n") if "Leonardo da Vinci" in sentence][0]
mention
```

    0. Art (Score: 1.0)
    1. Paint (Score: 0.896)
    2. Renaissance art (Score: 0.88)
    3. Painting (Score: 0.874)
    4. Renaissance (Score: 0.846)





    'In Europe, after the Middle Ages, there was a "Renaissance" which means "rebirth". People rediscovered science and artists were allowed to paint subjects other than religious subjects. People like Michelangelo and Leonardo da Vinci still painted religious pictures, but they also now could paint mythological pictures too. These artists also invented perspective where things in the distance look smaller in the picture. This was new because in the Middle Ages people would paint all the figures close up and just overlapping each other. These artists used nudity regularly in their art.'



## HNSW Index

Up until now, we've been using the ``FLAT`` or "brute-force" index to run our queries. Redis also supports the ``HNSW`` index which is a fast, approximate index. The ``HNSW`` index is a graph-based index that uses a hierarchical navigable small world graph to store vectors. The ``HNSW`` index is a good choice for large datasets where you want to run approximate queries.

``HNSW`` will take longer to build and consume more memory for most cases than ``FLAT`` but will be faster to run queries on, especially for large datasets.

The following cells will show how to create an ``HNSW`` index and run queries with it using the same data as before.


key: _node_content
string_value:
{"id_": "0c9815d8-fc5d-4cb8-a3d9-c0a8893338a4", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "51426527-c022-42e4-8c98-3975aae6d412", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b435e1ec-66c3-4142-9eac-670a1e037161", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "58f452b3738cc433fa12b11e9929d538fb1cd110c791f3bbb358041a2563983c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "76200e01-105c-4ddc-b449-5698a64d9ba6", "node_type": "1", "metadata": {}, "hash": "351b8cb5df5b075c6e22471bf20d5a9100892b9360be4b27030167d3e9e7b9d0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16087, "end_char_idx": 18745, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
'In Europe, after the Middle Ages, there was a "Renaissance" which means "rebirth". People rediscovered science and artists were allowed to paint subjects other than religious subjects. People like Michelangelo and Leonardo da Vinci still painted religious pictures, but they also now could paint mythological pictures too. These artists also invented perspective where things in the distance look smaller in the picture. This was new because in the Middle Ages people would paint all the figures close up and just overlapping each other. These artists used nudity regularly in their art.'



## HNSW Index

Up until now, we've been using the ``FLAT`` or "brute-force" index to run our queries. Redis also supports the ``HNSW`` index which is a fast, approximate index. The ``HNSW`` index is a graph-based index that uses a hierarchical navigable small world graph to store vectors. The ``HNSW`` index is a good choice for large datasets where you want to run approximate queries.

``HNSW`` will take longer to build and consume more memory for most cases than ``FLAT`` but will be faster to run queries on, especially for large datasets.

The following cells will show how to create an ``HNSW`` index and run queries with it using the same data as before.


```python
# re-define RediSearch vector fields to use HNSW index
title_embedding = VectorField("title_vector",
    "HNSW", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER
    }
)
text_embedding = VectorField("content_vector",
    "HNSW", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER
    }
)
fields = [title, url, text, title_embedding, text_embedding]
```


```python
import time
# Check if index exists
HNSW_INDEX_NAME = INDEX_NAME+ "_HNSW"

try:
    redis_client.ft(HNSW_INDEX_NAME).info()
    print("Index already exists")
except:
    # Create RediSearch Index
    redis_client.ft(HNSW_INDEX_NAME).create_index(
        fields = fields,
        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)
    )

# since RediSearch creates the index in the background for existing documents, we will wait until
# indexing is complete before running our queries. Although this is not necessary for the first query,
# some queries may take longer to run if the index is not fully built. In general, Redis will perform
# best when adding new documents to existing indices rather than new indices on existing documents.
while redis_client.ft(HNSW_INDEX_NAME).info()["indexing"] == "1":
    time.sleep(5)
```


key: _node_content
string_value:
{"id_": "76200e01-105c-4ddc-b449-5698a64d9ba6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "51426527-c022-42e4-8c98-3975aae6d412", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0c9815d8-fc5d-4cb8-a3d9-c0a8893338a4", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "5fd1d54c4034a33ea4948fff888d957f1222c4beb545833fc730d76ab80db802", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8d95b9fe-bf1f-4257-b5f2-c088df64b242", "node_type": "1", "metadata": {}, "hash": "10d82b05be97afebf2ff236a04b28e8635da2b57e558f661bf4e3eb3f1ebfbd8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17346, "end_char_idx": 19231, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# re-define RediSearch vector fields to use HNSW index
title_embedding = VectorField("title_vector",
    "HNSW", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER
    }
)
text_embedding = VectorField("content_vector",
    "HNSW", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER
    }
)
fields = [title, url, text, title_embedding, text_embedding]
```


```python
import time
# Check if index exists
HNSW_INDEX_NAME = INDEX_NAME+ "_HNSW"

try:
    redis_client.ft(HNSW_INDEX_NAME).info()
    print("Index already exists")
except:
    # Create RediSearch Index
    redis_client.ft(HNSW_INDEX_NAME).create_index(
        fields = fields,
        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)
    )

# since RediSearch creates the index in the background for existing documents, we will wait until
# indexing is complete before running our queries. Although this is not necessary for the first query,
# some queries may take longer to run if the index is not fully built. In general, Redis will perform
# best when adding new documents to existing indices rather than new indices on existing documents.
while redis_client.ft(HNSW_INDEX_NAME).info()["indexing"] == "1":
    time.sleep(5)
```


```python
results = search_redis(redis_client, 'modern art in Europe', index_name=HNSW_INDEX_NAME, k=10)
```

    0. Western Europe (Score: 0.868)
    1. Northern Europe (Score: 0.855)
    2. Central Europe (Score: 0.843)
    3. European (Score: 0.841)
    4. Eastern Europe (Score: 0.839)
    5. Europe (Score: 0.839)
    6. Western European Union (Score: 0.837)
    7. Southern Europe (Score: 0.831)
    8. Western civilization (Score: 0.83)
    9. Council of Europe (Score: 0.827)


key: _node_content
string_value:
{"id_": "8d95b9fe-bf1f-4257-b5f2-c088df64b242", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "51426527-c022-42e4-8c98-3975aae6d412", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "76200e01-105c-4ddc-b449-5698a64d9ba6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "14e964cd6f8306781b21d904a8ca1fdc0f100b32727f22168a88be4a47dd63d4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "72828370-ce17-4c12-b72c-290e95abd186", "node_type": "1", "metadata": {}, "hash": "5b0d7e8666a5b807e3bbea3400b6775cad72d42af45bacc0f92b52f9d959d8db", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17885, "end_char_idx": 21190, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import time
# Check if index exists
HNSW_INDEX_NAME = INDEX_NAME+ "_HNSW"

try:
    redis_client.ft(HNSW_INDEX_NAME).info()
    print("Index already exists")
except:
    # Create RediSearch Index
    redis_client.ft(HNSW_INDEX_NAME).create_index(
        fields = fields,
        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)
    )

# since RediSearch creates the index in the background for existing documents, we will wait until
# indexing is complete before running our queries. Although this is not necessary for the first query,
# some queries may take longer to run if the index is not fully built. In general, Redis will perform
# best when adding new documents to existing indices rather than new indices on existing documents.
while redis_client.ft(HNSW_INDEX_NAME).info()["indexing"] == "1":
    time.sleep(5)
```


```python
results = search_redis(redis_client, 'modern art in Europe', index_name=HNSW_INDEX_NAME, k=10)
```

    0. Western Europe (Score: 0.868)
    1. Northern Europe (Score: 0.855)
    2. Central Europe (Score: 0.843)
    3. European (Score: 0.841)
    4. Eastern Europe (Score: 0.839)
    5. Europe (Score: 0.839)
    6. Western European Union (Score: 0.837)
    7. Southern Europe (Score: 0.831)
    8. Western civilization (Score: 0.83)
    9. Council of Europe (Score: 0.827)



```python
# compare the results of the HNSW index to the FLAT index and time both queries
def time_queries(iterations: int = 10):
    print(" ----- Flat Index ----- ")
    t0 = time.time()
    for i in range(iterations):
        results_flat = search_redis(redis_client, 'modern art in Europe', k=10, print_results=False)
    t0 = (time.time() - t0) / iterations
    results_flat = search_redis(redis_client, 'modern art in Europe', k=10, print_results=True)
    print(f"Flat index query time: {round(t0, 3)} seconds\n")
    time.sleep(1)
    print(" ----- HNSW Index ------ ")
    t1 = time.time()
    for i in range(iterations):
        results_hnsw = search_redis(redis_client, 'modern art in Europe', index_name=HNSW_INDEX_NAME, k=10, print_results=False)
    t1 = (time.time() - t1) / iterations
    results_hnsw = search_redis(redis_client, 'modern art in Europe', index_name=HNSW_INDEX_NAME, k=10, print_results=True)
    print(f"HNSW index query time: {round(t1, 3)} seconds")
    print(" ------------------------ ")
time_queries()
```

     ----- Flat Index ----- 
    0. Museum of Modern Art (Score: 0.875)
    1. Western Europe (Score: 0.867)
    2. Renaissance art (Score: 0.864)
    3. Pop art (Score: 0.861)
    4. Northern Europe (Score: 0.855)
    5. Hellenistic art (Score: 0.853)
    6. Modernist literature (Score: 0.847)
    7. Art film (Score: 0.843)
    8. Central Europe (Score: 0.843)
    9. Art (Score: 0.842)
    Flat index query time: 0.263 seconds
    
     ----- HNSW Index ------ 
    0. Western Europe (Score: 0.867)
    1. Northern Europe (Score: 0.855)
    2. Central Europe (Score: 0.843)
    3. European (Score: 0.841)
    4. Eastern Europe (Score: 0.839)
    5. Europe (Score: 0.839)
    6. Western European Union (Score: 0.837)
    7. Southern Europe (Score: 0.831)
    8. Western civilization (Score: 0.83)
    9. Council of Europe (Score: 0.827)
    HNSW index query time: 0.129 seconds
     ------------------------


key: _node_content
string_value:
{"id_": "72828370-ce17-4c12-b72c-290e95abd186", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "51426527-c022-42e4-8c98-3975aae6d412", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8d95b9fe-bf1f-4257-b5f2-c088df64b242", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4f6925f7dae29ce25a580ccd831642b52ee7b2c90aeefef5e0ad0e0893a662d3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21195, "end_char_idx": 21209, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb


key: _node_content
string_value:
{"id_": "7f1f4b00-eb0f-49da-b5af-74ee3e41e27e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6e33b7ce-d0d0-43c1-b2e3-577e4b2a79e0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "df93daf4-421f-48c7-a280-2841d4293dbf", "node_type": "1", "metadata": {}, "hash": "48771e548cd855d4c03fd7f47a090609cdb397dbaf2db6a8493b3df0c8026bfe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3212, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Running Hybrid VSS Queries with Redis and OpenAI

This notebook provides an introduction to using Redis as a vector database with OpenAI embeddings and running hybrid queries that combine VSS and lexical search using Redis Query and Search capability. Redis is a scalable, real-time database that can be used as a vector database when using the [RediSearch Module](https://oss.redislabs.com/redisearch/). The Redis Query and Search capability allows you to index and search for vectors in Redis. This notebook will show you how to use the Redis Query and Search to index and search for vectors created by using the OpenAI API and stored in Redis.

Hybrid queries combine vector similarity with traditional Redis Query and Search filtering capabilities on GEO, NUMERIC, TAG or TEXT data simplifying application code. A common example of a hybrid query in an e-commerce use case is to find items visually similar to a given query image limited to items available in a GEO location and within a price range.

## Prerequisites

Before we start this project, we need to set up the following:

* start a Redis database with RediSearch (redis-stack)
* install libraries
    * [Redis-py](https://github.com/redis/redis-py)
* get your [OpenAI API key](https://beta.openai.com/account/api-keys)

===========================================================

### Start Redis

To keep this example simple, we will use the Redis Stack docker container which we can start as follows

```bash
$ docker-compose up -d
```

This also includes the [RedisInsight](https://redis.com/redis-enterprise/redis-insight/) GUI for managing your Redis database which you can view at [http://localhost:8001](http://localhost:8001) once you start the docker container.

You're all set up and ready to go! Next, we import and create our client for communicating with the Redis database we just created.

## Install Requirements

Redis-Py is the python client for communicating with Redis. We will use this to communicate with our Redis-stack database. 


```python
! pip install redis pandas openai

```

    Defaulting to user installation because normal site-packages is not writeable
    Requirement already satisfied: redis in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (4.5.4)
    Requirement already satisfied: pandas in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (2.0.1)
    Requirement already satisfied: openai in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (0.27.6)
    Requirement already satisfied: async-timeout>=4.0.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from redis) (4.0.2)
    Requirement already satisfied: python-dateutil>=2.8.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2.8.2)
    Requirement already satisfied: pytz>=2020.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2023.3)
    Requirement already satisfied: tzdata>=2022.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2023.3)
    Requirement already satisfied: numpy>=1.20.3 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (1.23.


key: _node_content
string_value:
{"id_": "df93daf4-421f-48c7-a280-2841d4293dbf", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6e33b7ce-d0d0-43c1-b2e3-577e4b2a79e0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7f1f4b00-eb0f-49da-b5af-74ee3e41e27e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "5fe4391cf140269d5190fb08948899485054c02937a5d473a8137e5eb960c365", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5c934b31-301e-4cd5-9145-39991fbd4f2d", "node_type": "1", "metadata": {}, "hash": "b5bbc484afc986c9a18fe4c73a4cb013ff3346a538ef8364957199a81b3e8c43", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2024, "end_char_idx": 4613, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
! pip install redis pandas openai

```

    Defaulting to user installation because normal site-packages is not writeable
    Requirement already satisfied: redis in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (4.5.4)
    Requirement already satisfied: pandas in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (2.0.1)
    Requirement already satisfied: openai in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (0.27.6)
    Requirement already satisfied: async-timeout>=4.0.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from redis) (4.0.2)
    Requirement already satisfied: python-dateutil>=2.8.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2.8.2)
    Requirement already satisfied: pytz>=2020.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2023.3)
    Requirement already satisfied: tzdata>=2022.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2023.3)
    Requirement already satisfied: numpy>=1.20.3 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (1.23.4)
    Requirement already satisfied: requests>=2.20 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (2.28.1)
    Requirement already satisfied: tqdm in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (4.64.1)
    Requirement already satisfied: aiohttp in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (3.8.4)
    Requirement already satisfied: six>=1.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)
    Requirement already satisfied: charset-normalizer<3,>=2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2.1.1)
    Requirement already satisfied: idna<4,>=2.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (3.4)
    Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (1.26.12)
    Requirement already satisfied: certifi>=2017.4.17 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2022.9.24)
    Requirement already satisfied: attrs>=17.3.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (23.1.0)
    Requirement already satisfied: multidict<7.


key: _node_content
string_value:
{"id_": "5c934b31-301e-4cd5-9145-39991fbd4f2d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6e33b7ce-d0d0-43c1-b2e3-577e4b2a79e0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "df93daf4-421f-48c7-a280-2841d4293dbf", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "744a50b963670a7a5df564127bba56bd8a4e063cd0ffc345c62100d8c7ba9f6d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1c9e0a08-e792-4608-88df-7af2ee5782ed", "node_type": "1", "metadata": {}, "hash": "ac5d72e261ea856563d0f3e6c0751e53d40e3f5887c8bb35bd939b6770c12dd3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3349, "end_char_idx": 5584, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
28.1)
    Requirement already satisfied: tqdm in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (4.64.1)
    Requirement already satisfied: aiohttp in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (3.8.4)
    Requirement already satisfied: six>=1.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)
    Requirement already satisfied: charset-normalizer<3,>=2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2.1.1)
    Requirement already satisfied: idna<4,>=2.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (3.4)
    Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (1.26.12)
    Requirement already satisfied: certifi>=2017.4.17 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2022.9.24)
    Requirement already satisfied: attrs>=17.3.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (23.1.0)
    Requirement already satisfied: multidict<7.0,>=4.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (6.0.4)
    Requirement already satisfied: yarl<2.0,>=1.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.9.2)
    Requirement already satisfied: frozenlist>=1.1.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.3)
    Requirement already satisfied: aiosignal>=1.1.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.1)


===========================================================
## Prepare your OpenAI API key

The `OpenAI API key` is used for vectorization of query data.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it to your environment variables as `OPENAI_API_KEY` by using following command:


key: _node_content
string_value:
{"id_": "1c9e0a08-e792-4608-88df-7af2ee5782ed", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6e33b7ce-d0d0-43c1-b2e3-577e4b2a79e0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5c934b31-301e-4cd5-9145-39991fbd4f2d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "9578ab9e6236f83de7fa47ba03515812c76a4c810c3b8d428fc89e0e4e5377b6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "251f32d5-dded-40e7-92ab-b5cf29c580e7", "node_type": "1", "metadata": {}, "hash": "54d7a83cada1ca318c4286170ec41e73ccba6ca043acce6f116fe4b991a1a070", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4233, "end_char_idx": 6312, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
20->openai) (1.26.12)
    Requirement already satisfied: certifi>=2017.4.17 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2022.9.24)
    Requirement already satisfied: attrs>=17.3.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (23.1.0)
    Requirement already satisfied: multidict<7.0,>=4.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (6.0.4)
    Requirement already satisfied: yarl<2.0,>=1.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.9.2)
    Requirement already satisfied: frozenlist>=1.1.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.3)
    Requirement already satisfied: aiosignal>=1.1.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.1)


===========================================================
## Prepare your OpenAI API key

The `OpenAI API key` is used for vectorization of query data.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it to your environment variables as `OPENAI_API_KEY` by using following command:


```python
# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.
import os
import openai

os.environ["OPENAI_API_KEY"] = '<YOUR_OPENAI_API_KEY>'

if os.getenv("OPENAI_API_KEY") is not None:
    openai.api_key = os.getenv("OPENAI_API_KEY")
    print ("OPENAI_API_KEY is ready")
else:
    print ("OPENAI_API_KEY environment variable not found")

```

    OPENAI_API_KEY is ready


## Load data

In this section we'll load and clean an ecommerce dataset. We'll generate embeddings using OpenAI and use this data to create an index in Redis and then search for similar vectors.


key: _node_content
string_value:
{"id_": "251f32d5-dded-40e7-92ab-b5cf29c580e7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6e33b7ce-d0d0-43c1-b2e3-577e4b2a79e0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1c9e0a08-e792-4608-88df-7af2ee5782ed", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "06c8e9ab77553ba89a1202b8133db929cd1eff6c8af40ef7192183c2bdfa8535", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9794f529-694e-491d-aeed-b3865f62c0fe", "node_type": "1", "metadata": {}, "hash": "dd753ee1a3c82a2e076ed454a1897d74cbf852bbf64769b6524519050158bef7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4761, "end_char_idx": 7685, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0,>=1.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.9.2)
    Requirement already satisfied: frozenlist>=1.1.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.3)
    Requirement already satisfied: aiosignal>=1.1.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.1)


===========================================================
## Prepare your OpenAI API key

The `OpenAI API key` is used for vectorization of query data.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it to your environment variables as `OPENAI_API_KEY` by using following command:


```python
# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.
import os
import openai

os.environ["OPENAI_API_KEY"] = '<YOUR_OPENAI_API_KEY>'

if os.getenv("OPENAI_API_KEY") is not None:
    openai.api_key = os.getenv("OPENAI_API_KEY")
    print ("OPENAI_API_KEY is ready")
else:
    print ("OPENAI_API_KEY environment variable not found")

```

    OPENAI_API_KEY is ready


## Load data

In this section we'll load and clean an ecommerce dataset. We'll generate embeddings using OpenAI and use this data to create an index in Redis and then search for similar vectors.


```python
import pandas as pd
import numpy as np
from typing import List

from utils.embeddings_utils import (
    get_embeddings,
    distances_from_embeddings,
    tsne_components_from_embeddings,
    chart_from_components,
    indices_of_nearest_neighbors_from_distances,
)

EMBEDDING_MODEL = "text-embedding-3-small"

# load in data and clean data types and drop null rows
df = pd.read_csv("../../data/styles_2k.csv", on_bad_lines='skip')
df.dropna(inplace=True)
df["year"] = df["year"].astype(int)
df.info()

# print dataframe
n_examples = 5
df.head(n_examples)

```

    <class 'pandas.core.frame.DataFrame'>
    Index: 1978 entries, 0 to 1998
    Data columns (total 10 columns):
     #   Column              Non-Null Count  Dtype 
    ---  ------              --------------  ----- 
     0   id                  1978 non-null   int64 
     1   gender              1978 non-null   object
     2   masterCategory      1978 non-null   object
     3   subCategory         1978 non-null   object
     4   articleType         1978 non-null   object
     5   baseColour          1978 non-null   object
     6   season              1978 non-null   object
     7   year                1978 non-null   int64 
     8   usage               1978 non-null   object
     9   productDisplayName  1978 non-null   object
    dtypes: int64(2), object(8)
    memory usage: 170.0+ KB


key: _node_content
string_value:
{"id_": "9794f529-694e-491d-aeed-b3865f62c0fe", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6e33b7ce-d0d0-43c1-b2e3-577e4b2a79e0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "251f32d5-dded-40e7-92ab-b5cf29c580e7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "30ae67437e33a6d7035398517d20fdec8b1741cec4c93e4501ac7dc277bdf4d0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2dde1eaa-01de-4218-8f26-03ed09f9270b", "node_type": "1", "metadata": {}, "hash": "8a1838f53f0ebecc073805ab5138940bc791f72e8342f728f05fa6c5cae09e4e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6315, "end_char_idx": 9727, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import pandas as pd
import numpy as np
from typing import List

from utils.embeddings_utils import (
    get_embeddings,
    distances_from_embeddings,
    tsne_components_from_embeddings,
    chart_from_components,
    indices_of_nearest_neighbors_from_distances,
)

EMBEDDING_MODEL = "text-embedding-3-small"

# load in data and clean data types and drop null rows
df = pd.read_csv("../../data/styles_2k.csv", on_bad_lines='skip')
df.dropna(inplace=True)
df["year"] = df["year"].astype(int)
df.info()

# print dataframe
n_examples = 5
df.head(n_examples)

```

    <class 'pandas.core.frame.DataFrame'>
    Index: 1978 entries, 0 to 1998
    Data columns (total 10 columns):
     #   Column              Non-Null Count  Dtype 
    ---  ------              --------------  ----- 
     0   id                  1978 non-null   int64 
     1   gender              1978 non-null   object
     2   masterCategory      1978 non-null   object
     3   subCategory         1978 non-null   object
     4   articleType         1978 non-null   object
     5   baseColour          1978 non-null   object
     6   season              1978 non-null   object
     7   year                1978 non-null   int64 
     8   usage               1978 non-null   object
     9   productDisplayName  1978 non-null   object
    dtypes: int64(2), object(8)
    memory usage: 170.0+ KB





<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>gender</th>
      <th>masterCategory</th>
      <th>subCategory</th>
      <th>articleType</th>
      <th>baseColour</th>
      <th>season</th>
      <th>year</th>
      <th>usage</th>
      <th>productDisplayName</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>15970</td>
      <td>Men</td>
      <td>Apparel</td>
      <td>Topwear</td>
      <td>Shirts</td>
      <td>Navy Blue</td>
      <td>Fall</td>
      <td>2011</td>
      <td>Casual</td>
      <td>Turtle Check Men Navy Blue Shirt</td>
    </tr>
    <tr>
      <th>1</th>
      <td>39386</td>
      <td>Men</td>
      <td>Apparel</td>
      <td>Bottomwear</td>
      <td>Jeans</td>
      <td>Blue</td>
      <td>Summer</td>
      <td>2012</td>
      <td>Casual</td>
      <td>Peter England Men Party Blue Jeans</td>
    </tr>
    <tr>
      <th>2</th>
      <td>59263</td>
      <td>Women</td>
      <td>Accessories</td>
      <td>Watches</td>
      <td>Watches</td>
      <td>Silver</td>
      <td>Winter</td>
      <td>2016</td>
      <td>Casual</td>
      <td>Titan Women Silver Watch</td>
    </tr>
    <tr>
      <th>3</th>
      <td>21379</td>
      <td>Men</td>
      <td>Apparel</td>
      <td>Bottomwear</td>
      <td>Track Pants</td>
      <td>Black</td>
      <td>Fall</td>
      <td>2011</td>
      <td>Casual</td>
      <td>Manchester United Men Solid Black Track Pants</td>
    </tr>
    <tr>
      <th>4</th>
      <td>53759</td>
      <td>Men</td>
      <td>Apparel</td>
      <td>Topwear</td>
      <td>Tshirts</td>
      <td>Grey</td>
      <td>Summer</td>
      <td>2012</td>
      <td>Casual</td>
      <td>Puma Men Grey T-shirt</td>
    </tr>
  </tbody>
</table>
</div>


key: _node_content
string_value:
{"id_": "2dde1eaa-01de-4218-8f26-03ed09f9270b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6e33b7ce-d0d0-43c1-b2e3-577e4b2a79e0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9794f529-694e-491d-aeed-b3865f62c0fe", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "9e0660a48e3bbab420a5fef8734244693b8114f102a002d8bfb8e036f58e5410", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "022dd967-8370-4dc9-84b1-9614c9f34af2", "node_type": "1", "metadata": {}, "hash": "fbc02fb4f6fbee58d7cafd7b61e743d803df335815c61765236654adeb6db225", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9732, "end_char_idx": 12404, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
df["product_text"] = df.apply(lambda row: f"name {row['productDisplayName']} category {row['masterCategory']} subcategory {row['subCategory']} color {row['baseColour']} gender {row['gender']}".lower(), axis=1)
df.rename({"id":"product_id"}, inplace=True, axis=1)

df.info()

```

    <class 'pandas.core.frame.DataFrame'>
    Index: 1978 entries, 0 to 1998
    Data columns (total 11 columns):
     #   Column              Non-Null Count  Dtype 
    ---  ------              --------------  ----- 
     0   product_id          1978 non-null   int64 
     1   gender              1978 non-null   object
     2   masterCategory      1978 non-null   object
     3   subCategory         1978 non-null   object
     4   articleType         1978 non-null   object
     5   baseColour          1978 non-null   object
     6   season              1978 non-null   object
     7   year                1978 non-null   int64 
     8   usage               1978 non-null   object
     9   productDisplayName  1978 non-null   object
     10  product_text        1978 non-null   object
    dtypes: int64(2), object(9)
    memory usage: 185.4+ KB



```python
# check out one of the texts we will use to create semantic embeddings
df["product_text"][0]

```




    'name turtle check men navy blue shirt category apparel subcategory topwear color navy blue gender men'



## Connect to Redis

Now that we have our Redis database running, we can connect to it using the Redis-py client. We will use the default host and port for the Redis database which is `localhost:6379`.




```python
import redis
from redis.commands.search.indexDefinition import (
    IndexDefinition,
    IndexType
)
from redis.commands.search.query import Query
from redis.commands.search.field import (
    TagField,
    NumericField,
    TextField,
    VectorField
)

REDIS_HOST =  "localhost"
REDIS_PORT = 6379
REDIS_PASSWORD = "" # default for passwordless Redis

# Connect to Redis
redis_client = redis.Redis(
    host=REDIS_HOST,
    port=REDIS_PORT,
    password=REDIS_PASSWORD
)
redis_client.ping()

```




    True



## Creating a Search Index in Redis

The below cells will show how to specify and create a search index in Redis. We will:

1. Set some constants for defining our index like the distance metric and the index name
2. Define the index schema with RediSearch fields
3. Create the index


```python
# Constants
INDEX_NAME = "product_embeddings"           # name of the search index
PREFIX = "doc"                            # prefix for the document keys
DISTANCE_METRIC = "L2"                # distance metric for the vectors (ex. COSINE, IP, L2)
NUMBER_OF_VECTORS = len(df)

```


key: _node_content
string_value:
{"id_": "022dd967-8370-4dc9-84b1-9614c9f34af2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6e33b7ce-d0d0-43c1-b2e3-577e4b2a79e0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2dde1eaa-01de-4218-8f26-03ed09f9270b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1afbe544ed834c5a9480b442b36c737d26e2ffc587253647767748a8f67870ee", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1bf9de4a-1274-4eae-9dc8-5359eed19026", "node_type": "1", "metadata": {}, "hash": "e9dcab3fc94bd79bc76d8419f0f6229bdb90ebe00c4b7b45ecd1a5172bdbd1c7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10875, "end_char_idx": 13801, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# check out one of the texts we will use to create semantic embeddings
df["product_text"][0]

```




    'name turtle check men navy blue shirt category apparel subcategory topwear color navy blue gender men'



## Connect to Redis

Now that we have our Redis database running, we can connect to it using the Redis-py client. We will use the default host and port for the Redis database which is `localhost:6379`.




```python
import redis
from redis.commands.search.indexDefinition import (
    IndexDefinition,
    IndexType
)
from redis.commands.search.query import Query
from redis.commands.search.field import (
    TagField,
    NumericField,
    TextField,
    VectorField
)

REDIS_HOST =  "localhost"
REDIS_PORT = 6379
REDIS_PASSWORD = "" # default for passwordless Redis

# Connect to Redis
redis_client = redis.Redis(
    host=REDIS_HOST,
    port=REDIS_PORT,
    password=REDIS_PASSWORD
)
redis_client.ping()

```




    True



## Creating a Search Index in Redis

The below cells will show how to specify and create a search index in Redis. We will:

1. Set some constants for defining our index like the distance metric and the index name
2. Define the index schema with RediSearch fields
3. Create the index


```python
# Constants
INDEX_NAME = "product_embeddings"           # name of the search index
PREFIX = "doc"                            # prefix for the document keys
DISTANCE_METRIC = "L2"                # distance metric for the vectors (ex. COSINE, IP, L2)
NUMBER_OF_VECTORS = len(df)

```


```python
# Define RediSearch fields for each of the columns in the dataset
name = TextField(name="productDisplayName")
category = TagField(name="masterCategory")
articleType = TagField(name="articleType")
gender = TagField(name="gender")
season = TagField(name="season")
year = NumericField(name="year")
text_embedding = VectorField("product_vector",
    "FLAT", {
        "TYPE": "FLOAT32",
        "DIM": 1536,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": NUMBER_OF_VECTORS,
    }
)
fields = [name, category, articleType, gender, season, year, text_embedding]

```


```python
# Check if index exists
try:
    redis_client.ft(INDEX_NAME).info()
    print("Index already exists")
except:
    # Create RediSearch Index
    redis_client.ft(INDEX_NAME).create_index(
        fields = fields,
        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)
)

```

## Generate OpenAI Embeddings and Load Documents into the Index

Now that we have a search index, we can load documents into it. We will use the dataframe containing the styles dataset loaded previously. In Redis, either the HASH or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The cells below will show how to get OpenAI embeddings for the different products and load documents into the index.


key: _node_content
string_value:
{"id_": "1bf9de4a-1274-4eae-9dc8-5359eed19026", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6e33b7ce-d0d0-43c1-b2e3-577e4b2a79e0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "022dd967-8370-4dc9-84b1-9614c9f34af2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e8278f65a88c4513f9c039734366cf0f8c8577dbfb2f6ff4c395eeb7dc481a00", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d48e5d65-a5f4-44fa-b15e-ff5b7051d616", "node_type": "1", "metadata": {}, "hash": "0cc7cfab9ef3a6ccc1ae9dd760ea15783bb28972036b85c80f10369ce4a5c891", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12113, "end_char_idx": 15336, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Constants
INDEX_NAME = "product_embeddings"           # name of the search index
PREFIX = "doc"                            # prefix for the document keys
DISTANCE_METRIC = "L2"                # distance metric for the vectors (ex. COSINE, IP, L2)
NUMBER_OF_VECTORS = len(df)

```


```python
# Define RediSearch fields for each of the columns in the dataset
name = TextField(name="productDisplayName")
category = TagField(name="masterCategory")
articleType = TagField(name="articleType")
gender = TagField(name="gender")
season = TagField(name="season")
year = NumericField(name="year")
text_embedding = VectorField("product_vector",
    "FLAT", {
        "TYPE": "FLOAT32",
        "DIM": 1536,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": NUMBER_OF_VECTORS,
    }
)
fields = [name, category, articleType, gender, season, year, text_embedding]

```


```python
# Check if index exists
try:
    redis_client.ft(INDEX_NAME).info()
    print("Index already exists")
except:
    # Create RediSearch Index
    redis_client.ft(INDEX_NAME).create_index(
        fields = fields,
        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)
)

```

## Generate OpenAI Embeddings and Load Documents into the Index

Now that we have a search index, we can load documents into it. We will use the dataframe containing the styles dataset loaded previously. In Redis, either the HASH or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The cells below will show how to get OpenAI embeddings for the different products and load documents into the index.


```python
# Use OpenAI get_embeddings batch requests to speed up embedding creation
def embeddings_batch_request(documents: pd.DataFrame):
    records = documents.to_dict("records")
    print("Records to process: ", len(records))
    product_vectors = []
    docs = []
    batchsize = 1000

    for idx,doc in enumerate(records,start=1):
        # create byte vectors
        docs.append(doc["product_text"])
        if idx % batchsize == 0:
            product_vectors += get_embeddings(docs, EMBEDDING_MODEL)
            docs.clear()
            print("Vectors processed ", len(product_vectors), end='\r')
    product_vectors += get_embeddings(docs, EMBEDDING_MODEL)
    print("Vectors processed ", len(product_vectors), end='\r')
    return product_vectors

```


```python
def index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):
    product_vectors = embeddings_batch_request(documents)
    records = documents.to_dict("records")
    batchsize = 500

    # Use Redis pipelines to batch calls and save on round trip network communication
    pipe = client.pipeline()
    for idx,doc in enumerate(records,start=1):
        key = f"{prefix}:{str(doc['product_id'])}"

        # create byte vectors
        text_embedding = np.array((product_vectors[idx-1]), dtype=np.float32).tobytes()

        # replace list of floats with byte vectors
        doc["product_vector"] = text_embedding

        pipe.hset(key, mapping = doc)
        if idx % batchsize == 0:
            pipe.execute()
    pipe.execute()

```


key: _node_content
string_value:
{"id_": "d48e5d65-a5f4-44fa-b15e-ff5b7051d616", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6e33b7ce-d0d0-43c1-b2e3-577e4b2a79e0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1bf9de4a-1274-4eae-9dc8-5359eed19026", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "40bc1472f58d584aa6abb178eab99bf1c28ea2d78df3eef34313406bae03dd87", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "27c24dcd-4cb9-4be6-af0c-389d96688ed2", "node_type": "1", "metadata": {}, "hash": "84210844c53e5d61d8ed50964d4be8ebf098a4d84e7d0e7bbbafc30037758387", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13804, "end_char_idx": 16042, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


