# embedding_metadata

key: _node_content
string_value:
{"id_": "6420e2cc-8203-4f93-9d24-e0701e68c45b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b726e3c9-eb22-4b81-a44b-f6d00daa5d4b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "5e7cf39b2939fc387f82f1b1edcd54492de9e9c10b9794bd1c595ccf56724885", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aef7dd39-207a-42c9-8625-97dec779cb2f", "node_type": "1", "metadata": {}, "hash": "e31dbdf83f0b175955f34de70464687d224d388059d362c5fb12d62947de6bde", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23206, "end_char_idx": 25404, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
is the eighth month of the year ...</td>
      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>
      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>6</td>
      <td>https://simple.wikipedia.org/wiki/Art</td>
      <td>Art</td>
      <td>Art is a creative activity that expresses imag...</td>
      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>
      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>
      <td>2</td>
    </tr>
    <tr>
      <th>3</th>
      <td>8</td>
      <td>https://simple.wikipedia.org/wiki/A</td>
      <td>A</td>
      <td>A or a is the first letter of the English alph...</td>
      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>
      <td>[0.024894846603274345, -0.022186409682035446, ...</td>
      <td>3</td>
    </tr>
    <tr>
      <th>4</th>
      <td>9</td>
      <td>https://simple.wikipedia.org/wiki/Air</td>
      <td>Air</td>
      <td>Air refers to the Earth's atmosphere. Air is a...</td>
      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>
      <td>[0.021524671465158463, 0.018522677943110466, -...</td>
      <td>4</td>
    </tr>
  </tbody>
</table>
</div>




```python
# Read vectors from strings back into a list
article_df['title_vector'] = article_df.title_vector.apply(literal_eval)
article_df['content_vector'] = article_df.content_vector.apply(literal_eval)

# Set vector_id to be a string
article_df['vector_id'] = article_df['vector_id'].apply(str)
```


```python
article_df.info(show_counts=True)
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 25000 entries, 0 to 24999
    Data columns (total 7 columns):
     #   Column          Non-Null Count  Dtype 
    ---  ------          --------------  ----- 
     0   id              25000 non-null  int64 
     1   url             25000 non-null  object
     2   title           25000 non-null  object
     3   text            25000 non-null  object
     4   title_vector    25000 non-null  object
     5   content_vector  25000 non-null  object
     6   vector_id       25000 non-null  object
    dtypes: int64(1), object(6)
    memory usage: 1.3+ MB


key: _node_content
string_value:
{"id_": "aef7dd39-207a-42c9-8625-97dec779cb2f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6420e2cc-8203-4f93-9d24-e0701e68c45b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "2ece711683c6310cd604958b8cd5be14dbba53c8c66e3636993a862eb40a39da", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "af718734-5c1f-444a-b8b0-52c0f41af662", "node_type": "1", "metadata": {}, "hash": "6915acaf8b4f99ccbaf24180f03eb4a16e0105699601e93c153d4568a5fff1dd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24237, "end_char_idx": 27234, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Air is a...</td>
      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>
      <td>[0.021524671465158463, 0.018522677943110466, -...</td>
      <td>4</td>
    </tr>
  </tbody>
</table>
</div>




```python
# Read vectors from strings back into a list
article_df['title_vector'] = article_df.title_vector.apply(literal_eval)
article_df['content_vector'] = article_df.content_vector.apply(literal_eval)

# Set vector_id to be a string
article_df['vector_id'] = article_df['vector_id'].apply(str)
```


```python
article_df.info(show_counts=True)
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 25000 entries, 0 to 24999
    Data columns (total 7 columns):
     #   Column          Non-Null Count  Dtype 
    ---  ------          --------------  ----- 
     0   id              25000 non-null  int64 
     1   url             25000 non-null  object
     2   title           25000 non-null  object
     3   text            25000 non-null  object
     4   title_vector    25000 non-null  object
     5   content_vector  25000 non-null  object
     6   vector_id       25000 non-null  object
    dtypes: int64(1), object(6)
    memory usage: 1.3+ MB


# Chroma

We'll index these embedded documents in a vector database and search them. The first option we'll look at is **Chroma**, an easy to use open-source self-hosted in-memory vector database, designed for working with embeddings together with LLMs. 

In this section, we will:
- Instantiate the Chroma client
- Create collections for each class of embedding 
- Query each collection 

### Instantiate the Chroma client

Create the Chroma client. By default, Chroma is ephemeral and runs in memory. 
However, you can easily set up a persistent configuration which writes to disk.


```python
chroma_client = chromadb.EphemeralClient() # Equivalent to chromadb.Client(), ephemeral.
# Uncomment for persistent client
# chroma_client = chromadb.PersistentClient()
```

### Create collections

Chroma collections allow you to store and filter with arbitrary metadata, making it easy to query subsets of the embedded data. 

Chroma is already integrated with OpenAI's embedding functions. The best way to use them is on construction of a collection, as follows.
Alternatively, you can 'bring your own embeddings'. More information can be found [here](https://docs.trychroma.com/embeddings)


```python
from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction

# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.

# Note. alternatively you can set a temporary env variable like this:
# os.environ["OPENAI_API_KEY"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

if os.getenv("OPENAI_API_KEY") is not None:
    openai.api_key = os.getenv("OPENAI_API_KEY")
    print ("OPENAI_API_KEY is ready")
else:
    print ("OPENAI_API_KEY environment variable not found")


key: _node_content
string_value:
{"id_": "af718734-5c1f-444a-b8b0-52c0f41af662", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aef7dd39-207a-42c9-8625-97dec779cb2f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "d856fba364552c3d638a374629a47bba05767b95bc0837ae152492f23757fee6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e514a9be-ec80-4132-835c-95515201e90a", "node_type": "1", "metadata": {}, "hash": "4cab9f149edcde8976c9c43075473e4fd728f89cf4451a534e42faddc7ac53e0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25407, "end_char_idx": 28948, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Chroma

We'll index these embedded documents in a vector database and search them. The first option we'll look at is **Chroma**, an easy to use open-source self-hosted in-memory vector database, designed for working with embeddings together with LLMs. 

In this section, we will:
- Instantiate the Chroma client
- Create collections for each class of embedding 
- Query each collection 

### Instantiate the Chroma client

Create the Chroma client. By default, Chroma is ephemeral and runs in memory. 
However, you can easily set up a persistent configuration which writes to disk.


```python
chroma_client = chromadb.EphemeralClient() # Equivalent to chromadb.Client(), ephemeral.
# Uncomment for persistent client
# chroma_client = chromadb.PersistentClient()
```

### Create collections

Chroma collections allow you to store and filter with arbitrary metadata, making it easy to query subsets of the embedded data. 

Chroma is already integrated with OpenAI's embedding functions. The best way to use them is on construction of a collection, as follows.
Alternatively, you can 'bring your own embeddings'. More information can be found [here](https://docs.trychroma.com/embeddings)


```python
from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction

# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.

# Note. alternatively you can set a temporary env variable like this:
# os.environ["OPENAI_API_KEY"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

if os.getenv("OPENAI_API_KEY") is not None:
    openai.api_key = os.getenv("OPENAI_API_KEY")
    print ("OPENAI_API_KEY is ready")
else:
    print ("OPENAI_API_KEY environment variable not found")


embedding_function = OpenAIEmbeddingFunction(api_key=os.environ.get('OPENAI_API_KEY'), model_name=EMBEDDING_MODEL)

wikipedia_content_collection = chroma_client.create_collection(name='wikipedia_content', embedding_function=embedding_function)
wikipedia_title_collection = chroma_client.create_collection(name='wikipedia_titles', embedding_function=embedding_function)
```

    OPENAI_API_KEY is ready


### Populate the collections

Chroma collections allow you to populate, and filter on, whatever metadata you like. Chroma can also store the text alongside the vectors, and return everything in a single `query` call, when this is more convenient. 

For this use-case, we'll just store the embeddings and IDs, and use these to index the original dataframe. 


```python
# Add the content vectors
wikipedia_content_collection.add(
    ids=article_df.vector_id.tolist(),
    embeddings=article_df.content_vector.tolist(),
)

# Add the title vectors
wikipedia_title_collection.add(
    ids=article_df.vector_id.tolist(),
    embeddings=article_df.title_vector.tolist(),
)
```

### Search the collections

Chroma handles embedding queries for you if an embedding function is set, like in this example.


```python
def query_collection(collection, query, max_results, dataframe):
    results = collection.query(query_texts=query, n_results=max_results, include=['distances']) 
    df = pd.DataFrame({
                'id':results['ids'][0], 
                'score':results['distances'][0],
                'title': dataframe[dataframe.vector_id.isin(results['ids'][0])]['title'],
                'content': dataframe[dataframe.vector_id.isin(results['ids'][0])]['text'],
                })
    
    return df
```


key: _node_content
string_value:
{"id_": "e514a9be-ec80-4132-835c-95515201e90a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "af718734-5c1f-444a-b8b0-52c0f41af662", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "a5064641359a1e2fbf00e5df2f2ed51559f34313de1413b20581dc9b1f00528b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "186230b9-d662-477c-a848-e7afe7297cff", "node_type": "1", "metadata": {}, "hash": "3b60c49431a6444e1de36ffa84f1f5ac907c937da763d81595ffcda70ae57123", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27237, "end_char_idx": 29153, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
embedding_function = OpenAIEmbeddingFunction(api_key=os.environ.get('OPENAI_API_KEY'), model_name=EMBEDDING_MODEL)

wikipedia_content_collection = chroma_client.create_collection(name='wikipedia_content', embedding_function=embedding_function)
wikipedia_title_collection = chroma_client.create_collection(name='wikipedia_titles', embedding_function=embedding_function)
```

    OPENAI_API_KEY is ready


### Populate the collections

Chroma collections allow you to populate, and filter on, whatever metadata you like. Chroma can also store the text alongside the vectors, and return everything in a single `query` call, when this is more convenient. 

For this use-case, we'll just store the embeddings and IDs, and use these to index the original dataframe. 


```python
# Add the content vectors
wikipedia_content_collection.add(
    ids=article_df.vector_id.tolist(),
    embeddings=article_df.content_vector.tolist(),
)

# Add the title vectors
wikipedia_title_collection.add(
    ids=article_df.vector_id.tolist(),
    embeddings=article_df.title_vector.tolist(),
)
```

### Search the collections

Chroma handles embedding queries for you if an embedding function is set, like in this example.


```python
def query_collection(collection, query, max_results, dataframe):
    results = collection.query(query_texts=query, n_results=max_results, include=['distances']) 
    df = pd.DataFrame({
                'id':results['ids'][0], 
                'score':results['distances'][0],
                'title': dataframe[dataframe.vector_id.isin(results['ids'][0])]['title'],
                'content': dataframe[dataframe.vector_id.isin(results['ids'][0])]['text'],
                })
    
    return df
```


```python
title_query_result = query_collection(
    collection=wikipedia_title_collection,
    query="modern art in Europe",
    max_results=10,
    dataframe=article_df
)
title_query_result.head()
```


key: _node_content
string_value:
{"id_": "186230b9-d662-477c-a848-e7afe7297cff", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e514a9be-ec80-4132-835c-95515201e90a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f707ac8cbb10a5cc5e3139209e55fd723b86a39e6894247a51d6b44bef087599", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1fae1473-358c-4a40-8221-d51d93698a8a", "node_type": "1", "metadata": {}, "hash": "8261dc1b40eb4d98829a6e56f7c6e2ebddb0e2ea3c57980015bc72ad04bd8391", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27641, "end_char_idx": 30512, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Populate the collections

Chroma collections allow you to populate, and filter on, whatever metadata you like. Chroma can also store the text alongside the vectors, and return everything in a single `query` call, when this is more convenient. 

For this use-case, we'll just store the embeddings and IDs, and use these to index the original dataframe. 


```python
# Add the content vectors
wikipedia_content_collection.add(
    ids=article_df.vector_id.tolist(),
    embeddings=article_df.content_vector.tolist(),
)

# Add the title vectors
wikipedia_title_collection.add(
    ids=article_df.vector_id.tolist(),
    embeddings=article_df.title_vector.tolist(),
)
```

### Search the collections

Chroma handles embedding queries for you if an embedding function is set, like in this example.


```python
def query_collection(collection, query, max_results, dataframe):
    results = collection.query(query_texts=query, n_results=max_results, include=['distances']) 
    df = pd.DataFrame({
                'id':results['ids'][0], 
                'score':results['distances'][0],
                'title': dataframe[dataframe.vector_id.isin(results['ids'][0])]['title'],
                'content': dataframe[dataframe.vector_id.isin(results['ids'][0])]['text'],
                })
    
    return df
```


```python
title_query_result = query_collection(
    collection=wikipedia_title_collection,
    query="modern art in Europe",
    max_results=10,
    dataframe=article_df
)
title_query_result.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>score</th>
      <th>title</th>
      <th>content</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>2</th>
      <td>23266</td>
      <td>0.249646</td>
      <td>Art</td>
      <td>Art is a creative activity that expresses imag...</td>
    </tr>
    <tr>
      <th>11777</th>
      <td>15436</td>
      <td>0.271688</td>
      <td>Hellenistic art</td>
      <td>The art of the Hellenistic time (from 400 B.C....</td>
    </tr>
    <tr>
      <th>12178</th>
      <td>23265</td>
      <td>0.279306</td>
      <td>Byzantine art</td>
      <td>Byzantine art is a form of Christian Greek art...</td>
    </tr>
    <tr>
      <th>13215</th>
      <td>11777</td>
      <td>0.294415</td>
      <td>Art film</td>
      <td>Art films are a type of movie that is very dif...</td>
    </tr>
    <tr>
      <th>15436</th>
      <td>22108</td>
      <td>0.305937</td>
      <td>Renaissance art</td>
      <td>Many of the most famous and best-loved works o...</td>
    </tr>
  </tbody>
</table>
</div>


key: _node_content
string_value:
{"id_": "1fae1473-358c-4a40-8221-d51d93698a8a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7c54ae26-1c8c-487f-bf17-dd7249ab7b6c", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "358148240794022cd40de030ff5b55b1ca1a25fbd1e78d82f880f37d127090a0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "186230b9-d662-477c-a848-e7afe7297cff", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/Using_Chroma_for_embeddings_search.ipynb", "title": "Using Chroma for Embeddings Search", "tokens": 10702, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "15f1d4fcb09d1998d0efb4b49cbb3c4072d5dd45a01649a0304ada84c6a7d110", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30517, "end_char_idx": 32346, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
content_query_result = query_collection(
    collection=wikipedia_content_collection,
    query="Famous battles in Scottish history",
    max_results=10,
    dataframe=article_df
)
content_query_result.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>score</th>
      <th>title</th>
      <th>content</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>2923</th>
      <td>13135</td>
      <td>0.261328</td>
      <td>1651</td>
      <td>\n\nEvents \n January 1 – Charles II crowned K...</td>
    </tr>
    <tr>
      <th>3694</th>
      <td>13571</td>
      <td>0.277058</td>
      <td>Stirling</td>
      <td>Stirling () is a city in the middle of Scotlan...</td>
    </tr>
    <tr>
      <th>6248</th>
      <td>2923</td>
      <td>0.294823</td>
      <td>841</td>
      <td>\n\nEvents \n June 25: Battle of Fontenay – Lo...</td>
    </tr>
    <tr>
      <th>6297</th>
      <td>13568</td>
      <td>0.300756</td>
      <td>1746</td>
      <td>\n\nEvents \n January 8 – Bonnie Prince Charli...</td>
    </tr>
    <tr>
      <th>11702</th>
      <td>11708</td>
      <td>0.307572</td>
      <td>William Wallace</td>
      <td>William Wallace was a Scottish knight who foug...</td>
    </tr>
  </tbody>
</table>
</div>



Now that you've got a basic embeddings search running, you can [hop over to the Chroma docs](https://docs.trychroma.com/usage-guide#using-where-filters) to learn more about how to add filters to your query, update/delete data in your collections, and deploy Chroma.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb


key: _node_content
string_value:
{"id_": "0ec70657-4275-4e5e-af3d-2cbec0f22732", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d052b78d-de76-46ee-ae30-cd00595572c2", "node_type": "1", "metadata": {}, "hash": "2a683a846ba84360e06402d30d14238dd58a09a52d66cff465ff09bba1bf294b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3231, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Robust Question Answering with Chroma and OpenAI 

This notebook guides you step-by-step through answering questions about a collection of data, using [Chroma](https://trychroma.com), an open-source embeddings database, along with OpenAI's [text embeddings](https://platform.openai.com/docs/guides/embeddings/use-cases) and [chat completion](https://platform.openai.com/docs/guides/chat) API's. 

Additionally, this notebook demonstrates some of the tradeoffs in making a question answering system more robust. As we shall see, *simple querying doesn't always create the best results*! 

## Question Answering with LLMs

Large language models (LLMs) like OpenAI's ChatGPT can be used to answer questions about data that the model may not have been trained on, or have access to. For example;

- Personal data like e-mails and notes
- Highly specialized data like archival or legal documents
- Newly created data like recent news stories

In order to overcome this limitation, we can use a data store which is amenable to querying in natural language, just like the LLM itself. An embeddings store like Chroma represents documents as [embeddings](https://openai.com/blog/introducing-text-and-code-embeddings), alongside the documents themselves. 

By embedding a text query, Chroma can find relevant documents, which we can then pass to the LLM to answer our question. We'll show detailed examples and variants of this approach. 

# Setup and preliminaries

First we make sure the python dependencies we need are installed. 


```python
%pip install -qU openai chromadb pandas
```

    Note: you may need to restart the kernel to use updated packages.


We use OpenAI's API's throughout this notebook. You can get an API key from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys)

You can add your API key as an environment variable by executing the command `export OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx` in a terminal. Note that you will need to reload the notebook if the environment variable wasn't set yet. Alternatively, you can set it in the notebook, see below. 


```python
import os

# Uncomment the following line to set the environment variable in the notebook
# os.environ["OPENAI_API_KEY"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

if os.getenv("OPENAI_API_KEY") is not None:
    print("OPENAI_API_KEY is ready")
    import openai
    openai.api_key = os.getenv("OPENAI_API_KEY")
else:
    print("OPENAI_API_KEY environment variable not found")
```

    OPENAI_API_KEY is ready


# Dataset

Throughout this notebook, we use the [SciFact dataset](https://github.com/allenai/scifact). This is a curated dataset of expert annotated scientific claims, with an accompanying text corpus of paper titles and abstracts. Each claim may be supported, contradicted, or not have enough evidence either way, according to the documents in the corpus. 

Having the corpus available as ground-truth allows us to investigate how well the following approaches to LLM question answering perform. 


```python
# Load the claim dataset
import pandas as pd

data_path = '../../data'

claim_df = pd.read_json(f'{data_path}/scifact_claims.jsonl', lines=True)
claim_df.head()
```


key: _node_content
string_value:
{"id_": "d052b78d-de76-46ee-ae30-cd00595572c2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0ec70657-4275-4e5e-af3d-2cbec0f22732", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e5dad5a280180581156d07349f86803495a87fe18d2cc48935707f2548b0f9ca", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f118969f-d730-4335-8618-5897da4e0a3c", "node_type": "1", "metadata": {}, "hash": "266c03c3164aa015d5aee257ec93c88a5e2583aa158dfe4fe1195110e1268966", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1528, "end_char_idx": 4626, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
%pip install -qU openai chromadb pandas
```

    Note: you may need to restart the kernel to use updated packages.


We use OpenAI's API's throughout this notebook. You can get an API key from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys)

You can add your API key as an environment variable by executing the command `export OPENAI_API_KEY=sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx` in a terminal. Note that you will need to reload the notebook if the environment variable wasn't set yet. Alternatively, you can set it in the notebook, see below. 


```python
import os

# Uncomment the following line to set the environment variable in the notebook
# os.environ["OPENAI_API_KEY"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

if os.getenv("OPENAI_API_KEY") is not None:
    print("OPENAI_API_KEY is ready")
    import openai
    openai.api_key = os.getenv("OPENAI_API_KEY")
else:
    print("OPENAI_API_KEY environment variable not found")
```

    OPENAI_API_KEY is ready


# Dataset

Throughout this notebook, we use the [SciFact dataset](https://github.com/allenai/scifact). This is a curated dataset of expert annotated scientific claims, with an accompanying text corpus of paper titles and abstracts. Each claim may be supported, contradicted, or not have enough evidence either way, according to the documents in the corpus. 

Having the corpus available as ground-truth allows us to investigate how well the following approaches to LLM question answering perform. 


```python
# Load the claim dataset
import pandas as pd

data_path = '../../data'

claim_df = pd.read_json(f'{data_path}/scifact_claims.jsonl', lines=True)
claim_df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>claim</th>
      <th>evidence</th>
      <th>cited_doc_ids</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>0-dimensional biomaterials show inductive prop...</td>
      <td>{}</td>
      <td>[31715818]</td>
    </tr>
    <tr>
      <th>1</th>
      <td>3</td>
      <td>1,000 genomes project enables mapping of genet...</td>
      <td>{'14717500': [{'sentences': [2, 5], 'label': '...</td>
      <td>[14717500]</td>
    </tr>
    <tr>
      <th>2</th>
      <td>5</td>
      <td>1/2000 in UK have abnormal PrP positivity.</td>
      <td>{'13734012': [{'sentences': [4], 'label': 'SUP...</td>
      <td>[13734012]</td>
    </tr>
    <tr>
      <th>3</th>
      <td>13</td>
      <td>5% of perinatal mortality is due to low birth ...</td>
      <td>{}</td>
      <td>[1606628]</td>
    </tr>
    <tr>
      <th>4</th>
      <td>36</td>
      <td>A deficiency of vitamin B12 increases blood le...</td>
      <td>{}</td>
      <td>[5152028, 11705328]</td>
    </tr>
  </tbody>
</table>
</div>


key: _node_content
string_value:
{"id_": "f118969f-d730-4335-8618-5897da4e0a3c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d052b78d-de76-46ee-ae30-cd00595572c2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "68c24ecd4ade173279c9351091e77c19fb71d14bb1aae504c4aac90a6a212f07", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a21fb14e-af3a-4fcc-9ecc-40fd6cc10fb6", "node_type": "1", "metadata": {}, "hash": "355fbc1e7a71331fb5247d3623d9b8fb0538274ed8dba5cc363dcba2fd1c0cbf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4630, "end_char_idx": 7216, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Just asking the model

GPT-3.5 was trained on a large amount of scientific information. As a baseline, we'd like to understand what the model already knows without any further context. This will allow us to calibrate overall performance. 

We construct an appropriate prompt, with some example facts, then query the model with each claim in the dataset. We ask the model to assess a claim as 'True', 'False', or 'NEE' if there is not enough evidence one way or the other. 


```python
def build_prompt(claim):
    return [
        {"role": "system", "content": "I will ask you to assess a scientific claim. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence."},
        {"role": "user", "content": f"""        
Example:

Claim:
0-dimensional biomaterials show inductive properties.

Assessment:
False

Claim:
1/2000 in UK have abnormal PrP positivity.

Assessment:
True

Claim:
Aspirin inhibits the production of PGE2.

Assessment:
False

End of examples. Assess the following claim:

Claim:
{claim}

Assessment:
"""}
    ]


def assess_claims(claims):
    responses = []
    # Query the OpenAI API
    for claim in claims:
        response = openai.ChatCompletion.create(
            model='gpt-3.5-turbo',
            messages=build_prompt(claim),
            max_tokens=3,
        )
        # Strip any punctuation or whitespace from the response
        responses.append(response.choices[0].message.content.strip('., '))

    return responses
```

We sample 100 claims from the dataset


```python
# Let's take a look at 100 claims
samples = claim_df.sample(50)

claims = samples['claim'].tolist() 

```

We evaluate the ground-truth according to the dataset. From the dataset description, each claim is either supported or contradicted by the evidence, or else there isn't enough evidence either way. 


```python
def get_groundtruth(evidence):
    groundtruth = []
    for e in evidence:
        # Evidence is empty 
        if len(e) == 0:
            groundtruth.append('NEE')
        else:
            # In this dataset, all evidence for a given claim is consistent, either SUPPORT or CONTRADICT
            if list(e.values())[0][0]['label'] == 'SUPPORT':
                groundtruth.append('True')
            else:
                groundtruth.append('False')
    return groundtruth
```


```python
evidence = samples['evidence'].tolist()
groundtruth = get_groundtruth(evidence)
```

We also output the confusion matrix, comparing the model's assessments with the ground truth, in an easy to read table.


key: _node_content
string_value:
{"id_": "a21fb14e-af3a-4fcc-9ecc-40fd6cc10fb6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f118969f-d730-4335-8618-5897da4e0a3c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1353bc944e06942d6f92ed95dc0e4a2aa4d95ff79e82e6bcf6c69da7b0ff93fb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4f52e5c9-4f28-4552-9cdb-9becebc0e331", "node_type": "1", "metadata": {}, "hash": "2585e3d072f75550210dd85dba3e086b3331ed428f25dc567392fffc97703a33", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5728, "end_char_idx": 8574, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
def assess_claims(claims):
    responses = []
    # Query the OpenAI API
    for claim in claims:
        response = openai.ChatCompletion.create(
            model='gpt-3.5-turbo',
            messages=build_prompt(claim),
            max_tokens=3,
        )
        # Strip any punctuation or whitespace from the response
        responses.append(response.choices[0].message.content.strip('., '))

    return responses
```

We sample 100 claims from the dataset


```python
# Let's take a look at 100 claims
samples = claim_df.sample(50)

claims = samples['claim'].tolist() 

```

We evaluate the ground-truth according to the dataset. From the dataset description, each claim is either supported or contradicted by the evidence, or else there isn't enough evidence either way. 


```python
def get_groundtruth(evidence):
    groundtruth = []
    for e in evidence:
        # Evidence is empty 
        if len(e) == 0:
            groundtruth.append('NEE')
        else:
            # In this dataset, all evidence for a given claim is consistent, either SUPPORT or CONTRADICT
            if list(e.values())[0][0]['label'] == 'SUPPORT':
                groundtruth.append('True')
            else:
                groundtruth.append('False')
    return groundtruth
```


```python
evidence = samples['evidence'].tolist()
groundtruth = get_groundtruth(evidence)
```

We also output the confusion matrix, comparing the model's assessments with the ground truth, in an easy to read table. 


```python
def confusion_matrix(inferred, groundtruth):
    assert len(inferred) == len(groundtruth)
    confusion = {
        'True': {'True': 0, 'False': 0, 'NEE': 0},
        'False': {'True': 0, 'False': 0, 'NEE': 0},
        'NEE': {'True': 0, 'False': 0, 'NEE': 0},
    }
    for i, g in zip(inferred, groundtruth):
        confusion[i][g] += 1

    # Pretty print the confusion matrix
    print('\tGroundtruth')
    print('\tTrue\tFalse\tNEE')
    for i in confusion:
        print(i, end='\t')
        for g in confusion[i]:
            print(confusion[i][g], end='\t')
        print()

    return confusion
```

We ask the model to directly assess the claims, without additional context. 


```python
gpt_inferred = assess_claims(claims)
confusion_matrix(gpt_inferred, groundtruth)
```

    	Groundtruth
    	True	False	NEE
    True	15	5	14	
    False	0	2	1	
    NEE	3	3	7	





    {'True': {'True': 15, 'False': 5, 'NEE': 14},
     'False': {'True': 0, 'False': 2, 'NEE': 1},
     'NEE': {'True': 3, 'False': 3, 'NEE': 7}}



## Results

From these results we see that the LLM is strongly biased to assess claims as true, even when they are false, and also tends to assess false claims as not having enough evidence. Note that 'not enough evidence' is with respect to the model's assessment of the claim in a vacuum, without additional context.


key: _node_content
string_value:
{"id_": "4f52e5c9-4f28-4552-9cdb-9becebc0e331", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a21fb14e-af3a-4fcc-9ecc-40fd6cc10fb6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "65855cb9c8543815167a0c2902bf9fc5a5f8d6bf925dfee8644a570afe3d979b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d8da2860-024b-46fd-a3ec-f02d9593ff75", "node_type": "1", "metadata": {}, "hash": "dbcd26f0518ab002681e4ec6edc0956a9caf6d5ea8419af30b57c73648576cf6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7919, "end_char_idx": 8951, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
gpt_inferred = assess_claims(claims)
confusion_matrix(gpt_inferred, groundtruth)
```

    	Groundtruth
    	True	False	NEE
    True	15	5	14	
    False	0	2	1	
    NEE	3	3	7	





    {'True': {'True': 15, 'False': 5, 'NEE': 14},
     'False': {'True': 0, 'False': 2, 'NEE': 1},
     'NEE': {'True': 3, 'False': 3, 'NEE': 7}}



## Results

From these results we see that the LLM is strongly biased to assess claims as true, even when they are false, and also tends to assess false claims as not having enough evidence. Note that 'not enough evidence' is with respect to the model's assessment of the claim in a vacuum, without additional context.


# Adding context 

We now add the additional context available from the corpus of paper titles and abstracts. This section shows how to load a text corpus into Chroma, using OpenAI text embeddings. 

First, we load the text corpus. 


```python
# Load the corpus into a dataframe
corpus_df = pd.read_json(f'{data_path}/scifact_corpus.jsonl', lines=True)
corpus_df.head()
```


key: _node_content
string_value:
{"id_": "d8da2860-024b-46fd-a3ec-f02d9593ff75", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4f52e5c9-4f28-4552-9cdb-9becebc0e331", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e2705f8b9280fe72f564b5247c116abc6f1866ebcd2e6ea120aec0d1c98e5e3a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ee3df3ca-4837-4112-8d96-59f341717590", "node_type": "1", "metadata": {}, "hash": "76197a26079d0781a448b68ca74f4b47cd67db1631ef0f0f6dc0a516ac4e0d45", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7919, "end_char_idx": 10477, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
gpt_inferred = assess_claims(claims)
confusion_matrix(gpt_inferred, groundtruth)
```

    	Groundtruth
    	True	False	NEE
    True	15	5	14	
    False	0	2	1	
    NEE	3	3	7	





    {'True': {'True': 15, 'False': 5, 'NEE': 14},
     'False': {'True': 0, 'False': 2, 'NEE': 1},
     'NEE': {'True': 3, 'False': 3, 'NEE': 7}}



## Results

From these results we see that the LLM is strongly biased to assess claims as true, even when they are false, and also tends to assess false claims as not having enough evidence. Note that 'not enough evidence' is with respect to the model's assessment of the claim in a vacuum, without additional context.


# Adding context 

We now add the additional context available from the corpus of paper titles and abstracts. This section shows how to load a text corpus into Chroma, using OpenAI text embeddings. 

First, we load the text corpus. 


```python
# Load the corpus into a dataframe
corpus_df = pd.read_json(f'{data_path}/scifact_corpus.jsonl', lines=True)
corpus_df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>doc_id</th>
      <th>title</th>
      <th>abstract</th>
      <th>structured</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>4983</td>
      <td>Microstructural development of human newborn c...</td>
      <td>[Alterations of the architecture of cerebral w...</td>
      <td>False</td>
    </tr>
    <tr>
      <th>1</th>
      <td>5836</td>
      <td>Induction of myelodysplasia by myeloid-derived...</td>
      <td>[Myelodysplastic syndromes (MDS) are age-depen...</td>
      <td>False</td>
    </tr>
    <tr>
      <th>2</th>
      <td>7912</td>
      <td>BC1 RNA, the transcript from a master gene for...</td>
      <td>[ID elements are short interspersed elements (...</td>
      <td>False</td>
    </tr>
    <tr>
      <th>3</th>
      <td>18670</td>
      <td>The DNA Methylome of Human Peripheral Blood Mo...</td>
      <td>[DNA methylation plays an important role in bi...</td>
      <td>False</td>
    </tr>
    <tr>
      <th>4</th>
      <td>19238</td>
      <td>The human myelin basic protein gene is include...</td>
      <td>[Two human Golli (for gene expressed in the ol...</td>
      <td>False</td>
    </tr>
  </tbody>
</table>
</div>


key: _node_content
string_value:
{"id_": "ee3df3ca-4837-4112-8d96-59f341717590", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d8da2860-024b-46fd-a3ec-f02d9593ff75", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "70b42dac93d846ba1ddf24ce895f1bb40d71e03fc8f46791f42dfca237e69716", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "84a65079-cd27-4ed9-88af-b687742dbfdd", "node_type": "1", "metadata": {}, "hash": "4141e1ba657640c16d1e19319fd2889088058dddfcd42bf528197d0f5098e009", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10481, "end_char_idx": 13640, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Loading the corpus into Chroma

The next step is to load the corpus into Chroma. Given an embedding function, Chroma will automatically handle embedding each document, and will store it alongside its text and metadata, making it simple to query.

We instantiate a (ephemeral) Chroma client, and create a collection for the SciFact title and abstract corpus. 
Chroma can also be instantiated in a persisted configuration; learn more at the [Chroma docs](https://docs.trychroma.com/usage-guide?lang=py). 


```python
import chromadb
from chromadb.utils.embedding_functions import OpenAIEmbeddingFunction

# We initialize an embedding function, and provide it to the collection.
embedding_function = OpenAIEmbeddingFunction(api_key=os.getenv("OPENAI_API_KEY"))

chroma_client = chromadb.Client() # Ephemeral by default
scifact_corpus_collection = chroma_client.create_collection(name='scifact_corpus', embedding_function=embedding_function)
```

    Running Chroma using direct local API.
    Using DuckDB in-memory for database. Data will be transient.


Next we load the corpus into Chroma. Because this data loading is memory intensive, we recommend using a batched loading scheme in batches of 50-1000. For this example it should take just over one minute for the entire corpus. It's being embedded in the background, automatically, using the `embedding_function` we specified earlier.


```python
batch_size = 100

for i in range(0, len(corpus_df), batch_size):
    batch_df = corpus_df[i:i+batch_size]
    scifact_corpus_collection.add(
        ids=batch_df['doc_id'].apply(lambda x: str(x)).tolist(), # Chroma takes string IDs.
        documents=(batch_df['title'] + '. ' + batch_df['abstract'].apply(lambda x: ' '.join(x))).to_list(), # We concatenate the title and abstract.
        metadatas=[{"structured": structured} for structured in batch_df['structured'].to_list()] # We also store the metadata, though we don't use it in this example.
    )
```

## Retrieving context

Next we retrieve documents from the corpus which may be relevant to each claim in our sample. We want to provide these as context to the LLM for evaluating the claims. We retrieve the 3 most relevant documents for each claim, according to the embedding distance. 


```python
claim_query_result = scifact_corpus_collection.query(query_texts=claims, include=['documents', 'distances'], n_results=3)
```

We create a new prompt, this time taking into account the additional context we retrieve from the corpus. 


```python
def build_prompt_with_context(claim, context):
    return [{'role': 'system', 'content': "I will ask you to assess whether a particular scientific claim, based on evidence provided. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence."}, 
            {'role': 'user', 'content': f""""
The evidence is the following:

{' '.join(context)}

Assess the following claim on the basis of the evidence. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence. Do not output any other text. 

Claim:
{claim}

Assessment:
"""}]


key: _node_content
string_value:
{"id_": "84a65079-cd27-4ed9-88af-b687742dbfdd", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ee3df3ca-4837-4112-8d96-59f341717590", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4c5a5cca7fac41d07e1d3db95a8568244b9a239bf3b430fd4ebb19a8dfa5a1cc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "96666921-c260-4430-99c3-9e0485855e52", "node_type": "1", "metadata": {}, "hash": "0628ac2ec49633ccfa9cdc83edaaf9ad092c43126c21199ecdd621dbcda51b1d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12733, "end_char_idx": 15739, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
claim_query_result = scifact_corpus_collection.query(query_texts=claims, include=['documents', 'distances'], n_results=3)
```

We create a new prompt, this time taking into account the additional context we retrieve from the corpus. 


```python
def build_prompt_with_context(claim, context):
    return [{'role': 'system', 'content': "I will ask you to assess whether a particular scientific claim, based on evidence provided. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence."}, 
            {'role': 'user', 'content': f""""
The evidence is the following:

{' '.join(context)}

Assess the following claim on the basis of the evidence. Output only the text 'True' if the claim is true, 'False' if the claim is false, or 'NEE' if there's not enough evidence. Do not output any other text. 

Claim:
{claim}

Assessment:
"""}]


def assess_claims_with_context(claims, contexts):
    responses = []
    # Query the OpenAI API
    for claim, context in zip(claims, contexts):
        # If no evidence is provided, return NEE
        if len(context) == 0:
            responses.append('NEE')
            continue
        response = openai.ChatCompletion.create(
            model='gpt-3.5-turbo',
            messages=build_prompt_with_context(claim=claim, context=context),
            max_tokens=3,
        )
        # Strip any punctuation or whitespace from the response
        responses.append(response.choices[0].message.content.strip('., '))

    return responses
```

Then ask the model to evaluate the claims with the retrieved context. 


```python
gpt_with_context_evaluation = assess_claims_with_context(claims, claim_query_result['documents'])
confusion_matrix(gpt_with_context_evaluation, groundtruth)
```

    	Groundtruth
    	True	False	NEE
    True	16	2	8	
    False	1	6	5	
    NEE	1	2	9	





    {'True': {'True': 16, 'False': 2, 'NEE': 8},
     'False': {'True': 1, 'False': 6, 'NEE': 5},
     'NEE': {'True': 1, 'False': 2, 'NEE': 9}}



## Results

We see that the model is a lot less likely to evaluate a False claim as true (2 instances VS 5 previously), but that claims without enough evidence are still often assessed as True or False.

Taking a look at the retrieved documents, we see that they are sometimes not relevant to the claim - this causes the model to be confused by the extra information, and it may decide that sufficient evidence is present, even when the information is irrelevant. This happens because we always ask for the 3 'most' relevant documents, but these might not be relevant at all beyond a certain point.  

## Filtering context on relevance

Along with the documents themselves, Chroma returns a distance score. We can try thresholding on distance, so that fewer irrelevant documents make it into the context we provide the model. 

If, after filtering on the threshold, no context documents remain, we bypass the model and simply return that there is not enough evidence.


key: _node_content
string_value:
{"id_": "96666921-c260-4430-99c3-9e0485855e52", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "84a65079-cd27-4ed9-88af-b687742dbfdd", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a2034b8a20f8446978940ff03e81095fa0b76e455ed1e0671ad3cbf96ccbc2f1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fa59e943-f7dc-4b8a-9271-1af2e38a2e4f", "node_type": "1", "metadata": {}, "hash": "0debb42270b184f1c2c79545a873540128782d62b2012be14f939979e8a2e7a5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14361, "end_char_idx": 16808, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
gpt_with_context_evaluation = assess_claims_with_context(claims, claim_query_result['documents'])
confusion_matrix(gpt_with_context_evaluation, groundtruth)
```

    	Groundtruth
    	True	False	NEE
    True	16	2	8	
    False	1	6	5	
    NEE	1	2	9	





    {'True': {'True': 16, 'False': 2, 'NEE': 8},
     'False': {'True': 1, 'False': 6, 'NEE': 5},
     'NEE': {'True': 1, 'False': 2, 'NEE': 9}}



## Results

We see that the model is a lot less likely to evaluate a False claim as true (2 instances VS 5 previously), but that claims without enough evidence are still often assessed as True or False.

Taking a look at the retrieved documents, we see that they are sometimes not relevant to the claim - this causes the model to be confused by the extra information, and it may decide that sufficient evidence is present, even when the information is irrelevant. This happens because we always ask for the 3 'most' relevant documents, but these might not be relevant at all beyond a certain point.  

## Filtering context on relevance

Along with the documents themselves, Chroma returns a distance score. We can try thresholding on distance, so that fewer irrelevant documents make it into the context we provide the model. 

If, after filtering on the threshold, no context documents remain, we bypass the model and simply return that there is not enough evidence. 


```python
def filter_query_result(query_result, distance_threshold=0.25):
# For each query result, retain only the documents whose distance is below the threshold
    for ids, docs, distances in zip(query_result['ids'], query_result['documents'], query_result['distances']):
        for i in range(len(ids)-1, -1, -1):
            if distances[i] > distance_threshold:
                ids.pop(i)
                docs.pop(i)
                distances.pop(i)
    return query_result

```


```python
filtered_claim_query_result = filter_query_result(claim_query_result)
```

Now we assess the claims using this cleaner context. 


```python
gpt_with_filtered_context_evaluation = assess_claims_with_context(claims, filtered_claim_query_result['documents'])
confusion_matrix(gpt_with_filtered_context_evaluation, groundtruth)
```

    	Groundtruth
    	True	False	NEE
    True	10	2	1	
    False	0	2	1	
    NEE	8	6	20	





    {'True': {'True': 10, 'False': 2, 'NEE': 1},
     'False': {'True': 0, 'False': 2, 'NEE': 1},
     'NEE': {'True': 8, 'False': 6, 'NEE': 20}}


key: _node_content
string_value:
{"id_": "fa59e943-f7dc-4b8a-9271-1af2e38a2e4f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "96666921-c260-4430-99c3-9e0485855e52", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4cf8d2f39e5c8c2728a6728861bd1206e9ac12b6b3652c3e533cad19a75c30c1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "be9edb9d-8e8a-4bed-85a3-3594eee5bdc6", "node_type": "1", "metadata": {}, "hash": "725e8f3d5074fb077f7affdec9d8e9d486e4728c381e176262d779f22aaa5afd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15743, "end_char_idx": 18984, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def filter_query_result(query_result, distance_threshold=0.25):
# For each query result, retain only the documents whose distance is below the threshold
    for ids, docs, distances in zip(query_result['ids'], query_result['documents'], query_result['distances']):
        for i in range(len(ids)-1, -1, -1):
            if distances[i] > distance_threshold:
                ids.pop(i)
                docs.pop(i)
                distances.pop(i)
    return query_result

```


```python
filtered_claim_query_result = filter_query_result(claim_query_result)
```

Now we assess the claims using this cleaner context. 


```python
gpt_with_filtered_context_evaluation = assess_claims_with_context(claims, filtered_claim_query_result['documents'])
confusion_matrix(gpt_with_filtered_context_evaluation, groundtruth)
```

    	Groundtruth
    	True	False	NEE
    True	10	2	1	
    False	0	2	1	
    NEE	8	6	20	





    {'True': {'True': 10, 'False': 2, 'NEE': 1},
     'False': {'True': 0, 'False': 2, 'NEE': 1},
     'NEE': {'True': 8, 'False': 6, 'NEE': 20}}



## Results

The model now assesses many fewer claims as True or False when there is not enough evidence present. However, it now biases away from certainty. Most claims are now assessed as having not enough evidence, because a large fraction of them are filtered out by the distance threshold. It's possible to tune the distance threshold to find the optimal operating point, but this can be difficult, and is dataset and embedding model dependent. 

# Hypothetical Document Embeddings: Using hallucinations productively

We want to be able to retrieve relevant documents, without retrieving less relevant ones which might confuse the model. One way to accomplish this is to improve the retrieval query. 

Until now, we have queried the dataset using _claims_ which are single sentence statements, while the corpus contains _abstracts_ describing a scientific paper. Intuitively, while these might be related, there are significant differences in their structure and meaning. These differences are encoded by the embedding model, and so influence the distances between the query and the most relevant results. 

We can overcome this by leveraging the power of LLMs to generate relevant text. While the facts might be hallucinated, the content and structure of the documents the models generate is more similar to the documents in our corpus, than the queries are. This could lead to better queries and hence better results. 

This approach is called [Hypothetical Document Embeddings (HyDE)](https://arxiv.org/abs/2212.10496), and has been shown to be quite good at the retrieval task. It should help us bring more relevant information into the context, without polluting it.  

TL;DR:
- you get much better matches when you embed whole abstracts rather than single sentences
- but claims are usually single sentences
- So HyDE shows that using GPT3 to expand claims into hallucinated abstracts and then searching based on those abstracts works (claims -> abstracts -> results) better than searching directly (claims -> results)

First, we use in-context examples to prompt the model to generate documents similar to what's in the corpus, for each claim we want to assess.


key: _node_content
string_value:
{"id_": "be9edb9d-8e8a-4bed-85a3-3594eee5bdc6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fa59e943-f7dc-4b8a-9271-1af2e38a2e4f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "16d3c426bc313b207cfe8f5e8df5dc47b41e085a89aa8ae9c8e5e4e3edc801e9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dd6cb8da-0cfb-4656-9008-ab517c7c8712", "node_type": "1", "metadata": {}, "hash": "3ce52a35ffd3ed9ac882558ad576d3512cb63f01ed926b3e9164054a92403d5f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18988, "end_char_idx": 22163, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def build_hallucination_prompt(claim):
    return [{'role': 'system', 'content': """I will ask you to write an abstract for a scientific paper which supports or refutes a given claim. It should be written in scientific language, include a title. Output only one abstract, then stop.
    
    An Example:

    Claim:
    A high microerythrocyte count raises vulnerability to severe anemia in homozygous alpha (+)- thalassemia trait subjects.

    Abstract:
    BACKGROUND The heritable haemoglobinopathy alpha(+)-thalassaemia is caused by the reduced synthesis of alpha-globin chains that form part of normal adult haemoglobin (Hb). Individuals homozygous for alpha(+)-thalassaemia have microcytosis and an increased erythrocyte count. Alpha(+)-thalassaemia homozygosity confers considerable protection against severe malaria, including severe malarial anaemia (SMA) (Hb concentration < 50 g/l), but does not influence parasite count. We tested the hypothesis that the erythrocyte indices associated with alpha(+)-thalassaemia homozygosity provide a haematological benefit during acute malaria.   
    METHODS AND FINDINGS Data from children living on the north coast of Papua New Guinea who had participated in a case-control study of the protection afforded by alpha(+)-thalassaemia against severe malaria were reanalysed to assess the genotype-specific reduction in erythrocyte count and Hb levels associated with acute malarial disease. We observed a reduction in median erythrocyte count of approximately 1.5 x 10(12)/l in all children with acute falciparum malaria relative to values in community children (p < 0.001). We developed a simple mathematical model of the linear relationship between Hb concentration and erythrocyte count. This model predicted that children homozygous for alpha(+)-thalassaemia lose less Hb than children of normal genotype for a reduction in erythrocyte count of >1.1 x 10(12)/l as a result of the reduced mean cell Hb in homozygous alpha(+)-thalassaemia. In addition, children homozygous for alpha(+)-thalassaemia require a 10% greater reduction in erythrocyte count than children of normal genotype (p = 0.02) for Hb concentration to fall to 50 g/l, the cutoff for SMA. We estimated that the haematological profile in children homozygous for alpha(+)-thalassaemia reduces the risk of SMA during acute malaria compared to children of normal genotype (relative risk 0.52; 95% confidence interval [CI] 0.24-1.12, p = 0.09).   
    CONCLUSIONS The increased erythrocyte count and microcytosis in children homozygous for alpha(+)-thalassaemia may contribute substantially to their protection against SMA. A lower concentration of Hb per erythrocyte and a larger population of erythrocytes may be a biologically advantageous strategy against the significant reduction in erythrocyte count that occurs during acute infection with the malaria parasite Plasmodium falciparum. This haematological profile may reduce the risk of anaemia by other Plasmodium species, as well as other causes of anaemia. Other host polymorphisms that induce an increased erythrocyte count and microcytosis may confer a similar advantage.

    End of example.


key: _node_content
string_value:
{"id_": "dd6cb8da-0cfb-4656-9008-ab517c7c8712", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "be9edb9d-8e8a-4bed-85a3-3594eee5bdc6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c725155626775468ccff264bfb16d4b7caaee7ebebe6cc2f91f148bf87509b1d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "16beaa96-bd45-4f5c-93a9-471daf9703b9", "node_type": "1", "metadata": {}, "hash": "89ee72a4c1a935544a9770be131bcf7707e09d9833f115e340c7d1adc505a3f5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20622, "end_char_idx": 23668, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We developed a simple mathematical model of the linear relationship between Hb concentration and erythrocyte count. This model predicted that children homozygous for alpha(+)-thalassaemia lose less Hb than children of normal genotype for a reduction in erythrocyte count of >1.1 x 10(12)/l as a result of the reduced mean cell Hb in homozygous alpha(+)-thalassaemia. In addition, children homozygous for alpha(+)-thalassaemia require a 10% greater reduction in erythrocyte count than children of normal genotype (p = 0.02) for Hb concentration to fall to 50 g/l, the cutoff for SMA. We estimated that the haematological profile in children homozygous for alpha(+)-thalassaemia reduces the risk of SMA during acute malaria compared to children of normal genotype (relative risk 0.52; 95% confidence interval [CI] 0.24-1.12, p = 0.09).   
    CONCLUSIONS The increased erythrocyte count and microcytosis in children homozygous for alpha(+)-thalassaemia may contribute substantially to their protection against SMA. A lower concentration of Hb per erythrocyte and a larger population of erythrocytes may be a biologically advantageous strategy against the significant reduction in erythrocyte count that occurs during acute infection with the malaria parasite Plasmodium falciparum. This haematological profile may reduce the risk of anaemia by other Plasmodium species, as well as other causes of anaemia. Other host polymorphisms that induce an increased erythrocyte count and microcytosis may confer a similar advantage.

    End of example. 
    
    """}, {'role': 'user', 'content': f""""
    Perform the task for the following claim.

    Claim:
    {claim}

    Abstract:
    """}]


def hallucinate_evidence(claims):
    # Query the OpenAI API
    responses = []
    # Query the OpenAI API
    for claim in claims:
        response = openai.ChatCompletion.create(
            model='gpt-3.5-turbo',
            messages=build_hallucination_prompt(claim),
        )
        responses.append(response.choices[0].message.content)
    return responses
```

We hallucinate a document for each claim.

*NB: This can take a while, about 30m for 100 claims*. You can reduce the number of claims we want to assess to get results more quickly. 


```python
hallucinated_evidence = hallucinate_evidence(claims)
```

We use the hallucinated documents as queries into the corpus, and filter the results using the same distance threshold. 


```python
hallucinated_query_result = scifact_corpus_collection.query(query_texts=hallucinated_evidence, include=['documents', 'distances'], n_results=3)
filtered_hallucinated_query_result = filter_query_result(hallucinated_query_result)
```

We then ask the model to assess the claims, using the new context. 


```python
gpt_with_hallucinated_context_evaluation = assess_claims_with_context(claims, filtered_hallucinated_query_result['documents'])
confusion_matrix(gpt_with_hallucinated_context_evaluation, groundtruth)
```

    	Groundtruth
    	True	False	NEE
    True	15	2	5	
    False	1	5	4	
    NEE	2	3	13


key: _node_content
string_value:
{"id_": "16beaa96-bd45-4f5c-93a9-471daf9703b9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fbfe1fcf-38c8-4c66-af50-49fe591c21b7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f091756cf8191f24a52abb31a64c07351c6f762ddd837e559317f8659282bec", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dd6cb8da-0cfb-4656-9008-ab517c7c8712", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/chroma/hyde-with-chroma-and-openai.ipynb", "title": "Robust Question Answering with Chroma and OpenAI", "tokens": 6044, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b2a33432a960093a8a6bfd74a74c1ea6ffb5ad60bb14553270d5bc28d560f2ea", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 22148, "end_char_idx": 24502, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
End of example. 
    
    """}, {'role': 'user', 'content': f""""
    Perform the task for the following claim.

    Claim:
    {claim}

    Abstract:
    """}]


def hallucinate_evidence(claims):
    # Query the OpenAI API
    responses = []
    # Query the OpenAI API
    for claim in claims:
        response = openai.ChatCompletion.create(
            model='gpt-3.5-turbo',
            messages=build_hallucination_prompt(claim),
        )
        responses.append(response.choices[0].message.content)
    return responses
```

We hallucinate a document for each claim.

*NB: This can take a while, about 30m for 100 claims*. You can reduce the number of claims we want to assess to get results more quickly. 


```python
hallucinated_evidence = hallucinate_evidence(claims)
```

We use the hallucinated documents as queries into the corpus, and filter the results using the same distance threshold. 


```python
hallucinated_query_result = scifact_corpus_collection.query(query_texts=hallucinated_evidence, include=['documents', 'distances'], n_results=3)
filtered_hallucinated_query_result = filter_query_result(hallucinated_query_result)
```

We then ask the model to assess the claims, using the new context. 


```python
gpt_with_hallucinated_context_evaluation = assess_claims_with_context(claims, filtered_hallucinated_query_result['documents'])
confusion_matrix(gpt_with_hallucinated_context_evaluation, groundtruth)
```

    	Groundtruth
    	True	False	NEE
    True	15	2	5	
    False	1	5	4	
    NEE	2	3	13	





    {'True': {'True': 15, 'False': 2, 'NEE': 5},
     'False': {'True': 1, 'False': 5, 'NEE': 4},
     'NEE': {'True': 2, 'False': 3, 'NEE': 13}}



## Results

Combining HyDE with a simple distance threshold leads to a significant improvement. The model no longer biases assessing claims as True, nor toward their not being enough evidence. It also correctly assesses when there isn't enough evidence more often.

# Conclusion

Equipping LLMs with a context based on a corpus of documents is a powerful technique for bringing the general reasoning and natural language interactions of LLMs to your own data. However, it's important to know that naive query and retrieval may not produce the best possible results! Ultimately understanding the data will help get the most out of the retrieval based question-answering approach.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb


key: _node_content
string_value:
{"id_": "5da032a0-baaa-4a90-ae69-00f333102cda", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f95f57ee-b1d7-4445-800f-5521d01819f0", "node_type": "1", "metadata": {}, "hash": "088acc09ac6d00e2f69d60f7eaec540a731a22f958b8c7d25f609591117eb096", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3626, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Using Redis for Embeddings Search

This notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.

### What is a Vector Database

A vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.

### Why use a Vector Database

Vector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.


### Demo Flow
The demo flow is:
- **Setup**: Import packages and set any required variables
- **Load data**: Load a dataset and embed it using OpenAI embeddings
- **Redis**
    - *Setup*: Set up the Redis-Py client. For more details go [here](https://github.com/redis/redis-py)
    - *Index Data*: Create the search index for vector search and hybrid search (vector + full-text search) on all available fields.
    - *Search Data*: Run a few example queries with various goals in mind.

Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.

## Setup

Import the required libraries and set the embedding model that we'd like to use.


```python
# We'll need to install the Redis client
!pip install redis

#Install wget to pull zip file
!pip install wget
```


```python
import openai

from typing import List, Iterator
import pandas as pd
import numpy as np
import os
import wget
from ast import literal_eval

# Redis client library for Python
import redis

# I've set this to our new embeddings model, this can be changed to the embedding model of your choice
EMBEDDING_MODEL = "text-embedding-3-small"

# Ignore unclosed SSL socket warnings - optional in case you get these errors
import warnings

warnings.filterwarnings(action="ignore", message="unclosed", category=ResourceWarning)
warnings.filterwarnings("ignore", category=DeprecationWarning) 
```

## Load data

In this section we'll load embedded data that we've prepared previous to this session.


```python
embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```


```python
import zipfile
with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip","r") as zip_ref:
    zip_ref.extractall("../data")
```


```python
article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')
```


```python
article_df.head()
```


key: _node_content
string_value:
{"id_": "f95f57ee-b1d7-4445-800f-5521d01819f0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5da032a0-baaa-4a90-ae69-00f333102cda", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "6cb0e5791b2675568000a729aced58af9bb81fb4436908ee7d55e1b5282e3bf8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "95ca9e9e-50be-410b-94c2-1a3be3e9b92b", "node_type": "1", "metadata": {}, "hash": "55d75d734c6c465942a4bbe99c9e2550ca1dde90d5af183d906f3fc23ae149c3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2312, "end_char_idx": 4642, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# We'll need to install the Redis client
!pip install redis

#Install wget to pull zip file
!pip install wget
```


```python
import openai

from typing import List, Iterator
import pandas as pd
import numpy as np
import os
import wget
from ast import literal_eval

# Redis client library for Python
import redis

# I've set this to our new embeddings model, this can be changed to the embedding model of your choice
EMBEDDING_MODEL = "text-embedding-3-small"

# Ignore unclosed SSL socket warnings - optional in case you get these errors
import warnings

warnings.filterwarnings(action="ignore", message="unclosed", category=ResourceWarning)
warnings.filterwarnings("ignore", category=DeprecationWarning) 
```

## Load data

In this section we'll load embedded data that we've prepared previous to this session.


```python
embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```


```python
import zipfile
with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip","r") as zip_ref:
    zip_ref.extractall("../data")
```


```python
article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')
```


```python
article_df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>url</th>
      <th>title</th>
      <th>text</th>
      <th>title_vector</th>
      <th>content_vector</th>
      <th>vector_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>https://simple.wikipedia.org/wiki/April</td>
      <td>April</td>
      <td>April is the fourth month of the year in the J...</td>
      <td>[0.001009464613161981, -0.020700545981526375, ...</td>
      <td>[-0.011253940872848034, -0.013491976074874401,...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>https://simple.wikipedia.org/wiki/August</td>
      <td>August</td>
      <td>August (Aug.)


key: _node_content
string_value:
{"id_": "95ca9e9e-50be-410b-94c2-1a3be3e9b92b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f95f57ee-b1d7-4445-800f-5521d01819f0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a689ea010c1bc5251888130f4573694442c5ee0d666e0b4fb260dcaf8e91fe5e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8a68892d-dd5c-48aa-a03d-041dcbb591e7", "node_type": "1", "metadata": {}, "hash": "18e7b60db5538910c0b015b09df347c6a085ba812f72bb29fa3eeccb225ebb90", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3495, "end_char_idx": 5673, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: _node_content
string_value:
{"id_": "8a68892d-dd5c-48aa-a03d-041dcbb591e7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "95ca9e9e-50be-410b-94c2-1a3be3e9b92b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ba98c8b990e273960d99a1a8b9c6fac5aa624908bdcb9d71eb7037bfc7172e54", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c15a8f4c-6528-4401-9590-8362e85e5977", "node_type": "1", "metadata": {}, "hash": "647a53a8136e9b26b4b4b4cefb65311c24b5a2c31beb16c74dc5582c5bf676be", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4643, "end_char_idx": 6972, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
is the eighth month of the year ...</td>
      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>
      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>6</td>
      <td>https://simple.wikipedia.org/wiki/Art</td>
      <td>Art</td>
      <td>Art is a creative activity that expresses imag...</td>
      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>
      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>
      <td>2</td>
    </tr>
    <tr>
      <th>3</th>
      <td>8</td>
      <td>https://simple.wikipedia.org/wiki/A</td>
      <td>A</td>
      <td>A or a is the first letter of the English alph...</td>
      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>
      <td>[0.024894846603274345, -0.022186409682035446, ...</td>
      <td>3</td>
    </tr>
    <tr>
      <th>4</th>
      <td>9</td>
      <td>https://simple.wikipedia.org/wiki/Air</td>
      <td>Air</td>
      <td>Air refers to the Earth's atmosphere. Air is a...</td>
      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>
      <td>[0.021524671465158463, 0.018522677943110466, -...</td>
      <td>4</td>
    </tr>
  </tbody>
</table>
</div>




```python
# Read vectors from strings back into a list
article_df['title_vector'] = article_df.title_vector.apply(literal_eval)
article_df['content_vector'] = article_df.content_vector.apply(literal_eval)

# Set vector_id to be a string
article_df['vector_id'] = article_df['vector_id'].apply(str)
```


```python
article_df.info(show_counts=True)
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 25000 entries, 0 to 24999
    Data columns (total 7 columns):
     #   Column          Non-Null Count  Dtype 
    ---  ------          --------------  ----- 
     0   id              25000 non-null  int64 
     1   url             25000 non-null  object
     2   title           25000 non-null  object
     3   text            25000 non-null  object
     4   title_vector    25000 non-null  object
     5   content_vector  25000 non-null  object
     6   vector_id       25000 non-null  object
    dtypes: int64(1), object(6)
    memory usage: 1.3+ MB


# Redis

The next vector database covered in this tutorial is **[Redis](https://redis.io)**. You most likely already know Redis.


key: _node_content
string_value:
{"id_": "c15a8f4c-6528-4401-9590-8362e85e5977", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8a68892d-dd5c-48aa-a03d-041dcbb591e7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c28052e04f52063da569dfeb941e12b763aed2cd054bf995ed1bacca8fbfbb15", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0730bc98-a0b2-480e-a975-95037cfe6fec", "node_type": "1", "metadata": {}, "hash": "06777b422b8a962b9e3d3037f35aa5facb415c0cb6155bf970990a0f6ca82fbf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5674, "end_char_idx": 8400, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Air is a...</td>
      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>
      <td>[0.021524671465158463, 0.018522677943110466, -...</td>
      <td>4</td>
    </tr>
  </tbody>
</table>
</div>




```python
# Read vectors from strings back into a list
article_df['title_vector'] = article_df.title_vector.apply(literal_eval)
article_df['content_vector'] = article_df.content_vector.apply(literal_eval)

# Set vector_id to be a string
article_df['vector_id'] = article_df['vector_id'].apply(str)
```


```python
article_df.info(show_counts=True)
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 25000 entries, 0 to 24999
    Data columns (total 7 columns):
     #   Column          Non-Null Count  Dtype 
    ---  ------          --------------  ----- 
     0   id              25000 non-null  int64 
     1   url             25000 non-null  object
     2   title           25000 non-null  object
     3   text            25000 non-null  object
     4   title_vector    25000 non-null  object
     5   content_vector  25000 non-null  object
     6   vector_id       25000 non-null  object
    dtypes: int64(1), object(6)
    memory usage: 1.3+ MB


# Redis

The next vector database covered in this tutorial is **[Redis](https://redis.io)**. You most likely already know Redis. What you might not be aware of is the [RediSearch module](https://github.com/RediSearch/RediSearch). Enterprises have been using Redis with the RediSearch module for years now across all major cloud providers, Redis Cloud, and on premise. Recently, the Redis team added vector storage and search capability to this module in addition to the features RediSearch already had.

Given the large ecosystem around Redis, there are most likely client libraries in the language you need. You can use any standard Redis client library to run RediSearch commands, but it's easiest to use a library that wraps the RediSearch API. Below are a few examples, but you can find more client libraries [here](https://redis.io/resources/clients/).

| Project | Language | License | Author | Stars |
|----------|---------|--------|---------|-------|
| [jedis][jedis-url] | Java | MIT | [Redis][redis-url] | ![Stars][jedis-stars] |
| [redis-py][redis-py-url] | Python | MIT | [Redis][redis-url] | ![Stars][redis-py-stars] |
| [node-redis][node-redis-url] | Node.js | MIT | [Redis][redis-url] | ![Stars][node-redis-stars] |
| [nredisstack][nredisstack-url] | .NET | MIT | [Redis][redis-url] | ![Stars][nredisstack-stars] |
| [redisearch-go][redisearch-go-url] | Go | BSD | [Redis][redisearch-go-author] | [![redisearch-go-stars]][redisearch-go-url] |
| [redisearch-api-rs][redisearch-api-rs-url] | Rust | BSD | [Redis][redisearch-api-rs-author] | [!


key: _node_content
string_value:
{"id_": "0730bc98-a0b2-480e-a975-95037cfe6fec", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c15a8f4c-6528-4401-9590-8362e85e5977", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1ca1cfe187425171a779b3643ef7c9357ba9a22e6d91c49e34c0351af54af32d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "16df6f21-0d61-45d9-8ddb-3d6b3418eb1f", "node_type": "1", "metadata": {}, "hash": "845a012f52e37a98b828c0f17f31fdfd1a68f9a6186ab60e3678e5e6f272f198", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6937, "end_char_idx": 10175, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You most likely already know Redis. What you might not be aware of is the [RediSearch module](https://github.com/RediSearch/RediSearch). Enterprises have been using Redis with the RediSearch module for years now across all major cloud providers, Redis Cloud, and on premise. Recently, the Redis team added vector storage and search capability to this module in addition to the features RediSearch already had.

Given the large ecosystem around Redis, there are most likely client libraries in the language you need. You can use any standard Redis client library to run RediSearch commands, but it's easiest to use a library that wraps the RediSearch API. Below are a few examples, but you can find more client libraries [here](https://redis.io/resources/clients/).

| Project | Language | License | Author | Stars |
|----------|---------|--------|---------|-------|
| [jedis][jedis-url] | Java | MIT | [Redis][redis-url] | ![Stars][jedis-stars] |
| [redis-py][redis-py-url] | Python | MIT | [Redis][redis-url] | ![Stars][redis-py-stars] |
| [node-redis][node-redis-url] | Node.js | MIT | [Redis][redis-url] | ![Stars][node-redis-stars] |
| [nredisstack][nredisstack-url] | .NET | MIT | [Redis][redis-url] | ![Stars][nredisstack-stars] |
| [redisearch-go][redisearch-go-url] | Go | BSD | [Redis][redisearch-go-author] | [![redisearch-go-stars]][redisearch-go-url] |
| [redisearch-api-rs][redisearch-api-rs-url] | Rust | BSD | [Redis][redisearch-api-rs-author] | [![redisearch-api-rs-stars]][redisearch-api-rs-url] |

[redis-url]: https://redis.com

[redis-py-url]: https://github.com/redis/redis-py
[redis-py-stars]: https://img.shields.io/github/stars/redis/redis-py.svg?style=social&amp;label=Star&amp;maxAge=2592000
[redis-py-package]: https://pypi.python.org/pypi/redis

[jedis-url]: https://github.com/redis/jedis
[jedis-stars]: https://img.shields.io/github/stars/redis/jedis.svg?style=social&amp;label=Star&amp;maxAge=2592000
[Jedis-package]: https://search.maven.org/artifact/redis.clients/jedis

[nredisstack-url]: https://github.com/redis/nredisstack
[nredisstack-stars]: https://img.shields.io/github/stars/redis/nredisstack.svg?style=social&amp;label=Star&amp;maxAge=2592000
[nredisstack-package]: https://www.nuget.org/packages/nredisstack/

[node-redis-url]: https://github.com/redis/node-redis
[node-redis-stars]: https://img.shields.io/github/stars/redis/node-redis.svg?style=social&amp;label=Star&amp;maxAge=2592000
[node-redis-package]: https://www.npmjs.com/package/redis

[redis-om-python-url]: https://github.com/redis/redis-om-python
[redis-om-python-author]: https://redis.com
[redis-om-python-stars]: https://img.shields.io/github/stars/redis/redis-om-python.svg?style=social&amp;label=Star&amp;maxAge=2592000

[redisearch-go-url]: https://github.com/RediSearch/redisearch-go
[redisearch-go-author]: https://redis.com
[redisearch-go-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-go.svg?style=social&amp;label=Star&amp;maxAge=2592000

[redisearch-api-rs-url]: https://github.com/RediSearch/redisearch-api-rs
[redisearch-api-rs-author]: https://redis.com
[redisearch-api-rs-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-api-rs.svg?style=social&amp;label=Star&amp;maxAge=2592000


key: _node_content
string_value:
{"id_": "16df6f21-0d61-45d9-8ddb-3d6b3418eb1f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0730bc98-a0b2-480e-a975-95037cfe6fec", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b39f91b81ebb87ad5a5b4803bbe38240f481907ca6a60356b248d670b3652a4a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "26673c34-bef4-488c-94a2-b82669a7107c", "node_type": "1", "metadata": {}, "hash": "4b9ab54977f54ba8b00795c25cff3079b45023950e8f5130d45a560a1d613543", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10178, "end_char_idx": 13003, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In the below cells, we will walk you through using Redis as a vector database. Since many of you are likely already used to the Redis API, this should be familiar to most.

## Setup

There are many ways to deploy Redis with RediSearch. The easiest way to get started is to use Docker, but there are are many potential options for deployment. For other deployment options, see the [redis directory](./redis) in this repo.

For this tutorial, we will use Redis Stack on Docker.

Start a version of Redis with RediSearch (Redis Stack) by running the following docker command

```bash
$ cd redis
$ docker compose up -d
```
This also includes the [RedisInsight](https://redis.com/redis-enterprise/redis-insight/) GUI for managing your Redis database which you can view at [http://localhost:8001](http://localhost:8001) once you start the docker container.

You're all set up and ready to go! Next, we import and create our client for communicating with the Redis database we just created.


```python
import redis
from redis.commands.search.indexDefinition import (
    IndexDefinition,
    IndexType
)
from redis.commands.search.query import Query
from redis.commands.search.field import (
    TextField,
    VectorField
)

REDIS_HOST =  "localhost"
REDIS_PORT = 6379
REDIS_PASSWORD = "" # default for passwordless Redis

# Connect to Redis
redis_client = redis.Redis(
    host=REDIS_HOST,
    port=REDIS_PORT,
    password=REDIS_PASSWORD
)
redis_client.ping()
```




    True



## Creating a Search Index

The below cells will show how to specify and create a search index in Redis. We will

1. Set some constants for defining our index like the distance metric and the index name
2. Define the index schema with RediSearch fields
3. Create the index



```python
# Constants
VECTOR_DIM = len(article_df['title_vector'][0]) # length of the vectors
VECTOR_NUMBER = len(article_df)                 # initial number of vectors
INDEX_NAME = "embeddings-index"                 # name of the search index
PREFIX = "doc"                                  # prefix for the document keys
DISTANCE_METRIC = "COSINE"                      # distance metric for the vectors (ex. COSINE, IP, L2)
```


```python
# Define RediSearch fields for each of the columns in the dataset
title = TextField(name="title")
url = TextField(name="url")
text = TextField(name="text")
title_embedding = VectorField("title_vector",
    "FLAT", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER,
    }
)
text_embedding = VectorField("content_vector",
    "FLAT", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER,
    }
)
fields = [title, url, text, title_embedding, text_embedding]
```


key: _node_content
string_value:
{"id_": "26673c34-bef4-488c-94a2-b82669a7107c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "16df6f21-0d61-45d9-8ddb-3d6b3418eb1f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d43be6244787ff80f97e206d0363aebc1629a61e741af83d693c19e759c9d6ac", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a2813576-adcd-4560-9642-1bd0cb006199", "node_type": "1", "metadata": {}, "hash": "39ab50975f0a16a65a7b2763f6d87b336d84a7e1a0abc07e558b57eddd8b6425", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11647, "end_char_idx": 14591, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
True



## Creating a Search Index

The below cells will show how to specify and create a search index in Redis. We will

1. Set some constants for defining our index like the distance metric and the index name
2. Define the index schema with RediSearch fields
3. Create the index



```python
# Constants
VECTOR_DIM = len(article_df['title_vector'][0]) # length of the vectors
VECTOR_NUMBER = len(article_df)                 # initial number of vectors
INDEX_NAME = "embeddings-index"                 # name of the search index
PREFIX = "doc"                                  # prefix for the document keys
DISTANCE_METRIC = "COSINE"                      # distance metric for the vectors (ex. COSINE, IP, L2)
```


```python
# Define RediSearch fields for each of the columns in the dataset
title = TextField(name="title")
url = TextField(name="url")
text = TextField(name="text")
title_embedding = VectorField("title_vector",
    "FLAT", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER,
    }
)
text_embedding = VectorField("content_vector",
    "FLAT", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER,
    }
)
fields = [title, url, text, title_embedding, text_embedding]
```


```python
# Check if index exists
try:
    redis_client.ft(INDEX_NAME).info()
    print("Index already exists")
except:
    # Create RediSearch Index
    redis_client.ft(INDEX_NAME).create_index(
        fields = fields,
        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)
    )
```

## Load Documents into the Index

Now that we have a search index, we can load documents into it. We will use the same documents we used in the previous examples. In Redis, either the Hash or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The below cells will show how to load documents into the index.


```python
def index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):
    records = documents.to_dict("records")
    for doc in records:
        key = f"{prefix}:{str(doc['id'])}"

        # create byte vectors for title and content
        title_embedding = np.array(doc["title_vector"], dtype=np.float32).tobytes()
        content_embedding = np.array(doc["content_vector"], dtype=np.float32).tobytes()

        # replace list of floats with byte vectors
        doc["title_vector"] = title_embedding
        doc["content_vector"] = content_embedding

        client.hset(key, mapping = doc)
```


```python
index_documents(redis_client, PREFIX, article_df)
print(f"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}")
```

    Loaded 25000 documents in Redis search index with name: embeddings-index


key: _node_content
string_value:
{"id_": "a2813576-adcd-4560-9642-1bd0cb006199", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "26673c34-bef4-488c-94a2-b82669a7107c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "dc23a3b876b5ee4ec3874bfa359efa6e05ce21be0c374f5d89fde4da54009918", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d3f14ea9-3f39-4556-8d9e-b710e32649c3", "node_type": "1", "metadata": {}, "hash": "14c41fb6388a9b347e8d726c9e619ceb3216ba48c7a37f4be7844bc2417bedb4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13006, "end_char_idx": 15568, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Check if index exists
try:
    redis_client.ft(INDEX_NAME).info()
    print("Index already exists")
except:
    # Create RediSearch Index
    redis_client.ft(INDEX_NAME).create_index(
        fields = fields,
        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)
    )
```

## Load Documents into the Index

Now that we have a search index, we can load documents into it. We will use the same documents we used in the previous examples. In Redis, either the Hash or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The below cells will show how to load documents into the index.


```python
def index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):
    records = documents.to_dict("records")
    for doc in records:
        key = f"{prefix}:{str(doc['id'])}"

        # create byte vectors for title and content
        title_embedding = np.array(doc["title_vector"], dtype=np.float32).tobytes()
        content_embedding = np.array(doc["content_vector"], dtype=np.float32).tobytes()

        # replace list of floats with byte vectors
        doc["title_vector"] = title_embedding
        doc["content_vector"] = content_embedding

        client.hset(key, mapping = doc)
```


```python
index_documents(redis_client, PREFIX, article_df)
print(f"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}")
```

    Loaded 25000 documents in Redis search index with name: embeddings-index


## Running Search Queries

Now that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database. Each example will demonstrate specific features to keep in mind when developing your search application with Redis.

1. **Return Fields**: You can specify which fields you want to return in the search results. This is useful if you only want to return a subset of the fields in your documents and doesn't require a separate call to retrieve documents. In the below example, we will only return the `title` field in the search results.
2. **Hybrid Search**: You can combine vector search with any of the other RediSearch fields for hybrid search such as full text search, tag, geo, and numeric. In the below example, we will combine vector search with full text search.


key: _node_content
string_value:
{"id_": "d3f14ea9-3f39-4556-8d9e-b710e32649c3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a2813576-adcd-4560-9642-1bd0cb006199", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7ed0b7e400e1aecc8f3b033407fd9a2ae028bb8bab93f18befcf9df90ba71122", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f5a70c4d-ebc4-48e0-97c8-aae3dd590af8", "node_type": "1", "metadata": {}, "hash": "5765ce945710b5e3b8c7b151bd19436eaaff407dedd1df6a8bf9fdfd431e2c4a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13720, "end_char_idx": 16762, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):
    records = documents.to_dict("records")
    for doc in records:
        key = f"{prefix}:{str(doc['id'])}"

        # create byte vectors for title and content
        title_embedding = np.array(doc["title_vector"], dtype=np.float32).tobytes()
        content_embedding = np.array(doc["content_vector"], dtype=np.float32).tobytes()

        # replace list of floats with byte vectors
        doc["title_vector"] = title_embedding
        doc["content_vector"] = content_embedding

        client.hset(key, mapping = doc)
```


```python
index_documents(redis_client, PREFIX, article_df)
print(f"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}")
```

    Loaded 25000 documents in Redis search index with name: embeddings-index


## Running Search Queries

Now that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database. Each example will demonstrate specific features to keep in mind when developing your search application with Redis.

1. **Return Fields**: You can specify which fields you want to return in the search results. This is useful if you only want to return a subset of the fields in your documents and doesn't require a separate call to retrieve documents. In the below example, we will only return the `title` field in the search results.
2. **Hybrid Search**: You can combine vector search with any of the other RediSearch fields for hybrid search such as full text search, tag, geo, and numeric. In the below example, we will combine vector search with full text search.



```python
def search_redis(
    redis_client: redis.Redis,
    user_query: str,
    index_name: str = "embeddings-index",
    vector_field: str = "title_vector",
    return_fields: list = ["title", "url", "text", "vector_score"],
    hybrid_fields = "*",
    k: int = 20,
) -> List[dict]:

    # Creates embedding vector from user query
    embedded_query = openai.Embedding.create(input=user_query,
                                            model=EMBEDDING_MODEL,
                                            )["data"][0]['embedding']

    # Prepare the Query
    base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'
    query = (
        Query(base_query)
         .return_fields(*return_fields)
         .sort_by("vector_score")
         .paging(0, k)
         .dialect(2)
    )
    params_dict = {"vector": np.array(embedded_query).astype(dtype=np.float32).tobytes()}

    # perform vector search
    results = redis_client.ft(index_name).search(query, params_dict)
    for i, article in enumerate(results.docs):
        score = 1 - float(article.vector_score)
        print(f"{i}. {article.title} (Score: {round(score ,3) })")
    return results.docs
```


key: _node_content
string_value:
{"id_": "f5a70c4d-ebc4-48e0-97c8-aae3dd590af8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d3f14ea9-3f39-4556-8d9e-b710e32649c3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e09f9719e2068d70221001567bc4fa69fb93f8b47e0b190d1879427f99fdb0e6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b5d89480-daae-4ed4-b9df-d701333f0aa6", "node_type": "1", "metadata": {}, "hash": "b87ee3e2b78c1e1e0d083d238aa850d7def6c3e19e6b872ed42bcf58c8d1f8d0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15572, "end_char_idx": 18252, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def search_redis(
    redis_client: redis.Redis,
    user_query: str,
    index_name: str = "embeddings-index",
    vector_field: str = "title_vector",
    return_fields: list = ["title", "url", "text", "vector_score"],
    hybrid_fields = "*",
    k: int = 20,
) -> List[dict]:

    # Creates embedding vector from user query
    embedded_query = openai.Embedding.create(input=user_query,
                                            model=EMBEDDING_MODEL,
                                            )["data"][0]['embedding']

    # Prepare the Query
    base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'
    query = (
        Query(base_query)
         .return_fields(*return_fields)
         .sort_by("vector_score")
         .paging(0, k)
         .dialect(2)
    )
    params_dict = {"vector": np.array(embedded_query).astype(dtype=np.float32).tobytes()}

    # perform vector search
    results = redis_client.ft(index_name).search(query, params_dict)
    for i, article in enumerate(results.docs):
        score = 1 - float(article.vector_score)
        print(f"{i}. {article.title} (Score: {round(score ,3) })")
    return results.docs
```


```python
# For using OpenAI to generate query embedding
openai.api_key = os.getenv("OPENAI_API_KEY", "sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")
results = search_redis(redis_client, 'modern art in Europe', k=10)
```

    0. Museum of Modern Art (Score: 0.875)
    1. Western Europe (Score: 0.867)
    2. Renaissance art (Score: 0.864)
    3. Pop art (Score: 0.86)
    4. Northern Europe (Score: 0.855)
    5. Hellenistic art (Score: 0.853)
    6. Modernist literature (Score: 0.847)
    7. Art film (Score: 0.843)
    8. Central Europe (Score: 0.843)
    9. European (Score: 0.841)



```python
results = search_redis(redis_client, 'Famous battles in Scottish history', vector_field='content_vector', k=10)
```

    0. Battle of Bannockburn (Score: 0.869)
    1. Wars of Scottish Independence (Score: 0.861)
    2. 1651 (Score: 0.853)
    3. First War of Scottish Independence (Score: 0.85)
    4. Robert I of Scotland (Score: 0.846)
    5. 841 (Score: 0.844)
    6. 1716 (Score: 0.844)
    7. 1314 (Score: 0.837)
    8. 1263 (Score: 0.836)
    9. William Wallace (Score: 0.835)


## Hybrid Queries with Redis

The previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the below example, we will combine vector search with full text search.


```python
def create_hybrid_field(field_name: str, value: str) -> str:
    return f'@{field_name}:"{value}"'
```


key: _node_content
string_value:
{"id_": "b5d89480-daae-4ed4-b9df-d701333f0aa6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f5a70c4d-ebc4-48e0-97c8-aae3dd590af8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0502f54a747773e52cc858c77c9c8a4fcec263117b8cfac90516a862cf82df60", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "19643cb5-4125-4fd6-a148-ccccc1f62d3a", "node_type": "1", "metadata": {}, "hash": "6cb263833ce2db73fc6ea420ea725838d71a38887aa9f8c5fbb0f7d90ed6e5ce", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17354, "end_char_idx": 20462, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
results = search_redis(redis_client, 'Famous battles in Scottish history', vector_field='content_vector', k=10)
```

    0. Battle of Bannockburn (Score: 0.869)
    1. Wars of Scottish Independence (Score: 0.861)
    2. 1651 (Score: 0.853)
    3. First War of Scottish Independence (Score: 0.85)
    4. Robert I of Scotland (Score: 0.846)
    5. 841 (Score: 0.844)
    6. 1716 (Score: 0.844)
    7. 1314 (Score: 0.837)
    8. 1263 (Score: 0.836)
    9. William Wallace (Score: 0.835)


## Hybrid Queries with Redis

The previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the below example, we will combine vector search with full text search.


```python
def create_hybrid_field(field_name: str, value: str) -> str:
    return f'@{field_name}:"{value}"'
```


```python
# search the content vector for articles about famous battles in Scottish history and only include results with Scottish in the title
results = search_redis(redis_client,
                       "Famous battles in Scottish history",
                       vector_field="title_vector",
                       k=5,
                       hybrid_fields=create_hybrid_field("title", "Scottish")
                       )
```

    0. First War of Scottish Independence (Score: 0.892)
    1. Wars of Scottish Independence (Score: 0.889)
    2. Second War of Scottish Independence (Score: 0.879)
    3. List of Scottish monarchs (Score: 0.873)
    4. Scottish Borders (Score: 0.863)



```python
# run a hybrid query for articles about Art in the title vector and only include results with the phrase "Leonardo da Vinci" in the text
results = search_redis(redis_client,
                       "Art",
                       vector_field="title_vector",
                       k=5,
                       hybrid_fields=create_hybrid_field("text", "Leonardo da Vinci")
                       )

# find specific mention of Leonardo da Vinci in the text that our full-text-search query returned
mention = [sentence for sentence in results[0].text.split("\n") if "Leonardo da Vinci" in sentence][0]
mention
```

    0. Art (Score: 1.0)
    1. Paint (Score: 0.896)
    2. Renaissance art (Score: 0.88)
    3. Painting (Score: 0.874)
    4. Renaissance (Score: 0.846)





    'In Europe, after the Middle Ages, there was a "Renaissance" which means "rebirth". People rediscovered science and artists were allowed to paint subjects other than religious subjects. People like Michelangelo and Leonardo da Vinci still painted religious pictures, but they also now could paint mythological pictures too. These artists also invented perspective where things in the distance look smaller in the picture. This was new because in the Middle Ages people would paint all the figures close up and just overlapping each other. These artists used nudity regularly in their art.'



For more example with Redis as a vector database, see the README and examples within the ``vector_databases/redis`` directory of this repository


key: _node_content
string_value:
{"id_": "19643cb5-4125-4fd6-a148-ccccc1f62d3a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "44faa2e7-58cb-4f5b-852e-4f211cd17acd", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f34df684b767fa97c5f53afb64f2f5556af0cc7429cba4b02b9d1bea84c3a927", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b5d89480-daae-4ed4-b9df-d701333f0aa6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/Using_Redis_for_embeddings_search.ipynb", "title": "Using Redis for Embeddings Search", "tokens": 5244, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1455475e46fc448513bd9d1ee7df7b5daaf42774e6fae9058747d1ec6747a942", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18942, "end_char_idx": 20479, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# run a hybrid query for articles about Art in the title vector and only include results with the phrase "Leonardo da Vinci" in the text
results = search_redis(redis_client,
                       "Art",
                       vector_field="title_vector",
                       k=5,
                       hybrid_fields=create_hybrid_field("text", "Leonardo da Vinci")
                       )

# find specific mention of Leonardo da Vinci in the text that our full-text-search query returned
mention = [sentence for sentence in results[0].text.split("\n") if "Leonardo da Vinci" in sentence][0]
mention
```

    0. Art (Score: 1.0)
    1. Paint (Score: 0.896)
    2. Renaissance art (Score: 0.88)
    3. Painting (Score: 0.874)
    4. Renaissance (Score: 0.846)





    'In Europe, after the Middle Ages, there was a "Renaissance" which means "rebirth". People rediscovered science and artists were allowed to paint subjects other than religious subjects. People like Michelangelo and Leonardo da Vinci still painted religious pictures, but they also now could paint mythological pictures too. These artists also invented perspective where things in the distance look smaller in the picture. This was new because in the Middle Ages people would paint all the figures close up and just overlapping each other. These artists used nudity regularly in their art.'



For more example with Redis as a vector database, see the README and examples within the ``vector_databases/redis`` directory of this repository


```python

```


key: _node_content
string_value:
{"id_": "f3c97012-a116-4e56-9a79-37e3e6e861fb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2aa7946d-a00f-4f81-bdd6-e48e11274ff7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b08d2ce2199cb0ef05b32554508f11d39da0689c732dce096a9cbf3dd901eca0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "968cd1e8-9d08-41c1-a291-1f6c231698b0", "node_type": "1", "metadata": {}, "hash": "60252f48545fffa75c59a4eb516fd84fc4c1ac8dcd6a08004bbf9cef8f9bfd57", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2501, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Redis

### What is Redis?

Most developers from a web services background are probably familiar with Redis. At it's core, Redis is an open-source key-value store that can be used as a cache, message broker, and database. Developers choice Redis because it is fast, has a large ecosystem of client libraries, and has been deployed by major enterprises for years.

In addition to the traditional uses of Redis. Redis also provides [Redis Modules](https://redis.io/modules) which are a way to extend Redis with new capabilities, commands and data types. Example modules include [RedisJSON](https://redis.io/docs/stack/json/), [RedisTimeSeries](https://redis.io/docs/stack/timeseries/), [RedisBloom](https://redis.io/docs/stack/bloom/) and [RediSearch](https://redis.io/docs/stack/search/).


### Deployment options

There are a number of ways to deploy Redis. For local development, the quickest method is to use the [Redis Stack docker container](https://hub.docker.com/r/redis/redis-stack) which we will use here. Redis Stack contains a number of Redis modules that can be used together to create a fast, multi-model data store and query engine.

For production use cases, The easiest way to get started is to use the [Redis Cloud](https://redislabs.com/redis-enterprise-cloud/overview/) service. Redis Cloud is a fully managed Redis service. You can also deploy Redis on your own infrastructure using [Redis Enterprise](https://redislabs.com/redis-enterprise/overview/). Redis Enterprise is a fully managed Redis service that can be deployed in kubernetes, on-premises or in the cloud.

Additionally, every major cloud provider ([AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-e6y7ork67pjwg?sr=0-2&ref_=beagle&applicationId=AWSMPContessa), [Google Marketplace](https://console.cloud.google.com/marketplace/details/redislabs-public/redis-enterprise?pli=1), or [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/garantiadata.redis_enterprise_1sp_public_preview?tab=Overview)) offers Redis Enterprise in a marketplace offering.


### What is RediSearch?

RediSearch is a [Redis module](https://redis.io/modules) that provides querying, secondary indexing, full-text search and vector search for Redis. To use RediSearch, you first declare indexes on your Redis data. You can then use the RediSearch clients to query that data. For more information on the feature set of RediSearch, see the [RediSearch documentation](https://redis.io/docs/stack/search/).


key: _node_content
string_value:
{"id_": "968cd1e8-9d08-41c1-a291-1f6c231698b0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2aa7946d-a00f-4f81-bdd6-e48e11274ff7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b08d2ce2199cb0ef05b32554508f11d39da0689c732dce096a9cbf3dd901eca0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f3c97012-a116-4e56-9a79-37e3e6e861fb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "80a34cf61b078186617c84d636e9a15d7ce53212072a42d702fa9297589564f7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "33bfa666-f704-4905-b5d8-30be7abd894f", "node_type": "1", "metadata": {}, "hash": "cb7257b683082a9e2a155f60bb6de86a717353be30ded39f860e2f3d5be40b32", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 791, "end_char_idx": 3909, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Deployment options

There are a number of ways to deploy Redis. For local development, the quickest method is to use the [Redis Stack docker container](https://hub.docker.com/r/redis/redis-stack) which we will use here. Redis Stack contains a number of Redis modules that can be used together to create a fast, multi-model data store and query engine.

For production use cases, The easiest way to get started is to use the [Redis Cloud](https://redislabs.com/redis-enterprise-cloud/overview/) service. Redis Cloud is a fully managed Redis service. You can also deploy Redis on your own infrastructure using [Redis Enterprise](https://redislabs.com/redis-enterprise/overview/). Redis Enterprise is a fully managed Redis service that can be deployed in kubernetes, on-premises or in the cloud.

Additionally, every major cloud provider ([AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-e6y7ork67pjwg?sr=0-2&ref_=beagle&applicationId=AWSMPContessa), [Google Marketplace](https://console.cloud.google.com/marketplace/details/redislabs-public/redis-enterprise?pli=1), or [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/garantiadata.redis_enterprise_1sp_public_preview?tab=Overview)) offers Redis Enterprise in a marketplace offering.


### What is RediSearch?

RediSearch is a [Redis module](https://redis.io/modules) that provides querying, secondary indexing, full-text search and vector search for Redis. To use RediSearch, you first declare indexes on your Redis data. You can then use the RediSearch clients to query that data. For more information on the feature set of RediSearch, see the [RediSearch documentation](https://redis.io/docs/stack/search/).


### Features

RediSearch uses compressed, inverted indexes for fast indexing with a low memory footprint. RediSearch indexes enhance Redis by providing exact-phrase matching, fuzzy search, and numeric filtering, among many other features. Such as:

* Full-Text indexing of multiple fields in Redis hashes
* Incremental indexing without performance loss
* Vector similarity search
* Document ranking (using [tf-idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf), with optional user-provided weights)
* Field weighting
* Complex boolean queries with AND, OR, and NOT operators
* Prefix matching, fuzzy matching, and exact-phrase queries
* Support for [double-metaphone phonetic matching](https://redis.io/docs/stack/search/reference/phonetic_matching/)
* Auto-complete suggestions (with fuzzy prefix suggestions)
* Stemming-based query expansion in [many languages](https://redis.io/docs/stack/search/reference/stemming/) (using [Snowball](http://snowballstem.org/))
* Support for Chinese-language tokenization and querying (using [Friso](https://github.com/lionsoul2014/friso))
* Numeric filters and ranges
* Geospatial searches using [Redis geospatial indexing](/commands/georadius)
* A powerful aggregations engine
* Supports for all utf-8 encoded text
* Retrieve full documents, selected fields, or only the document IDs
* Sorting results (for example, by creation date)
* JSON support through RedisJSON


key: _node_content
string_value:
{"id_": "33bfa666-f704-4905-b5d8-30be7abd894f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2aa7946d-a00f-4f81-bdd6-e48e11274ff7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b08d2ce2199cb0ef05b32554508f11d39da0689c732dce096a9cbf3dd901eca0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "968cd1e8-9d08-41c1-a291-1f6c231698b0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "bd34788ffe83d6511cd0a9ff2503d8fab821af0fb6a3c6c53271ad57793cbc8d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bbad043e-7781-46ab-b542-079d2921150e", "node_type": "1", "metadata": {}, "hash": "dc5a05be55dbfc5b2b58c5484e6582f2da7e8cd8fdf126f134a2b850ab78a20c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2504, "end_char_idx": 6474, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Features

RediSearch uses compressed, inverted indexes for fast indexing with a low memory footprint. RediSearch indexes enhance Redis by providing exact-phrase matching, fuzzy search, and numeric filtering, among many other features. Such as:

* Full-Text indexing of multiple fields in Redis hashes
* Incremental indexing without performance loss
* Vector similarity search
* Document ranking (using [tf-idf](https://en.wikipedia.org/wiki/Tf%E2%80%93idf), with optional user-provided weights)
* Field weighting
* Complex boolean queries with AND, OR, and NOT operators
* Prefix matching, fuzzy matching, and exact-phrase queries
* Support for [double-metaphone phonetic matching](https://redis.io/docs/stack/search/reference/phonetic_matching/)
* Auto-complete suggestions (with fuzzy prefix suggestions)
* Stemming-based query expansion in [many languages](https://redis.io/docs/stack/search/reference/stemming/) (using [Snowball](http://snowballstem.org/))
* Support for Chinese-language tokenization and querying (using [Friso](https://github.com/lionsoul2014/friso))
* Numeric filters and ranges
* Geospatial searches using [Redis geospatial indexing](/commands/georadius)
* A powerful aggregations engine
* Supports for all utf-8 encoded text
* Retrieve full documents, selected fields, or only the document IDs
* Sorting results (for example, by creation date)
* JSON support through RedisJSON


### Clients

Given the large ecosystem around Redis, there are most likely client libraries in the language you need. You can use any standard Redis client library to run RediSearch commands, but it's easiest to use a library that wraps the RediSearch API. Below are a few examples, but you can find more client libraries [here](https://redis.io/resources/clients/).

| Project | Language | License | Author | Stars |
|----------|---------|--------|---------|-------|
| [jedis][jedis-url] | Java | MIT | [Redis][redis-url] | ![Stars][jedis-stars] |
| [redis-py][redis-py-url] | Python | MIT | [Redis][redis-url] | ![Stars][redis-py-stars] |
| [node-redis][node-redis-url] | Node.js | MIT | [Redis][redis-url] | ![Stars][node-redis-stars] |
| [nredisstack][nredisstack-url] | .NET | MIT | [Redis][redis-url] | ![Stars][nredisstack-stars] |

[redis-url]: https://redis.com

[redis-py-url]: https://github.com/redis/redis-py
[redis-py-stars]: https://img.shields.io/github/stars/redis/redis-py.svg?style=social&amp;label=Star&amp;maxAge=2592000
[redis-py-package]: https://pypi.python.org/pypi/redis

[jedis-url]: https://github.com/redis/jedis
[jedis-stars]: https://img.shields.io/github/stars/redis/jedis.svg?style=social&amp;label=Star&amp;maxAge=2592000
[Jedis-package]: https://search.maven.org/artifact/redis.clients/jedis

[nredisstack-url]: https://github.com/redis/nredisstack
[nredisstack-stars]: https://img.shields.io/github/stars/redis/nredisstack.svg?style=social&amp;label=Star&amp;maxAge=2592000
[nredisstack-package]: https://www.nuget.org/packages/nredisstack/

[node-redis-url]: https://github.com/redis/node-redis
[node-redis-stars]: https://img.shields.io/github/stars/redis/node-redis.svg?style=social&amp;label=Star&amp;maxAge=2592000
[node-redis-package]: https://www.npmjs.com/package/redis

[redis-om-python-url]: https://github.com/redis/redis-om-python
[redis-om-python-author]: https://redis.com
[redis-om-python-stars]: https://img.shields.io/github/stars/redis/redis-om-python.svg?style=social&amp;label=Star&amp;maxAge=2592000

[redisearch-go-url]: https://github.com/RediSearch/redisearch-go
[redisearch-go-author]: https://redis.com
[redisearch-go-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-go.svg?style=social&amp;label=Star&amp;maxAge=2592000

[redisearch-api-rs-url]: https://github.com/RediSearch/redisearch-api-rs
[redisearch-api-rs-author]: https://redis.com
[redisearch-api-rs-stars]: https://img.shields.io/github/stars/RediSearch/redisearch-api-rs.svg?style=social&amp;label=Star&amp;maxAge=2592000


key: _node_content
string_value:
{"id_": "bbad043e-7781-46ab-b542-079d2921150e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2aa7946d-a00f-4f81-bdd6-e48e11274ff7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b08d2ce2199cb0ef05b32554508f11d39da0689c732dce096a9cbf3dd901eca0", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "33bfa666-f704-4905-b5d8-30be7abd894f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/README.ipynb", "title": "Redis", "tokens": 2363, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "9b21f1fabb3bcd7cfc5b7c63fb1a4bee3ce73426fe0a6272cc4f44e2fa77c13e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6477, "end_char_idx": 9609, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Deployment Options

There are many ways to deploy Redis with RediSearch. The easiest way to get started is to use Docker, but there are are many potential options for deployment such as

- [Redis Cloud](https://redis.com/redis-enterprise-cloud/overview/)
- Cloud marketplaces: [AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-e6y7ork67pjwg?sr=0-2&ref_=beagle&applicationId=AWSMPContessa), [Google Marketplace](https://console.cloud.google.com/marketplace/details/redislabs-public/redis-enterprise?pli=1), or [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/garantiadata.redis_enterprise_1sp_public_preview?tab=Overview)
- On-premise: [Redis Enterprise Software](https://redis.com/redis-enterprise-software/overview/)
- Kubernetes: [Redis Enterprise Software on Kubernetes](https://docs.redis.com/latest/kubernetes/)
- [Docker (RediSearch)](https://hub.docker.com/r/redislabs/redisearch)
- [Docker (Redis Stack)](https://hub.docker.com/r/redis/redis-stack)


### Cluster support

RediSearch has a distributed cluster version that scales to billions of documents across hundreds of servers. At the moment, distributed RediSearch is available as part of [Redis Enterprise Cloud](https://redis.com/redis-enterprise-cloud/overview/) and [Redis Enterprise Software](https://redis.com/redis-enterprise-software/overview/).

See [RediSearch on Redis Enterprise](https://redis.com/modules/redisearch/) for more information.

### Examples

- [Product Search](https://github.com/RedisVentures/redis-product-search) - eCommerce product search (with image and text)
- [Product Recommendations with DocArray / Jina](https://github.com/jina-ai/product-recommendation-redis-docarray) - Content-based product recommendations example with Redis and DocArray.
- [Redis VSS in RecSys](https://github.com/RedisVentures/Redis-Recsys) - 3 end-to-end Redis & NVIDIA Merlin Recommendation System Architectures.
- [Azure OpenAI Embeddings Q&A](https://github.com/ruoccofabrizio/azure-open-ai-embeddings-qna) - OpenAI and Redis as a Q&A service on Azure.
- [ArXiv Paper Search](https://github.com/RedisVentures/redis-arXiv-search) - Semantic search over arXiv scholarly papers


### More Resources

For more information on how to use Redis as a vector database, check out the following resources:

- [Redis Vector Similarity Docs](https://redis.io/docs/stack/search/reference/vectors/) - Redis official docs for Vector Search.
- [Redis-py Search Docs](https://redis.readthedocs.io/en/latest/redismodules.html#redisearch-commands) - Redis-py client library docs for RediSearch.
- [Vector Similarity Search: From Basics to Production](https://mlops.community/vector-similarity-search-from-basics-to-production/) - Introductory blog post to VSS and Redis as a VectorDB.
- [AI-Powered Document Search](https://datasciencedojo.com/blog/ai-powered-document-search/) - Blog post covering AI Powered Document Search Use Cases & Architectures.
- [Vector Database Benchmarks](https://jina.ai/news/benchmark-vector-search-databases-with-one-million-data/) - Jina AI VectorDB benchmarks comparing Redis against others.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb


key: _node_content
string_value:
{"id_": "0b710ae6-bb0b-41de-8014-33a33a86d175", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1ea90e9d-d264-451b-82cf-59f0972d5d1b", "node_type": "1", "metadata": {}, "hash": "d570b3617e31f41433e5b1d8fd413e18a61b6b06e39832d89f4fc2cfe4ddd3fe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3027, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Using Redis as a Vector Database with OpenAI

This notebook provides an introduction to using Redis as a vector database with OpenAI embeddings. Redis is a scalable, real-time database that can be used as a vector database when using the [RediSearch Module](https://oss.redislabs.com/redisearch/). The RediSearch module allows you to index and search for vectors in Redis. This notebook will show you how to use the RediSearch module to index and search for vectors created by using the OpenAI API and stored in Redis.

### What is Redis?

Most developers from a web services background are probably familiar with Redis. At it's core, Redis is an open-source key-value store that can be used as a cache, message broker, and database. Developers choice Redis because it is fast, has a large ecosystem of client libraries, and has been deployed by major enterprises for years.

In addition to the traditional uses of Redis. Redis also provides [Redis Modules](https://redis.io/modules) which are a way to extend Redis with new data types and commands. Example modules include [RedisJSON](https://redis.io/docs/stack/json/), [RedisTimeSeries](https://redis.io/docs/stack/timeseries/), [RedisBloom](https://redis.io/docs/stack/bloom/) and [RediSearch](https://redis.io/docs/stack/search/).

### What is RediSearch?

RediSearch is a [Redis module](https://redis.io/modules) that provides querying, secondary indexing, full-text search and vector search for Redis. To use RediSearch, you first declare indexes on your Redis data. You can then use the RediSearch clients to query that data. For more information on the feature set of RediSearch, see the [README](./README.md) or the [RediSearch documentation](https://redis.io/docs/stack/search/).

### Deployment options

There are a number of ways to deploy Redis. For local development, the quickest method is to use the [Redis Stack docker container](https://hub.docker.com/r/redis/redis-stack) which we will use here. Redis Stack contains a number of Redis modules that can be used together to create a fast, multi-model data store and query engine.

For production use cases, The easiest way to get started is to use the [Redis Cloud](https://redislabs.com/redis-enterprise-cloud/overview/) service. Redis Cloud is a fully managed Redis service. You can also deploy Redis on your own infrastructure using [Redis Enterprise](https://redislabs.com/redis-enterprise/overview/). Redis Enterprise is a fully managed Redis service that can be deployed in kubernetes, on-premises or in the cloud.

Additionally, every major cloud provider ([AWS Marketplace](https://aws.amazon.com/marketplace/pp/prodview-e6y7ork67pjwg?sr=0-2&ref_=beagle&applicationId=AWSMPContessa), [Google Marketplace](https://console.cloud.google.com/marketplace/details/redislabs-public/redis-enterprise?pli=1), or [Azure Marketplace](https://azuremarketplace.microsoft.com/en-us/marketplace/apps/garantiadata.redis_enterprise_1sp_public_preview?tab=Overview)) offers Redis Enterprise in a marketplace offering.


key: _node_content
string_value:
{"id_": "1ea90e9d-d264-451b-82cf-59f0972d5d1b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0b710ae6-bb0b-41de-8014-33a33a86d175", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f12a9bd3e27a18166f0027474e77a1d5d03b7a5c1c41d4a3e85aefd6888a96b3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dd124470-5e0f-48ca-8856-95a2b0fea4fa", "node_type": "1", "metadata": {}, "hash": "2a79989f01b54851ba761c4f72e140fa361cc4be1f7d2f59f6bff5623f6332ec", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3031, "end_char_idx": 5765, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Prerequisites

Before we start this project, we need to set up the following:

* start a Redis database with RediSearch (redis-stack)
* install libraries
    * [Redis-py](https://github.com/redis/redis-py)
* get your [OpenAI API key](https://beta.openai.com/account/api-keys)

===========================================================

### Start Redis

To keep this example simple, we will use the Redis Stack docker container which we can start as follows

```bash
$ docker-compose up -d
```

This also includes the [RedisInsight](https://redis.com/redis-enterprise/redis-insight/) GUI for managing your Redis database which you can view at [http://localhost:8001](http://localhost:8001) once you start the docker container.

You're all set up and ready to go! Next, we import and create our client for communicating with the Redis database we just created.

## Install Requirements

Redis-Py is the python client for communicating with Redis. We will use this to communicate with our Redis-stack database. 


```python
! pip install redis wget pandas openai
```

===========================================================
## Prepare your OpenAI API key

The `OpenAI API key` is used for vectorization of query data.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it to your environment variables as `OPENAI_API_KEY` by using following command:


```python
! export OPENAI_API_KEY="your API key"
```


```python
# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.
import os
import openai

# Note. alternatively you can set a temporary env variable like this:
# os.environ["OPENAI_API_KEY"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

if os.getenv("OPENAI_API_KEY") is not None:
    openai.api_key = os.getenv("OPENAI_API_KEY")
    print ("OPENAI_API_KEY is ready")
else:
    print ("OPENAI_API_KEY environment variable not found")
```

    OPENAI_API_KEY is ready


## Load data

In this section we'll load embedded data that has already been converted into vectors. We'll use this data to create an index in Redis and then search for similar vectors.


```python
import sys
import numpy as np
import pandas as pd
from typing import List

# use helper function in nbutils.py to download and read the data
# this should take from 5-10 min to run
if os.getcwd() not in sys.path:
    sys.path.append(os.getcwd())
import nbutils

nbutils.download_wikipedia_data()
data = nbutils.read_wikipedia_data()

data.head()
```

    File Downloaded


key: _node_content
string_value:
{"id_": "dd124470-5e0f-48ca-8856-95a2b0fea4fa", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1ea90e9d-d264-451b-82cf-59f0972d5d1b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "9f09fccf3f16dd31a15ad373413be01a4367092852784baecb0265e294d78a38", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dea63753-3038-4d5f-a0ba-86a5a1250f95", "node_type": "1", "metadata": {}, "hash": "8cd0a9c9c8bed21afa4c25e5abe679fb75283551f60ad360179fa48e2e466a3e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4518, "end_char_idx": 6782, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
! export OPENAI_API_KEY="your API key"
```


```python
# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.
import os
import openai

# Note. alternatively you can set a temporary env variable like this:
# os.environ["OPENAI_API_KEY"] = 'sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx'

if os.getenv("OPENAI_API_KEY") is not None:
    openai.api_key = os.getenv("OPENAI_API_KEY")
    print ("OPENAI_API_KEY is ready")
else:
    print ("OPENAI_API_KEY environment variable not found")
```

    OPENAI_API_KEY is ready


## Load data

In this section we'll load embedded data that has already been converted into vectors. We'll use this data to create an index in Redis and then search for similar vectors.


```python
import sys
import numpy as np
import pandas as pd
from typing import List

# use helper function in nbutils.py to download and read the data
# this should take from 5-10 min to run
if os.getcwd() not in sys.path:
    sys.path.append(os.getcwd())
import nbutils

nbutils.download_wikipedia_data()
data = nbutils.read_wikipedia_data()

data.head()
```

    File Downloaded





<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>url</th>
      <th>title</th>
      <th>text</th>
      <th>title_vector</th>
      <th>content_vector</th>
      <th>vector_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>https://simple.wikipedia.org/wiki/April</td>
      <td>April</td>
      <td>April is the fourth month of the year in the J...</td>
      <td>[0.001009464613161981, -0.020700545981526375, ...</td>
      <td>[-0.011253940872848034, -0.013491976074874401,...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>https://simple.wikipedia.org/wiki/August</td>
      <td>August</td>
      <td>August (Aug.)


key: _node_content
string_value:
{"id_": "dea63753-3038-4d5f-a0ba-86a5a1250f95", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dd124470-5e0f-48ca-8856-95a2b0fea4fa", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "04631ed923641cbd6b693a87c9df5d6bd071760be3fd82d5cbe7553c01255924", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "17015c72-b4a3-4065-8476-80f181acc532", "node_type": "1", "metadata": {}, "hash": "a38d440fd46e3d8ec15186b41e85bf1941379cf88d54866f98815a4eabb1d814", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5771, "end_char_idx": 7813, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>url</th>
      <th>title</th>
      <th>text</th>
      <th>title_vector</th>
      <th>content_vector</th>
      <th>vector_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>https://simple.wikipedia.org/wiki/April</td>
      <td>April</td>
      <td>April is the fourth month of the year in the J...</td>
      <td>[0.001009464613161981, -0.020700545981526375, ...</td>
      <td>[-0.011253940872848034, -0.013491976074874401,...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>https://simple.wikipedia.org/wiki/August</td>
      <td>August</td>
      <td>August (Aug.) is the eighth month of the year ...</td>
      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>
      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>6</td>
      <td>https://simple.wikipedia.org/wiki/Art</td>
      <td>Art</td>
      <td>Art is a creative activity that expresses imag...</td>
      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>
      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>
      <td>2</td>
    </tr>
    <tr>
      <th>3</th>
      <td>8</td>
      <td>https://simple.wikipedia.org/wiki/A</td>
      <td>A</td>
      <td>A or a is the first letter of the English alph...</td>
      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>
      <td>[0.024894846603274345, -0.022186409682035446, ...</td>
      <td>3</td>
    </tr>
    <tr>
      <th>4</th>
      <td>9</td>
      <td>https://simple.wikipedia.org/wiki/Air</td>
      <td>Air</td>
      <td>Air refers to the Earth's atmosphere.


key: _node_content
string_value:
{"id_": "17015c72-b4a3-4065-8476-80f181acc532", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dea63753-3038-4d5f-a0ba-86a5a1250f95", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d90c715b264776998131e75bb9ab0ca7a5ba3cccd1fef786e7d83411ca4327d6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2fddba5b-ffde-45bf-8a4b-f1270ff11663", "node_type": "1", "metadata": {}, "hash": "a608461bed49331890805a388d9965aafc86c90bcef890b0aa3e983daf0e5a7a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6783, "end_char_idx": 8997, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
is the eighth month of the year ...</td>
      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>
      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>6</td>
      <td>https://simple.wikipedia.org/wiki/Art</td>
      <td>Art</td>
      <td>Art is a creative activity that expresses imag...</td>
      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>
      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>
      <td>2</td>
    </tr>
    <tr>
      <th>3</th>
      <td>8</td>
      <td>https://simple.wikipedia.org/wiki/A</td>
      <td>A</td>
      <td>A or a is the first letter of the English alph...</td>
      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>
      <td>[0.024894846603274345, -0.022186409682035446, ...</td>
      <td>3</td>
    </tr>
    <tr>
      <th>4</th>
      <td>9</td>
      <td>https://simple.wikipedia.org/wiki/Air</td>
      <td>Air</td>
      <td>Air refers to the Earth's atmosphere. Air is a...</td>
      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>
      <td>[0.021524671465158463, 0.018522677943110466, -...</td>
      <td>4</td>
    </tr>
  </tbody>
</table>
</div>



## Connect to Redis

Now that we have our Redis database running, we can connect to it using the Redis-py client. We will use the default host and port for the Redis database which is `localhost:6379`.




```python
import redis
from redis.commands.search.indexDefinition import (
    IndexDefinition,
    IndexType
)
from redis.commands.search.query import Query
from redis.commands.search.field import (
    TextField,
    VectorField
)

REDIS_HOST =  "localhost"
REDIS_PORT = 6379
REDIS_PASSWORD = "" # default for passwordless Redis

# Connect to Redis
redis_client = redis.Redis(
    host=REDIS_HOST,
    port=REDIS_PORT,
    password=REDIS_PASSWORD
)
redis_client.ping()
```




    True



## Creating a Search Index in Redis

The below cells will show how to specify and create a search index in Redis. We will:

1. Set some constants for defining our index like the distance metric and the index name
2. Define the index schema with RediSearch fields
3. Create the index


key: _node_content
string_value:
{"id_": "2fddba5b-ffde-45bf-8a4b-f1270ff11663", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "17015c72-b4a3-4065-8476-80f181acc532", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "57e2fb0a9a329b51ab6508b9ccae3e2d0b863a3afb9dafffdb8c8f2ff91376e5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7f977bc2-aba7-41e1-9748-73c0965650d2", "node_type": "1", "metadata": {}, "hash": "2aef452339ec5229e17ec0e24316586af693499bb31ac7542bce86a3639e450d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7814, "end_char_idx": 10752, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Air is a...</td>
      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>
      <td>[0.021524671465158463, 0.018522677943110466, -...</td>
      <td>4</td>
    </tr>
  </tbody>
</table>
</div>



## Connect to Redis

Now that we have our Redis database running, we can connect to it using the Redis-py client. We will use the default host and port for the Redis database which is `localhost:6379`.




```python
import redis
from redis.commands.search.indexDefinition import (
    IndexDefinition,
    IndexType
)
from redis.commands.search.query import Query
from redis.commands.search.field import (
    TextField,
    VectorField
)

REDIS_HOST =  "localhost"
REDIS_PORT = 6379
REDIS_PASSWORD = "" # default for passwordless Redis

# Connect to Redis
redis_client = redis.Redis(
    host=REDIS_HOST,
    port=REDIS_PORT,
    password=REDIS_PASSWORD
)
redis_client.ping()
```




    True



## Creating a Search Index in Redis

The below cells will show how to specify and create a search index in Redis. We will:

1. Set some constants for defining our index like the distance metric and the index name
2. Define the index schema with RediSearch fields
3. Create the index


```python
# Constants
VECTOR_DIM = len(data['title_vector'][0]) # length of the vectors
VECTOR_NUMBER = len(data)                 # initial number of vectors
INDEX_NAME = "embeddings-index"           # name of the search index
PREFIX = "doc"                            # prefix for the document keys
DISTANCE_METRIC = "COSINE"                # distance metric for the vectors (ex. COSINE, IP, L2)
```


```python
# Define RediSearch fields for each of the columns in the dataset
title = TextField(name="title")
url = TextField(name="url")
text = TextField(name="text")
title_embedding = VectorField("title_vector",
    "FLAT", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER,
    }
)
text_embedding = VectorField("content_vector",
    "FLAT", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER,
    }
)
fields = [title, url, text, title_embedding, text_embedding]
```


```python
# Check if index exists
try:
    redis_client.ft(INDEX_NAME).info()
    print("Index already exists")
except:
    # Create RediSearch Index
    redis_client.ft(INDEX_NAME).create_index(
        fields = fields,
        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)
)
```

## Load Documents into the Index

Now that we have a search index, we can load documents into it. We will use the same documents we used in the previous examples. In Redis, either the HASH or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The below cells will show how to load documents into the index.


key: _node_content
string_value:
{"id_": "7f977bc2-aba7-41e1-9748-73c0965650d2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2fddba5b-ffde-45bf-8a4b-f1270ff11663", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e1132ed715785cd6b2613ba802c4e18e1417baabdde0c713f7f0e56e917b3ce2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "12d86870-d5cf-49c9-bfa3-7a0f422f7735", "node_type": "1", "metadata": {}, "hash": "4a0d2119b71c60a4b5306fee9f383594b7e682068d2543563c32df6bf2bd7628", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9403, "end_char_idx": 11963, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Define RediSearch fields for each of the columns in the dataset
title = TextField(name="title")
url = TextField(name="url")
text = TextField(name="text")
title_embedding = VectorField("title_vector",
    "FLAT", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER,
    }
)
text_embedding = VectorField("content_vector",
    "FLAT", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER,
    }
)
fields = [title, url, text, title_embedding, text_embedding]
```


```python
# Check if index exists
try:
    redis_client.ft(INDEX_NAME).info()
    print("Index already exists")
except:
    # Create RediSearch Index
    redis_client.ft(INDEX_NAME).create_index(
        fields = fields,
        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)
)
```

## Load Documents into the Index

Now that we have a search index, we can load documents into it. We will use the same documents we used in the previous examples. In Redis, either the HASH or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The below cells will show how to load documents into the index.


```python
def index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):
    records = documents.to_dict("records")
    for doc in records:
        key = f"{prefix}:{str(doc['id'])}"

        # create byte vectors for title and content
        title_embedding = np.array(doc["title_vector"], dtype=np.float32).tobytes()
        content_embedding = np.array(doc["content_vector"], dtype=np.float32).tobytes()

        # replace list of floats with byte vectors
        doc["title_vector"] = title_embedding
        doc["content_vector"] = content_embedding

        client.hset(key, mapping = doc)
```


```python
index_documents(redis_client, PREFIX, data)
print(f"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}")
```

    Loaded 25000 documents in Redis search index with name: embeddings-index


## Simple Vector Search Queries with OpenAI Query Embeddings

Now that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database.


key: _node_content
string_value:
{"id_": "12d86870-d5cf-49c9-bfa3-7a0f422f7735", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7f977bc2-aba7-41e1-9748-73c0965650d2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0d347f4355c6b4eedb1ffc51149d5c237b5cc4d80796b7aa1925d1ee41b7dc1d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "eba201ef-4444-40c6-9cb6-9e40fc0e8a0d", "node_type": "1", "metadata": {}, "hash": "0bb1516120e00d61d536bb174f6c12726c16166080eef2402122c96f1832e3ab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10755, "end_char_idx": 13727, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):
    records = documents.to_dict("records")
    for doc in records:
        key = f"{prefix}:{str(doc['id'])}"

        # create byte vectors for title and content
        title_embedding = np.array(doc["title_vector"], dtype=np.float32).tobytes()
        content_embedding = np.array(doc["content_vector"], dtype=np.float32).tobytes()

        # replace list of floats with byte vectors
        doc["title_vector"] = title_embedding
        doc["content_vector"] = content_embedding

        client.hset(key, mapping = doc)
```


```python
index_documents(redis_client, PREFIX, data)
print(f"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}")
```

    Loaded 25000 documents in Redis search index with name: embeddings-index


## Simple Vector Search Queries with OpenAI Query Embeddings

Now that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database.


```python
def search_redis(
    redis_client: redis.Redis,
    user_query: str,
    index_name: str = "embeddings-index",
    vector_field: str = "title_vector",
    return_fields: list = ["title", "url", "text", "vector_score"],
    hybrid_fields = "*",
    k: int = 20,
    print_results: bool = True,
) -> List[dict]:

    # Creates embedding vector from user query
    embedded_query = openai.Embedding.create(input=user_query,
                                            model="text-embedding-3-small",
                                            )["data"][0]['embedding']

    # Prepare the Query
    base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'
    query = (
        Query(base_query)
         .return_fields(*return_fields)
         .sort_by("vector_score")
         .paging(0, k)
         .dialect(2)
    )
    params_dict = {"vector": np.array(embedded_query).astype(dtype=np.float32).tobytes()}

    # perform vector search
    results = redis_client.ft(index_name).search(query, params_dict)
    if print_results:
        for i, article in enumerate(results.docs):
            score = 1 - float(article.vector_score)
            print(f"{i}. {article.title} (Score: {round(score ,3) })")
    return results.docs
```


```python
# For using OpenAI to generate query embedding
results = search_redis(redis_client, 'modern art in Europe', k=10)
```

    0. Museum of Modern Art (Score: 0.875)
    1. Western Europe (Score: 0.868)
    2. Renaissance art (Score: 0.864)
    3. Pop art (Score: 0.86)
    4. Northern Europe (Score: 0.855)
    5. Hellenistic art (Score: 0.853)
    6. Modernist literature (Score: 0.847)
    7. Art film (Score: 0.843)
    8. Central Europe (Score: 0.843)
    9. European (Score: 0.841)


key: _node_content
string_value:
{"id_": "eba201ef-4444-40c6-9cb6-9e40fc0e8a0d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "12d86870-d5cf-49c9-bfa3-7a0f422f7735", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "6b75a9d29ddd5a7b652a6f859b49e020105eeb98194063047412457a444b936d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "979144aa-e737-4c29-bb31-60f1386e3fa2", "node_type": "1", "metadata": {}, "hash": "559d12af48b71ced4c4154ceb046b72a950acd30db6df770a4129b5e21b40cc3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13234, "end_char_idx": 15300, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# For using OpenAI to generate query embedding
results = search_redis(redis_client, 'modern art in Europe', k=10)
```

    0. Museum of Modern Art (Score: 0.875)
    1. Western Europe (Score: 0.868)
    2. Renaissance art (Score: 0.864)
    3. Pop art (Score: 0.86)
    4. Northern Europe (Score: 0.855)
    5. Hellenistic art (Score: 0.853)
    6. Modernist literature (Score: 0.847)
    7. Art film (Score: 0.843)
    8. Central Europe (Score: 0.843)
    9. European (Score: 0.841)



```python
results = search_redis(redis_client, 'Famous battles in Scottish history', vector_field='content_vector', k=10)
```

    0. Battle of Bannockburn (Score: 0.869)
    1. Wars of Scottish Independence (Score: 0.861)
    2. 1651 (Score: 0.853)
    3. First War of Scottish Independence (Score: 0.85)
    4. Robert I of Scotland (Score: 0.846)
    5. 841 (Score: 0.844)
    6. 1716 (Score: 0.844)
    7. 1314 (Score: 0.837)
    8. 1263 (Score: 0.836)
    9. William Wallace (Score: 0.835)


## Hybrid Queries with Redis

The previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the below example, we will combine vector search with full text search.


```python
def create_hybrid_field(field_name: str, value: str) -> str:
    return f'@{field_name}:"{value}"'

# search the content vector for articles about famous battles in Scottish history and only include results with Scottish in the title
results = search_redis(redis_client,
                       "Famous battles in Scottish history",
                       vector_field="title_vector",
                       k=5,
                       hybrid_fields=create_hybrid_field("title", "Scottish")
                       )
```

    0. First War of Scottish Independence (Score: 0.892)
    1. Wars of Scottish Independence (Score: 0.889)
    2. Second War of Scottish Independence (Score: 0.879)
    3. List of Scottish monarchs (Score: 0.873)
    4. Scottish Borders (Score: 0.863)


key: _node_content
string_value:
{"id_": "979144aa-e737-4c29-bb31-60f1386e3fa2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "eba201ef-4444-40c6-9cb6-9e40fc0e8a0d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "5707566de7a33eb1fdb41d81cafaaf0426027e4ac6f0d64985a0d55f326d5dca", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "047a5976-c7ea-4c8e-ad9c-321e735451e8", "node_type": "1", "metadata": {}, "hash": "3f934ea4c249ac489b58c50e639a9dec48926fc9d23d9da862fa9e7d9a05eab7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14227, "end_char_idx": 17343, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Hybrid Queries with Redis

The previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the below example, we will combine vector search with full text search.


```python
def create_hybrid_field(field_name: str, value: str) -> str:
    return f'@{field_name}:"{value}"'

# search the content vector for articles about famous battles in Scottish history and only include results with Scottish in the title
results = search_redis(redis_client,
                       "Famous battles in Scottish history",
                       vector_field="title_vector",
                       k=5,
                       hybrid_fields=create_hybrid_field("title", "Scottish")
                       )
```

    0. First War of Scottish Independence (Score: 0.892)
    1. Wars of Scottish Independence (Score: 0.889)
    2. Second War of Scottish Independence (Score: 0.879)
    3. List of Scottish monarchs (Score: 0.873)
    4. Scottish Borders (Score: 0.863)



```python
# run a hybrid query for articles about Art in the title vector and only include results with the phrase "Leonardo da Vinci" in the text
results = search_redis(redis_client,
                       "Art",
                       vector_field="title_vector",
                       k=5,
                       hybrid_fields=create_hybrid_field("text", "Leonardo da Vinci")
                       )

# find specific mention of Leonardo da Vinci in the text that our full-text-search query returned
mention = [sentence for sentence in results[0].text.split("\n") if "Leonardo da Vinci" in sentence][0]
mention
```

    0. Art (Score: 1.0)
    1. Paint (Score: 0.896)
    2. Renaissance art (Score: 0.88)
    3. Painting (Score: 0.874)
    4. Renaissance (Score: 0.846)





    'In Europe, after the Middle Ages, there was a "Renaissance" which means "rebirth". People rediscovered science and artists were allowed to paint subjects other than religious subjects. People like Michelangelo and Leonardo da Vinci still painted religious pictures, but they also now could paint mythological pictures too. These artists also invented perspective where things in the distance look smaller in the picture. This was new because in the Middle Ages people would paint all the figures close up and just overlapping each other. These artists used nudity regularly in their art.'



## HNSW Index

Up until now, we've been using the ``FLAT`` or "brute-force" index to run our queries. Redis also supports the ``HNSW`` index which is a fast, approximate index. The ``HNSW`` index is a graph-based index that uses a hierarchical navigable small world graph to store vectors. The ``HNSW`` index is a good choice for large datasets where you want to run approximate queries.

``HNSW`` will take longer to build and consume more memory for most cases than ``FLAT`` but will be faster to run queries on, especially for large datasets.

The following cells will show how to create an ``HNSW`` index and run queries with it using the same data as before.


key: _node_content
string_value:
{"id_": "047a5976-c7ea-4c8e-ad9c-321e735451e8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "979144aa-e737-4c29-bb31-60f1386e3fa2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "58f452b3738cc433fa12b11e9929d538fb1cd110c791f3bbb358041a2563983c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1033a569-4384-4cc4-aee7-13f0d7e2a51c", "node_type": "1", "metadata": {}, "hash": "351b8cb5df5b075c6e22471bf20d5a9100892b9360be4b27030167d3e9e7b9d0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16087, "end_char_idx": 18745, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
'In Europe, after the Middle Ages, there was a "Renaissance" which means "rebirth". People rediscovered science and artists were allowed to paint subjects other than religious subjects. People like Michelangelo and Leonardo da Vinci still painted religious pictures, but they also now could paint mythological pictures too. These artists also invented perspective where things in the distance look smaller in the picture. This was new because in the Middle Ages people would paint all the figures close up and just overlapping each other. These artists used nudity regularly in their art.'



## HNSW Index

Up until now, we've been using the ``FLAT`` or "brute-force" index to run our queries. Redis also supports the ``HNSW`` index which is a fast, approximate index. The ``HNSW`` index is a graph-based index that uses a hierarchical navigable small world graph to store vectors. The ``HNSW`` index is a good choice for large datasets where you want to run approximate queries.

``HNSW`` will take longer to build and consume more memory for most cases than ``FLAT`` but will be faster to run queries on, especially for large datasets.

The following cells will show how to create an ``HNSW`` index and run queries with it using the same data as before.


```python
# re-define RediSearch vector fields to use HNSW index
title_embedding = VectorField("title_vector",
    "HNSW", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER
    }
)
text_embedding = VectorField("content_vector",
    "HNSW", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER
    }
)
fields = [title, url, text, title_embedding, text_embedding]
```


```python
import time
# Check if index exists
HNSW_INDEX_NAME = INDEX_NAME+ "_HNSW"

try:
    redis_client.ft(HNSW_INDEX_NAME).info()
    print("Index already exists")
except:
    # Create RediSearch Index
    redis_client.ft(HNSW_INDEX_NAME).create_index(
        fields = fields,
        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)
    )

# since RediSearch creates the index in the background for existing documents, we will wait until
# indexing is complete before running our queries. Although this is not necessary for the first query,
# some queries may take longer to run if the index is not fully built. In general, Redis will perform
# best when adding new documents to existing indices rather than new indices on existing documents.
while redis_client.ft(HNSW_INDEX_NAME).info()["indexing"] == "1":
    time.sleep(5)
```


key: _node_content
string_value:
{"id_": "1033a569-4384-4cc4-aee7-13f0d7e2a51c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "047a5976-c7ea-4c8e-ad9c-321e735451e8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "5fd1d54c4034a33ea4948fff888d957f1222c4beb545833fc730d76ab80db802", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5baaa791-caac-44c5-b4ef-cbed9455f3eb", "node_type": "1", "metadata": {}, "hash": "10d82b05be97afebf2ff236a04b28e8635da2b57e558f661bf4e3eb3f1ebfbd8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17346, "end_char_idx": 19231, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# re-define RediSearch vector fields to use HNSW index
title_embedding = VectorField("title_vector",
    "HNSW", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER
    }
)
text_embedding = VectorField("content_vector",
    "HNSW", {
        "TYPE": "FLOAT32",
        "DIM": VECTOR_DIM,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": VECTOR_NUMBER
    }
)
fields = [title, url, text, title_embedding, text_embedding]
```


```python
import time
# Check if index exists
HNSW_INDEX_NAME = INDEX_NAME+ "_HNSW"

try:
    redis_client.ft(HNSW_INDEX_NAME).info()
    print("Index already exists")
except:
    # Create RediSearch Index
    redis_client.ft(HNSW_INDEX_NAME).create_index(
        fields = fields,
        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)
    )

# since RediSearch creates the index in the background for existing documents, we will wait until
# indexing is complete before running our queries. Although this is not necessary for the first query,
# some queries may take longer to run if the index is not fully built. In general, Redis will perform
# best when adding new documents to existing indices rather than new indices on existing documents.
while redis_client.ft(HNSW_INDEX_NAME).info()["indexing"] == "1":
    time.sleep(5)
```


```python
results = search_redis(redis_client, 'modern art in Europe', index_name=HNSW_INDEX_NAME, k=10)
```

    0. Western Europe (Score: 0.868)
    1. Northern Europe (Score: 0.855)
    2. Central Europe (Score: 0.843)
    3. European (Score: 0.841)
    4. Eastern Europe (Score: 0.839)
    5. Europe (Score: 0.839)
    6. Western European Union (Score: 0.837)
    7. Southern Europe (Score: 0.831)
    8. Western civilization (Score: 0.83)
    9. Council of Europe (Score: 0.827)


key: _node_content
string_value:
{"id_": "5baaa791-caac-44c5-b4ef-cbed9455f3eb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1033a569-4384-4cc4-aee7-13f0d7e2a51c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "14e964cd6f8306781b21d904a8ca1fdc0f100b32727f22168a88be4a47dd63d4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cd8cf02d-f35c-42dd-9c4f-939e8cd95a34", "node_type": "1", "metadata": {}, "hash": "5b0d7e8666a5b807e3bbea3400b6775cad72d42af45bacc0f92b52f9d959d8db", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17885, "end_char_idx": 21190, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import time
# Check if index exists
HNSW_INDEX_NAME = INDEX_NAME+ "_HNSW"

try:
    redis_client.ft(HNSW_INDEX_NAME).info()
    print("Index already exists")
except:
    # Create RediSearch Index
    redis_client.ft(HNSW_INDEX_NAME).create_index(
        fields = fields,
        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)
    )

# since RediSearch creates the index in the background for existing documents, we will wait until
# indexing is complete before running our queries. Although this is not necessary for the first query,
# some queries may take longer to run if the index is not fully built. In general, Redis will perform
# best when adding new documents to existing indices rather than new indices on existing documents.
while redis_client.ft(HNSW_INDEX_NAME).info()["indexing"] == "1":
    time.sleep(5)
```


```python
results = search_redis(redis_client, 'modern art in Europe', index_name=HNSW_INDEX_NAME, k=10)
```

    0. Western Europe (Score: 0.868)
    1. Northern Europe (Score: 0.855)
    2. Central Europe (Score: 0.843)
    3. European (Score: 0.841)
    4. Eastern Europe (Score: 0.839)
    5. Europe (Score: 0.839)
    6. Western European Union (Score: 0.837)
    7. Southern Europe (Score: 0.831)
    8. Western civilization (Score: 0.83)
    9. Council of Europe (Score: 0.827)



```python
# compare the results of the HNSW index to the FLAT index and time both queries
def time_queries(iterations: int = 10):
    print(" ----- Flat Index ----- ")
    t0 = time.time()
    for i in range(iterations):
        results_flat = search_redis(redis_client, 'modern art in Europe', k=10, print_results=False)
    t0 = (time.time() - t0) / iterations
    results_flat = search_redis(redis_client, 'modern art in Europe', k=10, print_results=True)
    print(f"Flat index query time: {round(t0, 3)} seconds\n")
    time.sleep(1)
    print(" ----- HNSW Index ------ ")
    t1 = time.time()
    for i in range(iterations):
        results_hnsw = search_redis(redis_client, 'modern art in Europe', index_name=HNSW_INDEX_NAME, k=10, print_results=False)
    t1 = (time.time() - t1) / iterations
    results_hnsw = search_redis(redis_client, 'modern art in Europe', index_name=HNSW_INDEX_NAME, k=10, print_results=True)
    print(f"HNSW index query time: {round(t1, 3)} seconds")
    print(" ------------------------ ")
time_queries()
```

     ----- Flat Index ----- 
    0. Museum of Modern Art (Score: 0.875)
    1. Western Europe (Score: 0.867)
    2. Renaissance art (Score: 0.864)
    3. Pop art (Score: 0.861)
    4. Northern Europe (Score: 0.855)
    5. Hellenistic art (Score: 0.853)
    6. Modernist literature (Score: 0.847)
    7. Art film (Score: 0.843)
    8. Central Europe (Score: 0.843)
    9. Art (Score: 0.842)
    Flat index query time: 0.263 seconds
    
     ----- HNSW Index ------ 
    0. Western Europe (Score: 0.867)
    1. Northern Europe (Score: 0.855)
    2. Central Europe (Score: 0.843)
    3. European (Score: 0.841)
    4. Eastern Europe (Score: 0.839)
    5. Europe (Score: 0.839)
    6. Western European Union (Score: 0.837)
    7. Southern Europe (Score: 0.831)
    8. Western civilization (Score: 0.83)
    9. Council of Europe (Score: 0.827)
    HNSW index query time: 0.129 seconds
     ------------------------


key: _node_content
string_value:
{"id_": "cd8cf02d-f35c-42dd-9c4f-939e8cd95a34", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e0f3c006-85d6-4d74-90c3-543c7afc102d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7fb7c0407c068fa1452d21e547d6daed28f81451eb8accae5126ee4e986a125c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5baaa791-caac-44c5-b4ef-cbed9455f3eb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/getting-started-with-redis-and-openai.ipynb", "title": "Using Redis as a Vector Database with OpenAI", "tokens": 5520, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4f6925f7dae29ce25a580ccd831642b52ee7b2c90aeefef5e0ad0e0893a662d3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21195, "end_char_idx": 21209, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb


key: _node_content
string_value:
{"id_": "9244bddd-3360-4744-a991-864322a572d9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b5843b81-291b-40bc-b969-8753ef31e079", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ffe37e45-f015-476f-ad6b-d69d47540510", "node_type": "1", "metadata": {}, "hash": "48771e548cd855d4c03fd7f47a090609cdb397dbaf2db6a8493b3df0c8026bfe", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3212, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Running Hybrid VSS Queries with Redis and OpenAI

This notebook provides an introduction to using Redis as a vector database with OpenAI embeddings and running hybrid queries that combine VSS and lexical search using Redis Query and Search capability. Redis is a scalable, real-time database that can be used as a vector database when using the [RediSearch Module](https://oss.redislabs.com/redisearch/). The Redis Query and Search capability allows you to index and search for vectors in Redis. This notebook will show you how to use the Redis Query and Search to index and search for vectors created by using the OpenAI API and stored in Redis.

Hybrid queries combine vector similarity with traditional Redis Query and Search filtering capabilities on GEO, NUMERIC, TAG or TEXT data simplifying application code. A common example of a hybrid query in an e-commerce use case is to find items visually similar to a given query image limited to items available in a GEO location and within a price range.

## Prerequisites

Before we start this project, we need to set up the following:

* start a Redis database with RediSearch (redis-stack)
* install libraries
    * [Redis-py](https://github.com/redis/redis-py)
* get your [OpenAI API key](https://beta.openai.com/account/api-keys)

===========================================================

### Start Redis

To keep this example simple, we will use the Redis Stack docker container which we can start as follows

```bash
$ docker-compose up -d
```

This also includes the [RedisInsight](https://redis.com/redis-enterprise/redis-insight/) GUI for managing your Redis database which you can view at [http://localhost:8001](http://localhost:8001) once you start the docker container.

You're all set up and ready to go! Next, we import and create our client for communicating with the Redis database we just created.

## Install Requirements

Redis-Py is the python client for communicating with Redis. We will use this to communicate with our Redis-stack database. 


```python
! pip install redis pandas openai

```

    Defaulting to user installation because normal site-packages is not writeable
    Requirement already satisfied: redis in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (4.5.4)
    Requirement already satisfied: pandas in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (2.0.1)
    Requirement already satisfied: openai in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (0.27.6)
    Requirement already satisfied: async-timeout>=4.0.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from redis) (4.0.2)
    Requirement already satisfied: python-dateutil>=2.8.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2.8.2)
    Requirement already satisfied: pytz>=2020.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2023.3)
    Requirement already satisfied: tzdata>=2022.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2023.3)
    Requirement already satisfied: numpy>=1.20.3 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (1.23.


key: _node_content
string_value:
{"id_": "ffe37e45-f015-476f-ad6b-d69d47540510", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b5843b81-291b-40bc-b969-8753ef31e079", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9244bddd-3360-4744-a991-864322a572d9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "5fe4391cf140269d5190fb08948899485054c02937a5d473a8137e5eb960c365", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "12c53a10-67c9-44f2-90ac-aeded8d4afa7", "node_type": "1", "metadata": {}, "hash": "b5bbc484afc986c9a18fe4c73a4cb013ff3346a538ef8364957199a81b3e8c43", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2024, "end_char_idx": 4613, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
! pip install redis pandas openai

```

    Defaulting to user installation because normal site-packages is not writeable
    Requirement already satisfied: redis in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (4.5.4)
    Requirement already satisfied: pandas in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (2.0.1)
    Requirement already satisfied: openai in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (0.27.6)
    Requirement already satisfied: async-timeout>=4.0.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from redis) (4.0.2)
    Requirement already satisfied: python-dateutil>=2.8.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2.8.2)
    Requirement already satisfied: pytz>=2020.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2023.3)
    Requirement already satisfied: tzdata>=2022.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (2023.3)
    Requirement already satisfied: numpy>=1.20.3 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from pandas) (1.23.4)
    Requirement already satisfied: requests>=2.20 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (2.28.1)
    Requirement already satisfied: tqdm in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (4.64.1)
    Requirement already satisfied: aiohttp in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (3.8.4)
    Requirement already satisfied: six>=1.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)
    Requirement already satisfied: charset-normalizer<3,>=2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2.1.1)
    Requirement already satisfied: idna<4,>=2.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (3.4)
    Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (1.26.12)
    Requirement already satisfied: certifi>=2017.4.17 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2022.9.24)
    Requirement already satisfied: attrs>=17.3.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (23.1.0)
    Requirement already satisfied: multidict<7.


key: _node_content
string_value:
{"id_": "12c53a10-67c9-44f2-90ac-aeded8d4afa7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b5843b81-291b-40bc-b969-8753ef31e079", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ffe37e45-f015-476f-ad6b-d69d47540510", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "744a50b963670a7a5df564127bba56bd8a4e063cd0ffc345c62100d8c7ba9f6d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2b0c37c8-8365-4d42-a287-214b22ed14f8", "node_type": "1", "metadata": {}, "hash": "ac5d72e261ea856563d0f3e6c0751e53d40e3f5887c8bb35bd939b6770c12dd3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3349, "end_char_idx": 5584, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
28.1)
    Requirement already satisfied: tqdm in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (4.64.1)
    Requirement already satisfied: aiohttp in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from openai) (3.8.4)
    Requirement already satisfied: six>=1.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)
    Requirement already satisfied: charset-normalizer<3,>=2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2.1.1)
    Requirement already satisfied: idna<4,>=2.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (3.4)
    Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (1.26.12)
    Requirement already satisfied: certifi>=2017.4.17 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2022.9.24)
    Requirement already satisfied: attrs>=17.3.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (23.1.0)
    Requirement already satisfied: multidict<7.0,>=4.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (6.0.4)
    Requirement already satisfied: yarl<2.0,>=1.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.9.2)
    Requirement already satisfied: frozenlist>=1.1.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.3)
    Requirement already satisfied: aiosignal>=1.1.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.1)


===========================================================
## Prepare your OpenAI API key

The `OpenAI API key` is used for vectorization of query data.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it to your environment variables as `OPENAI_API_KEY` by using following command:


key: _node_content
string_value:
{"id_": "2b0c37c8-8365-4d42-a287-214b22ed14f8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b5843b81-291b-40bc-b969-8753ef31e079", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "12c53a10-67c9-44f2-90ac-aeded8d4afa7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "9578ab9e6236f83de7fa47ba03515812c76a4c810c3b8d428fc89e0e4e5377b6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "53dbc9fb-40c8-4841-8fe5-6c7e939d7379", "node_type": "1", "metadata": {}, "hash": "54d7a83cada1ca318c4286170ec41e73ccba6ca043acce6f116fe4b991a1a070", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4233, "end_char_idx": 6312, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
20->openai) (1.26.12)
    Requirement already satisfied: certifi>=2017.4.17 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from requests>=2.20->openai) (2022.9.24)
    Requirement already satisfied: attrs>=17.3.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (23.1.0)
    Requirement already satisfied: multidict<7.0,>=4.5 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (6.0.4)
    Requirement already satisfied: yarl<2.0,>=1.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.9.2)
    Requirement already satisfied: frozenlist>=1.1.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.3)
    Requirement already satisfied: aiosignal>=1.1.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.1)


===========================================================
## Prepare your OpenAI API key

The `OpenAI API key` is used for vectorization of query data.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it to your environment variables as `OPENAI_API_KEY` by using following command:


```python
# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.
import os
import openai

os.environ["OPENAI_API_KEY"] = '<YOUR_OPENAI_API_KEY>'

if os.getenv("OPENAI_API_KEY") is not None:
    openai.api_key = os.getenv("OPENAI_API_KEY")
    print ("OPENAI_API_KEY is ready")
else:
    print ("OPENAI_API_KEY environment variable not found")

```

    OPENAI_API_KEY is ready


## Load data

In this section we'll load and clean an ecommerce dataset. We'll generate embeddings using OpenAI and use this data to create an index in Redis and then search for similar vectors.


key: _node_content
string_value:
{"id_": "53dbc9fb-40c8-4841-8fe5-6c7e939d7379", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b5843b81-291b-40bc-b969-8753ef31e079", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2b0c37c8-8365-4d42-a287-214b22ed14f8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "06c8e9ab77553ba89a1202b8133db929cd1eff6c8af40ef7192183c2bdfa8535", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "545e5dae-826e-4ade-9f6c-b30391c0a264", "node_type": "1", "metadata": {}, "hash": "dd753ee1a3c82a2e076ed454a1897d74cbf852bbf64769b6524519050158bef7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4761, "end_char_idx": 7685, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0,>=1.0 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.9.2)
    Requirement already satisfied: frozenlist>=1.1.1 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.3)
    Requirement already satisfied: aiosignal>=1.1.2 in /Users/michael.yuan/Library/Python/3.9/lib/python/site-packages (from aiohttp->openai) (1.3.1)


===========================================================
## Prepare your OpenAI API key

The `OpenAI API key` is used for vectorization of query data.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it to your environment variables as `OPENAI_API_KEY` by using following command:


```python
# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.
import os
import openai

os.environ["OPENAI_API_KEY"] = '<YOUR_OPENAI_API_KEY>'

if os.getenv("OPENAI_API_KEY") is not None:
    openai.api_key = os.getenv("OPENAI_API_KEY")
    print ("OPENAI_API_KEY is ready")
else:
    print ("OPENAI_API_KEY environment variable not found")

```

    OPENAI_API_KEY is ready


## Load data

In this section we'll load and clean an ecommerce dataset. We'll generate embeddings using OpenAI and use this data to create an index in Redis and then search for similar vectors.


```python
import pandas as pd
import numpy as np
from typing import List

from utils.embeddings_utils import (
    get_embeddings,
    distances_from_embeddings,
    tsne_components_from_embeddings,
    chart_from_components,
    indices_of_nearest_neighbors_from_distances,
)

EMBEDDING_MODEL = "text-embedding-3-small"

# load in data and clean data types and drop null rows
df = pd.read_csv("../../data/styles_2k.csv", on_bad_lines='skip')
df.dropna(inplace=True)
df["year"] = df["year"].astype(int)
df.info()

# print dataframe
n_examples = 5
df.head(n_examples)

```

    <class 'pandas.core.frame.DataFrame'>
    Index: 1978 entries, 0 to 1998
    Data columns (total 10 columns):
     #   Column              Non-Null Count  Dtype 
    ---  ------              --------------  ----- 
     0   id                  1978 non-null   int64 
     1   gender              1978 non-null   object
     2   masterCategory      1978 non-null   object
     3   subCategory         1978 non-null   object
     4   articleType         1978 non-null   object
     5   baseColour          1978 non-null   object
     6   season              1978 non-null   object
     7   year                1978 non-null   int64 
     8   usage               1978 non-null   object
     9   productDisplayName  1978 non-null   object
    dtypes: int64(2), object(8)
    memory usage: 170.0+ KB


key: _node_content
string_value:
{"id_": "545e5dae-826e-4ade-9f6c-b30391c0a264", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b5843b81-291b-40bc-b969-8753ef31e079", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "53dbc9fb-40c8-4841-8fe5-6c7e939d7379", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "30ae67437e33a6d7035398517d20fdec8b1741cec4c93e4501ac7dc277bdf4d0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e56ef2bc-a8ee-4934-b4ad-b45c2857ec9e", "node_type": "1", "metadata": {}, "hash": "8a1838f53f0ebecc073805ab5138940bc791f72e8342f728f05fa6c5cae09e4e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6315, "end_char_idx": 9727, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import pandas as pd
import numpy as np
from typing import List

from utils.embeddings_utils import (
    get_embeddings,
    distances_from_embeddings,
    tsne_components_from_embeddings,
    chart_from_components,
    indices_of_nearest_neighbors_from_distances,
)

EMBEDDING_MODEL = "text-embedding-3-small"

# load in data and clean data types and drop null rows
df = pd.read_csv("../../data/styles_2k.csv", on_bad_lines='skip')
df.dropna(inplace=True)
df["year"] = df["year"].astype(int)
df.info()

# print dataframe
n_examples = 5
df.head(n_examples)

```

    <class 'pandas.core.frame.DataFrame'>
    Index: 1978 entries, 0 to 1998
    Data columns (total 10 columns):
     #   Column              Non-Null Count  Dtype 
    ---  ------              --------------  ----- 
     0   id                  1978 non-null   int64 
     1   gender              1978 non-null   object
     2   masterCategory      1978 non-null   object
     3   subCategory         1978 non-null   object
     4   articleType         1978 non-null   object
     5   baseColour          1978 non-null   object
     6   season              1978 non-null   object
     7   year                1978 non-null   int64 
     8   usage               1978 non-null   object
     9   productDisplayName  1978 non-null   object
    dtypes: int64(2), object(8)
    memory usage: 170.0+ KB





<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>gender</th>
      <th>masterCategory</th>
      <th>subCategory</th>
      <th>articleType</th>
      <th>baseColour</th>
      <th>season</th>
      <th>year</th>
      <th>usage</th>
      <th>productDisplayName</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>15970</td>
      <td>Men</td>
      <td>Apparel</td>
      <td>Topwear</td>
      <td>Shirts</td>
      <td>Navy Blue</td>
      <td>Fall</td>
      <td>2011</td>
      <td>Casual</td>
      <td>Turtle Check Men Navy Blue Shirt</td>
    </tr>
    <tr>
      <th>1</th>
      <td>39386</td>
      <td>Men</td>
      <td>Apparel</td>
      <td>Bottomwear</td>
      <td>Jeans</td>
      <td>Blue</td>
      <td>Summer</td>
      <td>2012</td>
      <td>Casual</td>
      <td>Peter England Men Party Blue Jeans</td>
    </tr>
    <tr>
      <th>2</th>
      <td>59263</td>
      <td>Women</td>
      <td>Accessories</td>
      <td>Watches</td>
      <td>Watches</td>
      <td>Silver</td>
      <td>Winter</td>
      <td>2016</td>
      <td>Casual</td>
      <td>Titan Women Silver Watch</td>
    </tr>
    <tr>
      <th>3</th>
      <td>21379</td>
      <td>Men</td>
      <td>Apparel</td>
      <td>Bottomwear</td>
      <td>Track Pants</td>
      <td>Black</td>
      <td>Fall</td>
      <td>2011</td>
      <td>Casual</td>
      <td>Manchester United Men Solid Black Track Pants</td>
    </tr>
    <tr>
      <th>4</th>
      <td>53759</td>
      <td>Men</td>
      <td>Apparel</td>
      <td>Topwear</td>
      <td>Tshirts</td>
      <td>Grey</td>
      <td>Summer</td>
      <td>2012</td>
      <td>Casual</td>
      <td>Puma Men Grey T-shirt</td>
    </tr>
  </tbody>
</table>
</div>


key: _node_content
string_value:
{"id_": "e56ef2bc-a8ee-4934-b4ad-b45c2857ec9e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b5843b81-291b-40bc-b969-8753ef31e079", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "545e5dae-826e-4ade-9f6c-b30391c0a264", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "9e0660a48e3bbab420a5fef8734244693b8114f102a002d8bfb8e036f58e5410", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "821a352b-4613-4062-933a-39ea0ea10b12", "node_type": "1", "metadata": {}, "hash": "fbc02fb4f6fbee58d7cafd7b61e743d803df335815c61765236654adeb6db225", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9732, "end_char_idx": 12404, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
df["product_text"] = df.apply(lambda row: f"name {row['productDisplayName']} category {row['masterCategory']} subcategory {row['subCategory']} color {row['baseColour']} gender {row['gender']}".lower(), axis=1)
df.rename({"id":"product_id"}, inplace=True, axis=1)

df.info()

```

    <class 'pandas.core.frame.DataFrame'>
    Index: 1978 entries, 0 to 1998
    Data columns (total 11 columns):
     #   Column              Non-Null Count  Dtype 
    ---  ------              --------------  ----- 
     0   product_id          1978 non-null   int64 
     1   gender              1978 non-null   object
     2   masterCategory      1978 non-null   object
     3   subCategory         1978 non-null   object
     4   articleType         1978 non-null   object
     5   baseColour          1978 non-null   object
     6   season              1978 non-null   object
     7   year                1978 non-null   int64 
     8   usage               1978 non-null   object
     9   productDisplayName  1978 non-null   object
     10  product_text        1978 non-null   object
    dtypes: int64(2), object(9)
    memory usage: 185.4+ KB



```python
# check out one of the texts we will use to create semantic embeddings
df["product_text"][0]

```




    'name turtle check men navy blue shirt category apparel subcategory topwear color navy blue gender men'



## Connect to Redis

Now that we have our Redis database running, we can connect to it using the Redis-py client. We will use the default host and port for the Redis database which is `localhost:6379`.




```python
import redis
from redis.commands.search.indexDefinition import (
    IndexDefinition,
    IndexType
)
from redis.commands.search.query import Query
from redis.commands.search.field import (
    TagField,
    NumericField,
    TextField,
    VectorField
)

REDIS_HOST =  "localhost"
REDIS_PORT = 6379
REDIS_PASSWORD = "" # default for passwordless Redis

# Connect to Redis
redis_client = redis.Redis(
    host=REDIS_HOST,
    port=REDIS_PORT,
    password=REDIS_PASSWORD
)
redis_client.ping()

```




    True



## Creating a Search Index in Redis

The below cells will show how to specify and create a search index in Redis. We will:

1. Set some constants for defining our index like the distance metric and the index name
2. Define the index schema with RediSearch fields
3. Create the index


```python
# Constants
INDEX_NAME = "product_embeddings"           # name of the search index
PREFIX = "doc"                            # prefix for the document keys
DISTANCE_METRIC = "L2"                # distance metric for the vectors (ex. COSINE, IP, L2)
NUMBER_OF_VECTORS = len(df)

```


key: _node_content
string_value:
{"id_": "821a352b-4613-4062-933a-39ea0ea10b12", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b5843b81-291b-40bc-b969-8753ef31e079", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e56ef2bc-a8ee-4934-b4ad-b45c2857ec9e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1afbe544ed834c5a9480b442b36c737d26e2ffc587253647767748a8f67870ee", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2f99cc15-1539-4d7e-b1fa-eae44a9df176", "node_type": "1", "metadata": {}, "hash": "e9dcab3fc94bd79bc76d8419f0f6229bdb90ebe00c4b7b45ecd1a5172bdbd1c7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10875, "end_char_idx": 13801, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# check out one of the texts we will use to create semantic embeddings
df["product_text"][0]

```




    'name turtle check men navy blue shirt category apparel subcategory topwear color navy blue gender men'



## Connect to Redis

Now that we have our Redis database running, we can connect to it using the Redis-py client. We will use the default host and port for the Redis database which is `localhost:6379`.




```python
import redis
from redis.commands.search.indexDefinition import (
    IndexDefinition,
    IndexType
)
from redis.commands.search.query import Query
from redis.commands.search.field import (
    TagField,
    NumericField,
    TextField,
    VectorField
)

REDIS_HOST =  "localhost"
REDIS_PORT = 6379
REDIS_PASSWORD = "" # default for passwordless Redis

# Connect to Redis
redis_client = redis.Redis(
    host=REDIS_HOST,
    port=REDIS_PORT,
    password=REDIS_PASSWORD
)
redis_client.ping()

```




    True



## Creating a Search Index in Redis

The below cells will show how to specify and create a search index in Redis. We will:

1. Set some constants for defining our index like the distance metric and the index name
2. Define the index schema with RediSearch fields
3. Create the index


```python
# Constants
INDEX_NAME = "product_embeddings"           # name of the search index
PREFIX = "doc"                            # prefix for the document keys
DISTANCE_METRIC = "L2"                # distance metric for the vectors (ex. COSINE, IP, L2)
NUMBER_OF_VECTORS = len(df)

```


```python
# Define RediSearch fields for each of the columns in the dataset
name = TextField(name="productDisplayName")
category = TagField(name="masterCategory")
articleType = TagField(name="articleType")
gender = TagField(name="gender")
season = TagField(name="season")
year = NumericField(name="year")
text_embedding = VectorField("product_vector",
    "FLAT", {
        "TYPE": "FLOAT32",
        "DIM": 1536,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": NUMBER_OF_VECTORS,
    }
)
fields = [name, category, articleType, gender, season, year, text_embedding]

```


```python
# Check if index exists
try:
    redis_client.ft(INDEX_NAME).info()
    print("Index already exists")
except:
    # Create RediSearch Index
    redis_client.ft(INDEX_NAME).create_index(
        fields = fields,
        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)
)

```

## Generate OpenAI Embeddings and Load Documents into the Index

Now that we have a search index, we can load documents into it. We will use the dataframe containing the styles dataset loaded previously. In Redis, either the HASH or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The cells below will show how to get OpenAI embeddings for the different products and load documents into the index.


key: _node_content
string_value:
{"id_": "2f99cc15-1539-4d7e-b1fa-eae44a9df176", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b5843b81-291b-40bc-b969-8753ef31e079", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "821a352b-4613-4062-933a-39ea0ea10b12", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e8278f65a88c4513f9c039734366cf0f8c8577dbfb2f6ff4c395eeb7dc481a00", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "609bf07a-ecf3-47d9-8f04-a8dac51b515d", "node_type": "1", "metadata": {}, "hash": "0cc7cfab9ef3a6ccc1ae9dd760ea15783bb28972036b85c80f10369ce4a5c891", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12113, "end_char_idx": 15336, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Constants
INDEX_NAME = "product_embeddings"           # name of the search index
PREFIX = "doc"                            # prefix for the document keys
DISTANCE_METRIC = "L2"                # distance metric for the vectors (ex. COSINE, IP, L2)
NUMBER_OF_VECTORS = len(df)

```


```python
# Define RediSearch fields for each of the columns in the dataset
name = TextField(name="productDisplayName")
category = TagField(name="masterCategory")
articleType = TagField(name="articleType")
gender = TagField(name="gender")
season = TagField(name="season")
year = NumericField(name="year")
text_embedding = VectorField("product_vector",
    "FLAT", {
        "TYPE": "FLOAT32",
        "DIM": 1536,
        "DISTANCE_METRIC": DISTANCE_METRIC,
        "INITIAL_CAP": NUMBER_OF_VECTORS,
    }
)
fields = [name, category, articleType, gender, season, year, text_embedding]

```


```python
# Check if index exists
try:
    redis_client.ft(INDEX_NAME).info()
    print("Index already exists")
except:
    # Create RediSearch Index
    redis_client.ft(INDEX_NAME).create_index(
        fields = fields,
        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)
)

```

## Generate OpenAI Embeddings and Load Documents into the Index

Now that we have a search index, we can load documents into it. We will use the dataframe containing the styles dataset loaded previously. In Redis, either the HASH or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The cells below will show how to get OpenAI embeddings for the different products and load documents into the index.


```python
# Use OpenAI get_embeddings batch requests to speed up embedding creation
def embeddings_batch_request(documents: pd.DataFrame):
    records = documents.to_dict("records")
    print("Records to process: ", len(records))
    product_vectors = []
    docs = []
    batchsize = 1000

    for idx,doc in enumerate(records,start=1):
        # create byte vectors
        docs.append(doc["product_text"])
        if idx % batchsize == 0:
            product_vectors += get_embeddings(docs, EMBEDDING_MODEL)
            docs.clear()
            print("Vectors processed ", len(product_vectors), end='\r')
    product_vectors += get_embeddings(docs, EMBEDDING_MODEL)
    print("Vectors processed ", len(product_vectors), end='\r')
    return product_vectors

```


```python
def index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):
    product_vectors = embeddings_batch_request(documents)
    records = documents.to_dict("records")
    batchsize = 500

    # Use Redis pipelines to batch calls and save on round trip network communication
    pipe = client.pipeline()
    for idx,doc in enumerate(records,start=1):
        key = f"{prefix}:{str(doc['product_id'])}"

        # create byte vectors
        text_embedding = np.array((product_vectors[idx-1]), dtype=np.float32).tobytes()

        # replace list of floats with byte vectors
        doc["product_vector"] = text_embedding

        pipe.hset(key, mapping = doc)
        if idx % batchsize == 0:
            pipe.execute()
    pipe.execute()

```


key: _node_content
string_value:
{"id_": "609bf07a-ecf3-47d9-8f04-a8dac51b515d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b5843b81-291b-40bc-b969-8753ef31e079", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2f99cc15-1539-4d7e-b1fa-eae44a9df176", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "40bc1472f58d584aa6abb178eab99bf1c28ea2d78df3eef34313406bae03dd87", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1d5bd9c6-6df1-4b00-be90-e21cbc24efd5", "node_type": "1", "metadata": {}, "hash": "84210844c53e5d61d8ed50964d4be8ebf098a4d84e7d0e7bbbafc30037758387", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13804, "end_char_idx": 16042, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Use OpenAI get_embeddings batch requests to speed up embedding creation
def embeddings_batch_request(documents: pd.DataFrame):
    records = documents.to_dict("records")
    print("Records to process: ", len(records))
    product_vectors = []
    docs = []
    batchsize = 1000

    for idx,doc in enumerate(records,start=1):
        # create byte vectors
        docs.append(doc["product_text"])
        if idx % batchsize == 0:
            product_vectors += get_embeddings(docs, EMBEDDING_MODEL)
            docs.clear()
            print("Vectors processed ", len(product_vectors), end='\r')
    product_vectors += get_embeddings(docs, EMBEDDING_MODEL)
    print("Vectors processed ", len(product_vectors), end='\r')
    return product_vectors

```


```python
def index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):
    product_vectors = embeddings_batch_request(documents)
    records = documents.to_dict("records")
    batchsize = 500

    # Use Redis pipelines to batch calls and save on round trip network communication
    pipe = client.pipeline()
    for idx,doc in enumerate(records,start=1):
        key = f"{prefix}:{str(doc['product_id'])}"

        # create byte vectors
        text_embedding = np.array((product_vectors[idx-1]), dtype=np.float32).tobytes()

        # replace list of floats with byte vectors
        doc["product_vector"] = text_embedding

        pipe.hset(key, mapping = doc)
        if idx % batchsize == 0:
            pipe.execute()
    pipe.execute()

```


```python
%%time
index_documents(redis_client, PREFIX, df)
print(f"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}")

```

    Records to process:  1978
    Loaded 1978 documents in Redis search index with name: product_embeddings
    CPU times: user 619 ms, sys: 78.9 ms, total: 698 ms
    Wall time: 3.34 s


## Simple Vector Search Queries with OpenAI Query Embeddings

Now that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database.


key: _node_content
string_value:
{"id_": "1d5bd9c6-6df1-4b00-be90-e21cbc24efd5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b5843b81-291b-40bc-b969-8753ef31e079", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "609bf07a-ecf3-47d9-8f04-a8dac51b515d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "48f85030f130ddfd49167074cb483835c5791c1fde7766fdad30bba72e99583a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c5e7253b-9db6-4fd5-a9ce-69d90550c1bc", "node_type": "1", "metadata": {}, "hash": "427afe109f8013c071a1ccf83d173d235e01dba5add7f8ea8e5834734dbe578c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14571, "end_char_idx": 17385, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):
    product_vectors = embeddings_batch_request(documents)
    records = documents.to_dict("records")
    batchsize = 500

    # Use Redis pipelines to batch calls and save on round trip network communication
    pipe = client.pipeline()
    for idx,doc in enumerate(records,start=1):
        key = f"{prefix}:{str(doc['product_id'])}"

        # create byte vectors
        text_embedding = np.array((product_vectors[idx-1]), dtype=np.float32).tobytes()

        # replace list of floats with byte vectors
        doc["product_vector"] = text_embedding

        pipe.hset(key, mapping = doc)
        if idx % batchsize == 0:
            pipe.execute()
    pipe.execute()

```


```python
%%time
index_documents(redis_client, PREFIX, df)
print(f"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}")

```

    Records to process:  1978
    Loaded 1978 documents in Redis search index with name: product_embeddings
    CPU times: user 619 ms, sys: 78.9 ms, total: 698 ms
    Wall time: 3.34 s


## Simple Vector Search Queries with OpenAI Query Embeddings

Now that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database.


```python
def search_redis(
    redis_client: redis.Redis,
    user_query: str,
    index_name: str = "product_embeddings",
    vector_field: str = "product_vector",
    return_fields: list = ["productDisplayName", "masterCategory", "gender", "season", "year", "vector_score"],
    hybrid_fields = "*",
    k: int = 20,
    print_results: bool = True,
) -> List[dict]:

    # Use OpenAI to create embedding vector from user query
    embedded_query = openai.Embedding.create(input=user_query,
                                            model="text-embedding-3-small",
                                            )["data"][0]['embedding']

    # Prepare the Query
    base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'
    query = (
        Query(base_query)
         .return_fields(*return_fields)
         .sort_by("vector_score")
         .paging(0, k)
         .dialect(2)
    )
    params_dict = {"vector": np.array(embedded_query).astype(dtype=np.float32).tobytes()}

    # perform vector search
    results = redis_client.ft(index_name).search(query, params_dict)
    if print_results:
        for i, product in enumerate(results.docs):
            score = 1 - float(product.vector_score)
            print(f"{i}. {product.productDisplayName} (Score: {round(score ,3) })")
    return results.docs

```


key: _node_content
string_value:
{"id_": "c5e7253b-9db6-4fd5-a9ce-69d90550c1bc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b5843b81-291b-40bc-b969-8753ef31e079", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1d5bd9c6-6df1-4b00-be90-e21cbc24efd5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e5f4b246adb17cb52327fd550023c0f81321cba47858ce209a16501ba1bdd521", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4fdbb533-646c-4e84-9cbc-0d142334c8f3", "node_type": "1", "metadata": {}, "hash": "e98b9963d8996e68db99fea6328ba8f9af5d69fb5bd4690d91718a7e655aae6c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15702, "end_char_idx": 18333, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Simple Vector Search Queries with OpenAI Query Embeddings

Now that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database.


```python
def search_redis(
    redis_client: redis.Redis,
    user_query: str,
    index_name: str = "product_embeddings",
    vector_field: str = "product_vector",
    return_fields: list = ["productDisplayName", "masterCategory", "gender", "season", "year", "vector_score"],
    hybrid_fields = "*",
    k: int = 20,
    print_results: bool = True,
) -> List[dict]:

    # Use OpenAI to create embedding vector from user query
    embedded_query = openai.Embedding.create(input=user_query,
                                            model="text-embedding-3-small",
                                            )["data"][0]['embedding']

    # Prepare the Query
    base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'
    query = (
        Query(base_query)
         .return_fields(*return_fields)
         .sort_by("vector_score")
         .paging(0, k)
         .dialect(2)
    )
    params_dict = {"vector": np.array(embedded_query).astype(dtype=np.float32).tobytes()}

    # perform vector search
    results = redis_client.ft(index_name).search(query, params_dict)
    if print_results:
        for i, product in enumerate(results.docs):
            score = 1 - float(product.vector_score)
            print(f"{i}. {product.productDisplayName} (Score: {round(score ,3) })")
    return results.docs

```


```python
# Execute a simple vector search in Redis
results = search_redis(redis_client, 'man blue jeans', k=10)

```

    0. John Players Men Blue Jeans (Score: 0.791)
    1. Lee Men Tino Blue Jeans (Score: 0.775)
    2. Peter England Men Party Blue Jeans (Score: 0.763)
    3. Lee Men Blue Chicago Fit Jeans (Score: 0.761)
    4. Lee Men Blue Chicago Fit Jeans (Score: 0.761)
    5. French Connection Men Blue Jeans (Score: 0.74)
    6. Locomotive Men Washed Blue Jeans (Score: 0.739)
    7. Locomotive Men Washed Blue Jeans (Score: 0.739)
    8. Do U Speak Green Men Blue Shorts (Score: 0.736)
    9. Palm Tree Kids Boy Washed Blue Jeans (Score: 0.732)


## Hybrid Queries with Redis

The previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the example below, we will combine vector search with full text search.


key: _node_content
string_value:
{"id_": "4fdbb533-646c-4e84-9cbc-0d142334c8f3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b5843b81-291b-40bc-b969-8753ef31e079", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c5e7253b-9db6-4fd5-a9ce-69d90550c1bc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "cf746d4a575f77ef1cef83437989d7403667a80fb3c2e57ed2daca5e97bcc396", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f722227e-4dd0-4924-861b-aad660f2d925", "node_type": "1", "metadata": {}, "hash": "78139ef8cfd93f539797b1e3f18d032888c1a0f0ac35a4e06eea55ce811870b3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17388, "end_char_idx": 20144, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Execute a simple vector search in Redis
results = search_redis(redis_client, 'man blue jeans', k=10)

```

    0. John Players Men Blue Jeans (Score: 0.791)
    1. Lee Men Tino Blue Jeans (Score: 0.775)
    2. Peter England Men Party Blue Jeans (Score: 0.763)
    3. Lee Men Blue Chicago Fit Jeans (Score: 0.761)
    4. Lee Men Blue Chicago Fit Jeans (Score: 0.761)
    5. French Connection Men Blue Jeans (Score: 0.74)
    6. Locomotive Men Washed Blue Jeans (Score: 0.739)
    7. Locomotive Men Washed Blue Jeans (Score: 0.739)
    8. Do U Speak Green Men Blue Shorts (Score: 0.736)
    9. Palm Tree Kids Boy Washed Blue Jeans (Score: 0.732)


## Hybrid Queries with Redis

The previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the example below, we will combine vector search with full text search.


```python
# improve search quality by adding hybrid query for "man blue jeans" in the product vector combined with a phrase search for "blue jeans"
results = search_redis(redis_client,
                       "man blue jeans",
                       vector_field="product_vector",
                       k=10,
                       hybrid_fields='@productDisplayName:"blue jeans"'
                       )

```

    0. John Players Men Blue Jeans (Score: 0.791)
    1. Lee Men Tino Blue Jeans (Score: 0.775)
    2. Peter England Men Party Blue Jeans (Score: 0.763)
    3. French Connection Men Blue Jeans (Score: 0.74)
    4. Locomotive Men Washed Blue Jeans (Score: 0.739)
    5. Locomotive Men Washed Blue Jeans (Score: 0.739)
    6. Palm Tree Kids Boy Washed Blue Jeans (Score: 0.732)
    7. Denizen Women Blue Jeans (Score: 0.725)
    8. Jealous 21 Women Washed Blue Jeans (Score: 0.713)
    9. Jealous 21 Women Washed Blue Jeans (Score: 0.713)



```python
# hybrid query for shirt in the product vector and only include results with the phrase "slim fit" in the title
results = search_redis(redis_client,
                       "shirt",
                       vector_field="product_vector",
                       k=10,
                       hybrid_fields='@productDisplayName:"slim fit"'
                       )

```

    0. Basics Men White Slim Fit Striped Shirt (Score: 0.633)
    1. ADIDAS Men's Slim Fit White T-shirt (Score: 0.628)
    2. Basics Men Blue Slim Fit Checked Shirt (Score: 0.627)
    3. Basics Men Blue Slim Fit Checked Shirt (Score: 0.627)
    4. Basics Men Red Slim Fit Checked Shirt (Score: 0.623)
    5. Basics Men Navy Slim Fit Checked Shirt (Score: 0.613)
    6. Lee Rinse Navy Blue Slim Fit Jeans (Score: 0.558)
    7. Tokyo Talkies Women Navy Slim Fit Jeans (Score: 0.552)


key: _node_content
string_value:
{"id_": "f722227e-4dd0-4924-861b-aad660f2d925", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b5843b81-291b-40bc-b969-8753ef31e079", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4fdbb533-646c-4e84-9cbc-0d142334c8f3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c51836e9e492cc2dcab5a96243cd2ce8bbe288e2176e587c036c7d82aeb615d3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "168e7d3e-48e9-4d5c-9299-34b9f52fe13f", "node_type": "1", "metadata": {}, "hash": "6542bcbadd9014a90bbf608532b3c81be147d74bc42ba28447573347a75ab055", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19288, "end_char_idx": 21168, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# hybrid query for shirt in the product vector and only include results with the phrase "slim fit" in the title
results = search_redis(redis_client,
                       "shirt",
                       vector_field="product_vector",
                       k=10,
                       hybrid_fields='@productDisplayName:"slim fit"'
                       )

```

    0. Basics Men White Slim Fit Striped Shirt (Score: 0.633)
    1. ADIDAS Men's Slim Fit White T-shirt (Score: 0.628)
    2. Basics Men Blue Slim Fit Checked Shirt (Score: 0.627)
    3. Basics Men Blue Slim Fit Checked Shirt (Score: 0.627)
    4. Basics Men Red Slim Fit Checked Shirt (Score: 0.623)
    5. Basics Men Navy Slim Fit Checked Shirt (Score: 0.613)
    6. Lee Rinse Navy Blue Slim Fit Jeans (Score: 0.558)
    7. Tokyo Talkies Women Navy Slim Fit Jeans (Score: 0.552)



```python
# hybrid query for watch in the product vector and only include results with the tag "Accessories" in the masterCategory field
results = search_redis(redis_client,
                       "watch",
                       vector_field="product_vector",
                       k=10,
                       hybrid_fields='@masterCategory:{Accessories}'
                       )

```

    0. Titan Women Gold Watch (Score: 0.544)
    1. Being Human Men Grey Dial Blue Strap Watch (Score: 0.544)
    2. Police Men Black Dial Watch PL12170JSB (Score: 0.544)
    3. Titan Men Black Watch (Score: 0.543)
    4. Police Men Black Dial Chronograph Watch PL12777JS-02M (Score: 0.542)
    5. CASIO Youth Series Digital Men Black Small Dial Digital Watch W-210-1CVDF I065 (Score: 0.542)
    6. Titan Women Silver Watch (Score: 0.542)
    7. Police Men Black Dial Watch PL12778MSU-61 (Score: 0.541)
    8. Titan Raga Women Gold Watch (Score: 0.539)
    9. ADIDAS Original Men Black Dial Chronograph Watch ADH2641 (Score: 0.539)


key: _node_content
string_value:
{"id_": "168e7d3e-48e9-4d5c-9299-34b9f52fe13f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b5843b81-291b-40bc-b969-8753ef31e079", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f722227e-4dd0-4924-861b-aad660f2d925", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "713d92df7d088ee255358a31d994d5f66c8e34280a0486bae7f840591398c8f3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cd666e18-03d3-4299-bd7e-954c5ce35223", "node_type": "1", "metadata": {}, "hash": "02816a573c7830234a50742354c3079de7e0a8d731e89b6d8555aa2988d7a767", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20148, "end_char_idx": 22057, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# hybrid query for watch in the product vector and only include results with the tag "Accessories" in the masterCategory field
results = search_redis(redis_client,
                       "watch",
                       vector_field="product_vector",
                       k=10,
                       hybrid_fields='@masterCategory:{Accessories}'
                       )

```

    0. Titan Women Gold Watch (Score: 0.544)
    1. Being Human Men Grey Dial Blue Strap Watch (Score: 0.544)
    2. Police Men Black Dial Watch PL12170JSB (Score: 0.544)
    3. Titan Men Black Watch (Score: 0.543)
    4. Police Men Black Dial Chronograph Watch PL12777JS-02M (Score: 0.542)
    5. CASIO Youth Series Digital Men Black Small Dial Digital Watch W-210-1CVDF I065 (Score: 0.542)
    6. Titan Women Silver Watch (Score: 0.542)
    7. Police Men Black Dial Watch PL12778MSU-61 (Score: 0.541)
    8. Titan Raga Women Gold Watch (Score: 0.539)
    9. ADIDAS Original Men Black Dial Chronograph Watch ADH2641 (Score: 0.539)



```python
# hybrid query for sandals in the product vector and only include results within the 2011-2012 year range
results = search_redis(redis_client,
                       "sandals",
                       vector_field="product_vector",
                       k=10,
                       hybrid_fields='@year:[2011 2012]'
                       )

```

    0. Enroute Teens Orange Sandals (Score: 0.701)
    1. Fila Men Camper Brown Sandals (Score: 0.692)
    2. Clarks Men Black Leather Closed Sandals (Score: 0.691)
    3. Coolers Men Black Sandals (Score: 0.69)
    4. Coolers Men Black Sandals (Score: 0.69)
    5. Enroute Teens Brown Sandals (Score: 0.69)
    6. Crocs Dora Boots Pink Sandals (Score: 0.69)
    7. Enroute Men Leather Black Sandals (Score: 0.685)
    8. ADIDAS Men Navy Blue Benton Sandals (Score: 0.684)
    9. Coolers Men Black Sports Sandals (Score: 0.684)


key: _node_content
string_value:
{"id_": "cd666e18-03d3-4299-bd7e-954c5ce35223", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b5843b81-291b-40bc-b969-8753ef31e079", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "168e7d3e-48e9-4d5c-9299-34b9f52fe13f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d6d813c5fd8f9e2217fa334a0f9edc30614d28810ae15bee8a5538f4764658e8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21172, "end_char_idx": 23790, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# hybrid query for sandals in the product vector and only include results within the 2011-2012 year range
results = search_redis(redis_client,
                       "sandals",
                       vector_field="product_vector",
                       k=10,
                       hybrid_fields='@year:[2011 2012]'
                       )

```

    0. Enroute Teens Orange Sandals (Score: 0.701)
    1. Fila Men Camper Brown Sandals (Score: 0.692)
    2. Clarks Men Black Leather Closed Sandals (Score: 0.691)
    3. Coolers Men Black Sandals (Score: 0.69)
    4. Coolers Men Black Sandals (Score: 0.69)
    5. Enroute Teens Brown Sandals (Score: 0.69)
    6. Crocs Dora Boots Pink Sandals (Score: 0.69)
    7. Enroute Men Leather Black Sandals (Score: 0.685)
    8. ADIDAS Men Navy Blue Benton Sandals (Score: 0.684)
    9. Coolers Men Black Sports Sandals (Score: 0.684)



```python
# hybrid query for sandals in the product vector and only include results within the 2011-2012 year range from the summer season
results = search_redis(redis_client,
                       "blue sandals",
                       vector_field="product_vector",
                       k=10,
                       hybrid_fields='(@year:[2011 2012] @season:{Summer})'
                       )

```

    0. ADIDAS Men Navy Blue Benton Sandals (Score: 0.691)
    1. Enroute Teens Brown Sandals (Score: 0.681)
    2. ADIDAS Women's Adi Groove Blue Flip Flop (Score: 0.672)
    3. Enroute Women Turquoise Blue Flats (Score: 0.671)
    4. Red Tape Men Black Sandals (Score: 0.67)
    5. Enroute Teens Orange Sandals (Score: 0.661)
    6. Vans Men Blue Era Scilla Plaid Shoes (Score: 0.658)
    7. FILA Men Aruba Navy Blue Sandal (Score: 0.657)
    8. Quiksilver Men Blue Flip Flops (Score: 0.656)
    9. Reebok Men Navy Twist Sandals (Score: 0.656)



```python
# hybrid query for a brown belt filtering results by a year (NUMERIC) with a specific article types (TAG) and with a brand name (TEXT)
results = search_redis(redis_client,
                       "brown belt",
                       vector_field="product_vector",
                       k=10,
                       hybrid_fields='(@year:[2012 2012] @articleType:{Shirts | Belts} @productDisplayName:"Wrangler")'
                       )

```

    0. Wrangler Men Leather Brown Belt (Score: 0.67)
    1. Wrangler Women Black Belt (Score: 0.639)
    2. Wrangler Men Green Striped Shirt (Score: 0.575)
    3. Wrangler Men Purple Striped Shirt (Score: 0.549)
    4. Wrangler Men Griffith White Shirt (Score: 0.543)
    5. Wrangler Women Stella Green Shirt (Score: 0.542)


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb


key: _node_content
string_value:
{"id_": "29496040-7325-4d75-a2b4-e932a53c236e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "6d3bac1e156fc9f35a5a0494df7d73ead8a1a3d7fd0e67ee56c3acbc3bb6a6fd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5f2b8edb-023c-484d-9dbb-f6d04314afcc", "node_type": "1", "metadata": {}, "hash": "9577e2f6f7b9db4edb4ac6f9c29af58f8407f3420a92bfa98d400fc6e6a1e7eb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3175, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Redis as a Context Store with OpenAI Chat
This notebook demonstrates how to use Redis as high-speed context memory with ChatGPT.

## Prerequisites
* Redis instance with the Redis Search and Redis JSON modules
* Redis-py client lib
* OpenAI Python client lib
* OpenAI API key

## Installation
Install Python modules necessary for the examples.


```python
! pip install redis openai python-dotenv openai[datalib]
```

## OpenAI API Key
Create a .env file and add your OpenAI key to it


```python
OPENAI_API_KEY=your_key
```

## OpenAI Setup
Key load + helper function for chat completion


```python
import openai
import os
from dotenv import load_dotenv

load_dotenv()
openai.api_key = os.getenv("OPENAI_API_KEY")

def get_completion(prompt, model="gpt-3.5-turbo"):
    messages = [{"role": "user", "content": prompt}]
    response = openai.ChatCompletion.create(
        model=model,
        messages=messages,
        temperature=0, 
    )
    return response.choices[0].message["content"]
```

## Experiment - Chat Completion on a Topic outside of the Model's Knowledge Cutoff Date
Gpt-3.5-turbo was trained on data up to Sep 2021.  Let's ask it a question about something that is beyond that date.  In this case, the FTX/Sam Bankman-Fried scandal.


```python
prompt = "Is Sam Bankman-Fried's company, FTX, considered a well-managed company?"
response = get_completion(prompt)
print(response)
```

## Incomplete Information
An unfortunate behavior of these AI systems is the system will provide a confident-sounding response - even when the system is not confident with its result.  One way to mitigate this is prompt re-engineering, as seen below.


```python
prompt ="Is Sam Bankman-Fried's company, FTX, considered a well-managed company?  If you don't know for certain, say unknown."
response = get_completion(prompt)
print(response)
```

## Additional Context
Another way to combat incomplete information is to give the system more information such that it can make intelligent decisions vs guessing.  We'll use Redis as the source for that additional context.  We'll pull in business news articles from after the GPT knowledge cut-off date such that the system will have a better understanding of how FTX was actually managed. 

## Start the Redis Stack Docker container


```python
! docker compose up -d
```

## Connect Redis client


```python
from redis import from_url

REDIS_URL = 'redis://localhost:6379'
client = from_url(REDIS_URL)
client.ping()
```




    True



## Create Index
[FT.CREATE](https://redis.io/commands/ft.create/)


```python
from redis.commands.search.field import TextField, VectorField
from redis.commands.search.indexDefinition import IndexDefinition, IndexType

schema = [ VectorField('$.vector', 
            "FLAT", 
            {   "TYPE": 'FLOAT32', 
                "DIM": 1536, 
                "DISTANCE_METRIC": "COSINE"
            },  as_name='vector' ),
            TextField('$.content', as_name='content')
        ]
idx_def = IndexDefinition(index_type=IndexType.JSON, prefix=['doc:'])
try: 
    client.ft('idx').dropindex()
except:
    pass
client.ft('idx').create_index(schema, definition=idx_def)
```




    b'OK'


key: _node_content
string_value:
{"id_": "5f2b8edb-023c-484d-9dbb-f6d04314afcc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "6d3bac1e156fc9f35a5a0494df7d73ead8a1a3d7fd0e67ee56c3acbc3bb6a6fd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "29496040-7325-4d75-a2b4-e932a53c236e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "5a3b3720e58160761893cfb720af22c43f7f35ac661406fd3470cf8d8e5622dc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2e2087b1-a10c-4a0c-a002-ae1a6f514d74", "node_type": "1", "metadata": {}, "hash": "6f8b77ba77826d25bb17bac139a8a74e96a27b2ea3b399b86bb720d397d6930f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1658, "end_char_idx": 4564, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
prompt ="Is Sam Bankman-Fried's company, FTX, considered a well-managed company?  If you don't know for certain, say unknown."
response = get_completion(prompt)
print(response)
```

## Additional Context
Another way to combat incomplete information is to give the system more information such that it can make intelligent decisions vs guessing.  We'll use Redis as the source for that additional context.  We'll pull in business news articles from after the GPT knowledge cut-off date such that the system will have a better understanding of how FTX was actually managed. 

## Start the Redis Stack Docker container


```python
! docker compose up -d
```

## Connect Redis client


```python
from redis import from_url

REDIS_URL = 'redis://localhost:6379'
client = from_url(REDIS_URL)
client.ping()
```




    True



## Create Index
[FT.CREATE](https://redis.io/commands/ft.create/)


```python
from redis.commands.search.field import TextField, VectorField
from redis.commands.search.indexDefinition import IndexDefinition, IndexType

schema = [ VectorField('$.vector', 
            "FLAT", 
            {   "TYPE": 'FLOAT32', 
                "DIM": 1536, 
                "DISTANCE_METRIC": "COSINE"
            },  as_name='vector' ),
            TextField('$.content', as_name='content')
        ]
idx_def = IndexDefinition(index_type=IndexType.JSON, prefix=['doc:'])
try: 
    client.ft('idx').dropindex()
except:
    pass
client.ft('idx').create_index(schema, definition=idx_def)
```




    b'OK'



## Load Data Files into Redis as JSON Objects with Text and Vector Fields
[Redis JSON](https://redis.io/docs/stack/json/)


```python
import os
import openai

directory = './assets/'
model='text-embedding-3-small'
i = 1
for file in os.listdir(directory):
    with open(os.path.join(directory, file)) as f:
        content = f.read()
        vector = openai.Embedding.create(input = [content], model = model)['data'][0]['embedding']
        client.json().set(f'doc:{i}', '$', {'content': content, 'vector': vector})
    i += 1
```

## Embed the Question and Perform VSS to find the most relevant document
[KNN Search](https://redis.io/docs/stack/search/reference/vectors/#knn-search)


```python
from redis.commands.search.query import Query
import numpy as np

vec = np.array(openai.Embedding.create(input = [prompt], model = model)['data'][0]['embedding'], dtype=np.float32).tobytes()
q = Query('*=>[KNN 1 @vector $query_vec AS vector_score]')\
    .sort_by('vector_score')\
    .return_fields('content')\
    .dialect(2)    
params = {"query_vec": vec}

context = client.ft('idx').search(q, query_params=params).docs[0].content
print(context)
```

    Embattled Crypto Exchange FTX Files for Bankruptcy
    
    Nov. 11, 2022
    On Monday, Sam Bankman-Fried, the chief executive of the cryptocurrency exchange FTX, took to Twitter to reassure his customers: “FTX is fine,” he wrote.


key: _node_content
string_value:
{"id_": "2e2087b1-a10c-4a0c-a002-ae1a6f514d74", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "6d3bac1e156fc9f35a5a0494df7d73ead8a1a3d7fd0e67ee56c3acbc3bb6a6fd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5f2b8edb-023c-484d-9dbb-f6d04314afcc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "6451b0596e8c21f3360f5bb5f4fec5580b03019a334d4b4fb5491ed6d06a6a7e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a50ae065-0609-40ea-8610-d3df116fb916", "node_type": "1", "metadata": {}, "hash": "a70e33542e45a8ab605868decd6c71edd5f94f50b9e50309a56f0b591bcebcbd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3170, "end_char_idx": 6457, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
b'OK'



## Load Data Files into Redis as JSON Objects with Text and Vector Fields
[Redis JSON](https://redis.io/docs/stack/json/)


```python
import os
import openai

directory = './assets/'
model='text-embedding-3-small'
i = 1
for file in os.listdir(directory):
    with open(os.path.join(directory, file)) as f:
        content = f.read()
        vector = openai.Embedding.create(input = [content], model = model)['data'][0]['embedding']
        client.json().set(f'doc:{i}', '$', {'content': content, 'vector': vector})
    i += 1
```

## Embed the Question and Perform VSS to find the most relevant document
[KNN Search](https://redis.io/docs/stack/search/reference/vectors/#knn-search)


```python
from redis.commands.search.query import Query
import numpy as np

vec = np.array(openai.Embedding.create(input = [prompt], model = model)['data'][0]['embedding'], dtype=np.float32).tobytes()
q = Query('*=>[KNN 1 @vector $query_vec AS vector_score]')\
    .sort_by('vector_score')\
    .return_fields('content')\
    .dialect(2)    
params = {"query_vec": vec}

context = client.ft('idx').search(q, query_params=params).docs[0].content
print(context)
```

    Embattled Crypto Exchange FTX Files for Bankruptcy
    
    Nov. 11, 2022
    On Monday, Sam Bankman-Fried, the chief executive of the cryptocurrency exchange FTX, took to Twitter to reassure his customers: “FTX is fine,” he wrote. “Assets are fine.”
    
    On Friday, FTX announced that it was filing for bankruptcy, capping an extraordinary week of corporate drama that has upended crypto markets, sent shock waves through an industry struggling to gain mainstream credibility and sparked government investigations that could lead to more damaging revelations or even criminal charges.
    
    In a statement on Twitter, the company said that Mr. Bankman-Fried had resigned, with John J. Ray III, a corporate turnaround specialist, taking over as chief executive.
    
    The speed of FTX’s downfall has left crypto insiders stunned. Just days ago, Mr. Bankman-Fried was considered one of the smartest leaders in the crypto industry, an influential figure in Washington who was lobbying to shape regulations. And FTX was widely viewed as one of the most stable and responsible companies in the freewheeling, loosely regulated crypto industry.
    
    “Here we are, with one of the richest people in the world, his net worth dropping to zero, his business dropping to zero,” said Jared Ellias, a bankruptcy professor at Harvard Law School. “The velocity of this failure is just unbelievable.”
    
    Now, the bankruptcy has set up a rush among investors and customers to salvage funds from what remains of FTX. A surge of customers tried to withdraw funds from the platform this week, and the company couldn’t meet the demand. The exchange owes as much as $8 billion, according to people familiar with its finances.
    
    FTX’s collapse has destabilized the crypto industry, which was already reeling from a crash in the spring that drained $1 trillion from the market. The prices of the leading cryptocurrencies, Bitcoin and Ether, have plummeted. The crypto lender BlockFi, which was closely entangled with FTX, announced on Thursday that it was suspending operations as a result of FTX’s collapse.
    
    Mr.


key: _node_content
string_value:
{"id_": "a50ae065-0609-40ea-8610-d3df116fb916", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "6d3bac1e156fc9f35a5a0494df7d73ead8a1a3d7fd0e67ee56c3acbc3bb6a6fd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2e2087b1-a10c-4a0c-a002-ae1a6f514d74", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "eb7ff24ccd44940a3d01f070b8d41e100da98198fc5c46dcbfb18dc4b3b7ea42", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0735e0ed-7e0a-48f7-bc04-399ef4ea6006", "node_type": "1", "metadata": {}, "hash": "b2153be666f5197b61503fa31fcdd4b832fd34f27f3c69be0b1bde62eaba416f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4565, "end_char_idx": 8177, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
“Assets are fine.”
    
    On Friday, FTX announced that it was filing for bankruptcy, capping an extraordinary week of corporate drama that has upended crypto markets, sent shock waves through an industry struggling to gain mainstream credibility and sparked government investigations that could lead to more damaging revelations or even criminal charges.
    
    In a statement on Twitter, the company said that Mr. Bankman-Fried had resigned, with John J. Ray III, a corporate turnaround specialist, taking over as chief executive.
    
    The speed of FTX’s downfall has left crypto insiders stunned. Just days ago, Mr. Bankman-Fried was considered one of the smartest leaders in the crypto industry, an influential figure in Washington who was lobbying to shape regulations. And FTX was widely viewed as one of the most stable and responsible companies in the freewheeling, loosely regulated crypto industry.
    
    “Here we are, with one of the richest people in the world, his net worth dropping to zero, his business dropping to zero,” said Jared Ellias, a bankruptcy professor at Harvard Law School. “The velocity of this failure is just unbelievable.”
    
    Now, the bankruptcy has set up a rush among investors and customers to salvage funds from what remains of FTX. A surge of customers tried to withdraw funds from the platform this week, and the company couldn’t meet the demand. The exchange owes as much as $8 billion, according to people familiar with its finances.
    
    FTX’s collapse has destabilized the crypto industry, which was already reeling from a crash in the spring that drained $1 trillion from the market. The prices of the leading cryptocurrencies, Bitcoin and Ether, have plummeted. The crypto lender BlockFi, which was closely entangled with FTX, announced on Thursday that it was suspending operations as a result of FTX’s collapse.
    
    Mr. Bankman-Fried was backed by some of the highest-profile venture capital investors in Silicon Valley, including Sequoia Capital and Lightspeed Venture Partners. Some of those investors, facing questions about how closely they scrutinized FTX before they put money into it, have said that their nine-figure investments in the crypto exchange are now essentially worthless.
    
    The company’s demise has also set off a reckoning over risky practices that have become pervasive in crypto, an industry that was founded partly as a corrective to the type of dangerous financial engineering that caused the 2008 economic crisis.
    
    “I’m really sorry, again, that we ended up here,” Mr. Bankman-Fried said on Twitter on Friday. “Hopefully this can bring some amount of transparency, trust, and governance.”
    
    The bankruptcy filing marks the start of what will probably be months or even years of legal fallout, as lawyers try to work out whether the exchange can ever continue to operate in some form and customers demand compensation. FTX is already the target of investigations by the Securities and Exchange Commission and the Justice Department, with investigators focused on whether the company improperly used customer funds to prop up Alameda Research, a trading firm that Mr. Bankman-Fried also founded.
    
    The bankruptcy filing included FTX, its U.S. arm and Alameda. According to a bare-bones legal filing in U.S. Bankruptcy Court in Delaware, FTX has assets valued between $10 billion and $50 billion, with the size of its liabilities in the same range. The company has more than 100,000 creditors, the filing said.
    
    The bankruptcy is a stunning fall from grace for the 30-year-old Mr.


key: _node_content
string_value:
{"id_": "0735e0ed-7e0a-48f7-bc04-399ef4ea6006", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "6d3bac1e156fc9f35a5a0494df7d73ead8a1a3d7fd0e67ee56c3acbc3bb6a6fd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a50ae065-0609-40ea-8610-d3df116fb916", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c4c8b339bbc1e5b65179e10d1e4840c3827c0fd3ad3343412335d2e6fabc1d4f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5a5a074a-4e2e-4356-9fa5-f85f0c9e2e3b", "node_type": "1", "metadata": {}, "hash": "12e28056c6f8a916c4f0b7be06f8de47e4a3bcb67333d3d611275c27ff046d25", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6293, "end_char_idx": 9656, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The crypto lender BlockFi, which was closely entangled with FTX, announced on Thursday that it was suspending operations as a result of FTX’s collapse.
    
    Mr. Bankman-Fried was backed by some of the highest-profile venture capital investors in Silicon Valley, including Sequoia Capital and Lightspeed Venture Partners. Some of those investors, facing questions about how closely they scrutinized FTX before they put money into it, have said that their nine-figure investments in the crypto exchange are now essentially worthless.
    
    The company’s demise has also set off a reckoning over risky practices that have become pervasive in crypto, an industry that was founded partly as a corrective to the type of dangerous financial engineering that caused the 2008 economic crisis.
    
    “I’m really sorry, again, that we ended up here,” Mr. Bankman-Fried said on Twitter on Friday. “Hopefully this can bring some amount of transparency, trust, and governance.”
    
    The bankruptcy filing marks the start of what will probably be months or even years of legal fallout, as lawyers try to work out whether the exchange can ever continue to operate in some form and customers demand compensation. FTX is already the target of investigations by the Securities and Exchange Commission and the Justice Department, with investigators focused on whether the company improperly used customer funds to prop up Alameda Research, a trading firm that Mr. Bankman-Fried also founded.
    
    The bankruptcy filing included FTX, its U.S. arm and Alameda. According to a bare-bones legal filing in U.S. Bankruptcy Court in Delaware, FTX has assets valued between $10 billion and $50 billion, with the size of its liabilities in the same range. The company has more than 100,000 creditors, the filing said.
    
    The bankruptcy is a stunning fall from grace for the 30-year-old Mr. Bankman-Fried, who cultivated a reputation as a boy genius with a host of endearing quirks, including a habit of sleeping on a beanbag at the office. At one point, he was one of the richest people in the industry, with an estimated fortune of $24 billion. He hobnobbed with actors, professional athletes and former world leaders.
    
    Mr. Bankman-Fried’s crypto empire had an elaborate structure. The bankruptcy filing lists more than 130 corporate entities affiliated with FTX and Alameda. But as of June, FTX had only about 300 employees, a point of pride for Mr. Bankman-Fried, who said he had resisted calls from venture investors to hire more staff.
    
    “We told them additional employees added too quickly were net negative,” Mr. Bankman-Fried said on Twitter in June. “They could take it or leave it.”
    
    Unusually for a major start-up, none of FTX’s investors had seats on the board, which instead consisted of Mr. Bankman-Fried, another FTX executive and a lawyer in Antigua and Barbuda.
    
    FTX and Alameda were based in the Bahamas, where Mr. Bankman-Fried and a small circle of top executives called most of the shots and lived together in a luxury resort. Officially, Alameda was run by Caroline Ellison, a former trader for the hedge fund Jane Street, but Mr. Bankman-Fried was heavily involved, contributing to the decision-making on big trades, according to a person familiar with the matter.
    
    In addition to Mr. Bankman-Fried and Ms.


key: _node_content
string_value:
{"id_": "5a5a074a-4e2e-4356-9fa5-f85f0c9e2e3b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "6d3bac1e156fc9f35a5a0494df7d73ead8a1a3d7fd0e67ee56c3acbc3bb6a6fd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0735e0ed-7e0a-48f7-bc04-399ef4ea6006", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e2f0a939b9ab2e26d1d5aa1c4cddaf516e2c29675aedfbe82a8a1ae27427cde7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bd1752b7-7b38-4201-823f-8d1f6db8dde2", "node_type": "1", "metadata": {}, "hash": "f0785849910c8ffbd3a0e71ac91ea9380511878168d243920912b8a520b60862", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8038, "end_char_idx": 11334, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The company has more than 100,000 creditors, the filing said.
    
    The bankruptcy is a stunning fall from grace for the 30-year-old Mr. Bankman-Fried, who cultivated a reputation as a boy genius with a host of endearing quirks, including a habit of sleeping on a beanbag at the office. At one point, he was one of the richest people in the industry, with an estimated fortune of $24 billion. He hobnobbed with actors, professional athletes and former world leaders.
    
    Mr. Bankman-Fried’s crypto empire had an elaborate structure. The bankruptcy filing lists more than 130 corporate entities affiliated with FTX and Alameda. But as of June, FTX had only about 300 employees, a point of pride for Mr. Bankman-Fried, who said he had resisted calls from venture investors to hire more staff.
    
    “We told them additional employees added too quickly were net negative,” Mr. Bankman-Fried said on Twitter in June. “They could take it or leave it.”
    
    Unusually for a major start-up, none of FTX’s investors had seats on the board, which instead consisted of Mr. Bankman-Fried, another FTX executive and a lawyer in Antigua and Barbuda.
    
    FTX and Alameda were based in the Bahamas, where Mr. Bankman-Fried and a small circle of top executives called most of the shots and lived together in a luxury resort. Officially, Alameda was run by Caroline Ellison, a former trader for the hedge fund Jane Street, but Mr. Bankman-Fried was heavily involved, contributing to the decision-making on big trades, according to a person familiar with the matter.
    
    In addition to Mr. Bankman-Fried and Ms. Ellison, the circle of executives running FTX included Nishad Singh, FTX’s director of engineering, and Gary Wang, the chief technology officer. Few others had visibility into how the company was run: When the firm collapsed this week, lower-ranking employees were left confused and blindsided, according to people familiar with the matter. Mr. Singh and Ms. Ellison did not respond to requests for comment; Mr. Wang could not immediately be reached.
    
    As a crypto exchange, FTX provided a marketplace for customers to buy, sell and store a wide range of digital currencies. Most of its revenue stemmed from a risky type of trade — in which crypto investors borrowed money to make huge bets on the future prices of cryptocurrencies — that remains illegal in the United States. But Mr. Bankman-Fried also ran a smaller U.S. affiliate that offered more basic trading options.
    
    Mr. Bankman-Fried’s problems started over the weekend, when the chief executive of Binance, the largest crypto exchange, suggested publicly that FTX might be on shaky financial footing. A rush of customers tried to withdraw their crypto holdings from the platform, and FTX was unable to meet the demand.
    
    On Tuesday, Mr. Bankman-Fried said he had struck a deal to sell FTX to Binance. But after reviewing the company’s financial documents, Binance’s chief executive, Changpeng Zhao, pulled out of the agreement, leaving Mr. Bankman-Fried with limited options.
    
    In calls with investors and messages to employees this week, he apologized repeatedly and stressed that he was working hard to raise money and resolve the situation. But the hole was ultimately too big to fill.


key: _node_content
string_value:
{"id_": "bd1752b7-7b38-4201-823f-8d1f6db8dde2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "6d3bac1e156fc9f35a5a0494df7d73ead8a1a3d7fd0e67ee56c3acbc3bb6a6fd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5a5a074a-4e2e-4356-9fa5-f85f0c9e2e3b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "6ba1f9af98be74e352933606270277132c8d94db6dd4a4db2fea3e61455ee66a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "939e3568-6777-4074-ba3a-b0461627f973", "node_type": "1", "metadata": {}, "hash": "9ca998b635bf1b5ffd80e15f3d78bca5560b3ef1b0c58578bf8f7095e46337a9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9616, "end_char_idx": 13136, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In addition to Mr. Bankman-Fried and Ms. Ellison, the circle of executives running FTX included Nishad Singh, FTX’s director of engineering, and Gary Wang, the chief technology officer. Few others had visibility into how the company was run: When the firm collapsed this week, lower-ranking employees were left confused and blindsided, according to people familiar with the matter. Mr. Singh and Ms. Ellison did not respond to requests for comment; Mr. Wang could not immediately be reached.
    
    As a crypto exchange, FTX provided a marketplace for customers to buy, sell and store a wide range of digital currencies. Most of its revenue stemmed from a risky type of trade — in which crypto investors borrowed money to make huge bets on the future prices of cryptocurrencies — that remains illegal in the United States. But Mr. Bankman-Fried also ran a smaller U.S. affiliate that offered more basic trading options.
    
    Mr. Bankman-Fried’s problems started over the weekend, when the chief executive of Binance, the largest crypto exchange, suggested publicly that FTX might be on shaky financial footing. A rush of customers tried to withdraw their crypto holdings from the platform, and FTX was unable to meet the demand.
    
    On Tuesday, Mr. Bankman-Fried said he had struck a deal to sell FTX to Binance. But after reviewing the company’s financial documents, Binance’s chief executive, Changpeng Zhao, pulled out of the agreement, leaving Mr. Bankman-Fried with limited options.
    
    In calls with investors and messages to employees this week, he apologized repeatedly and stressed that he was working hard to raise money and resolve the situation. But the hole was ultimately too big to fill.
    
    FTX’s bankruptcy is the latest — and by far the biggest — in a series of bankruptcies that have shaken the crypto world this year. After a market crash in the spring, two crypto lending companies, Celsius Network and Voyager Digital, filed for bankruptcy, kicking off months of legal maneuvering over how their remaining assets should be divided. In an ironic twist, FTX had recently won an auction to buy Voyager’s remaining assets.
    
    As it enters its own bankruptcy process, FTX will be led by Mr. Ray, who has ample experience managing distressed situations. He helped manage Enron after the collapse of its business in an accounting fraud scandal in 2001. And he helped liquidate the trust of the subprime mortgage company ResCap after its 2012 bankruptcy.
    
    The bankruptcy proceedings may be only the beginning of Mr. Bankman-Fried’s legal troubles. Federal investigators are examining the relationship between FTX and Alameda, and customers are likely to file lawsuits.
    
    Mr. Bankman-Fried’s old allies have quickly abandoned him. On Thursday night, the team running the FTX Future Fund, a charitable group that Mr. Bankman-Fried bankrolled, announced that they were resigning.
    
    “We were shocked and immensely saddened to learn of the recent events at FTX,” they wrote in a statement. “We have fundamental questions about the legitimacy and integrity of the business operations that were funding the FTX Foundation and the Future Fund.”
    
    Not long ago, Mr. Bankman-Fried was performing a comedy routine onstage at a conference with Anthony Scaramucci, the former White House communications director and a business partner of FTX.
    
    “I’m disappointed,” Mr. Scaramucci said in an interview on CNBC on Friday. “Duped, I guess, is the right word.”


key: _node_content
string_value:
{"id_": "939e3568-6777-4074-ba3a-b0461627f973", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "6d3bac1e156fc9f35a5a0494df7d73ead8a1a3d7fd0e67ee56c3acbc3bb6a6fd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bd1752b7-7b38-4201-823f-8d1f6db8dde2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "88ed8b948a6ed03c7080424b9b66bb197ca47a6a79ef45dc776e710068f284a1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11290, "end_char_idx": 14107, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
But the hole was ultimately too big to fill.
    
    FTX’s bankruptcy is the latest — and by far the biggest — in a series of bankruptcies that have shaken the crypto world this year. After a market crash in the spring, two crypto lending companies, Celsius Network and Voyager Digital, filed for bankruptcy, kicking off months of legal maneuvering over how their remaining assets should be divided. In an ironic twist, FTX had recently won an auction to buy Voyager’s remaining assets.
    
    As it enters its own bankruptcy process, FTX will be led by Mr. Ray, who has ample experience managing distressed situations. He helped manage Enron after the collapse of its business in an accounting fraud scandal in 2001. And he helped liquidate the trust of the subprime mortgage company ResCap after its 2012 bankruptcy.
    
    The bankruptcy proceedings may be only the beginning of Mr. Bankman-Fried’s legal troubles. Federal investigators are examining the relationship between FTX and Alameda, and customers are likely to file lawsuits.
    
    Mr. Bankman-Fried’s old allies have quickly abandoned him. On Thursday night, the team running the FTX Future Fund, a charitable group that Mr. Bankman-Fried bankrolled, announced that they were resigning.
    
    “We were shocked and immensely saddened to learn of the recent events at FTX,” they wrote in a statement. “We have fundamental questions about the legitimacy and integrity of the business operations that were funding the FTX Foundation and the Future Fund.”
    
    Not long ago, Mr. Bankman-Fried was performing a comedy routine onstage at a conference with Anthony Scaramucci, the former White House communications director and a business partner of FTX.
    
    “I’m disappointed,” Mr. Scaramucci said in an interview on CNBC on Friday. “Duped, I guess, is the right word.”
    


## Repeat the Question to OpenAI with context
Now that we have relevant context, add that to the prompt to OpenAI and get a very different response.


```python
prompt = f"""
Using the information delimited by triple backticks, answer this question: Is Sam Bankman-Fried's company, FTX, considered a well-managed company?

Context: ```{context}```
"""

response = get_completion(prompt)
print(response)
```

    No, Sam Bankman-Fried's company FTX is not considered a well-managed company as it has filed for bankruptcy and owes as much as $8 billion to its creditors. The collapse of FTX has destabilized the crypto industry, and the company is already the target of investigations by the Securities and Exchange Commission and the Justice Department. FTX was widely viewed as one of the most stable and responsible companies in the freewheeling, loosely regulated crypto industry, but its risky practices have become pervasive in crypto, leading to a reckoning.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb


key: _node_content
string_value:
{"id_": "44253cf5-41e6-4829-83dd-469a8a7f8222", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "99729068d07d382e306777f280f8426e203ffde162e1024fe16ecf6dc8d9ed3f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d5c02822-d5bb-432f-8612-540f19631ea1", "node_type": "1", "metadata": {}, "hash": "8d2d0ca97243dde3ea8c9044ae310d031613be0ed046441b15d692edd9f715e9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 709, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Redis Vectors as JSON with OpenAI
This notebook expands on the other Redis OpenAI-cookbook examples with examples of how to use JSON with vectors.
[Storing Vectors in JSON](https://redis.io/docs/stack/search/reference/vectors/#storing-vectors-in-json)

## Prerequisites
* Redis instance with the Redis Search and Redis JSON modules
* Redis-py client lib
* OpenAI API key

## Installation
Install Python modules necessary for the examples.


```python
! pip install redis openai python-dotenv openai[datalib]
```

## OpenAI API Key
Create a .env file and add your OpenAI key to it


```python
OPENAI_API_KEY=your_key
```

## Create Text Vectors
Create embeddings (array of floats) of the news excerpts below.


key: _node_content
string_value:
{"id_": "d5c02822-d5bb-432f-8612-540f19631ea1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "99729068d07d382e306777f280f8426e203ffde162e1024fe16ecf6dc8d9ed3f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "44253cf5-41e6-4829-83dd-469a8a7f8222", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "6a8155b7745299f7a5071e046cc4daa80a18e71223de5ddda7af3c6464321136", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0c3bf501-be71-4cc7-8871-6a69598eb0fa", "node_type": "1", "metadata": {}, "hash": "b52d340c57370ee932a3f11059eb95a1125a10f80eaf12ffe964ec2d6c67af04", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3176, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Redis Vectors as JSON with OpenAI
This notebook expands on the other Redis OpenAI-cookbook examples with examples of how to use JSON with vectors.
[Storing Vectors in JSON](https://redis.io/docs/stack/search/reference/vectors/#storing-vectors-in-json)

## Prerequisites
* Redis instance with the Redis Search and Redis JSON modules
* Redis-py client lib
* OpenAI API key

## Installation
Install Python modules necessary for the examples.


```python
! pip install redis openai python-dotenv openai[datalib]
```

## OpenAI API Key
Create a .env file and add your OpenAI key to it


```python
OPENAI_API_KEY=your_key
```

## Create Text Vectors
Create embeddings (array of floats) of the news excerpts below.


```python
import openai
import os
from dotenv import load_dotenv

load_dotenv()
openai.api_key = os.getenv("OPENAI_API_KEY")

def get_vector(text, model="text-embedding-3-small"):
    text = text.replace("\n", " ")
    return openai.Embedding.create(input = [text], model = model)['data'][0]['embedding']

text_1 = """Japan narrowly escapes recession

Japan's economy teetered on the brink of a technical recession in the three months to September, figures show.

Revised figures indicated growth of just 0.1% - and a similar-sized contraction in the previous quarter. On an annual basis, the data suggests annual growth of just 0.2%, suggesting a much more hesitant recovery than had previously been thought. A common technical definition of a recession is two successive quarters of negative growth.
The government was keen to play down the worrying implications of the data. "I maintain the view that Japan's economy remains in a minor adjustment phase in an upward climb, and we will monitor developments carefully," said economy minister Heizo Takenaka. But in the face of the strengthening yen making exports less competitive and indications of weakening economic conditions ahead, observers were less sanguine. "It's painting a picture of a recovery... much patchier than previously thought," said Paul Sheard, economist at Lehman Brothers in Tokyo. Improvements in the job market apparently have yet to feed through to domestic demand, with private consumption up just 0.2% in the third quarter.
"""

text_2 = """Dibaba breaks 5,000m world record

Ethiopia's Tirunesh Dibaba set a new world record in winning the women's 5,000m at the Boston Indoor Games.

Dibaba won in 14 minutes 32.93 seconds to erase the previous world indoor mark of 14:39.29 set by another Ethiopian, Berhane Adera, in Stuttgart last year. But compatriot Kenenisa Bekele's record hopes were dashed when he miscounted his laps in the men's 3,000m and staged his sprint finish a lap too soon. Ireland's Alistair Cragg won in 7:39.89 as Bekele battled to second in 7:41.42. "I didn't want to sit back and get out-kicked," said Cragg. "So I kept on the pace. The plan was to go with 500m to go no matter what, but when Bekele made the mistake that was it. The race was mine." Sweden's Carolina Kluft, the Olympic heptathlon champion, and Slovenia's Jolanda Ceplak had winning performances, too. Kluft took the long jump at 6.63m, while Ceplak easily won the women's 800m in 2:01.52. 
"""


key: _node_content
string_value:
{"id_": "0c3bf501-be71-4cc7-8871-6a69598eb0fa", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "99729068d07d382e306777f280f8426e203ffde162e1024fe16ecf6dc8d9ed3f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d5c02822-d5bb-432f-8612-540f19631ea1", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3ee45f7ea78cfc54c622379b96f056bbfa593a639ec7bd1c44e7fc7f520ec0f8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e9a87d8a-de9c-43c0-8fa4-aba635075ca8", "node_type": "1", "metadata": {}, "hash": "8aa3fa81465b5cd8bfb62fd6cb659f9a29efc7516604b196434a77fe9e194f48", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3179, "end_char_idx": 6356, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
text_3 = """Google's toolbar sparks concern

Search engine firm Google has released a trial tool which is concerning some net users because it directs people to pre-selected commercial websites.

The AutoLink feature comes with Google's latest toolbar and provides links in a webpage to Amazon.com if it finds a book's ISBN number on the site. It also links to Google's map service, if there is an address, or to car firm Carfax, if there is a licence plate. Google said the feature, available only in the US, "adds useful links". But some users are concerned that Google's dominant position in the search engine market place could mean it would be giving a competitive edge to firms like Amazon.

AutoLink works by creating a link to a website based on information contained in a webpage - even if there is no link specified and whether or not the publisher of the page has given permission.

If a user clicks the AutoLink feature in the Google toolbar then a webpage with a book's unique ISBN number would link directly to Amazon's website. It could mean online libraries that list ISBN book numbers find they are directing users to Amazon.com whether they like it or not. Websites which have paid for advertising on their pages may also be directing people to rival services. Dan Gillmor, founder of Grassroots Media, which supports citizen-based media, said the tool was a "bad idea, and an unfortunate move by a company that is looking to continue its hypergrowth". In a statement Google said the feature was still only in beta, ie trial, stage and that the company welcomed feedback from users. It said: "The user can choose never to click on the AutoLink button, and web pages she views will never be modified. "In addition, the user can choose to disable the AutoLink feature entirely at any time."

The new tool has been compared to the Smart Tags feature from Microsoft by some users. It was widely criticised by net users and later dropped by Microsoft after concerns over trademark use were raised. Smart Tags allowed Microsoft to link any word on a web page to another site chosen by the company. Google said none of the companies which received AutoLinks had paid for the service. Some users said AutoLink would only be fair if websites had to sign up to allow the feature to work on their pages or if they received revenue for any "click through" to a commercial site. Cory Doctorow, European outreach coordinator for digital civil liberties group Electronic Fronter Foundation, said that Google should not be penalised for its market dominance. "Of course Google should be allowed to direct people to whatever proxies it chooses. "But as an end user I would want to know - 'Can I choose to use this service?, 'How much is Google being paid?', 'Can I substitute my own companies for the ones chosen by Google?'." Mr Doctorow said the only objection would be if users were forced into using AutoLink or "tricked into using the service".
"""

doc_1 = {"content": text_1, "vector": get_vector(text_1)}
doc_2 = {"content": text_2, "vector": get_vector(text_2)}
doc_3 = {"content": text_3, "vector": get_vector(text_3)}

```

## Start the Redis Stack Docker container


key: _node_content
string_value:
{"id_": "e9a87d8a-de9c-43c0-8fa4-aba635075ca8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "99729068d07d382e306777f280f8426e203ffde162e1024fe16ecf6dc8d9ed3f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0c3bf501-be71-4cc7-8871-6a69598eb0fa", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "10469fa35bff15c4642d88ed5984e3709dde32dc8b5e687b00602a88724a932c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a6635bf3-50f0-4fc8-934f-d36ffaaa55d7", "node_type": "1", "metadata": {}, "hash": "b9286c8669040e5a17b1e8e84fda5b717dfcc7b4eb9245ee5613eb8a88da4958", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6359, "end_char_idx": 8920, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
! docker compose up -d
```

    [1A[1B[0G[?25l[+] Running 0/0
     ⠿ Container redisjson-redis-1  Starting                                   [34m0.1s [0m
    [?25h[1A[1A[0G[?25l[+] Running 0/1
     ⠿ Container redisjson-redis-1  Starting                                   [34m0.2s [0m
    [?25h[1A[1A[0G[?25l[+] Running 0/1
     ⠿ Container redisjson-redis-1  Starting                                   [34m0.3s [0m
    [?25h[1A[1A[0G[?25l[+] Running 0/1
     ⠿ Container redisjson-redis-1  Starting                                   [34m0.4s [0m
    [?25h[1A[1A[0G[?25l[34m[+] Running 1/1[0m
     [32m✔[0m Container redisjson-redis-1  [32mStarted[0m                                    [34m0.4s [0m
    [?25h

## Connect Redis client


```python
from redis import from_url

REDIS_URL = 'redis://localhost:6379'
client = from_url(REDIS_URL)
client.ping()
```




    True



## Create Index
[FT.CREATE](https://redis.io/commands/ft.create/)


```python
from redis.commands.search.field import TextField, VectorField
from redis.commands.search.indexDefinition import IndexDefinition, IndexType

schema = [ VectorField('$.vector', 
            "FLAT", 
            {   "TYPE": 'FLOAT32', 
                "DIM": len(doc_1['vector']), 
                "DISTANCE_METRIC": "COSINE"
            },  as_name='vector' ),
            TextField('$.content', as_name='content')
        ]
idx_def = IndexDefinition(index_type=IndexType.JSON, prefix=['doc:'])
try: 
    client.ft('idx').dropindex()
except:
    pass
client.ft('idx').create_index(schema, definition=idx_def)
```




    b'OK'



## Load Data into Redis as JSON objects
[Redis JSON](https://redis.io/docs/stack/json/)


```python
client.json().set('doc:1', '$', doc_1)
client.json().set('doc:2', '$', doc_2)
client.json().set('doc:3', '$', doc_3)
```




    True



# Semantic Search
Given a sports-related article, search Redis via Vector Similarity Search (VSS) for similar articles.  
[KNN Search](https://redis.io/docs/stack/search/reference/vectors/#knn-search)


```python
from redis.commands.search.query import Query
import numpy as np

text_4 = """Radcliffe yet to answer GB call

Paula Radcliffe has been granted extra time to decide whether to compete in the World Cross-Country Championships.

The 31-year-old is concerned the event, which starts on 19 March in France, could upset her preparations for the London Marathon on 17 April. "There is no question that Paula would be a huge asset to the GB team," said Zara Hyde Peters of UK Athletics.


key: _node_content
string_value:
{"id_": "a6635bf3-50f0-4fc8-934f-d36ffaaa55d7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "99729068d07d382e306777f280f8426e203ffde162e1024fe16ecf6dc8d9ed3f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e9a87d8a-de9c-43c0-8fa4-aba635075ca8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8fd299fb1f0636a1758d69a5b3f4e143caac9560d8b6632210cd9ffd28a467db", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e2a738fa-afb3-4b30-8c5f-7318d745e30c", "node_type": "1", "metadata": {}, "hash": "431278fcc6f7763266c02451d5e8c6b681ddd24f0951f25d768679f2244321cb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7352, "end_char_idx": 10431, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from redis.commands.search.field import TextField, VectorField
from redis.commands.search.indexDefinition import IndexDefinition, IndexType

schema = [ VectorField('$.vector', 
            "FLAT", 
            {   "TYPE": 'FLOAT32', 
                "DIM": len(doc_1['vector']), 
                "DISTANCE_METRIC": "COSINE"
            },  as_name='vector' ),
            TextField('$.content', as_name='content')
        ]
idx_def = IndexDefinition(index_type=IndexType.JSON, prefix=['doc:'])
try: 
    client.ft('idx').dropindex()
except:
    pass
client.ft('idx').create_index(schema, definition=idx_def)
```




    b'OK'



## Load Data into Redis as JSON objects
[Redis JSON](https://redis.io/docs/stack/json/)


```python
client.json().set('doc:1', '$', doc_1)
client.json().set('doc:2', '$', doc_2)
client.json().set('doc:3', '$', doc_3)
```




    True



# Semantic Search
Given a sports-related article, search Redis via Vector Similarity Search (VSS) for similar articles.  
[KNN Search](https://redis.io/docs/stack/search/reference/vectors/#knn-search)


```python
from redis.commands.search.query import Query
import numpy as np

text_4 = """Radcliffe yet to answer GB call

Paula Radcliffe has been granted extra time to decide whether to compete in the World Cross-Country Championships.

The 31-year-old is concerned the event, which starts on 19 March in France, could upset her preparations for the London Marathon on 17 April. "There is no question that Paula would be a huge asset to the GB team," said Zara Hyde Peters of UK Athletics. "But she is working out whether she can accommodate the worlds without too much compromise in her marathon training." Radcliffe must make a decision by Tuesday - the deadline for team nominations. British team member Hayley Yelling said the team would understand if Radcliffe opted out of the event. "It would be fantastic to have Paula in the team," said the European cross-country champion. "But you have to remember that athletics is basically an individual sport and anything achieved for the team is a bonus. "She is not messing us around. We all understand the problem." Radcliffe was world cross-country champion in 2001 and 2002 but missed last year's event because of injury. In her absence, the GB team won bronze in Brussels.
"""

vec = np.array(get_vector(text_4), dtype=np.float32).tobytes()
q = Query('*=>[KNN 3 @vector $query_vec AS vector_score]')\
    .sort_by('vector_score')\
    .return_fields('vector_score', 'content')\
    .dialect(2)    
params = {"query_vec": vec}

results = client.ft('idx').search(q, query_params=params)
for doc in results.docs:
    print(f"distance:{round(float(doc['vector_score']),3)} content:{doc['content']}\n")

```

    distance:0.188 content:Dibaba breaks 5,000m world record
    
    Ethiopia's Tirunesh Dibaba set a new world record in winning the women's 5,000m at the Boston Indoor Games.
    
    Dibaba won in 14 minutes 32.93 seconds to erase the previous world indoor mark of 14:39.29 set by another Ethiopian, Berhane Adera, in Stuttgart last year.


key: _node_content
string_value:
{"id_": "e2a738fa-afb3-4b30-8c5f-7318d745e30c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "99729068d07d382e306777f280f8426e203ffde162e1024fe16ecf6dc8d9ed3f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a6635bf3-50f0-4fc8-934f-d36ffaaa55d7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "57b4cc0f16012c341723768f7c712d2abf80956c3ba593d954f49859643d3a7a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "032885cb-40fc-41b2-91aa-ad7c7dec7b53", "node_type": "1", "metadata": {}, "hash": "5559c858b990357d3a95ded10a53e69b4f160ff1ee6e0a09f415b29c5a2f59b6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8810, "end_char_idx": 11869, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"There is no question that Paula would be a huge asset to the GB team," said Zara Hyde Peters of UK Athletics. "But she is working out whether she can accommodate the worlds without too much compromise in her marathon training." Radcliffe must make a decision by Tuesday - the deadline for team nominations. British team member Hayley Yelling said the team would understand if Radcliffe opted out of the event. "It would be fantastic to have Paula in the team," said the European cross-country champion. "But you have to remember that athletics is basically an individual sport and anything achieved for the team is a bonus. "She is not messing us around. We all understand the problem." Radcliffe was world cross-country champion in 2001 and 2002 but missed last year's event because of injury. In her absence, the GB team won bronze in Brussels.
"""

vec = np.array(get_vector(text_4), dtype=np.float32).tobytes()
q = Query('*=>[KNN 3 @vector $query_vec AS vector_score]')\
    .sort_by('vector_score')\
    .return_fields('vector_score', 'content')\
    .dialect(2)    
params = {"query_vec": vec}

results = client.ft('idx').search(q, query_params=params)
for doc in results.docs:
    print(f"distance:{round(float(doc['vector_score']),3)} content:{doc['content']}\n")

```

    distance:0.188 content:Dibaba breaks 5,000m world record
    
    Ethiopia's Tirunesh Dibaba set a new world record in winning the women's 5,000m at the Boston Indoor Games.
    
    Dibaba won in 14 minutes 32.93 seconds to erase the previous world indoor mark of 14:39.29 set by another Ethiopian, Berhane Adera, in Stuttgart last year. But compatriot Kenenisa Bekele's record hopes were dashed when he miscounted his laps in the men's 3,000m and staged his sprint finish a lap too soon. Ireland's Alistair Cragg won in 7:39.89 as Bekele battled to second in 7:41.42. "I didn't want to sit back and get out-kicked," said Cragg. "So I kept on the pace. The plan was to go with 500m to go no matter what, but when Bekele made the mistake that was it. The race was mine." Sweden's Carolina Kluft, the Olympic heptathlon champion, and Slovenia's Jolanda Ceplak had winning performances, too. Kluft took the long jump at 6.63m, while Ceplak easily won the women's 800m in 2:01.52. 
    
    
    distance:0.268 content:Japan narrowly escapes recession
    
    Japan's economy teetered on the brink of a technical recession in the three months to September, figures show.
    
    Revised figures indicated growth of just 0.1% - and a similar-sized contraction in the previous quarter. On an annual basis, the data suggests annual growth of just 0.2%, suggesting a much more hesitant recovery than had previously been thought. A common technical definition of a recession is two successive quarters of negative growth.
    The government was keen to play down the worrying implications of the data. "I maintain the view that Japan's economy remains in a minor adjustment phase in an upward climb, and we will monitor developments carefully," said economy minister Heizo Takenaka.


key: _node_content
string_value:
{"id_": "032885cb-40fc-41b2-91aa-ad7c7dec7b53", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "99729068d07d382e306777f280f8426e203ffde162e1024fe16ecf6dc8d9ed3f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e2a738fa-afb3-4b30-8c5f-7318d745e30c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "bc16df6507a18d68081acc7e900d1e96e7f4abdd6d732cd2abb867f17134d11c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1154c004-1d55-4f78-8e32-9a171e316dff", "node_type": "1", "metadata": {}, "hash": "f5fd67aa2975f011b683e2f53bde92842adbc5e18bdc32524915cf8a9b43883f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10432, "end_char_idx": 13842, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
But compatriot Kenenisa Bekele's record hopes were dashed when he miscounted his laps in the men's 3,000m and staged his sprint finish a lap too soon. Ireland's Alistair Cragg won in 7:39.89 as Bekele battled to second in 7:41.42. "I didn't want to sit back and get out-kicked," said Cragg. "So I kept on the pace. The plan was to go with 500m to go no matter what, but when Bekele made the mistake that was it. The race was mine." Sweden's Carolina Kluft, the Olympic heptathlon champion, and Slovenia's Jolanda Ceplak had winning performances, too. Kluft took the long jump at 6.63m, while Ceplak easily won the women's 800m in 2:01.52. 
    
    
    distance:0.268 content:Japan narrowly escapes recession
    
    Japan's economy teetered on the brink of a technical recession in the three months to September, figures show.
    
    Revised figures indicated growth of just 0.1% - and a similar-sized contraction in the previous quarter. On an annual basis, the data suggests annual growth of just 0.2%, suggesting a much more hesitant recovery than had previously been thought. A common technical definition of a recession is two successive quarters of negative growth.
    The government was keen to play down the worrying implications of the data. "I maintain the view that Japan's economy remains in a minor adjustment phase in an upward climb, and we will monitor developments carefully," said economy minister Heizo Takenaka. But in the face of the strengthening yen making exports less competitive and indications of weakening economic conditions ahead, observers were less sanguine. "It's painting a picture of a recovery... much patchier than previously thought," said Paul Sheard, economist at Lehman Brothers in Tokyo. Improvements in the job market apparently have yet to feed through to domestic demand, with private consumption up just 0.2% in the third quarter.
    
    
    distance:0.287 content:Google's toolbar sparks concern
    
    Search engine firm Google has released a trial tool which is concerning some net users because it directs people to pre-selected commercial websites.
    
    The AutoLink feature comes with Google's latest toolbar and provides links in a webpage to Amazon.com if it finds a book's ISBN number on the site. It also links to Google's map service, if there is an address, or to car firm Carfax, if there is a licence plate. Google said the feature, available only in the US, "adds useful links". But some users are concerned that Google's dominant position in the search engine market place could mean it would be giving a competitive edge to firms like Amazon.
    
    AutoLink works by creating a link to a website based on information contained in a webpage - even if there is no link specified and whether or not the publisher of the page has given permission.
    
    If a user clicks the AutoLink feature in the Google toolbar then a webpage with a book's unique ISBN number would link directly to Amazon's website. It could mean online libraries that list ISBN book numbers find they are directing users to Amazon.com whether they like it or not. Websites which have paid for advertising on their pages may also be directing people to rival services. Dan Gillmor, founder of Grassroots Media, which supports citizen-based media, said the tool was a "bad idea, and an unfortunate move by a company that is looking to continue its hypergrowth".


key: _node_content
string_value:
{"id_": "1154c004-1d55-4f78-8e32-9a171e316dff", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "99729068d07d382e306777f280f8426e203ffde162e1024fe16ecf6dc8d9ed3f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "032885cb-40fc-41b2-91aa-ad7c7dec7b53", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d762a3230304081cc323a17581a7ad4edf344770d5366f40d6b7262f760f2d5b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9e36d66d-400a-4797-8d63-0ed1e299bc80", "node_type": "1", "metadata": {}, "hash": "59d57b77dacb9f73dc95a23572e4f5d54c8d60d40435da540189d34bac7c85e3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12029, "end_char_idx": 15330, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"It's painting a picture of a recovery... much patchier than previously thought," said Paul Sheard, economist at Lehman Brothers in Tokyo. Improvements in the job market apparently have yet to feed through to domestic demand, with private consumption up just 0.2% in the third quarter.
    
    
    distance:0.287 content:Google's toolbar sparks concern
    
    Search engine firm Google has released a trial tool which is concerning some net users because it directs people to pre-selected commercial websites.
    
    The AutoLink feature comes with Google's latest toolbar and provides links in a webpage to Amazon.com if it finds a book's ISBN number on the site. It also links to Google's map service, if there is an address, or to car firm Carfax, if there is a licence plate. Google said the feature, available only in the US, "adds useful links". But some users are concerned that Google's dominant position in the search engine market place could mean it would be giving a competitive edge to firms like Amazon.
    
    AutoLink works by creating a link to a website based on information contained in a webpage - even if there is no link specified and whether or not the publisher of the page has given permission.
    
    If a user clicks the AutoLink feature in the Google toolbar then a webpage with a book's unique ISBN number would link directly to Amazon's website. It could mean online libraries that list ISBN book numbers find they are directing users to Amazon.com whether they like it or not. Websites which have paid for advertising on their pages may also be directing people to rival services. Dan Gillmor, founder of Grassroots Media, which supports citizen-based media, said the tool was a "bad idea, and an unfortunate move by a company that is looking to continue its hypergrowth". In a statement Google said the feature was still only in beta, ie trial, stage and that the company welcomed feedback from users. It said: "The user can choose never to click on the AutoLink button, and web pages she views will never be modified. "In addition, the user can choose to disable the AutoLink feature entirely at any time."
    
    The new tool has been compared to the Smart Tags feature from Microsoft by some users. It was widely criticised by net users and later dropped by Microsoft after concerns over trademark use were raised. Smart Tags allowed Microsoft to link any word on a web page to another site chosen by the company. Google said none of the companies which received AutoLinks had paid for the service. Some users said AutoLink would only be fair if websites had to sign up to allow the feature to work on their pages or if they received revenue for any "click through" to a commercial site. Cory Doctorow, European outreach coordinator for digital civil liberties group Electronic Fronter Foundation, said that Google should not be penalised for its market dominance. "Of course Google should be allowed to direct people to whatever proxies it chooses. "But as an end user I would want to know - 'Can I choose to use this service?, 'How much is Google being paid?', 'Can I substitute my own companies for the ones chosen by Google?'." Mr Doctorow said the only objection would be if users were forced into using AutoLink or "tricked into using the service".


key: _node_content
string_value:
{"id_": "9e36d66d-400a-4797-8d63-0ed1e299bc80", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "99729068d07d382e306777f280f8426e203ffde162e1024fe16ecf6dc8d9ed3f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1154c004-1d55-4f78-8e32-9a171e316dff", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "28fb8baf1dfe08c713722cec1e8fa0ff5d6eb2746540db5b9990e7cd8471db95", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7a20b5e1-dd32-4e82-83b6-545f9bb9b0d4", "node_type": "1", "metadata": {}, "hash": "68b8a1c2f77389e0808b1e31504826bc9502b2ddd3dda8171adee4c77a9d1d76", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13547, "end_char_idx": 15715, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Websites which have paid for advertising on their pages may also be directing people to rival services. Dan Gillmor, founder of Grassroots Media, which supports citizen-based media, said the tool was a "bad idea, and an unfortunate move by a company that is looking to continue its hypergrowth". In a statement Google said the feature was still only in beta, ie trial, stage and that the company welcomed feedback from users. It said: "The user can choose never to click on the AutoLink button, and web pages she views will never be modified. "In addition, the user can choose to disable the AutoLink feature entirely at any time."
    
    The new tool has been compared to the Smart Tags feature from Microsoft by some users. It was widely criticised by net users and later dropped by Microsoft after concerns over trademark use were raised. Smart Tags allowed Microsoft to link any word on a web page to another site chosen by the company. Google said none of the companies which received AutoLinks had paid for the service. Some users said AutoLink would only be fair if websites had to sign up to allow the feature to work on their pages or if they received revenue for any "click through" to a commercial site. Cory Doctorow, European outreach coordinator for digital civil liberties group Electronic Fronter Foundation, said that Google should not be penalised for its market dominance. "Of course Google should be allowed to direct people to whatever proxies it chooses. "But as an end user I would want to know - 'Can I choose to use this service?, 'How much is Google being paid?', 'Can I substitute my own companies for the ones chosen by Google?'." Mr Doctorow said the only objection would be if users were forced into using AutoLink or "tricked into using the service".
    
    


## Hybrid Search
Use a combination of full text search and VSS to find a matching article.  For this scenario, we filter on a full text search of the term 'recession' and then find the KNN articles.  In this case, business-related.  Reminder document #1 was about a recession in Japan.
[Hybrid Queries](https://redis.io/docs/stack/search/reference/vectors/#hybrid-queries)


key: _node_content
string_value:
{"id_": "7a20b5e1-dd32-4e82-83b6-545f9bb9b0d4", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "99729068d07d382e306777f280f8426e203ffde162e1024fe16ecf6dc8d9ed3f", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9e36d66d-400a-4797-8d63-0ed1e299bc80", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0fc8f9c94e7ba5b44c6544dc9a67341e0bd3f820e1dcb615ed2a27e7470d700d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13973, "end_char_idx": 18915, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It said: "The user can choose never to click on the AutoLink button, and web pages she views will never be modified. "In addition, the user can choose to disable the AutoLink feature entirely at any time."
    
    The new tool has been compared to the Smart Tags feature from Microsoft by some users. It was widely criticised by net users and later dropped by Microsoft after concerns over trademark use were raised. Smart Tags allowed Microsoft to link any word on a web page to another site chosen by the company. Google said none of the companies which received AutoLinks had paid for the service. Some users said AutoLink would only be fair if websites had to sign up to allow the feature to work on their pages or if they received revenue for any "click through" to a commercial site. Cory Doctorow, European outreach coordinator for digital civil liberties group Electronic Fronter Foundation, said that Google should not be penalised for its market dominance. "Of course Google should be allowed to direct people to whatever proxies it chooses. "But as an end user I would want to know - 'Can I choose to use this service?, 'How much is Google being paid?', 'Can I substitute my own companies for the ones chosen by Google?'." Mr Doctorow said the only objection would be if users were forced into using AutoLink or "tricked into using the service".
    
    


## Hybrid Search
Use a combination of full text search and VSS to find a matching article.  For this scenario, we filter on a full text search of the term 'recession' and then find the KNN articles.  In this case, business-related.  Reminder document #1 was about a recession in Japan.
[Hybrid Queries](https://redis.io/docs/stack/search/reference/vectors/#hybrid-queries)


```python
text_5 = """Ethiopia's crop production up 24%

Ethiopia produced 14.27 million tonnes of crops in 2004, 24% higher than in 2003 and 21% more than the average of the past five years, a report says.

In 2003, crop production totalled 11.49 million tonnes, the joint report from the Food and Agriculture Organisation and the World Food Programme said. Good rains, increased use of fertilizers and improved seeds contributed to the rise in production. Nevertheless, 2.2 million Ethiopians will still need emergency assistance.

The report calculated emergency food requirements for 2005 to be 387,500 tonnes. On top of that, 89,000 tonnes of fortified blended food and vegetable oil for "targeted supplementary food distributions for a survival programme for children under five and pregnant and lactating women" will be needed.

In eastern and southern Ethiopia, a prolonged drought has killed crops and drained wells. Last year, a total of 965,000 tonnes of food assistance was needed to help seven million Ethiopians. The Food and Agriculture Organisation (FAO) recommend that the food assistance is bought locally. "Local purchase of cereals for food assistance programmes is recommended as far as possible, so as to assist domestic markets and farmers," said Henri Josserand, chief of FAO's Global Information and Early Warning System. Agriculture is the main economic activity in Ethiopia, representing 45% of gross domestic product. About 80% of Ethiopians depend directly or indirectly on agriculture.
"""

vec = np.array(get_vector(text_5), dtype=np.float32).tobytes()
q = Query('@content:recession => [KNN 3 @vector $query_vec AS vector_score]')\
    .sort_by('vector_score')\
    .return_fields('vector_score', 'content')\
    .dialect(2)    
params = {"query_vec": vec}

results = client.ft('idx').search(q, query_params=params)
for doc in results.docs:
    print(f"distance:{round(float(doc['vector_score']),3)} content:{doc['content']}\n")
```

    distance:0.241 content:Japan narrowly escapes recession
    
    Japan's economy teetered on the brink of a technical recession in the three months to September, figures show.
    
    Revised figures indicated growth of just 0.1% - and a similar-sized contraction in the previous quarter. On an annual basis, the data suggests annual growth of just 0.2%, suggesting a much more hesitant recovery than had previously been thought. A common technical definition of a recession is two successive quarters of negative growth.
    The government was keen to play down the worrying implications of the data. "I maintain the view that Japan's economy remains in a minor adjustment phase in an upward climb, and we will monitor developments carefully," said economy minister Heizo Takenaka. But in the face of the strengthening yen making exports less competitive and indications of weakening economic conditions ahead, observers were less sanguine. "It's painting a picture of a recovery... much patchier than previously thought," said Paul Sheard, economist at Lehman Brothers in Tokyo. Improvements in the job market apparently have yet to feed through to domestic demand, with private consumption up just 0.2% in the third quarter.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb


key: _node_content
string_value:
{"id_": "35b72222-942c-43bc-9b4d-3ce76aca6c90", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6acee902-5735-4d36-a385-6c034da1da8f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e5a8951b-460f-4ad3-9a97-7e1bfdcac648", "node_type": "1", "metadata": {}, "hash": "2a91c59f48d3a558669bf051ea0cccd23a42c6f47c8e1eb52f8a859f4aaef26f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3594, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Using Typesense for Embeddings Search

This notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.

### What is a Vector Database

A vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.

### Why use a Vector Database

Vector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.


### Demo Flow
The demo flow is:
- **Setup**: Import packages and set any required variables
- **Load data**: Load a dataset and embed it using OpenAI embeddings
- **Typesense**
    - *Setup*: Set up the Typesense Python client. For more details go [here](https://typesense.org/docs/0.24.0/api/)
    - *Index Data*: We'll create a collection and index it for both __titles__ and __content__.
    - *Search Data*: Run a few example queries with various goals in mind.

Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.

## Setup

Import the required libraries and set the embedding model that we'd like to use.


```python
# We'll need to install the Typesense client
!pip install typesense

#Install wget to pull zip file
!pip install wget
```


```python
import openai

from typing import List, Iterator
import pandas as pd
import numpy as np
import os
import wget
from ast import literal_eval

# Typesense's client library for Python
import typesense

# I've set this to our new embeddings model, this can be changed to the embedding model of your choice
EMBEDDING_MODEL = "text-embedding-3-small"

# Ignore unclosed SSL socket warnings - optional in case you get these errors
import warnings

warnings.filterwarnings(action="ignore", message="unclosed", category=ResourceWarning)
warnings.filterwarnings("ignore", category=DeprecationWarning) 
```

## Load data

In this section we'll load embedded data that we've prepared previous to this session.


```python
embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```


```python
import zipfile
with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip","r") as zip_ref:
    zip_ref.extractall("../data")
```


```python
article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')
```


key: _node_content
string_value:
{"id_": "e5a8951b-460f-4ad3-9a97-7e1bfdcac648", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6acee902-5735-4d36-a385-6c034da1da8f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "35b72222-942c-43bc-9b4d-3ce76aca6c90", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1e7a8fe70fec5033f5fb1fa17044acd396515dbfbfd25c4bbb8cf7dc94ee28cf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f0f769bb-6346-4d09-91dc-d41bbdefb70a", "node_type": "1", "metadata": {}, "hash": "55d75d734c6c465942a4bbe99c9e2550ca1dde90d5af183d906f3fc23ae149c3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2296, "end_char_idx": 4644, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# We'll need to install the Typesense client
!pip install typesense

#Install wget to pull zip file
!pip install wget
```


```python
import openai

from typing import List, Iterator
import pandas as pd
import numpy as np
import os
import wget
from ast import literal_eval

# Typesense's client library for Python
import typesense

# I've set this to our new embeddings model, this can be changed to the embedding model of your choice
EMBEDDING_MODEL = "text-embedding-3-small"

# Ignore unclosed SSL socket warnings - optional in case you get these errors
import warnings

warnings.filterwarnings(action="ignore", message="unclosed", category=ResourceWarning)
warnings.filterwarnings("ignore", category=DeprecationWarning) 
```

## Load data

In this section we'll load embedded data that we've prepared previous to this session.


```python
embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```


```python
import zipfile
with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip","r") as zip_ref:
    zip_ref.extractall("../data")
```


```python
article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')
```


```python
article_df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>url</th>
      <th>title</th>
      <th>text</th>
      <th>title_vector</th>
      <th>content_vector</th>
      <th>vector_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>https://simple.wikipedia.org/wiki/April</td>
      <td>April</td>
      <td>April is the fourth month of the year in the J...</td>
      <td>[0.001009464613161981, -0.020700545981526375, ...</td>
      <td>[-0.011253940872848034, -0.013491976074874401,...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>https://simple.wikipedia.org/wiki/August</td>
      <td>August</td>
      <td>August (Aug.)


key: _node_content
string_value:
{"id_": "f0f769bb-6346-4d09-91dc-d41bbdefb70a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6acee902-5735-4d36-a385-6c034da1da8f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e5a8951b-460f-4ad3-9a97-7e1bfdcac648", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7cbf504b5e1f5bcc109b5f23a7a06b52775e22eeb1e548ca9bf07e707252fa46", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c2fdce29-3320-4d9d-9803-e60cf3c3a12e", "node_type": "1", "metadata": {}, "hash": "0eb9f1fbcad7c87e0ae1cc80d55f22c888212d66f0901dbcd88725f848899c3c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3497, "end_char_idx": 5675, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: _node_content
string_value:
{"id_": "c2fdce29-3320-4d9d-9803-e60cf3c3a12e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6acee902-5735-4d36-a385-6c034da1da8f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f0f769bb-6346-4d09-91dc-d41bbdefb70a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3f9e1e2d7ec1d3927273924fe76b631250a45879929cf8e38a3c61766399c020", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8e0166d9-5243-4942-882e-6854a936ba4a", "node_type": "1", "metadata": {}, "hash": "3f7b4d59d71a4aeb15f37ff473c5aaf54e96b32c0390919255591521f059a058", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4645, "end_char_idx": 6843, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: _node_content
string_value:
{"id_": "8e0166d9-5243-4942-882e-6854a936ba4a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6acee902-5735-4d36-a385-6c034da1da8f", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c2fdce29-3320-4d9d-9803-e60cf3c3a12e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3df6f49c2f9b9707d27ab66c4db8b12984adf9523c41930532679c0bb71d7707", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "59c07658-6371-4227-8a1d-d8f23330c7eb", "node_type": "1", "metadata": {}, "hash": "084fafb325520eb33296139dcc48c0723a8178600ea83911ab4b1b154ceeaa1f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5676, "end_char_idx": 8300, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Air is a...</td>
      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>
      <td>[0.021524671465158463, 0.018522677943110466, -...</td>
      <td>4</td>
    </tr>
  </tbody>
</table>
</div>




```python
# Read vectors from strings back into a list
article_df['title_vector'] = article_df.title_vector.apply(literal_eval)
article_df['content_vector'] = article_df.content_vector.apply(literal_eval)

# Set vector_id to be a string
article_df['vector_id'] = article_df['vector_id'].apply(str)
```


```python
article_df.info(show_counts=True)
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 25000 entries, 0 to 24999
    Data columns (total 7 columns):
     #   Column          Non-Null Count  Dtype 
    ---  ------          --------------  ----- 
     0   id              25000 non-null  int64 
     1   url             25000 non-null  object
     2   title           25000 non-null  object
     3   text            25000 non-null  object
     4   title_vector    25000 non-null  object
     5   content_vector  25000 non-null  object
     6   vector_id       25000 non-null  object
    dtypes: int64(1), object(6)
    memory usage: 1.3+ MB


## Typesense

The next vector store we'll look at is [Typesense](https://typesense.org/), which is an open source, in-memory search engine, that you can either self-host or run on [Typesense Cloud](https://cloud.typesense.org).

Typesense focuses on performance by storing the entire index in RAM (with a backup on disk) and also focuses on providing an out-of-the-box developer experience by simplifying available options and setting good defaults. It also lets you combine attribute-based filtering together with vector queries.

For this example, we will set up a local docker-based Typesense server, index our vectors in Typesense and then do some nearest-neighbor search queries. If you use Typesense Cloud, you can skip the docker setup part and just obtain the hostname and API keys from your cluster dashboard.

### Setup

To run Typesense locally, you'll need [Docker](https://www.docker.com/). Following the instructions contained in the Typesense documentation [here](https://typesense.org/docs/guide/install-typesense.html#docker-compose), we created an example docker-compose.yml file in this repo saved at [./typesense/docker-compose.yml](./typesense/docker-compose.yml).

After starting Docker, you can start Typesense locally by navigating to the `examples/vector_databases/typesense/` directory and running `docker-compose up -d`.

The default API key is set to `xyz` in the Docker compose file, and the default Typesense port to `8108`.


