# embedding_metadata

key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb


key: _node_content
string_value:
{"id_": "c492d33d-7fe9-4ee6-9924-6d770bf81bf0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "60bcb85d-5e6e-4ab0-a9fc-5187130928dd", "node_type": "1", "metadata": {}, "hash": "ea33eec6ce7fe9964cdec82a5a01d5024ec7994955bd2f2cb55509a0cc3dd62d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3520, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Using Weaviate for Embeddings Search

This notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.

### What is a Vector Database

A vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.

### Why use a Vector Database

Vector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.


### Demo Flow
The demo flow is:
- **Setup**: Import packages and set any required variables
- **Load data**: Load a dataset and embed it using OpenAI embeddings
- **Weaviate**
    - *Setup*: Here we'll set up the Python client for Weaviate. For more details go [here](https://weaviate.io/developers/weaviate/current/client-libraries/python.html)
    - *Index Data*: We'll create an index with __title__ search vectors in it
    - *Search Data*: We'll run a few searches to confirm it works

Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.

## Setup

Import the required libraries and set the embedding model that we'd like to use.


```python
# We'll need to install the Weaviate client
!pip install weaviate-client

#Install wget to pull zip file
!pip install wget
```


```python
import openai

from typing import List, Iterator
import pandas as pd
import numpy as np
import os
import wget
from ast import literal_eval

# Weaviate's client library for Python
import weaviate

# I've set this to our new embeddings model, this can be changed to the embedding model of your choice
EMBEDDING_MODEL = "text-embedding-3-small"

# Ignore unclosed SSL socket warnings - optional in case you get these errors
import warnings

warnings.filterwarnings(action="ignore", message="unclosed", category=ResourceWarning)
warnings.filterwarnings("ignore", category=DeprecationWarning) 
```

## Load data

In this section we'll load embedded data that we've prepared previous to this session.


```python
embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```


```python
import zipfile
with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip","r") as zip_ref:
    zip_ref.extractall("../data")
```


key: _node_content
string_value:
{"id_": "60bcb85d-5e6e-4ab0-a9fc-5187130928dd", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c492d33d-7fe9-4ee6-9924-6d770bf81bf0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "c55639b4894da83201da04224277c0e2b3d41084bb7df908bfe8894b45c129de", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d1e3c7cd-30c3-402d-a9c5-b8bbb99209ec", "node_type": "1", "metadata": {}, "hash": "55d75d734c6c465942a4bbe99c9e2550ca1dde90d5af183d906f3fc23ae149c3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2319, "end_char_idx": 4670, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# We'll need to install the Weaviate client
!pip install weaviate-client

#Install wget to pull zip file
!pip install wget
```


```python
import openai

from typing import List, Iterator
import pandas as pd
import numpy as np
import os
import wget
from ast import literal_eval

# Weaviate's client library for Python
import weaviate

# I've set this to our new embeddings model, this can be changed to the embedding model of your choice
EMBEDDING_MODEL = "text-embedding-3-small"

# Ignore unclosed SSL socket warnings - optional in case you get these errors
import warnings

warnings.filterwarnings(action="ignore", message="unclosed", category=ResourceWarning)
warnings.filterwarnings("ignore", category=DeprecationWarning) 
```

## Load data

In this section we'll load embedded data that we've prepared previous to this session.


```python
embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```


```python
import zipfile
with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip","r") as zip_ref:
    zip_ref.extractall("../data")
```


```python
article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')
```


```python
article_df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>url</th>
      <th>title</th>
      <th>text</th>
      <th>title_vector</th>
      <th>content_vector</th>
      <th>vector_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>https://simple.wikipedia.org/wiki/April</td>
      <td>April</td>
      <td>April is the fourth month of the year in the J...</td>
      <td>[0.001009464613161981, -0.020700545981526375, ...</td>
      <td>[-0.011253940872848034, -0.013491976074874401,...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>https://simple.wikipedia.org/wiki/August</td>
      <td>August</td>
      <td>August (Aug.)


key: _node_content
string_value:
{"id_": "d1e3c7cd-30c3-402d-a9c5-b8bbb99209ec", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "60bcb85d-5e6e-4ab0-a9fc-5187130928dd", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "e52ab9730fc78a7c173e9f5cc18540aaa7e7c8a7178a7f4071ad2c19a019dbd5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1db9ed07-ec32-4a10-a0d9-b645f4631e1b", "node_type": "1", "metadata": {}, "hash": "0eb9f1fbcad7c87e0ae1cc80d55f22c888212d66f0901dbcd88725f848899c3c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3523, "end_char_idx": 5701, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: _node_content
string_value:
{"id_": "1db9ed07-ec32-4a10-a0d9-b645f4631e1b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d1e3c7cd-30c3-402d-a9c5-b8bbb99209ec", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "6e7f6ae44a256c97413473634539eeb442908ff2e22cf217d14a8da3e65baee4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "83fcc0ee-9cc4-4d03-9f26-bda3c77705c1", "node_type": "1", "metadata": {}, "hash": "56e2aa1bde5ed15fe8073faa23455e10c4418bcfff784ccff7bb57b05c25c908", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4671, "end_char_idx": 6869, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: _node_content
string_value:
{"id_": "83fcc0ee-9cc4-4d03-9f26-bda3c77705c1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1db9ed07-ec32-4a10-a0d9-b645f4631e1b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f120b37e3b2b0aa4b98e8e3c26ef91952e7bdbdc599479b8d831bafefaa73cdb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2f25fbd0-5d9b-4b0d-915c-8f4cde303639", "node_type": "1", "metadata": {}, "hash": "686d4ea9bbc87de3b274fe0ef6b95b9180e4ab82f9db022f1d6861c6c8def44d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5702, "end_char_idx": 7367, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Air is a...</td>
      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>
      <td>[0.021524671465158463, 0.018522677943110466, -...</td>
      <td>4</td>
    </tr>
  </tbody>
</table>
</div>




```python
# Read vectors from strings back into a list
article_df['title_vector'] = article_df.title_vector.apply(literal_eval)
article_df['content_vector'] = article_df.content_vector.apply(literal_eval)

# Set vector_id to be a string
article_df['vector_id'] = article_df['vector_id'].apply(str)
```


```python
article_df.info(show_counts=True)
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 25000 entries, 0 to 24999
    Data columns (total 7 columns):
     #   Column          Non-Null Count  Dtype 
    ---  ------          --------------  ----- 
     0   id              25000 non-null  int64 
     1   url             25000 non-null  object
     2   title           25000 non-null  object
     3   text            25000 non-null  object
     4   title_vector    25000 non-null  object
     5   content_vector  25000 non-null  object
     6   vector_id       25000 non-null  object
    dtypes: int64(1), object(6)
    memory usage: 1.3+ MB


## Weaviate

Another vector database option we'll explore is **Weaviate**, which offers both a managed, [SaaS](https://console.weaviate.io/) option, as well as a self-hosted [open source](https://github.com/weaviate/weaviate) option. As we've already looked at a cloud vector database, we'll try the self-hosted option here.

For this we will:
- Set up a local deployment of Weaviate
- Create indices in Weaviate
- Store our data there
- Fire some similarity search queries
- Try a real use case


key: _node_content
string_value:
{"id_": "2f25fbd0-5d9b-4b0d-915c-8f4cde303639", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "83fcc0ee-9cc4-4d03-9f26-bda3c77705c1", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "a11644a5aa3252312c197afde75ef80ddd957da6aec0ccc5c714d1cbb9311bda", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7ecd2798-5ea5-4e6e-a7c0-3eda213ec0b0", "node_type": "1", "metadata": {}, "hash": "90fecac5f3411c5c078a789fd288daf8432efa3aa262124fce90eb352c3ffb0b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5907, "end_char_idx": 9346, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Read vectors from strings back into a list
article_df['title_vector'] = article_df.title_vector.apply(literal_eval)
article_df['content_vector'] = article_df.content_vector.apply(literal_eval)

# Set vector_id to be a string
article_df['vector_id'] = article_df['vector_id'].apply(str)
```


```python
article_df.info(show_counts=True)
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 25000 entries, 0 to 24999
    Data columns (total 7 columns):
     #   Column          Non-Null Count  Dtype 
    ---  ------          --------------  ----- 
     0   id              25000 non-null  int64 
     1   url             25000 non-null  object
     2   title           25000 non-null  object
     3   text            25000 non-null  object
     4   title_vector    25000 non-null  object
     5   content_vector  25000 non-null  object
     6   vector_id       25000 non-null  object
    dtypes: int64(1), object(6)
    memory usage: 1.3+ MB


## Weaviate

Another vector database option we'll explore is **Weaviate**, which offers both a managed, [SaaS](https://console.weaviate.io/) option, as well as a self-hosted [open source](https://github.com/weaviate/weaviate) option. As we've already looked at a cloud vector database, we'll try the self-hosted option here.

For this we will:
- Set up a local deployment of Weaviate
- Create indices in Weaviate
- Store our data there
- Fire some similarity search queries
- Try a real use case


### Bring your own vectors approach
In this cookbook, we provide the data with already generated vectors. This is a good approach for scenarios, where your data is already vectorized.

### Automated vectorization with OpenAI module
For scenarios, where your data is not vectorized yet, you can delegate the vectorization task with OpenAI to Weaviate.
Weaviate offers a built-in module [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai), which takes care of the vectorization for you at:
* import
* for any CRUD operations
* for semantic search

Check out the [Getting Started with Weaviate and OpenAI module cookbook](./weaviate/getting-started-with-weaviate-and-openai.ipynb) to learn step by step how to import and vectorize data in one step.

### Setup

To run Weaviate locally, you'll need [Docker](https://www.docker.com/). Following the instructions contained in the Weaviate documentation [here](https://weaviate.io/developers/weaviate/installation/docker-compose), we created an example docker-compose.yml file in this repo saved at [./weaviate/docker-compose.yml](./weaviate/docker-compose.yml).

After starting Docker, you can start Weaviate locally by navigating to the `examples/vector_databases/weaviate/` directory and running `docker-compose up -d`.

#### SaaS
Alternatively you can use [Weaviate Cloud Service](https://console.weaviate.io/) (WCS) to create a free Weaviate cluster.
1. create a free account and/or login to [WCS](https://console.weaviate.io/)
2. create a `Weaviate Cluster` with the following settings:
    * Sandbox: `Sandbox Free`
    * Weaviate Version: Use default (latest)
    * OIDC Authentication: `Disabled`
3. your instance should be ready in a minute or two
4. make a note of the `Cluster Id`. The link will take you to the full path of your cluster (you will need it later to connect to it). It should be something like: `https://your-project-name-suffix.weaviate.network`


key: _node_content
string_value:
{"id_": "7ecd2798-5ea5-4e6e-a7c0-3eda213ec0b0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2f25fbd0-5d9b-4b0d-915c-8f4cde303639", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f779f991bb87dd122ed6185541715327cdd4094a4493ac98bc1847bec7429871", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dd540420-1137-471b-a2d0-352fc76a6f73", "node_type": "1", "metadata": {}, "hash": "982b6e72b18ea116823a5652081d817fc7c69f77f70b1f9af7be82eb4780ea38", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9350, "end_char_idx": 11140, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Option #1 - Self-hosted - Weaviate Open Source 
client = weaviate.Client(
    url="http://localhost:8080",
    additional_headers={
        "X-OpenAI-Api-Key": os.getenv("OPENAI_API_KEY")
    }
)
```


```python
# Option #2 - SaaS - (Weaviate Cloud Service)
client = weaviate.Client(
    url="https://your-wcs-instance-name.weaviate.network",
    additional_headers={
        "X-OpenAI-Api-Key": os.getenv("OPENAI_API_KEY")
    }
)
```


```python
client.is_ready()
```

### Index data

In Weaviate you create __schemas__ to capture each of the entities you will be searching. 

In this case we'll create a schema called **Article** with the **title** vector from above included for us to search by.

The next few steps closely follow the documentation Weaviate provides [here](https://weaviate.io/developers/weaviate/quickstart).



```python
# Clear up the schema, so that we can recreate it
client.schema.delete_all()
client.schema.get()

# Define the Schema object to use `text-embedding-3-small` on `title` and `content`, but skip it for `url`
article_schema = {
    "class": "Article",
    "description": "A collection of articles",
    "vectorizer": "text2vec-openai",
    "moduleConfig": {
        "text2vec-openai": {
          "model": "ada",
          "modelVersion": "002",
          "type": "text"
        }
    },
    "properties": [{
        "name": "title",
        "description": "Title of the article",
        "dataType": ["string"]
    },
    {
        "name": "content",
        "description": "Contents of the article",
        "dataType": ["text"],
        "moduleConfig": { "text2vec-openai": { "skip": True } }
    }]
}

# add the Article schema
client.schema.create_class(article_schema)

# get the schema to make sure it worked
client.schema.get()
```


key: _node_content
string_value:
{"id_": "dd540420-1137-471b-a2d0-352fc76a6f73", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7ecd2798-5ea5-4e6e-a7c0-3eda213ec0b0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "9f45cae6bbc7e759a2bc076005dfe583d9c22ddc2f2ba7a5e03e010806085aaa", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0147c603-7d13-4716-ae65-94af41b4ec2d", "node_type": "1", "metadata": {}, "hash": "e6ee35f80c0a64480f769fb7c0d160afc6cd88a0f192af479dec9b9d33e0cd37", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9564, "end_char_idx": 12819, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Option #2 - SaaS - (Weaviate Cloud Service)
client = weaviate.Client(
    url="https://your-wcs-instance-name.weaviate.network",
    additional_headers={
        "X-OpenAI-Api-Key": os.getenv("OPENAI_API_KEY")
    }
)
```


```python
client.is_ready()
```

### Index data

In Weaviate you create __schemas__ to capture each of the entities you will be searching. 

In this case we'll create a schema called **Article** with the **title** vector from above included for us to search by.

The next few steps closely follow the documentation Weaviate provides [here](https://weaviate.io/developers/weaviate/quickstart).



```python
# Clear up the schema, so that we can recreate it
client.schema.delete_all()
client.schema.get()

# Define the Schema object to use `text-embedding-3-small` on `title` and `content`, but skip it for `url`
article_schema = {
    "class": "Article",
    "description": "A collection of articles",
    "vectorizer": "text2vec-openai",
    "moduleConfig": {
        "text2vec-openai": {
          "model": "ada",
          "modelVersion": "002",
          "type": "text"
        }
    },
    "properties": [{
        "name": "title",
        "description": "Title of the article",
        "dataType": ["string"]
    },
    {
        "name": "content",
        "description": "Contents of the article",
        "dataType": ["text"],
        "moduleConfig": { "text2vec-openai": { "skip": True } }
    }]
}

# add the Article schema
client.schema.create_class(article_schema)

# get the schema to make sure it worked
client.schema.get()
```




    {'classes': [{'class': 'Article',
       'description': 'A collection of articles',
       'invertedIndexConfig': {'bm25': {'b': 0.75, 'k1': 1.2},
        'cleanupIntervalSeconds': 60,
        'stopwords': {'additions': None, 'preset': 'en', 'removals': None}},
       'moduleConfig': {'text2vec-openai': {'model': 'ada',
         'modelVersion': '002',
         'type': 'text',
         'vectorizeClassName': True}},
       'properties': [{'dataType': ['string'],
         'description': 'Title of the article',
         'moduleConfig': {'text2vec-openai': {'skip': False,
           'vectorizePropertyName': False}},
         'name': 'title',
         'tokenization': 'word'},
        {'dataType': ['text'],
         'description': 'Contents of the article',
         'moduleConfig': {'text2vec-openai': {'skip': True,
           'vectorizePropertyName': False}},
         'name': 'content',
         'tokenization': 'word'}],
       'replicationConfig': {'factor': 1},
       'shardingConfig': {'virtualPerPhysical': 128,
        'desiredCount': 1,
        'actualCount': 1,
        'desiredVirtualCount': 128,
        'actualVirtualCount': 128,
        'key': '_id',
        'strategy': 'hash',
        'function': 'murmur3'},
       'vectorIndexConfig': {'skip': False,
        'cleanupIntervalSeconds': 300,
        'maxConnections': 64,
        'efConstruction': 128,
        'ef': -1,
        'dynamicEfMin': 100,
        'dynamicEfMax': 500,
        'dynamicEfFactor': 8,
        'vectorCacheMaxObjects': 1000000000000,
        'flatSearchCutoff': 40000,
        'distance': 'cosine'},
       'vectorIndexType': 'hnsw',
       'vectorizer': 'text2vec-openai'}]}


key: _node_content
string_value:
{"id_": "0147c603-7d13-4716-ae65-94af41b4ec2d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dd540420-1137-471b-a2d0-352fc76a6f73", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "e30736ca499de12145dba1d2a701ae675399f0822f28ed30cdb00e7754731f67", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0b359bce-a9bc-41a5-aab2-6636dbc1e9ee", "node_type": "1", "metadata": {}, "hash": "b826d1f19ce073fb40a7dd4d7b173da3b4357cb1e4c3230cae13b71d465433f5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
### Step 1 - configure Weaviate Batch, which optimizes CRUD operations in bulk
# - starting batch size of 100
# - dynamically increase/decrease based on performance
# - add timeout retries if something goes wrong

client.batch.configure(
    batch_size=100,
    dynamic=True,
    timeout_retries=3,
)
```




    <weaviate.batch.crud_batch.Batch at 0x3f0ca0fa0>




```python
### Step 2 - import data

print("Uploading data with vectors to Article schema.")

counter=0

with client.batch as batch:
    for k,v in article_df.iterrows():
        
        # print update message every 100 objects        
        if (counter %100 == 0):
            print(f"Import {counter} / {len(article_df)} ")
        
        properties = {
            "title": v["title"],
            "content": v["text"]
        }
        
        vector = v["title_vector"]
        
        batch.add_data_object(properties, "Article", None, vector)
        counter = counter+1

print(f"Importing ({len(article_df)}) Articles complete")  
```

    Uploading data with vectors to Article schema.
    Import 0 / 25000 
    Import 100 / 25000 
    Import 200 / 25000 
    Import 300 / 25000 
    Import 400 / 25000 
    Import 500 / 25000 
    Import 600 / 25000 
    Import 700 / 25000 
    Import 800 / 25000 
    Import 900 / 25000 
    Import 1000 / 25000 
    Import 1100 / 25000 
    Import 1200 / 25000 
    Import 1300 / 25000 
    Import 1400 / 25000 
    Import 1500 / 25000 
    Import 1600 / 25000 
    Import 1700 / 25000 
    Import 1800 / 25000 
    Import 1900 / 25000 
    Import 2000 / 25000 
    Import 2100 / 25000 
    Import 2200 / 25000 
    Import 2300 / 25000 
    Import 2400 / 25000 
    Import 2500 / 25000 
    Import 2600 / 25000 
    Import 2700 / 25000 
    Import 2800 / 25000 
    Import 2900 / 25000 
    Import 3000 / 25000 
    Import 3100 / 25000 
    Import 3200 / 25000 
    Import 3300 / 25000 
    Import 3400 / 25000 
    Import 3500 / 25000 
    Import 3600 / 25000 
    Import 3700 / 25000 
    Import 3800 / 25000 
    Import 3900 / 25000 
    Import 4000 / 25000 
    Import 4100 / 25000 
    Import 4200 /


key: _node_content
string_value:
{"id_": "0b359bce-a9bc-41a5-aab2-6636dbc1e9ee", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0147c603-7d13-4716-ae65-94af41b4ec2d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "271697b691b536cf2b637b974e83f8fe5efec1df0923e46e23a840586a6cc2a6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1c0d5c2f-4353-407a-9858-11d6c9b3b483", "node_type": "1", "metadata": {}, "hash": "817b9687fbb19a9f5a55424392c1379fb2b6b748f1b47e451a691cded3994200", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14121, "end_char_idx": 15714, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Import 900 / 25000 
    Import 1000 / 25000 
    Import 1100 / 25000 
    Import 1200 / 25000 
    Import 1300 / 25000 
    Import 1400 / 25000 
    Import 1500 / 25000 
    Import 1600 / 25000 
    Import 1700 / 25000 
    Import 1800 / 25000 
    Import 1900 / 25000 
    Import 2000 / 25000 
    Import 2100 / 25000 
    Import 2200 / 25000 
    Import 2300 / 25000 
    Import 2400 / 25000 
    Import 2500 / 25000 
    Import 2600 / 25000 
    Import 2700 / 25000 
    Import 2800 / 25000 
    Import 2900 / 25000 
    Import 3000 / 25000 
    Import 3100 / 25000 
    Import 3200 / 25000 
    Import 3300 / 25000 
    Import 3400 / 25000 
    Import 3500 / 25000 
    Import 3600 / 25000 
    Import 3700 / 25000 
    Import 3800 / 25000 
    Import 3900 / 25000 
    Import 4000 / 25000 
    Import 4100 / 25000 
    Import 4200 / 25000 
    Import 4300 / 25000 
    Import 4400 / 25000 
    Import 4500 / 25000 
    Import 4600 / 25000 
    Import 4700 / 25000 
    Import 4800 / 25000 
    Import 4900 / 25000 
    Import 5000 / 25000 
    Import 5100 / 25000 
    Import 5200 / 25000 
    Import 5300 / 25000 
    Import 5400 / 25000 
    Import 5500 / 25000 
    Import 5600 / 25000 
    Import 5700 / 25000 
    Import 5800 / 25000 
    Import 5900 / 25000 
    Import 6000 / 25000 
    Import 6100 / 25000 
    Import 6200 / 25000 
    Import 6300 / 25000 
    Import 6400 / 25000 
    Import 6500 / 25000 
    Import 6600 / 25000 
    Import 6700 / 25000 
    Import 6800 / 25000 
    Import 6900 / 25000 
    Import 7000 / 25000 
    Import 7100 / 25000 
    Import 7200 / 25000


key: _node_content
string_value:
{"id_": "1c0d5c2f-4353-407a-9858-11d6c9b3b483", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0b359bce-a9bc-41a5-aab2-6636dbc1e9ee", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "ba8d1b913c114074043bf0fc3d43e18c361163b2f8e8b53a651b7623b2da5096", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b7b139fc-71e9-4a5c-9e58-649ad8ca4664", "node_type": "1", "metadata": {}, "hash": "4d2127694d0da5e7bdd5b1f8131be3765976240e348bcd183cd3fc898e763d42", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14895, "end_char_idx": 16487, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Import 4000 / 25000 
    Import 4100 / 25000 
    Import 4200 / 25000 
    Import 4300 / 25000 
    Import 4400 / 25000 
    Import 4500 / 25000 
    Import 4600 / 25000 
    Import 4700 / 25000 
    Import 4800 / 25000 
    Import 4900 / 25000 
    Import 5000 / 25000 
    Import 5100 / 25000 
    Import 5200 / 25000 
    Import 5300 / 25000 
    Import 5400 / 25000 
    Import 5500 / 25000 
    Import 5600 / 25000 
    Import 5700 / 25000 
    Import 5800 / 25000 
    Import 5900 / 25000 
    Import 6000 / 25000 
    Import 6100 / 25000 
    Import 6200 / 25000 
    Import 6300 / 25000 
    Import 6400 / 25000 
    Import 6500 / 25000 
    Import 6600 / 25000 
    Import 6700 / 25000 
    Import 6800 / 25000 
    Import 6900 / 25000 
    Import 7000 / 25000 
    Import 7100 / 25000 
    Import 7200 / 25000 
    Import 7300 / 25000 
    Import 7400 / 25000 
    Import 7500 / 25000 
    Import 7600 / 25000 
    Import 7700 / 25000 
    Import 7800 / 25000 
    Import 7900 / 25000 
    Import 8000 / 25000 
    Import 8100 / 25000 
    Import 8200 / 25000 
    Import 8300 / 25000 
    Import 8400 / 25000 
    Import 8500 / 25000 
    Import 8600 / 25000 
    Import 8700 / 25000 
    Import 8800 / 25000 
    Import 8900 / 25000 
    Import 9000 / 25000 
    Import 9100 / 25000 
    Import 9200 / 25000 
    Import 9300 / 25000 
    Import 9400 / 25000 
    Import 9500 / 25000 
    Import 9600 / 25000 
    Import 9700 / 25000 
    Import 9800 / 25000 
    Import 9900 / 25000 
    Import 10000 / 25000 
    Import 10100 / 25000 
    Import 10200 / 25000 
    Import 10300 /


key: _node_content
string_value:
{"id_": "b7b139fc-71e9-4a5c-9e58-649ad8ca4664", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1c0d5c2f-4353-407a-9858-11d6c9b3b483", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "81c436b5fc63bf39fb797bf9b534253189ac8076757f38595b5b23aa78d8cb5e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f78cea9a-a1f9-4ede-98dc-c05fa25968f3", "node_type": "1", "metadata": {}, "hash": "033d7a3b510026426048cd08515a88375fe8f2f51bb087c47a2fbccb5bf770f0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15652, "end_char_idx": 17273, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
7000 / 25000 
    Import 7100 / 25000 
    Import 7200 / 25000 
    Import 7300 / 25000 
    Import 7400 / 25000 
    Import 7500 / 25000 
    Import 7600 / 25000 
    Import 7700 / 25000 
    Import 7800 / 25000 
    Import 7900 / 25000 
    Import 8000 / 25000 
    Import 8100 / 25000 
    Import 8200 / 25000 
    Import 8300 / 25000 
    Import 8400 / 25000 
    Import 8500 / 25000 
    Import 8600 / 25000 
    Import 8700 / 25000 
    Import 8800 / 25000 
    Import 8900 / 25000 
    Import 9000 / 25000 
    Import 9100 / 25000 
    Import 9200 / 25000 
    Import 9300 / 25000 
    Import 9400 / 25000 
    Import 9500 / 25000 
    Import 9600 / 25000 
    Import 9700 / 25000 
    Import 9800 / 25000 
    Import 9900 / 25000 
    Import 10000 / 25000 
    Import 10100 / 25000 
    Import 10200 / 25000 
    Import 10300 / 25000 
    Import 10400 / 25000 
    Import 10500 / 25000 
    Import 10600 / 25000 
    Import 10700 / 25000 
    Import 10800 / 25000 
    Import 10900 / 25000 
    Import 11000 / 25000 
    Import 11100 / 25000 
    Import 11200 / 25000 
    Import 11300 / 25000 
    Import 11400 / 25000 
    Import 11500 / 25000 
    Import 11600 / 25000 
    Import 11700 / 25000 
    Import 11800 / 25000 
    Import 11900 / 25000 
    Import 12000 / 25000 
    Import 12100 / 25000 
    Import 12200 / 25000 
    Import 12300 / 25000 
    Import 12400 / 25000 
    Import 12500 / 25000 
    Import 12600 / 25000 
    Import 12700 / 25000 
    Import 12800 / 25000 
    Import 12900 / 25000 
    Import 13000 / 25000 
    Import 13100 / 25000 
    Import 13200 / 25000 
    Import 13300 / 25000


key: _node_content
string_value:
{"id_": "f78cea9a-a1f9-4ede-98dc-c05fa25968f3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b7b139fc-71e9-4a5c-9e58-649ad8ca4664", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "621fa2aa09ee190dac52a45e9acc7b262e75d0d8ee167ab7dd212d7d303a20c9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "635e3432-689c-478e-9195-c7925a7910ad", "node_type": "1", "metadata": {}, "hash": "3d8b43043e604db0040ed442e3c16eea436617c0f84150730e58c41785e4191f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16421, "end_char_idx": 18073, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Import 10100 / 25000 
    Import 10200 / 25000 
    Import 10300 / 25000 
    Import 10400 / 25000 
    Import 10500 / 25000 
    Import 10600 / 25000 
    Import 10700 / 25000 
    Import 10800 / 25000 
    Import 10900 / 25000 
    Import 11000 / 25000 
    Import 11100 / 25000 
    Import 11200 / 25000 
    Import 11300 / 25000 
    Import 11400 / 25000 
    Import 11500 / 25000 
    Import 11600 / 25000 
    Import 11700 / 25000 
    Import 11800 / 25000 
    Import 11900 / 25000 
    Import 12000 / 25000 
    Import 12100 / 25000 
    Import 12200 / 25000 
    Import 12300 / 25000 
    Import 12400 / 25000 
    Import 12500 / 25000 
    Import 12600 / 25000 
    Import 12700 / 25000 
    Import 12800 / 25000 
    Import 12900 / 25000 
    Import 13000 / 25000 
    Import 13100 / 25000 
    Import 13200 / 25000 
    Import 13300 / 25000 
    Import 13400 / 25000 
    Import 13500 / 25000 
    Import 13600 / 25000 
    Import 13700 / 25000 
    Import 13800 / 25000 
    Import 13900 / 25000 
    Import 14000 / 25000 
    Import 14100 / 25000 
    Import 14200 / 25000 
    Import 14300 / 25000 
    Import 14400 / 25000 
    Import 14500 / 25000 
    Import 14600 / 25000 
    Import 14700 / 25000 
    Import 14800 / 25000 
    Import 14900 / 25000 
    Import 15000 / 25000 
    Import 15100 / 25000 
    Import 15200 / 25000 
    Import 15300 / 25000 
    Import 15400 / 25000 
    Import 15500 / 25000 
    Import 15600 / 25000 
    Import 15700 / 25000 
    Import 15800 / 25000 
    Import 15900 / 25000 
    Import 16000 / 25000 
    Import 16100 / 25000 
    Import 16200 / 25000 
    Import 16300 / 25000 
    Import 16400 /


key: _node_content
string_value:
{"id_": "635e3432-689c-478e-9195-c7925a7910ad", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f78cea9a-a1f9-4ede-98dc-c05fa25968f3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "57d3473f8d99999e85bfa488103ac2d47cb7e6cfbe1e12574fcae90421bb3f65", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bfb3ddc0-dd4a-424a-8244-62970640de2a", "node_type": "1", "metadata": {}, "hash": "0fb7cc44d0b07d1653fa1a91c95ed4363d079296d63dbc1f2268b9bd5f39e198", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17208, "end_char_idx": 18859, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
13100 / 25000 
    Import 13200 / 25000 
    Import 13300 / 25000 
    Import 13400 / 25000 
    Import 13500 / 25000 
    Import 13600 / 25000 
    Import 13700 / 25000 
    Import 13800 / 25000 
    Import 13900 / 25000 
    Import 14000 / 25000 
    Import 14100 / 25000 
    Import 14200 / 25000 
    Import 14300 / 25000 
    Import 14400 / 25000 
    Import 14500 / 25000 
    Import 14600 / 25000 
    Import 14700 / 25000 
    Import 14800 / 25000 
    Import 14900 / 25000 
    Import 15000 / 25000 
    Import 15100 / 25000 
    Import 15200 / 25000 
    Import 15300 / 25000 
    Import 15400 / 25000 
    Import 15500 / 25000 
    Import 15600 / 25000 
    Import 15700 / 25000 
    Import 15800 / 25000 
    Import 15900 / 25000 
    Import 16000 / 25000 
    Import 16100 / 25000 
    Import 16200 / 25000 
    Import 16300 / 25000 
    Import 16400 / 25000 
    Import 16500 / 25000 
    Import 16600 / 25000 
    Import 16700 / 25000 
    Import 16800 / 25000 
    Import 16900 / 25000 
    Import 17000 / 25000 
    Import 17100 / 25000 
    Import 17200 / 25000 
    Import 17300 / 25000 
    Import 17400 / 25000 
    Import 17500 / 25000 
    Import 17600 / 25000 
    Import 17700 / 25000 
    Import 17800 / 25000 
    Import 17900 / 25000 
    Import 18000 / 25000 
    Import 18100 / 25000 
    Import 18200 / 25000 
    Import 18300 / 25000 
    Import 18400 / 25000 
    Import 18500 / 25000 
    Import 18600 / 25000 
    Import 18700 / 25000 
    Import 18800 / 25000 
    Import 18900 / 25000 
    Import 19000 / 25000 
    Import 19100 / 25000 
    Import 19200 / 25000 
    Import 19300 / 25000 
    Import 19400 / 25000


key: _node_content
string_value:
{"id_": "bfb3ddc0-dd4a-424a-8244-62970640de2a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "635e3432-689c-478e-9195-c7925a7910ad", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "ca51f04467adba54416ad3c9408e93bb0d4a68670b663aadeb447289c6fe5579", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bd015e80-525b-4ea2-a42e-20bd93442503", "node_type": "1", "metadata": {}, "hash": "a6e7db12c0c87c5baf1bbc57d8cf8993016903035c85c18fbf6369d1a533aa51", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18007, "end_char_idx": 19659, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Import 16200 / 25000 
    Import 16300 / 25000 
    Import 16400 / 25000 
    Import 16500 / 25000 
    Import 16600 / 25000 
    Import 16700 / 25000 
    Import 16800 / 25000 
    Import 16900 / 25000 
    Import 17000 / 25000 
    Import 17100 / 25000 
    Import 17200 / 25000 
    Import 17300 / 25000 
    Import 17400 / 25000 
    Import 17500 / 25000 
    Import 17600 / 25000 
    Import 17700 / 25000 
    Import 17800 / 25000 
    Import 17900 / 25000 
    Import 18000 / 25000 
    Import 18100 / 25000 
    Import 18200 / 25000 
    Import 18300 / 25000 
    Import 18400 / 25000 
    Import 18500 / 25000 
    Import 18600 / 25000 
    Import 18700 / 25000 
    Import 18800 / 25000 
    Import 18900 / 25000 
    Import 19000 / 25000 
    Import 19100 / 25000 
    Import 19200 / 25000 
    Import 19300 / 25000 
    Import 19400 / 25000 
    Import 19500 / 25000 
    Import 19600 / 25000 
    Import 19700 / 25000 
    Import 19800 / 25000 
    Import 19900 / 25000 
    Import 20000 / 25000 
    Import 20100 / 25000 
    Import 20200 / 25000 
    Import 20300 / 25000 
    Import 20400 / 25000 
    Import 20500 / 25000 
    Import 20600 / 25000 
    Import 20700 / 25000 
    Import 20800 / 25000 
    Import 20900 / 25000 
    Import 21000 / 25000 
    Import 21100 / 25000 
    Import 21200 / 25000 
    Import 21300 / 25000 
    Import 21400 / 25000 
    Import 21500 / 25000 
    Import 21600 / 25000 
    Import 21700 / 25000 
    Import 21800 / 25000 
    Import 21900 / 25000 
    Import 22000 / 25000 
    Import 22100 / 25000 
    Import 22200 / 25000 
    Import 22300 / 25000 
    Import 22400 / 25000 
    Import 22500 /


key: _node_content
string_value:
{"id_": "bd015e80-525b-4ea2-a42e-20bd93442503", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bfb3ddc0-dd4a-424a-8244-62970640de2a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "cae7e4e2a6ad6c634a9f4689751390472ccf53177a5ccb0a53bd64ad9eae131a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c517f4aa-cb2e-4198-a500-71e144fc54fa", "node_type": "1", "metadata": {}, "hash": "11ab5009488c061ec53d52b34bdfac3bb1ed7d14eccb9852acc725e337c4c2ee", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18794, "end_char_idx": 20330, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
19200 / 25000 
    Import 19300 / 25000 
    Import 19400 / 25000 
    Import 19500 / 25000 
    Import 19600 / 25000 
    Import 19700 / 25000 
    Import 19800 / 25000 
    Import 19900 / 25000 
    Import 20000 / 25000 
    Import 20100 / 25000 
    Import 20200 / 25000 
    Import 20300 / 25000 
    Import 20400 / 25000 
    Import 20500 / 25000 
    Import 20600 / 25000 
    Import 20700 / 25000 
    Import 20800 / 25000 
    Import 20900 / 25000 
    Import 21000 / 25000 
    Import 21100 / 25000 
    Import 21200 / 25000 
    Import 21300 / 25000 
    Import 21400 / 25000 
    Import 21500 / 25000 
    Import 21600 / 25000 
    Import 21700 / 25000 
    Import 21800 / 25000 
    Import 21900 / 25000 
    Import 22000 / 25000 
    Import 22100 / 25000 
    Import 22200 / 25000 
    Import 22300 / 25000 
    Import 22400 / 25000 
    Import 22500 / 25000 
    Import 22600 / 25000 
    Import 22700 / 25000 
    Import 22800 / 25000 
    Import 22900 / 25000 
    Import 23000 / 25000 
    Import 23100 / 25000 
    Import 23200 / 25000 
    Import 23300 / 25000 
    Import 23400 / 25000 
    Import 23500 / 25000 
    Import 23600 / 25000 
    Import 23700 / 25000 
    Import 23800 / 25000 
    Import 23900 / 25000 
    Import 24000 / 25000 
    Import 24100 / 25000 
    Import 24200 / 25000 
    Import 24300 / 25000 
    Import 24400 / 25000 
    Import 24500 / 25000 
    Import 24600 / 25000 
    Import 24700 / 25000 
    Import 24800 / 25000 
    Import 24900 / 25000 
    Importing (25000) Articles complete


key: _node_content
string_value:
{"id_": "c517f4aa-cb2e-4198-a500-71e144fc54fa", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bd015e80-525b-4ea2-a42e-20bd93442503", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "625d61e05c335b7167075913662b13c61ef48de834d0d89f4f524763099d1f07", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e75977e7-5a02-4de3-83c8-65d341fa0f44", "node_type": "1", "metadata": {}, "hash": "bb61926ac4774cd1afc875939707c911be5d9e348f15974cd3999ad48fa7ef96", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19450, "end_char_idx": 21244, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
/ 25000 
    Import 21800 / 25000 
    Import 21900 / 25000 
    Import 22000 / 25000 
    Import 22100 / 25000 
    Import 22200 / 25000 
    Import 22300 / 25000 
    Import 22400 / 25000 
    Import 22500 / 25000 
    Import 22600 / 25000 
    Import 22700 / 25000 
    Import 22800 / 25000 
    Import 22900 / 25000 
    Import 23000 / 25000 
    Import 23100 / 25000 
    Import 23200 / 25000 
    Import 23300 / 25000 
    Import 23400 / 25000 
    Import 23500 / 25000 
    Import 23600 / 25000 
    Import 23700 / 25000 
    Import 23800 / 25000 
    Import 23900 / 25000 
    Import 24000 / 25000 
    Import 24100 / 25000 
    Import 24200 / 25000 
    Import 24300 / 25000 
    Import 24400 / 25000 
    Import 24500 / 25000 
    Import 24600 / 25000 
    Import 24700 / 25000 
    Import 24800 / 25000 
    Import 24900 / 25000 
    Importing (25000) Articles complete



```python
# Test that all data has loaded – get object count
result = (
    client.query.aggregate("Article")
    .with_fields("meta { count }")
    .do()
)
print("Object count: ", result["data"]["Aggregate"]["Article"])
```

    Object count:  [{'meta': {'count': 25000}}]



```python
# Test one article has worked by checking one object
test_article = (
    client.query
    .get("Article", ["title", "content", "_additional {id}"])
    .with_limit(1)
    .do()
)["data"]["Get"]["Article"][0]

print(test_article["_additional"]["id"])
print(test_article["title"])
print(test_article["content"])
```

    000393f2-1182-4e3d-abcf-4217eda64be0
    Lago d'Origlio
    Lago d'Origlio is a lake in the municipality of Origlio, in Ticino, Switzerland.
    
    Lakes of Ticino


### Search data

As above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors


key: _node_content
string_value:
{"id_": "e75977e7-5a02-4de3-83c8-65d341fa0f44", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c517f4aa-cb2e-4198-a500-71e144fc54fa", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "09a3f50d4207592491d87f4a05a1e78015e0c33526c936b4f47f1ea9c8cd99da", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "97a93e0f-7579-4835-847c-f2a863e85d8a", "node_type": "1", "metadata": {}, "hash": "38ff07da86b966c01397abffc90605b7d2f0fd7a0fd9339a7963fa0a561b66e4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19983, "end_char_idx": 21840, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Import 23800 / 25000 
    Import 23900 / 25000 
    Import 24000 / 25000 
    Import 24100 / 25000 
    Import 24200 / 25000 
    Import 24300 / 25000 
    Import 24400 / 25000 
    Import 24500 / 25000 
    Import 24600 / 25000 
    Import 24700 / 25000 
    Import 24800 / 25000 
    Import 24900 / 25000 
    Importing (25000) Articles complete



```python
# Test that all data has loaded – get object count
result = (
    client.query.aggregate("Article")
    .with_fields("meta { count }")
    .do()
)
print("Object count: ", result["data"]["Aggregate"]["Article"])
```

    Object count:  [{'meta': {'count': 25000}}]



```python
# Test one article has worked by checking one object
test_article = (
    client.query
    .get("Article", ["title", "content", "_additional {id}"])
    .with_limit(1)
    .do()
)["data"]["Get"]["Article"][0]

print(test_article["_additional"]["id"])
print(test_article["title"])
print(test_article["content"])
```

    000393f2-1182-4e3d-abcf-4217eda64be0
    Lago d'Origlio
    Lago d'Origlio is a lake in the municipality of Origlio, in Ticino, Switzerland.
    
    Lakes of Ticino


### Search data

As above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors


```python
def query_weaviate(query, collection_name, top_k=20):

    # Creates embedding vector from user query
    embedded_query = openai.Embedding.create(
        input=query,
        model=EMBEDDING_MODEL,
    )["data"][0]['embedding']
    
    near_vector = {"vector": embedded_query}

    # Queries input schema with vectorised user query
    query_result = (
        client.query
        .get(collection_name, ["title", "content", "_additional {certainty distance}"])
        .with_near_vector(near_vector)
        .with_limit(top_k)
        .do()
    )
    
    return query_result
```


key: _node_content
string_value:
{"id_": "97a93e0f-7579-4835-847c-f2a863e85d8a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e75977e7-5a02-4de3-83c8-65d341fa0f44", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "249bf8feeb3f9b6a3645561d9a212799cd37d268aa110036d8bee434f516a0e6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a3d74bd0-7401-4469-a7b3-f0ff39fbf01b", "node_type": "1", "metadata": {}, "hash": "250117ee8578b85026bd11c00c797f387ab9033d560b419116d20bdf2b90426f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20295, "end_char_idx": 23310, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Importing (25000) Articles complete



```python
# Test that all data has loaded – get object count
result = (
    client.query.aggregate("Article")
    .with_fields("meta { count }")
    .do()
)
print("Object count: ", result["data"]["Aggregate"]["Article"])
```

    Object count:  [{'meta': {'count': 25000}}]



```python
# Test one article has worked by checking one object
test_article = (
    client.query
    .get("Article", ["title", "content", "_additional {id}"])
    .with_limit(1)
    .do()
)["data"]["Get"]["Article"][0]

print(test_article["_additional"]["id"])
print(test_article["title"])
print(test_article["content"])
```

    000393f2-1182-4e3d-abcf-4217eda64be0
    Lago d'Origlio
    Lago d'Origlio is a lake in the municipality of Origlio, in Ticino, Switzerland.
    
    Lakes of Ticino


### Search data

As above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors


```python
def query_weaviate(query, collection_name, top_k=20):

    # Creates embedding vector from user query
    embedded_query = openai.Embedding.create(
        input=query,
        model=EMBEDDING_MODEL,
    )["data"][0]['embedding']
    
    near_vector = {"vector": embedded_query}

    # Queries input schema with vectorised user query
    query_result = (
        client.query
        .get(collection_name, ["title", "content", "_additional {certainty distance}"])
        .with_near_vector(near_vector)
        .with_limit(top_k)
        .do()
    )
    
    return query_result
```


```python
query_result = query_weaviate("modern art in Europe", "Article")
counter = 0
for article in query_result["data"]["Get"]["Article"]:
    counter += 1
    print(f"{counter}. { article['title']} (Certainty: {round(article['_additional']['certainty'],3) }) (Distance: {round(article['_additional']['distance'],3) })")
```

    1. Museum of Modern Art (Certainty: 0.938) (Distance: 0.125)
    2. Western Europe (Certainty: 0.934) (Distance: 0.133)
    3. Renaissance art (Certainty: 0.932) (Distance: 0.136)
    4. Pop art (Certainty: 0.93) (Distance: 0.14)
    5. Northern Europe (Certainty: 0.927) (Distance: 0.145)
    6. Hellenistic art (Certainty: 0.926) (Distance: 0.147)
    7. Modernist literature (Certainty: 0.924) (Distance: 0.153)
    8. Art film (Certainty: 0.922) (Distance: 0.157)
    9. Central Europe (Certainty: 0.921) (Distance: 0.157)
    10. European (Certainty: 0.921) (Distance: 0.159)
    11. Art (Certainty: 0.921) (Distance: 0.159)
    12. Byzantine art (Certainty: 0.92) (Distance: 0.159)
    13. Postmodernism (Certainty: 0.92) (Distance: 0.16)
    14. Eastern Europe (Certainty: 0.92) (Distance: 0.161)
    15. Europe (Certainty: 0.919) (Distance: 0.161)
    16. Cubism (Certainty: 0.919) (Distance: 0.161)
    17. Impressionism (Certainty: 0.919) (Distance: 0.162)
    18. Bauhaus (Certainty: 0.919) (Distance: 0.162)
    19. Expressionism (Certainty: 0.918) (Distance: 0.163)
    20. Surrealism (Certainty: 0.918) (Distance: 0.163)


key: _node_content
string_value:
{"id_": "a3d74bd0-7401-4469-a7b3-f0ff39fbf01b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "97a93e0f-7579-4835-847c-f2a863e85d8a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "b398480b99d7dfdcf56b5a7ed2ab3f7418706900a7b94535f22bc9cae3bfd82c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1426d632-43e8-4e2a-8687-337dd6e9887a", "node_type": "1", "metadata": {}, "hash": "a1c9811b5079510a370a01716f8f65534cba6264be9dffb3c6b640ede9d73f23", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23314, "end_char_idx": 25369, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
query_result = query_weaviate("Famous battles in Scottish history", "Article")
counter = 0
for article in query_result["data"]["Get"]["Article"]:
    counter += 1
    print(f"{counter}. {article['title']} (Score: {round(article['_additional']['certainty'],3) })")
```

    1. Historic Scotland (Score: 0.946)
    2. First War of Scottish Independence (Score: 0.946)
    3. Battle of Bannockburn (Score: 0.946)
    4. Wars of Scottish Independence (Score: 0.944)
    5. Second War of Scottish Independence (Score: 0.94)
    6. List of Scottish monarchs (Score: 0.937)
    7. Scottish Borders (Score: 0.932)
    8. Braveheart (Score: 0.929)
    9. John of Scotland (Score: 0.929)
    10. Guardians of Scotland (Score: 0.926)
    11. Holyrood Abbey (Score: 0.925)
    12. Scottish (Score: 0.925)
    13. Scots (Score: 0.925)
    14. Robert I of Scotland (Score: 0.924)
    15. Scottish people (Score: 0.924)
    16. Edinburgh Castle (Score: 0.924)
    17. Alexander I of Scotland (Score: 0.924)
    18. Robert Burns (Score: 0.924)
    19. Battle of Bosworth Field (Score: 0.922)
    20. David II of Scotland (Score: 0.922)


### Let Weaviate handle vector embeddings

Weaviate has a [built-in module for OpenAI](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai), which takes care of the steps required to generate a vector embedding for your queries and any CRUD operations.

This allows you to run a vector query with the `with_near_text` filter, which uses your `OPEN_API_KEY`.


```python
def near_text_weaviate(query, collection_name):
    
    nearText = {
        "concepts": [query],
        "distance": 0.7,
    }

    properties = [
        "title", "content",
        "_additional {certainty distance}"
    ]

    query_result = (
        client.query
        .get(collection_name, properties)
        .with_near_text(nearText)
        .with_limit(20)
        .do()
    )["data"]["Get"][collection_name]
    
    print (f"Objects returned: {len(query_result)}")
    
    return query_result
```


key: _node_content
string_value:
{"id_": "1426d632-43e8-4e2a-8687-337dd6e9887a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a3d74bd0-7401-4469-a7b3-f0ff39fbf01b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "bc2b5b771cefc2c8cd8e20f27a169c60ad479e462f932e40b9f33696bc15eacb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c84c6052-b04b-4376-a149-82a46519965c", "node_type": "1", "metadata": {}, "hash": "4b1d951090cca7e1b4232cfac53d912cbda980def8226a40ef6711dc01592866", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24446, "end_char_idx": 26841, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Let Weaviate handle vector embeddings

Weaviate has a [built-in module for OpenAI](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai), which takes care of the steps required to generate a vector embedding for your queries and any CRUD operations.

This allows you to run a vector query with the `with_near_text` filter, which uses your `OPEN_API_KEY`.


```python
def near_text_weaviate(query, collection_name):
    
    nearText = {
        "concepts": [query],
        "distance": 0.7,
    }

    properties = [
        "title", "content",
        "_additional {certainty distance}"
    ]

    query_result = (
        client.query
        .get(collection_name, properties)
        .with_near_text(nearText)
        .with_limit(20)
        .do()
    )["data"]["Get"][collection_name]
    
    print (f"Objects returned: {len(query_result)}")
    
    return query_result
```


```python
query_result = near_text_weaviate("modern art in Europe","Article")
counter = 0
for article in query_result:
    counter += 1
    print(f"{counter}. { article['title']} (Certainty: {round(article['_additional']['certainty'],3) }) (Distance: {round(article['_additional']['distance'],3) })")
```

    Objects returned: 20
    1. Museum of Modern Art (Certainty: 0.938) (Distance: 0.125)
    2. Western Europe (Certainty: 0.934) (Distance: 0.133)
    3. Renaissance art (Certainty: 0.932) (Distance: 0.136)
    4. Pop art (Certainty: 0.93) (Distance: 0.14)
    5. Northern Europe (Certainty: 0.927) (Distance: 0.145)
    6. Hellenistic art (Certainty: 0.926) (Distance: 0.147)
    7. Modernist literature (Certainty: 0.923) (Distance: 0.153)
    8. Art film (Certainty: 0.922) (Distance: 0.157)
    9. Central Europe (Certainty: 0.921) (Distance: 0.157)
    10. European (Certainty: 0.921) (Distance: 0.159)
    11. Art (Certainty: 0.921) (Distance: 0.159)
    12. Byzantine art (Certainty: 0.92) (Distance: 0.159)
    13. Postmodernism (Certainty: 0.92) (Distance: 0.16)
    14. Eastern Europe (Certainty: 0.92) (Distance: 0.161)
    15. Europe (Certainty: 0.919) (Distance: 0.161)
    16. Cubism (Certainty: 0.919) (Distance: 0.161)
    17. Impressionism (Certainty: 0.919) (Distance: 0.162)
    18. Bauhaus (Certainty: 0.919) (Distance: 0.162)
    19. Surrealism (Certainty: 0.918) (Distance: 0.163)
    20. Expressionism (Certainty: 0.918) (Distance: 0.163)


key: _node_content
string_value:
{"id_": "c84c6052-b04b-4376-a149-82a46519965c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1426d632-43e8-4e2a-8687-337dd6e9887a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "b20848e46e6e78314699670dde401e1e11e702c782d7557679e84f0123be667c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 26845, "end_char_idx": 28495, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
query_result = near_text_weaviate("Famous battles in Scottish history","Article")
counter = 0
for article in query_result:
    counter += 1
    print(f"{counter}. { article['title']} (Certainty: {round(article['_additional']['certainty'],3) }) (Distance: {round(article['_additional']['distance'],3) })")
```

    Objects returned: 20
    1. Historic Scotland (Certainty: 0.946) (Distance: 0.107)
    2. First War of Scottish Independence (Certainty: 0.946) (Distance: 0.108)
    3. Battle of Bannockburn (Certainty: 0.946) (Distance: 0.109)
    4. Wars of Scottish Independence (Certainty: 0.944) (Distance: 0.111)
    5. Second War of Scottish Independence (Certainty: 0.94) (Distance: 0.121)
    6. List of Scottish monarchs (Certainty: 0.937) (Distance: 0.127)
    7. Scottish Borders (Certainty: 0.932) (Distance: 0.137)
    8. Braveheart (Certainty: 0.929) (Distance: 0.141)
    9. John of Scotland (Certainty: 0.929) (Distance: 0.142)
    10. Guardians of Scotland (Certainty: 0.926) (Distance: 0.148)
    11. Holyrood Abbey (Certainty: 0.925) (Distance: 0.15)
    12. Scottish (Certainty: 0.925) (Distance: 0.15)
    13. Scots (Certainty: 0.925) (Distance: 0.15)
    14. Robert I of Scotland (Certainty: 0.924) (Distance: 0.151)
    15. Scottish people (Certainty: 0.924) (Distance: 0.152)
    16. Edinburgh Castle (Certainty: 0.924) (Distance: 0.153)
    17. Alexander I of Scotland (Certainty: 0.924) (Distance: 0.153)
    18. Robert Burns (Certainty: 0.924) (Distance: 0.153)
    19. Battle of Bosworth Field (Certainty: 0.922) (Distance: 0.155)
    20. David II of Scotland (Certainty: 0.922) (Distance: 0.157)



```python

```


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb


key: _node_content
string_value:
{"id_": "5f14be55-b52b-49b9-9796-95effcb7c007", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "bce57ee6c084b1c18fd858209606446bc92a66767f7298d29c766e10979f850d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6a008aeb-6ee1-437f-9609-d42f0419cc69", "node_type": "1", "metadata": {}, "hash": "1cedf7ad193670c958567e54a7c96fd27833e6fdc3c016f35ac09d9afa067293", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2380, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Question Answering in Weaviate with OpenAI Q&A module

This notebook is prepared for a scenario where:
* Your data is not vectorized
* You want to run Q&A ([learn more](https://weaviate.io/developers/weaviate/modules/reader-generator-modules/qna-openai)) on your data based on the [OpenAI completions](https://beta.openai.com/docs/api-reference/completions) endpoint.
* You want to use Weaviate with the OpenAI module ([text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai)), to generate vector embeddings for you.

This notebook takes you through a simple flow to set up a Weaviate instance, connect to it (with OpenAI API key), configure data schema, import data (which will automatically generate vector embeddings for your data), and run question answering.

## What is Weaviate

Weaviate is an open-source vector search engine that stores data objects together with their vectors. This allows for combining vector search with structured filtering.

Weaviate uses KNN algorithms to create an vector-optimized index, which allows your queries to run extremely fast. Learn more [here](https://weaviate.io/blog/why-is-vector-search-so-fast).

Weaviate let you use your favorite ML-models, and scale seamlessly into billions of data objects.

### Deployment options

Whatever your scenario or production setup, Weaviate has an option for you. You can deploy Weaviate in the following setups:
* Self-hosted – you can deploy Weaviate with docker locally, or any server you want.
* SaaS – you can use [Weaviate Cloud Service (WCS)](https://console.weaviate.io/) to host your Weaviate instances.
* Hybrid-SaaS – you can deploy Weaviate in your own private Cloud Service 

### Programming languages

Weaviate offers four [client libraries](https://weaviate.io/developers/weaviate/client-libraries), which allow you to communicate from your apps:
* [Python](https://weaviate.io/developers/weaviate/client-libraries/python)
* [JavaScript](https://weaviate.io/developers/weaviate/client-libraries/javascript)
* [Java](https://weaviate.io/developers/weaviate/client-libraries/java)
* [Go](https://weaviate.io/developers/weaviate/client-libraries/go)

Additionally, Weaviate has a [REST layer](https://weaviate.io/developers/weaviate/api/rest/objects). Basically you can call Weaviate from any language that supports REST requests.


key: _node_content
string_value:
{"id_": "6a008aeb-6ee1-437f-9609-d42f0419cc69", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "bce57ee6c084b1c18fd858209606446bc92a66767f7298d29c766e10979f850d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5f14be55-b52b-49b9-9796-95effcb7c007", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "b00f052da40d4259422bac50fdc9af4c430b59ddfb7cd4333dda6495ec7105fb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "db2f00de-19ec-4277-a73c-a95a477af330", "node_type": "1", "metadata": {}, "hash": "edf9f6dd3cbbe94a584e4dd8c5d77fb413406bea18b3d7afe040bd62b0577c42", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 822, "end_char_idx": 3997, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## What is Weaviate

Weaviate is an open-source vector search engine that stores data objects together with their vectors. This allows for combining vector search with structured filtering.

Weaviate uses KNN algorithms to create an vector-optimized index, which allows your queries to run extremely fast. Learn more [here](https://weaviate.io/blog/why-is-vector-search-so-fast).

Weaviate let you use your favorite ML-models, and scale seamlessly into billions of data objects.

### Deployment options

Whatever your scenario or production setup, Weaviate has an option for you. You can deploy Weaviate in the following setups:
* Self-hosted – you can deploy Weaviate with docker locally, or any server you want.
* SaaS – you can use [Weaviate Cloud Service (WCS)](https://console.weaviate.io/) to host your Weaviate instances.
* Hybrid-SaaS – you can deploy Weaviate in your own private Cloud Service 

### Programming languages

Weaviate offers four [client libraries](https://weaviate.io/developers/weaviate/client-libraries), which allow you to communicate from your apps:
* [Python](https://weaviate.io/developers/weaviate/client-libraries/python)
* [JavaScript](https://weaviate.io/developers/weaviate/client-libraries/javascript)
* [Java](https://weaviate.io/developers/weaviate/client-libraries/java)
* [Go](https://weaviate.io/developers/weaviate/client-libraries/go)

Additionally, Weaviate has a [REST layer](https://weaviate.io/developers/weaviate/api/rest/objects). Basically you can call Weaviate from any language that supports REST requests.

## Demo Flow
The demo flow is:
- **Prerequisites Setup**: Create a Weaviate instance and install required libraries
- **Connect**: Connect to your Weaviate instance 
- **Schema Configuration**: Configure the schema of your data
    - *Note*: Here we can define which OpenAI Embedding Model to use
    - *Note*: Here we can configure which properties to index
- **Import data**: Load a demo dataset and import it into Weaviate
    - *Note*: The import process will automatically index your data - based on the configuration in the schema
    - *Note*: You don't need to explicitly vectorize your data, Weaviate will communicate with OpenAI to do it for you
- **Run Queries**: Query 
    - *Note*: You don't need to explicitly vectorize your queries, Weaviate will communicate with OpenAI to do it for you
    - *Note*: The `qna-openai` module automatically communicates with the OpenAI completions endpoint

Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases for question answering.

## OpenAI Module in Weaviate
All Weaviate instances come equipped with the [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) and the [qna-openai](https://weaviate.io/developers/weaviate/modules/reader-generator-modules/qna-openai) modules.

The first module is responsible for handling vectorization at import (or any CRUD operations) and when you run a search query. The second module communicates with the OpenAI completions endpoint.

### No need to manually vectorize data
This is great news for you.


key: _node_content
string_value:
{"id_": "db2f00de-19ec-4277-a73c-a95a477af330", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "bce57ee6c084b1c18fd858209606446bc92a66767f7298d29c766e10979f850d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6a008aeb-6ee1-437f-9609-d42f0419cc69", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "d2b67def3f2328706d8599bddb23231a1a707219dae6748e4f6b965e011930fe", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5fd9d549-527e-4705-8361-8f3405604ea0", "node_type": "1", "metadata": {}, "hash": "542895e94c9739af8d54a92c8b463901a65e9c5f6da37bdaeb93d209975e184e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2302, "end_char_idx": 5155, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Basically you can call Weaviate from any language that supports REST requests.

## Demo Flow
The demo flow is:
- **Prerequisites Setup**: Create a Weaviate instance and install required libraries
- **Connect**: Connect to your Weaviate instance 
- **Schema Configuration**: Configure the schema of your data
    - *Note*: Here we can define which OpenAI Embedding Model to use
    - *Note*: Here we can configure which properties to index
- **Import data**: Load a demo dataset and import it into Weaviate
    - *Note*: The import process will automatically index your data - based on the configuration in the schema
    - *Note*: You don't need to explicitly vectorize your data, Weaviate will communicate with OpenAI to do it for you
- **Run Queries**: Query 
    - *Note*: You don't need to explicitly vectorize your queries, Weaviate will communicate with OpenAI to do it for you
    - *Note*: The `qna-openai` module automatically communicates with the OpenAI completions endpoint

Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases for question answering.

## OpenAI Module in Weaviate
All Weaviate instances come equipped with the [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) and the [qna-openai](https://weaviate.io/developers/weaviate/modules/reader-generator-modules/qna-openai) modules.

The first module is responsible for handling vectorization at import (or any CRUD operations) and when you run a search query. The second module communicates with the OpenAI completions endpoint.

### No need to manually vectorize data
This is great news for you. With [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) you don't need to manually vectorize your data, as Weaviate will call OpenAI for you whenever necessary.

All you need to do is:
1. provide your OpenAI API Key – when you connected to the Weaviate Client
2. define which OpenAI vectorizer to use in your Schema

## Prerequisites

Before we start this project, we need setup the following:

* create a `Weaviate` instance
* install libraries
    * `weaviate-client`
    * `datasets`
    * `apache-beam`
* get your [OpenAI API key](https://beta.openai.com/account/api-keys)

===========================================================
### Create a Weaviate instance

To create a Weaviate instance we have 2 options:

1. (Recommended path) [Weaviate Cloud Service](https://console.weaviate.io/) – to host your Weaviate instance in the cloud. The free sandbox should be more than enough for this cookbook.
2. Install and run Weaviate locally with Docker.

#### Option 1 – WCS Installation Steps

Use [Weaviate Cloud Service](https://console.weaviate.io/) (WCS) to create a free Weaviate cluster.


key: _node_content
string_value:
{"id_": "5fd9d549-527e-4705-8361-8f3405604ea0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "bce57ee6c084b1c18fd858209606446bc92a66767f7298d29c766e10979f850d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "db2f00de-19ec-4277-a73c-a95a477af330", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "66d90faf06126c6a292063eb8d8ba420f9e7e9501d54f4d44dd32f3dc597156e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "96ca7da2-3517-40e5-b39d-004daeaf8cf8", "node_type": "1", "metadata": {}, "hash": "edb5b569c70b90f9bd00e5eff3a7319fe0e77a79206bc15a24c687e3432c8db8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3734, "end_char_idx": 6810, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The first module is responsible for handling vectorization at import (or any CRUD operations) and when you run a search query. The second module communicates with the OpenAI completions endpoint.

### No need to manually vectorize data
This is great news for you. With [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) you don't need to manually vectorize your data, as Weaviate will call OpenAI for you whenever necessary.

All you need to do is:
1. provide your OpenAI API Key – when you connected to the Weaviate Client
2. define which OpenAI vectorizer to use in your Schema

## Prerequisites

Before we start this project, we need setup the following:

* create a `Weaviate` instance
* install libraries
    * `weaviate-client`
    * `datasets`
    * `apache-beam`
* get your [OpenAI API key](https://beta.openai.com/account/api-keys)

===========================================================
### Create a Weaviate instance

To create a Weaviate instance we have 2 options:

1. (Recommended path) [Weaviate Cloud Service](https://console.weaviate.io/) – to host your Weaviate instance in the cloud. The free sandbox should be more than enough for this cookbook.
2. Install and run Weaviate locally with Docker.

#### Option 1 – WCS Installation Steps

Use [Weaviate Cloud Service](https://console.weaviate.io/) (WCS) to create a free Weaviate cluster.
1. create a free account and/or login to [WCS](https://console.weaviate.io/)
2. create a `Weaviate Cluster` with the following settings:
    * Sandbox: `Sandbox Free`
    * Weaviate Version: Use default (latest)
    * OIDC Authentication: `Disabled`
3. your instance should be ready in a minute or two
4. make a note of the `Cluster Id`. The link will take you to the full path of your cluster (you will need it later to connect to it). It should be something like: `https://your-project-name.weaviate.network` 

#### Option 2 – local Weaviate instance with Docker

Install and run Weaviate locally with Docker.
1. Download the [./docker-compose.yml](./docker-compose.yml) file
2. Then open your terminal, navigate to where your docker-compose.yml file is located, and start docker with: `docker-compose up -d`
3. Once this is ready, your instance should be available at [http://localhost:8080](http://localhost:8080)

Note. To shut down your docker instance you can call: `docker-compose down`

##### Learn more
To learn more, about using Weaviate with Docker see the [installation documentation](https://weaviate.io/developers/weaviate/installation/docker-compose).

===========================================================    
## Install required libraries

Before running this project make sure to have the following libraries:

### Weaviate Python client

The [Weaviate Python client](https://weaviate.io/developers/weaviate/client-libraries/python) allows you to communicate with your Weaviate instance from your Python project.

### datasets & apache-beam

To load sample data, you need the `datasets` library and its' dependency `apache-beam`.


key: _node_content
string_value:
{"id_": "96ca7da2-3517-40e5-b39d-004daeaf8cf8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "bce57ee6c084b1c18fd858209606446bc92a66767f7298d29c766e10979f850d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5fd9d549-527e-4705-8361-8f3405604ea0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "d40b596d19276ad4b0d2ef723da4f1a2098882086974004a72971e52b6bc9392", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8f3a000e-40f1-4432-9163-5c18d7691ef1", "node_type": "1", "metadata": {}, "hash": "c6071153842b32a8ff6bf387dde86cca1ac06f861683f8a8048c8964b5abf50f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5156, "end_char_idx": 7490, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
1. create a free account and/or login to [WCS](https://console.weaviate.io/)
2. create a `Weaviate Cluster` with the following settings:
    * Sandbox: `Sandbox Free`
    * Weaviate Version: Use default (latest)
    * OIDC Authentication: `Disabled`
3. your instance should be ready in a minute or two
4. make a note of the `Cluster Id`. The link will take you to the full path of your cluster (you will need it later to connect to it). It should be something like: `https://your-project-name.weaviate.network` 

#### Option 2 – local Weaviate instance with Docker

Install and run Weaviate locally with Docker.
1. Download the [./docker-compose.yml](./docker-compose.yml) file
2. Then open your terminal, navigate to where your docker-compose.yml file is located, and start docker with: `docker-compose up -d`
3. Once this is ready, your instance should be available at [http://localhost:8080](http://localhost:8080)

Note. To shut down your docker instance you can call: `docker-compose down`

##### Learn more
To learn more, about using Weaviate with Docker see the [installation documentation](https://weaviate.io/developers/weaviate/installation/docker-compose).

===========================================================    
## Install required libraries

Before running this project make sure to have the following libraries:

### Weaviate Python client

The [Weaviate Python client](https://weaviate.io/developers/weaviate/client-libraries/python) allows you to communicate with your Weaviate instance from your Python project.

### datasets & apache-beam

To load sample data, you need the `datasets` library and its' dependency `apache-beam`.


```python
# Install the Weaviate client for Python
!pip install weaviate-client>3.11.0

# Install datasets and apache-beam to load the sample datasets
!pip install datasets apache-beam
```

===========================================================
## Prepare your OpenAI API key

The `OpenAI API key` is used for vectorization of your data at import, and for queries.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it to your environment variables as `OPENAI_API_KEY`.


```python
# Export OpenAI API Key
!export OPENAI_API_KEY="your key"
```


key: _node_content
string_value:
{"id_": "8f3a000e-40f1-4432-9163-5c18d7691ef1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "bce57ee6c084b1c18fd858209606446bc92a66767f7298d29c766e10979f850d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "96ca7da2-3517-40e5-b39d-004daeaf8cf8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "7f61e3e73ebc22b37062dd36f70fd276ecb31ebe751a712fe1b58ebe9354b700", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "74e9b8af-46f1-4b36-8454-558d169afc8b", "node_type": "1", "metadata": {}, "hash": "88f96e7784f267ea98a6b6819a100ebb82099924dc4490279e1f8d07c048f3f6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5768, "end_char_idx": 8400, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
1. Download the [./docker-compose.yml](./docker-compose.yml) file
2. Then open your terminal, navigate to where your docker-compose.yml file is located, and start docker with: `docker-compose up -d`
3. Once this is ready, your instance should be available at [http://localhost:8080](http://localhost:8080)

Note. To shut down your docker instance you can call: `docker-compose down`

##### Learn more
To learn more, about using Weaviate with Docker see the [installation documentation](https://weaviate.io/developers/weaviate/installation/docker-compose).

===========================================================    
## Install required libraries

Before running this project make sure to have the following libraries:

### Weaviate Python client

The [Weaviate Python client](https://weaviate.io/developers/weaviate/client-libraries/python) allows you to communicate with your Weaviate instance from your Python project.

### datasets & apache-beam

To load sample data, you need the `datasets` library and its' dependency `apache-beam`.


```python
# Install the Weaviate client for Python
!pip install weaviate-client>3.11.0

# Install datasets and apache-beam to load the sample datasets
!pip install datasets apache-beam
```

===========================================================
## Prepare your OpenAI API key

The `OpenAI API key` is used for vectorization of your data at import, and for queries.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it to your environment variables as `OPENAI_API_KEY`.


```python
# Export OpenAI API Key
!export OPENAI_API_KEY="your key"
```


```python
# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.
import os

# Note. alternatively you can set a temporary env variable like this:
# os.environ['OPENAI_API_KEY'] = 'your-key-goes-here'

if os.getenv("OPENAI_API_KEY") is not None:
    print ("OPENAI_API_KEY is ready")
else:
    print ("OPENAI_API_KEY environment variable not found")
```

## Connect to your Weaviate instance

In this section, we will:

1. test env variable `OPENAI_API_KEY` – **make sure** you completed the step in [#Prepare-your-OpenAI-API-key](#Prepare-your-OpenAI-API-key)
2. connect to your Weaviate your `OpenAI API Key`
3. and test the client connection

### The client 

After this step, the `client` object will be used to perform all Weaviate-related operations.


key: _node_content
string_value:
{"id_": "74e9b8af-46f1-4b36-8454-558d169afc8b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "bce57ee6c084b1c18fd858209606446bc92a66767f7298d29c766e10979f850d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8f3a000e-40f1-4432-9163-5c18d7691ef1", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "bf8b9b161fe6b2a17113d853f6e624d1bc0ac810ab2c6d06578e33bf8f2d754e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c94705cf-1e81-4e92-89ca-88cb163d7f23", "node_type": "1", "metadata": {}, "hash": "4664c256fb8dd63a9972c509f70d3cd812cdfe63596fe35694a1b91fc0f9707c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6813, "end_char_idx": 9333, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Install the Weaviate client for Python
!pip install weaviate-client>3.11.0

# Install datasets and apache-beam to load the sample datasets
!pip install datasets apache-beam
```

===========================================================
## Prepare your OpenAI API key

The `OpenAI API key` is used for vectorization of your data at import, and for queries.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it to your environment variables as `OPENAI_API_KEY`.


```python
# Export OpenAI API Key
!export OPENAI_API_KEY="your key"
```


```python
# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.
import os

# Note. alternatively you can set a temporary env variable like this:
# os.environ['OPENAI_API_KEY'] = 'your-key-goes-here'

if os.getenv("OPENAI_API_KEY") is not None:
    print ("OPENAI_API_KEY is ready")
else:
    print ("OPENAI_API_KEY environment variable not found")
```

## Connect to your Weaviate instance

In this section, we will:

1. test env variable `OPENAI_API_KEY` – **make sure** you completed the step in [#Prepare-your-OpenAI-API-key](#Prepare-your-OpenAI-API-key)
2. connect to your Weaviate your `OpenAI API Key`
3. and test the client connection

### The client 

After this step, the `client` object will be used to perform all Weaviate-related operations.


```python
import weaviate
from datasets import load_dataset
import os

# Connect to your Weaviate instance
client = weaviate.Client(
    url="https://your-wcs-instance-name.weaviate.network/",
#   url="http://localhost:8080/",
    auth_client_secret=weaviate.auth.AuthApiKey(api_key="<YOUR-WEAVIATE-API-KEY>"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)
    additional_headers={
        "X-OpenAI-Api-Key": os.getenv("OPENAI_API_KEY")
    }
)

# Check if your instance is live and ready
# This should return `True`
client.is_ready()
```

# Schema

In this section, we will:
1. configure the data schema for your data
2. select OpenAI module

> This is the second and final step, which requires OpenAI specific configuration.
> After this step, the rest of instructions wlll only touch on Weaviate, as the OpenAI tasks will be handled automatically.


key: _node_content
string_value:
{"id_": "c94705cf-1e81-4e92-89ca-88cb163d7f23", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "bce57ee6c084b1c18fd858209606446bc92a66767f7298d29c766e10979f850d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "74e9b8af-46f1-4b36-8454-558d169afc8b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "a6b8983f7dc01bd13c03b2a31bf9295113b93f58ce3e33fd07f89c8bd668cf9d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f2239543-22b7-40f1-b169-5bd658798e3d", "node_type": "1", "metadata": {}, "hash": "83c5fd2f64583cc6965b52341845d5b99c7756e036529a54b66f8efb651a89b7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8403, "end_char_idx": 9933, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import weaviate
from datasets import load_dataset
import os

# Connect to your Weaviate instance
client = weaviate.Client(
    url="https://your-wcs-instance-name.weaviate.network/",
#   url="http://localhost:8080/",
    auth_client_secret=weaviate.auth.AuthApiKey(api_key="<YOUR-WEAVIATE-API-KEY>"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)
    additional_headers={
        "X-OpenAI-Api-Key": os.getenv("OPENAI_API_KEY")
    }
)

# Check if your instance is live and ready
# This should return `True`
client.is_ready()
```

# Schema

In this section, we will:
1. configure the data schema for your data
2. select OpenAI module

> This is the second and final step, which requires OpenAI specific configuration.
> After this step, the rest of instructions wlll only touch on Weaviate, as the OpenAI tasks will be handled automatically.


## What is a schema

In Weaviate you create __schemas__ to capture each of the entities you will be searching.

A schema is how you tell Weaviate:
* what embedding model should be used to vectorize the data
* what your data is made of (property names and types)
* which properties should be vectorized and indexed

In this cookbook we will use a dataset for `Articles`, which contains:
* `title`
* `content`
* `url`

We want to vectorize `title` and `content`, but not the `url`.

To vectorize and query the data, we will use `text-embedding-3-small`. For Q&A we will use `gpt-3.5-turbo-instruct`.


key: _node_content
string_value:
{"id_": "f2239543-22b7-40f1-b169-5bd658798e3d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "bce57ee6c084b1c18fd858209606446bc92a66767f7298d29c766e10979f850d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c94705cf-1e81-4e92-89ca-88cb163d7f23", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "773ff49c660ab0d628bb6b1de03e0ea57fa40cc0cb27fc825ead0c010911a9f4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ed33eea6-7489-41cc-90f4-d09897bd5e07", "node_type": "1", "metadata": {}, "hash": "65c244219ebf03383e791121f75bcbdf3b72a8343eea35f0ee347031a44f596e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8403, "end_char_idx": 11642, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import weaviate
from datasets import load_dataset
import os

# Connect to your Weaviate instance
client = weaviate.Client(
    url="https://your-wcs-instance-name.weaviate.network/",
#   url="http://localhost:8080/",
    auth_client_secret=weaviate.auth.AuthApiKey(api_key="<YOUR-WEAVIATE-API-KEY>"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)
    additional_headers={
        "X-OpenAI-Api-Key": os.getenv("OPENAI_API_KEY")
    }
)

# Check if your instance is live and ready
# This should return `True`
client.is_ready()
```

# Schema

In this section, we will:
1. configure the data schema for your data
2. select OpenAI module

> This is the second and final step, which requires OpenAI specific configuration.
> After this step, the rest of instructions wlll only touch on Weaviate, as the OpenAI tasks will be handled automatically.


## What is a schema

In Weaviate you create __schemas__ to capture each of the entities you will be searching.

A schema is how you tell Weaviate:
* what embedding model should be used to vectorize the data
* what your data is made of (property names and types)
* which properties should be vectorized and indexed

In this cookbook we will use a dataset for `Articles`, which contains:
* `title`
* `content`
* `url`

We want to vectorize `title` and `content`, but not the `url`.

To vectorize and query the data, we will use `text-embedding-3-small`. For Q&A we will use `gpt-3.5-turbo-instruct`.


```python
# Clear up the schema, so that we can recreate it
client.schema.delete_all()
client.schema.get()

# Define the Schema object to use `text-embedding-3-small` on `title` and `content`, but skip it for `url`
article_schema = {
    "class": "Article",
    "description": "A collection of articles",
    "vectorizer": "text2vec-openai",
    "moduleConfig": {
        "text2vec-openai": {
          "model": "ada",
          "modelVersion": "002",
          "type": "text"
        }, 
        "qna-openai": {
          "model": "gpt-3.5-turbo-instruct",
          "maxTokens": 16,
          "temperature": 0.0,
          "topP": 1,
          "frequencyPenalty": 0.0,
          "presencePenalty": 0.0
        }
    },
    "properties": [{
        "name": "title",
        "description": "Title of the article",
        "dataType": ["string"]
    },
    {
        "name": "content",
        "description": "Contents of the article",
        "dataType": ["text"]
    },
    {
        "name": "url",
        "description": "URL to the article",
        "dataType": ["string"],
        "moduleConfig": { "text2vec-openai": { "skip": True } }
    }]
}

# add the Article schema
client.schema.create_class(article_schema)

# get the schema to make sure it worked
client.schema.get()
```

## Import data

In this section we will:
1. load the Simple Wikipedia dataset
2. configure Weaviate Batch import (to make the import more efficient)
3. import the data into Weaviate

> Note: <br/>
> Like mentioned before. We don't need to manually vectorize the data.<br/>
> The [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) module will take care of that.


key: _node_content
string_value:
{"id_": "ed33eea6-7489-41cc-90f4-d09897bd5e07", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "bce57ee6c084b1c18fd858209606446bc92a66767f7298d29c766e10979f850d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f2239543-22b7-40f1-b169-5bd658798e3d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "dacddca736878c2d75b3705cb8c716b21cfd1e2b72f4111785b1133db707b174", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c8a5b3d3-16f9-46ac-b1b3-8f72277d9083", "node_type": "1", "metadata": {}, "hash": "49887c2bf6e2a01509dc4d4ef70138967631275af3f7740204460169d0d90914", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11645, "end_char_idx": 14637, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
### STEP 1 - load the dataset

from datasets import load_dataset
from typing import List, Iterator

# We'll use the datasets library to pull the Simple Wikipedia dataset for embedding
dataset = list(load_dataset("wikipedia", "20220301.simple")["train"])

# For testing, limited to 2.5k articles for demo purposes
dataset = dataset[:2_500]

# Limited to 25k articles for larger demo purposes
# dataset = dataset[:25_000]

# for free OpenAI acounts, you can use 50 objects
# dataset = dataset[:50]
```


```python
### Step 2 - configure Weaviate Batch, with
# - starting batch size of 100
# - dynamically increase/decrease based on performance
# - add timeout retries if something goes wrong

client.batch.configure(
    batch_size=10, 
    dynamic=True,
    timeout_retries=3,
#   callback=None,
)
```


```python
### Step 3 - import data

print("Importing Articles")

counter=0

with client.batch as batch:
    for article in dataset:
        if (counter %10 == 0):
            print(f"Import {counter} / {len(dataset)} ")

        properties = {
            "title": article["title"],
            "content": article["text"],
            "url": article["url"]
        }
        
        batch.add_data_object(properties, "Article")
        counter = counter+1

print("Importing Articles complete")
```


```python
# Test that all data has loaded – get object count
result = (
    client.query.aggregate("Article")
    .with_fields("meta { count }")
    .do()
)
print("Object count: ", result["data"]["Aggregate"]["Article"], "\n")
```


```python
# Test one article has worked by checking one object
test_article = (
    client.query
    .get("Article", ["title", "url", "content"])
    .with_limit(1)
    .do()
)["data"]["Get"]["Article"][0]

print(test_article['title'])
print(test_article['url'])
print(test_article['content'])
```

### Question Answering on the Data

As above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors


```python
def qna(query, collection_name):
    
    properties = [
        "title", "content", "url",
        "_additional { answer { hasAnswer property result startPosition endPosition } distance }"
    ]

    ask = {
        "question": query,
        "properties": ["content"]
    }

    result = (
        client.query
        .get(collection_name, properties)
        .with_ask(ask)
        .with_limit(1)
        .do()
    )
    
    # Check for errors
    if ("errors" in result):
        print ("\033[91mYou probably have run out of OpenAI API calls for the current minute – the limit is set at 60 per minute.")
        raise Exception(result["errors"][0]['message'])
    
    return result["data"]["Get"][collection_name]
```


```python
query_result = qna("Did Alanis Morissette win a Grammy?", "Article")

for i, article in enumerate(query_result):
    print(f"{i+1}. { article['_additional']['answer']['result']} (Distance: {round(article['_additional']['distance'],3) })")
```


key: _node_content
string_value:
{"id_": "c8a5b3d3-16f9-46ac-b1b3-8f72277d9083", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "bce57ee6c084b1c18fd858209606446bc92a66767f7298d29c766e10979f850d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ed33eea6-7489-41cc-90f4-d09897bd5e07", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "796eda1b591e267814ba27f72c0e01d35743b5d081e756ea8a3351c3cc088b7b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13192, "end_char_idx": 15233, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Test one article has worked by checking one object
test_article = (
    client.query
    .get("Article", ["title", "url", "content"])
    .with_limit(1)
    .do()
)["data"]["Get"]["Article"][0]

print(test_article['title'])
print(test_article['url'])
print(test_article['content'])
```

### Question Answering on the Data

As above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors


```python
def qna(query, collection_name):
    
    properties = [
        "title", "content", "url",
        "_additional { answer { hasAnswer property result startPosition endPosition } distance }"
    ]

    ask = {
        "question": query,
        "properties": ["content"]
    }

    result = (
        client.query
        .get(collection_name, properties)
        .with_ask(ask)
        .with_limit(1)
        .do()
    )
    
    # Check for errors
    if ("errors" in result):
        print ("\033[91mYou probably have run out of OpenAI API calls for the current minute – the limit is set at 60 per minute.")
        raise Exception(result["errors"][0]['message'])
    
    return result["data"]["Get"][collection_name]
```


```python
query_result = qna("Did Alanis Morissette win a Grammy?", "Article")

for i, article in enumerate(query_result):
    print(f"{i+1}. { article['_additional']['answer']['result']} (Distance: {round(article['_additional']['distance'],3) })")
```


```python
query_result = qna("What is the capital of China?", "Article")

for i, article in enumerate(query_result):
    if article['_additional']['answer']['hasAnswer'] == False:
      print('No answer found')
    else:
      print(f"{i+1}. { article['_additional']['answer']['result']} (Distance: {round(article['_additional']['distance'],3) })")
```

Thanks for following along, you're now equipped to set up your own vector databases and use embeddings to do all kinds of cool things - enjoy! For more complex use cases please continue to work through other cookbook examples in this repo.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb


key: _node_content
string_value:
{"id_": "bfd154fd-57cd-4620-a061-bdeb1a748be8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb", "title": "Question Answering with LangChain, Deep Lake, & OpenAI", "tokens": 1929, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bc204eab-7084-4001-939c-9b84cefb0dd4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb", "title": "Question Answering with LangChain, Deep Lake, & OpenAI", "tokens": 1929, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "76569fcab9e7e34ca4979f5fbb29e72dcc69ff53c162fcfcd9d7425dcfe74460", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a15b03dd-1415-4c38-b30a-8e8e5e540663", "node_type": "1", "metadata": {}, "hash": "665fc0e191604fb24e531a9081bb499ee93c0dfe8882853d9cca318743c8a157", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2582, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Question Answering with LangChain, Deep Lake, & OpenAI

This notebook shows how to implement a question answering system with LangChain, [Deep Lake](https://activeloop.ai/) as a vector store and OpenAI embeddings. We will take the following steps to achieve this:

1. Load a Deep Lake text dataset
2. Initialize a [Deep Lake vector store with LangChain](https://docs.activeloop.ai/tutorials/vector-store/deep-lake-vector-store-in-langchain)
3. Add text to the vector store
4. Run queries on the database
5. Done!

You can also follow other tutorials such as question answering over any type of data (PDFs, json, csv, text): [chatting with any data](https://www.activeloop.ai/resources/data-chad-an-ai-app-with-lang-chain-deep-lake-to-chat-with-any-data/) stored in Deep Lake, [code understanding](https://www.activeloop.ai/resources/lang-chain-gpt-4-for-code-understanding-twitter-algorithm/), or [question answering over PDFs](https://www.activeloop.ai/resources/ultimate-guide-to-lang-chain-deep-lake-build-chat-gpt-to-answer-questions-on-your-financial-data/), or [recommending songs](https://www.activeloop.ai/resources/3-ways-to-build-a-recommendation-engine-for-songs-with-lang-chain/).

## Install requirements
Let's install the following packages.


```python
!pip install deeplake langchain openai tiktoken
```

## Authentication
Provide your OpenAI API key here:


```python
import getpass
import os

os.environ['OPENAI_API_KEY'] = getpass.getpass()
```

    ··········


## Load a Deep Lake text dataset
We will use a 20000 sample subset of the [cohere-wikipedia-22](https://app.activeloop.ai/davitbun/cohere-wikipedia-22) dataset for this example.


```python
import deeplake

ds = deeplake.load("hub://activeloop/cohere-wikipedia-22-sample")
ds.summary()
```

    \

    Opening dataset in read-only mode as you don't have write permissions.


    -

    This dataset can be visualized in Jupyter Notebook by ds.visualize() or at https://app.activeloop.ai/activeloop/cohere-wikipedia-22-sample
    


    |

    hub://activeloop/cohere-wikipedia-22-sample loaded successfully.
    
    Dataset(path='hub://activeloop/cohere-wikipedia-22-sample', read_only=True, tensors=['ids', 'metadata', 'text'])
    
      tensor    htype     shape      dtype  compression
     -------   -------   -------    -------  ------- 
       ids      text    (20000, 1)    str     None   
     metadata   json    (20000, 1)    str     None   
       text     text    (20000, 1)    str     None   


    
 



Let's take a look at a few samples:


```python
ds[:3].text.data()["value"]
```


key: _node_content
string_value:
{"id_": "a15b03dd-1415-4c38-b30a-8e8e5e540663", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb", "title": "Question Answering with LangChain, Deep Lake, & OpenAI", "tokens": 1929, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bc204eab-7084-4001-939c-9b84cefb0dd4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb", "title": "Question Answering with LangChain, Deep Lake, & OpenAI", "tokens": 1929, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "76569fcab9e7e34ca4979f5fbb29e72dcc69ff53c162fcfcd9d7425dcfe74460", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bfd154fd-57cd-4620-a061-bdeb1a748be8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb", "title": "Question Answering with LangChain, Deep Lake, & OpenAI", "tokens": 1929, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c6b5c44e116c47d8b67e904d4c15a1535a18a62061fb9321ba2c5fca966d3280", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6192ef69-2550-4bbe-baa2-c6c51f35a891", "node_type": "1", "metadata": {}, "hash": "1e87b575b3af86e056ade500aac8a05d17c75fc87052e2cc872f7722a79b02aa", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1260, "end_char_idx": 4053, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
!pip install deeplake langchain openai tiktoken
```

## Authentication
Provide your OpenAI API key here:


```python
import getpass
import os

os.environ['OPENAI_API_KEY'] = getpass.getpass()
```

    ··········


## Load a Deep Lake text dataset
We will use a 20000 sample subset of the [cohere-wikipedia-22](https://app.activeloop.ai/davitbun/cohere-wikipedia-22) dataset for this example.


```python
import deeplake

ds = deeplake.load("hub://activeloop/cohere-wikipedia-22-sample")
ds.summary()
```

    \

    Opening dataset in read-only mode as you don't have write permissions.


    -

    This dataset can be visualized in Jupyter Notebook by ds.visualize() or at https://app.activeloop.ai/activeloop/cohere-wikipedia-22-sample
    


    |

    hub://activeloop/cohere-wikipedia-22-sample loaded successfully.
    
    Dataset(path='hub://activeloop/cohere-wikipedia-22-sample', read_only=True, tensors=['ids', 'metadata', 'text'])
    
      tensor    htype     shape      dtype  compression
     -------   -------   -------    -------  ------- 
       ids      text    (20000, 1)    str     None   
     metadata   json    (20000, 1)    str     None   
       text     text    (20000, 1)    str     None   


    
 



Let's take a look at a few samples:


```python
ds[:3].text.data()["value"]
```




    ['The 24-hour clock is a way of telling the time in which the day runs from midnight to midnight and is divided into 24 hours, numbered from 0 to 23. It does not use a.m. or p.m. This system is also referred to (only in the US and the English speaking parts of Canada) as military time or (only in the United Kingdom and now very rarely) as continental time. In some parts of the world, it is called railway time. Also, the international standard notation of time (ISO 8601) is based on this format.',
     'A time in the 24-hour clock is written in the form hours:minutes (for example, 01:23), or hours:minutes:seconds (01:23:45). Numbers under 10 have a zero in front (called a leading zero); e.g. 09:07. Under the 24-hour clock system, the day begins at midnight, 00:00, and the last minute of the day begins at 23:59 and ends at 24:00, which is identical to 00:00 of the following day. 12:00 can only be mid-day. Midnight is called 24:00 and is used to mean the end of the day and 00:00 is used to mean the beginning of the day. For example, you would say "Tuesday at 24:00" and "Wednesday at 00:00" to mean exactly the same time.',
     'However, the US military prefers not to say 24:00 - they do not like to have two names for the same thing, so they always say "23:59", which is one minute before midnight.']



## LangChain's Deep Lake vector store
Let's define a `dataset_path`, this is where your Deep Lake vector store will house the text embeddings.


key: _node_content
string_value:
{"id_": "6192ef69-2550-4bbe-baa2-c6c51f35a891", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb", "title": "Question Answering with LangChain, Deep Lake, & OpenAI", "tokens": 1929, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bc204eab-7084-4001-939c-9b84cefb0dd4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb", "title": "Question Answering with LangChain, Deep Lake, & OpenAI", "tokens": 1929, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "76569fcab9e7e34ca4979f5fbb29e72dcc69ff53c162fcfcd9d7425dcfe74460", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a15b03dd-1415-4c38-b30a-8e8e5e540663", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb", "title": "Question Answering with LangChain, Deep Lake, & OpenAI", "tokens": 1929, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "147446e28dc2dda934924343163481d233b5ff132967a3a9c058449cd6eeb261", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "92c20786-3480-4fc5-9b05-4d935e29e4a5", "node_type": "1", "metadata": {}, "hash": "4bdf07fab2d3c01768d1becc4b92455275dd6dfcf47ea16a1d27faff9254bc11", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2591, "end_char_idx": 5106, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
['The 24-hour clock is a way of telling the time in which the day runs from midnight to midnight and is divided into 24 hours, numbered from 0 to 23. It does not use a.m. or p.m. This system is also referred to (only in the US and the English speaking parts of Canada) as military time or (only in the United Kingdom and now very rarely) as continental time. In some parts of the world, it is called railway time. Also, the international standard notation of time (ISO 8601) is based on this format.',
     'A time in the 24-hour clock is written in the form hours:minutes (for example, 01:23), or hours:minutes:seconds (01:23:45). Numbers under 10 have a zero in front (called a leading zero); e.g. 09:07. Under the 24-hour clock system, the day begins at midnight, 00:00, and the last minute of the day begins at 23:59 and ends at 24:00, which is identical to 00:00 of the following day. 12:00 can only be mid-day. Midnight is called 24:00 and is used to mean the end of the day and 00:00 is used to mean the beginning of the day. For example, you would say "Tuesday at 24:00" and "Wednesday at 00:00" to mean exactly the same time.',
     'However, the US military prefers not to say 24:00 - they do not like to have two names for the same thing, so they always say "23:59", which is one minute before midnight.']



## LangChain's Deep Lake vector store
Let's define a `dataset_path`, this is where your Deep Lake vector store will house the text embeddings.


```python
dataset_path = 'wikipedia-embeddings-deeplake'
```

We will setup OpenAI's `text-embedding-3-small` as our embedding function and initialize a Deep Lake vector store at `dataset_path`...


```python
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import DeepLake

embedding = OpenAIEmbeddings(model="text-embedding-3-small")
db = DeepLake(dataset_path, embedding=embedding, overwrite=True)
```

    




... and populate it with samples, one batch at a time, using the `add_texts` method.


```python
from tqdm.auto import tqdm

batch_size = 100

nsamples = 10  # for testing. Replace with len(ds) to append everything
for i in tqdm(range(0, nsamples, batch_size)):
    # find end of batch
    i_end = min(nsamples, i + batch_size)

    batch = ds[i:i_end]
    id_batch = batch.ids.data()["value"]
    text_batch = batch.text.data()["value"]
    meta_batch = batch.metadata.data()["value"]

    db.add_texts(text_batch, metadatas=meta_batch, ids=id_batch)
```


      0%|          | 0/1 [00:00<?, ?it/s]


key: _node_content
string_value:
{"id_": "92c20786-3480-4fc5-9b05-4d935e29e4a5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb", "title": "Question Answering with LangChain, Deep Lake, & OpenAI", "tokens": 1929, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bc204eab-7084-4001-939c-9b84cefb0dd4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb", "title": "Question Answering with LangChain, Deep Lake, & OpenAI", "tokens": 1929, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "76569fcab9e7e34ca4979f5fbb29e72dcc69ff53c162fcfcd9d7425dcfe74460", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6192ef69-2550-4bbe-baa2-c6c51f35a891", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb", "title": "Question Answering with LangChain, Deep Lake, & OpenAI", "tokens": 1929, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "107c6c0001137429fe98d5b765f46665cbfaf9bab4b331b7d655cd75d3d63d18", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dbfaa266-9e53-47a2-b7d4-aa27de380d2b", "node_type": "1", "metadata": {}, "hash": "01a916c1e98d52a1067a32ef516e8f4040ebcc2d303042aa5d6c85f7c03a37c3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3911, "end_char_idx": 6573, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## LangChain's Deep Lake vector store
Let's define a `dataset_path`, this is where your Deep Lake vector store will house the text embeddings.


```python
dataset_path = 'wikipedia-embeddings-deeplake'
```

We will setup OpenAI's `text-embedding-3-small` as our embedding function and initialize a Deep Lake vector store at `dataset_path`...


```python
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import DeepLake

embedding = OpenAIEmbeddings(model="text-embedding-3-small")
db = DeepLake(dataset_path, embedding=embedding, overwrite=True)
```

    




... and populate it with samples, one batch at a time, using the `add_texts` method.


```python
from tqdm.auto import tqdm

batch_size = 100

nsamples = 10  # for testing. Replace with len(ds) to append everything
for i in tqdm(range(0, nsamples, batch_size)):
    # find end of batch
    i_end = min(nsamples, i + batch_size)

    batch = ds[i:i_end]
    id_batch = batch.ids.data()["value"]
    text_batch = batch.text.data()["value"]
    meta_batch = batch.metadata.data()["value"]

    db.add_texts(text_batch, metadatas=meta_batch, ids=id_batch)
```


      0%|          | 0/1 [00:00<?, ?it/s]


    
    creating embeddings:   0%|          | 0/1 [00:00<?, ?it/s][A
    creating embeddings: 100%|██████████| 1/1 [00:02<00:00,  2.11s/it]
    
    100%|██████████| 10/10 [00:00<00:00, 462.42it/s]

    Dataset(path='wikipedia-embeddings-deeplake', tensors=['text', 'metadata', 'embedding', 'id'])
    
      tensor      htype      shape      dtype  compression
      -------    -------    -------    -------  ------- 
       text       text      (10, 1)      str     None   
     metadata     json      (10, 1)      str     None   
     embedding  embedding  (10, 1536)  float32   None   
        id        text      (10, 1)      str     None   


    
    


## Run user queries on the database
The underlying Deep Lake dataset object is accessible through `db.vectorstore.dataset`, and the data structure can be summarized using `db.vectorstore.summary()`, which shows 4 tensors with 10 samples:


```python
db.vectorstore.summary()
```

    Dataset(path='wikipedia-embeddings-deeplake', tensors=['text', 'metadata', 'embedding', 'id'])
    
      tensor      htype      shape      dtype  compression
      -------    -------    -------    -------  ------- 
       text       text      (10, 1)      str     None   
     metadata     json      (10, 1)      str     None   
     embedding  embedding  (10, 1536)  float32   None   
        id        text      (10, 1)      str     None   


We will now setup QA on our vector store with GPT-3.5-Turbo as our LLM.


key: _node_content
string_value:
{"id_": "dbfaa266-9e53-47a2-b7d4-aa27de380d2b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb", "title": "Question Answering with LangChain, Deep Lake, & OpenAI", "tokens": 1929, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "bc204eab-7084-4001-939c-9b84cefb0dd4", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb", "title": "Question Answering with LangChain, Deep Lake, & OpenAI", "tokens": 1929, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "76569fcab9e7e34ca4979f5fbb29e72dcc69ff53c162fcfcd9d7425dcfe74460", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "92c20786-3480-4fc5-9b05-4d935e29e4a5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb", "title": "Question Answering with LangChain, Deep Lake, & OpenAI", "tokens": 1929, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "ba10dff9b081603c3aa4b69416f0c209fcd3c5f478ba9bed731a0a2d72de6b9c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5772, "end_char_idx": 7368, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Run user queries on the database
The underlying Deep Lake dataset object is accessible through `db.vectorstore.dataset`, and the data structure can be summarized using `db.vectorstore.summary()`, which shows 4 tensors with 10 samples:


```python
db.vectorstore.summary()
```

    Dataset(path='wikipedia-embeddings-deeplake', tensors=['text', 'metadata', 'embedding', 'id'])
    
      tensor      htype      shape      dtype  compression
      -------    -------    -------    -------  ------- 
       text       text      (10, 1)      str     None   
     metadata     json      (10, 1)      str     None   
     embedding  embedding  (10, 1536)  float32   None   
        id        text      (10, 1)      str     None   


We will now setup QA on our vector store with GPT-3.5-Turbo as our LLM.


```python
from langchain.chains import RetrievalQA
from langchain.chat_models import ChatOpenAI

# Re-load the vector store in case it's no longer initialized
# db = DeepLake(dataset_path = dataset_path, embedding_function=embedding)

qa = RetrievalQA.from_chain_type(llm=ChatOpenAI(model='gpt-3.5-turbo'), chain_type="stuff", retriever=db.as_retriever())
```

Let's try running a prompt and check the output. Internally, this API performs an embedding search to find the most relevant data to feed into the LLM context.


```python
query = 'Why does the military not say 24:00?'
qa.run(query)
```




    'The military prefers not to say 24:00 because they do not like to have two names for the same thing. Instead, they always say "23:59", which is one minute before midnight.'



Et voila!


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb


key: _node_content
string_value:
{"id_": "ae74189b-1e3a-4599-baa2-4a7d8145b8c5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "deeef3e2-06ef-4a06-ba05-093d97fc8631", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "580e111f11b68f838408ecf45ea83ca8276666ddf4be43d83f7cce79599e13d7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "77f12164-55f8-48ec-b2d9-d9b13f8b2c6a", "node_type": "1", "metadata": {}, "hash": "4499fe894d164c59e09bbf19662d344b0a123d2c8a08848b3eb524c62699e0f4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3394, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Using Tair as a vector database for OpenAI embeddings

This notebook guides you step by step on using Tair as a vector database for OpenAI embeddings.

This notebook presents an end-to-end process of:
1. Using precomputed embeddings created by OpenAI API.
2. Storing the embeddings in a cloud instance of Tair.
3. Converting raw text query to an embedding with OpenAI API.
4. Using Tair to perform the nearest neighbour search in the created collection.

### What is Tair

[Tair](https://www.alibabacloud.com/help/en/tair/latest/what-is-tair) is a cloud native in-memory database service that is developed by Alibaba Cloud. Tair is compatible with open source Redis and provides a variety of data models and enterprise-class capabilities to support your real-time online scenarios. Tair also introduces persistent memory-optimized instances that are based on the new non-volatile memory (NVM) storage medium. These instances can reduce costs by 30%, ensure data persistence, and provide almost the same performance as in-memory databases. Tair has been widely used in areas such as government affairs, finance, manufacturing, healthcare, and pan-Internet to meet their high-speed query and computing requirements.

[Tairvector](https://www.alibabacloud.com/help/en/tair/latest/tairvector) is an in-house data structure that provides high-performance real-time storage and retrieval of vectors. TairVector provides two indexing algorithms: Hierarchical Navigable Small World (HNSW) and Flat Search. Additionally, TairVector supports multiple distance functions, such as Euclidean distance, inner product, and Jaccard distance. Compared with traditional vector retrieval services, TairVector has the following advantages:
- Stores all data in memory and supports real-time index updates to reduce latency of read and write operations.
- Uses an optimized data structure in memory to better utilize storage capacity.
- Functions as an out-of-the-box data structure in a simple and efficient architecture without complex modules or dependencies.

### Deployment options

- Using [Tair Cloud Vector Database](https://www.alibabacloud.com/help/en/tair/latest/getting-started-overview). [Click here](https://www.alibabacloud.com/product/tair) to fast deploy it.


## Prerequisites

For the purposes of this exercise we need to prepare a couple of things:

1. Tair cloud server instance.
2. The 'tair' library to interact with the tair database.
3. An [OpenAI API key](https://beta.openai.com/account/api-keys).



### Install requirements

This notebook obviously requires the `openai` and `tair` packages, but there are also some other additional libraries we will use. The following command installs them all:



```python
! pip install openai redis tair pandas wget
```

    Looking in indexes: http://sg.mirrors.cloud.aliyuncs.com/pypi/simple/
    Requirement already satisfied: openai in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (0.28.0)
    Requirement already satisfied: redis in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (5.0.0)
    Requirement already satisfied: tair in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (1.3.6)
    Requirement already satisfied: pandas in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (2.1.0)
    Requirement already satisfied: wget in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (3.


key: _node_content
string_value:
{"id_": "77f12164-55f8-48ec-b2d9-d9b13f8b2c6a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "deeef3e2-06ef-4a06-ba05-093d97fc8631", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "580e111f11b68f838408ecf45ea83ca8276666ddf4be43d83f7cce79599e13d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ae74189b-1e3a-4599-baa2-4a7d8145b8c5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "351e4ac21b31a6f4a0aa90aadf605c3864f56eba6da0b338f6f74d1be5d79e84", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8a9278c7-cc77-4f89-9a92-6fcdec21ba83", "node_type": "1", "metadata": {}, "hash": "5bc63a6e595fbfccde05b3ab22b229885e126f38ab6efc199ae96ecd07995818", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2259, "end_char_idx": 4879, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Prerequisites

For the purposes of this exercise we need to prepare a couple of things:

1. Tair cloud server instance.
2. The 'tair' library to interact with the tair database.
3. An [OpenAI API key](https://beta.openai.com/account/api-keys).



### Install requirements

This notebook obviously requires the `openai` and `tair` packages, but there are also some other additional libraries we will use. The following command installs them all:



```python
! pip install openai redis tair pandas wget
```

    Looking in indexes: http://sg.mirrors.cloud.aliyuncs.com/pypi/simple/
    Requirement already satisfied: openai in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (0.28.0)
    Requirement already satisfied: redis in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (5.0.0)
    Requirement already satisfied: tair in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (1.3.6)
    Requirement already satisfied: pandas in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (2.1.0)
    Requirement already satisfied: wget in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (3.2)
    Requirement already satisfied: requests>=2.20 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from openai) (2.31.0)
    Requirement already satisfied: tqdm in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from openai) (4.66.1)
    Requirement already satisfied: aiohttp in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from openai) (3.8.5)
    Requirement already satisfied: async-timeout>=4.0.2 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from redis) (4.0.3)
    Requirement already satisfied: numpy>=1.22.4 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from pandas) (1.25.2)
    Requirement already satisfied: python-dateutil>=2.8.2 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from pandas) (2.8.2)
    Requirement already satisfied: pytz>=2020.1 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from pandas) (2023.3.post1)
    Requirement already satisfied: tzdata>=2022.1 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from pandas) (2023.3)
    Requirement already satisfied: six>=1.5 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)
    Requirement already satisfied: charset-normalizer<4,>=2 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from requests>=2.20->openai) (3.2.0)
    Requirement already satisfied: idna<4,>=2.5 in /root/anaconda3/envs/notebook/lib/python3.


key: _node_content
string_value:
{"id_": "8a9278c7-cc77-4f89-9a92-6fcdec21ba83", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "deeef3e2-06ef-4a06-ba05-093d97fc8631", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "580e111f11b68f838408ecf45ea83ca8276666ddf4be43d83f7cce79599e13d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "77f12164-55f8-48ec-b2d9-d9b13f8b2c6a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4b530a22cdca393a4850c28b9b4a7c91309a33c63e72f41de9b665159e23ccf1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c6f5245b-0add-4ad5-b028-9f8d93eecc75", "node_type": "1", "metadata": {}, "hash": "47052296c660883ef719340b963d733c780c9a9f671ff52b75566c53081a9126", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3617, "end_char_idx": 5966, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
10/site-packages (from openai) (4.66.1)
    Requirement already satisfied: aiohttp in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from openai) (3.8.5)
    Requirement already satisfied: async-timeout>=4.0.2 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from redis) (4.0.3)
    Requirement already satisfied: numpy>=1.22.4 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from pandas) (1.25.2)
    Requirement already satisfied: python-dateutil>=2.8.2 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from pandas) (2.8.2)
    Requirement already satisfied: pytz>=2020.1 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from pandas) (2023.3.post1)
    Requirement already satisfied: tzdata>=2022.1 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from pandas) (2023.3)
    Requirement already satisfied: six>=1.5 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from python-dateutil>=2.8.2->pandas) (1.16.0)
    Requirement already satisfied: charset-normalizer<4,>=2 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from requests>=2.20->openai) (3.2.0)
    Requirement already satisfied: idna<4,>=2.5 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from requests>=2.20->openai) (3.4)
    Requirement already satisfied: urllib3<3,>=1.21.1 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from requests>=2.20->openai) (2.0.4)
    Requirement already satisfied: certifi>=2017.4.17 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from requests>=2.20->openai) (2023.7.22)
    Requirement already satisfied: attrs>=17.3.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (22.1.0)
    Requirement already satisfied: multidict<7.0,>=4.5 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (6.0.4)
    Requirement already satisfied: yarl<2.0,>=1.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (1.9.2)
    Requirement already satisfied: frozenlist>=1.1.1 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (1.4.0)
    Requirement already satisfied: aiosignal>=1.1.2 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (1.3.


key: _node_content
string_value:
{"id_": "c6f5245b-0add-4ad5-b028-9f8d93eecc75", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "deeef3e2-06ef-4a06-ba05-093d97fc8631", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "580e111f11b68f838408ecf45ea83ca8276666ddf4be43d83f7cce79599e13d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8a9278c7-cc77-4f89-9a92-6fcdec21ba83", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "b38f9925bd12c9a978e97c634d93404d0eca19422f7d99f8c1d60fd6d90446ea", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f6be1629-9c3b-4d17-bd01-3c5420b4893b", "node_type": "1", "metadata": {}, "hash": "ddecbe1e8266fa8287858488e4cd9d0c0e36a1889069b505410949e3a7b967b2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4781, "end_char_idx": 7228, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
2.0)
    Requirement already satisfied: idna<4,>=2.5 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from requests>=2.20->openai) (3.4)
    Requirement already satisfied: urllib3<3,>=1.21.1 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from requests>=2.20->openai) (2.0.4)
    Requirement already satisfied: certifi>=2017.4.17 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from requests>=2.20->openai) (2023.7.22)
    Requirement already satisfied: attrs>=17.3.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (22.1.0)
    Requirement already satisfied: multidict<7.0,>=4.5 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (6.0.4)
    Requirement already satisfied: yarl<2.0,>=1.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (1.9.2)
    Requirement already satisfied: frozenlist>=1.1.1 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (1.4.0)
    Requirement already satisfied: aiosignal>=1.1.2 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (1.3.1)
    [33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m[33m
    [0m

### Prepare your OpenAI API key

The OpenAI API key is used for vectorization of the documents and queries.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it by getpass.


```python
import getpass
import openai

openai.api_key = getpass.getpass("Input your OpenAI API key:")
```

    Input your OpenAI API key:········


## Connect to Tair
First add it to your environment variables.

Connecting to a running instance of Tair server is easy with the official Python library.


```python
# The format of url: redis://[[username]:[password]]@localhost:6379/0
TAIR_URL = getpass.getpass("Input your tair url:")
```

    Input your tair url:········



```python
from tair import Tair as TairClient

# connect to tair from url and create a client

url = TAIR_URL
client = TairClient.from_url(url)
```

We can test the connection by ping:


```python
client.ping()
```




    True


key: _node_content
string_value:
{"id_": "f6be1629-9c3b-4d17-bd01-3c5420b4893b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "deeef3e2-06ef-4a06-ba05-093d97fc8631", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "580e111f11b68f838408ecf45ea83ca8276666ddf4be43d83f7cce79599e13d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c6f5245b-0add-4ad5-b028-9f8d93eecc75", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "521beac131894795033cb9954e8d80597ef89cae44aec7f76ee49d365546bebe", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1cfa168c-5d70-4892-804d-42c7374aee67", "node_type": "1", "metadata": {}, "hash": "7861b717487a28cfd6c218373ce41b27783c7a079b179d696215139b7a306359", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5776, "end_char_idx": 8801, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
10/site-packages (from aiohttp->openai) (1.4.0)
    Requirement already satisfied: aiosignal>=1.1.2 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (1.3.1)
    [33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m[33m
    [0m

### Prepare your OpenAI API key

The OpenAI API key is used for vectorization of the documents and queries.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it by getpass.


```python
import getpass
import openai

openai.api_key = getpass.getpass("Input your OpenAI API key:")
```

    Input your OpenAI API key:········


## Connect to Tair
First add it to your environment variables.

Connecting to a running instance of Tair server is easy with the official Python library.


```python
# The format of url: redis://[[username]:[password]]@localhost:6379/0
TAIR_URL = getpass.getpass("Input your tair url:")
```

    Input your tair url:········



```python
from tair import Tair as TairClient

# connect to tair from url and create a client

url = TAIR_URL
client = TairClient.from_url(url)
```

We can test the connection by ping:


```python
client.ping()
```




    True




```python
import wget

embeddings_url = "https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip"

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```

    100% [......................................................................] 698933052 / 698933052




    'vector_database_wikipedia_articles_embedded (1).zip'



The downloaded file has to then be extracted:


```python
import zipfile
import os
import re
import tempfile

current_directory = os.getcwd()
zip_file_path = os.path.join(current_directory, "vector_database_wikipedia_articles_embedded.zip")
output_directory = os.path.join(current_directory, "../../data")

with zipfile.ZipFile(zip_file_path, "r") as zip_ref:
    zip_ref.extractall(output_directory)


# check the csv file exist
file_name = "vector_database_wikipedia_articles_embedded.csv"
data_directory = os.path.join(current_directory, "../../data")
file_path = os.path.join(data_directory, file_name)


if os.path.exists(file_path):
    print(f"The file {file_name} exists in the data directory.")
else:
    print(f"The file {file_name} does not exist in the data directory.")

```

    The file vector_database_wikipedia_articles_embedded.csv exists in the data directory.


## Create Index

Tair stores data in indexes where each object is described by one key. Each key contains a vector and multiple attribute_keys.

We will start with creating two indexes, one for **title_vector** and one for **content_vector**, and then we will fill it with our precomputed embeddings.


key: _node_content
string_value:
{"id_": "1cfa168c-5d70-4892-804d-42c7374aee67", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "deeef3e2-06ef-4a06-ba05-093d97fc8631", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "580e111f11b68f838408ecf45ea83ca8276666ddf4be43d83f7cce79599e13d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f6be1629-9c3b-4d17-bd01-3c5420b4893b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "6b2dbdb060f547e026d93424d7b694941ff690c44607e05f24990c3e6737fe9c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "58c68be5-a64b-4177-90a6-c1cbc3ed51b5", "node_type": "1", "metadata": {}, "hash": "69f9fcc9835b4402090cf9c0a310a10f1878c514f15472471755c1357d4f6973", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7188, "end_char_idx": 9690, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
client.ping()
```




    True




```python
import wget

embeddings_url = "https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip"

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```

    100% [......................................................................] 698933052 / 698933052




    'vector_database_wikipedia_articles_embedded (1).zip'



The downloaded file has to then be extracted:


```python
import zipfile
import os
import re
import tempfile

current_directory = os.getcwd()
zip_file_path = os.path.join(current_directory, "vector_database_wikipedia_articles_embedded.zip")
output_directory = os.path.join(current_directory, "../../data")

with zipfile.ZipFile(zip_file_path, "r") as zip_ref:
    zip_ref.extractall(output_directory)


# check the csv file exist
file_name = "vector_database_wikipedia_articles_embedded.csv"
data_directory = os.path.join(current_directory, "../../data")
file_path = os.path.join(data_directory, file_name)


if os.path.exists(file_path):
    print(f"The file {file_name} exists in the data directory.")
else:
    print(f"The file {file_name} does not exist in the data directory.")

```

    The file vector_database_wikipedia_articles_embedded.csv exists in the data directory.


## Create Index

Tair stores data in indexes where each object is described by one key. Each key contains a vector and multiple attribute_keys.

We will start with creating two indexes, one for **title_vector** and one for **content_vector**, and then we will fill it with our precomputed embeddings.


```python
# set index parameters
index = "openai_test"
embedding_dim = 1536
distance_type = "L2"
index_type = "HNSW"
data_type = "FLOAT32"

# Create two indexes, one for title_vector and one for content_vector, skip if already exists
index_names = [index + "_title_vector", index+"_content_vector"]
for index_name in index_names:
    index_connection = client.tvs_get_index(index_name)
    if index_connection is not None:
        print("Index already exists")
    else:
        client.tvs_create_index(name=index_name, dim=embedding_dim, distance_type=distance_type,
                                index_type=index_type, data_type=data_type)
```

    Index already exists
    Index already exists


## Load data

In this section we are going to load the data prepared previous to this session, so you don't have to recompute the embeddings of Wikipedia articles with your own credits.


key: _node_content
string_value:
{"id_": "58c68be5-a64b-4177-90a6-c1cbc3ed51b5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "deeef3e2-06ef-4a06-ba05-093d97fc8631", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "580e111f11b68f838408ecf45ea83ca8276666ddf4be43d83f7cce79599e13d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1cfa168c-5d70-4892-804d-42c7374aee67", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "f669c48c506a41837d4dee8ccee89adfe03306655685e321542cc7934a36d887", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "eec43b51-6986-4111-b4bb-e8b0e4b8f293", "node_type": "1", "metadata": {}, "hash": "64efa858381f9586943366ef6a5e928a5863d0d445c19416a169a67bdfbc43fc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8022, "end_char_idx": 11104, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# check the csv file exist
file_name = "vector_database_wikipedia_articles_embedded.csv"
data_directory = os.path.join(current_directory, "../../data")
file_path = os.path.join(data_directory, file_name)


if os.path.exists(file_path):
    print(f"The file {file_name} exists in the data directory.")
else:
    print(f"The file {file_name} does not exist in the data directory.")

```

    The file vector_database_wikipedia_articles_embedded.csv exists in the data directory.


## Create Index

Tair stores data in indexes where each object is described by one key. Each key contains a vector and multiple attribute_keys.

We will start with creating two indexes, one for **title_vector** and one for **content_vector**, and then we will fill it with our precomputed embeddings.


```python
# set index parameters
index = "openai_test"
embedding_dim = 1536
distance_type = "L2"
index_type = "HNSW"
data_type = "FLOAT32"

# Create two indexes, one for title_vector and one for content_vector, skip if already exists
index_names = [index + "_title_vector", index+"_content_vector"]
for index_name in index_names:
    index_connection = client.tvs_get_index(index_name)
    if index_connection is not None:
        print("Index already exists")
    else:
        client.tvs_create_index(name=index_name, dim=embedding_dim, distance_type=distance_type,
                                index_type=index_type, data_type=data_type)
```

    Index already exists
    Index already exists


## Load data

In this section we are going to load the data prepared previous to this session, so you don't have to recompute the embeddings of Wikipedia articles with your own credits.


```python
import pandas as pd
from ast import literal_eval
# Path to your local CSV file
csv_file_path = '../../data/vector_database_wikipedia_articles_embedded.csv'
article_df = pd.read_csv(csv_file_path)

# Read vectors from strings back into a list
article_df['title_vector'] = article_df.title_vector.apply(literal_eval).values
article_df['content_vector'] = article_df.content_vector.apply(literal_eval).values

# add/update data to indexes
for i in range(len(article_df)):
    # add data to index with title_vector
    client.tvs_hset(index=index_names[0], key=article_df.id[i].item(), vector=article_df.title_vector[i], is_binary=False,
                    **{"url": article_df.url[i], "title": article_df.title[i], "text": article_df.text[i]})
    # add data to index with content_vector
    client.tvs_hset(index=index_names[1], key=article_df.id[i].item(), vector=article_df.content_vector[i], is_binary=False,
                    **{"url": article_df.url[i], "title": article_df.title[i], "text": article_df.text[i]})
```


```python
# Check the data count to make sure all the points have been stored
for index_name in index_names:
    stats = client.tvs_get_index(index_name)
    count = int(stats["current_record_count"]) - int(stats["delete_record_count"])
    print(f"Count in {index_name}:{count}")

```

    Count in openai_test_title_vector:25000
    Count in openai_test_content_vector:25000


key: _node_content
string_value:
{"id_": "eec43b51-6986-4111-b4bb-e8b0e4b8f293", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "deeef3e2-06ef-4a06-ba05-093d97fc8631", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "580e111f11b68f838408ecf45ea83ca8276666ddf4be43d83f7cce79599e13d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "58c68be5-a64b-4177-90a6-c1cbc3ed51b5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "d3bca0ccc557336a9a0fe9e3e58ee34248b7997102473d547192476a171d013f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a46b3750-4075-4537-a3dc-a21a20eee54e", "node_type": "1", "metadata": {}, "hash": "c947b9821467ab678ba52a52e85f01f94fc1cdc2f0ff54052031aa2f232fa4e6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9505, "end_char_idx": 12545, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Load data

In this section we are going to load the data prepared previous to this session, so you don't have to recompute the embeddings of Wikipedia articles with your own credits.


```python
import pandas as pd
from ast import literal_eval
# Path to your local CSV file
csv_file_path = '../../data/vector_database_wikipedia_articles_embedded.csv'
article_df = pd.read_csv(csv_file_path)

# Read vectors from strings back into a list
article_df['title_vector'] = article_df.title_vector.apply(literal_eval).values
article_df['content_vector'] = article_df.content_vector.apply(literal_eval).values

# add/update data to indexes
for i in range(len(article_df)):
    # add data to index with title_vector
    client.tvs_hset(index=index_names[0], key=article_df.id[i].item(), vector=article_df.title_vector[i], is_binary=False,
                    **{"url": article_df.url[i], "title": article_df.title[i], "text": article_df.text[i]})
    # add data to index with content_vector
    client.tvs_hset(index=index_names[1], key=article_df.id[i].item(), vector=article_df.content_vector[i], is_binary=False,
                    **{"url": article_df.url[i], "title": article_df.title[i], "text": article_df.text[i]})
```


```python
# Check the data count to make sure all the points have been stored
for index_name in index_names:
    stats = client.tvs_get_index(index_name)
    count = int(stats["current_record_count"]) - int(stats["delete_record_count"])
    print(f"Count in {index_name}:{count}")

```

    Count in openai_test_title_vector:25000
    Count in openai_test_content_vector:25000


## Search data

Once the data is put into Tair we will start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search. Since the precomputed embeddings were created with `text-embedding-3-small` OpenAI model, we also have to use it during search.



```python
def query_tair(client, query, vector_name="title_vector", top_k=5):

    # Creates embedding vector from user query
    embedded_query = openai.Embedding.create(
        input= query,
        model="text-embedding-3-small",
    )["data"][0]['embedding']
    embedded_query = np.array(embedded_query)

    # search for the top k approximate nearest neighbors of vector in an index
    query_result = client.tvs_knnsearch(index=index+"_"+vector_name, k=top_k, vector=embedded_query)

    return query_result
```


```python
import openai
import numpy as np

query_result = query_tair(client=client, query="modern art in Europe", vector_name="title_vector")
for i in range(len(query_result)):
    title = client.tvs_hmget(index+"_"+"content_vector", query_result[i][0].decode('utf-8'), "title")
    print(f"{i + 1}. {title[0].decode('utf-8')} (Distance: {round(query_result[i][1],3)})")
```

    1. Museum of Modern Art (Distance: 0.125)
    2. Western Europe (Distance: 0.133)
    3. Renaissance art (Distance: 0.136)
    4. Pop art (Distance: 0.14)
    5. Northern Europe (Distance: 0.145)


key: _node_content
string_value:
{"id_": "a46b3750-4075-4537-a3dc-a21a20eee54e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "deeef3e2-06ef-4a06-ba05-093d97fc8631", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "580e111f11b68f838408ecf45ea83ca8276666ddf4be43d83f7cce79599e13d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "eec43b51-6986-4111-b4bb-e8b0e4b8f293", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/Getting_started_with_Tair_and_OpenAI.ipynb", "title": "Using Tair as a vector database for OpenAI embeddings", "tokens": 3360, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "6b14a4a657daaa4a804a0d9415ca615e63f2821e46c07373c3e174ad5816924a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11107, "end_char_idx": 13207, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Search data

Once the data is put into Tair we will start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search. Since the precomputed embeddings were created with `text-embedding-3-small` OpenAI model, we also have to use it during search.



```python
def query_tair(client, query, vector_name="title_vector", top_k=5):

    # Creates embedding vector from user query
    embedded_query = openai.Embedding.create(
        input= query,
        model="text-embedding-3-small",
    )["data"][0]['embedding']
    embedded_query = np.array(embedded_query)

    # search for the top k approximate nearest neighbors of vector in an index
    query_result = client.tvs_knnsearch(index=index+"_"+vector_name, k=top_k, vector=embedded_query)

    return query_result
```


```python
import openai
import numpy as np

query_result = query_tair(client=client, query="modern art in Europe", vector_name="title_vector")
for i in range(len(query_result)):
    title = client.tvs_hmget(index+"_"+"content_vector", query_result[i][0].decode('utf-8'), "title")
    print(f"{i + 1}. {title[0].decode('utf-8')} (Distance: {round(query_result[i][1],3)})")
```

    1. Museum of Modern Art (Distance: 0.125)
    2. Western Europe (Distance: 0.133)
    3. Renaissance art (Distance: 0.136)
    4. Pop art (Distance: 0.14)
    5. Northern Europe (Distance: 0.145)



```python
# This time we'll query using content vector
query_result = query_tair(client=client, query="Famous battles in Scottish history", vector_name="content_vector")
for i in range(len(query_result)):
    title = client.tvs_hmget(index+"_"+"content_vector", query_result[i][0].decode('utf-8'), "title")
    print(f"{i + 1}. {title[0].decode('utf-8')} (Distance: {round(query_result[i][1],3)})")
```

    1. Battle of Bannockburn (Distance: 0.131)
    2. Wars of Scottish Independence (Distance: 0.139)
    3. 1651 (Distance: 0.147)
    4. First War of Scottish Independence (Distance: 0.15)
    5. Robert I of Scotland (Distance: 0.154)



```python

```


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb


key: _node_content
string_value:
{"id_": "722dabd6-78bd-4691-a9d3-565ef0947cc4", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f69e0113-07fc-44a9-8b99-9c24002c1d71", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c275cad15baaccbe5fe38077467806f30fbde690ef94207aa07ed32e8f11a7b6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9ffb441b-608c-418e-8702-10816a20c711", "node_type": "1", "metadata": {}, "hash": "2b48771f4ca9bbf4b81538d05c3fe158dfc7192264d096ce829878087c71b699", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2854, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Question Answering with Langchain, Tair and OpenAI
This notebook presents how to implement a Question Answering system with Langchain, Tair as a knowledge based and OpenAI embeddings. If you are not familiar with Tair, it’s better to check out the [Getting_started_with_Tair_and_OpenAI.ipynb](Getting_started_with_Tair_and_OpenAI.ipynb) notebook.

This notebook presents an end-to-end process of:
- Calculating the embeddings with OpenAI API.
- Storing the embeddings in an Tair instance to build a knowledge base.
- Converting raw text query to an embedding with OpenAI API.
- Using Tair to perform the nearest neighbour search in the created collection to find some context.
- Asking LLM to find the answer in a given context.

All the steps will be simplified to calling some corresponding Langchain methods.

## Prerequisites
For the purposes of this exercise we need to prepare a couple of things:
[Tair cloud instance](https://www.alibabacloud.com/help/en/tair/latest/what-is-tair).
[Langchain](https://github.com/hwchase17/langchain) as a framework.
An OpenAI API key.

### Install requirements
This notebook requires the following Python packages: `openai`, `tiktoken`, `langchain` and `tair`.
- `openai` provides convenient access to the OpenAI API.
- `tiktoken` is a fast BPE tokeniser for use with OpenAI's models.
- `langchain` helps us to build applications with LLM more easily.
- `tair` library is used to interact with the tair vector database.


```python
! pip install openai tiktoken langchain tair 
```

    Looking in indexes: http://sg.mirrors.cloud.aliyuncs.com/pypi/simple/
    Requirement already satisfied: openai in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (0.28.0)
    Requirement already satisfied: tiktoken in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (0.4.0)
    Requirement already satisfied: langchain in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (0.0.281)
    Requirement already satisfied: tair in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (1.3.6)
    Requirement already satisfied: requests>=2.20 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from openai) (2.31.0)
    Requirement already satisfied: tqdm in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from openai) (4.66.1)
    Requirement already satisfied: aiohttp in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from openai) (3.8.5)
    Requirement already satisfied: regex>=2022.1.18 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from tiktoken) (2023.8.8)
    Requirement already satisfied: PyYAML>=5.3 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from langchain) (6.0.1)
    Requirement already satisfied: SQLAlchemy<3,>=1.4 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from langchain) (2.0.


key: _node_content
string_value:
{"id_": "9ffb441b-608c-418e-8702-10816a20c711", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f69e0113-07fc-44a9-8b99-9c24002c1d71", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c275cad15baaccbe5fe38077467806f30fbde690ef94207aa07ed32e8f11a7b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "722dabd6-78bd-4691-a9d3-565ef0947cc4", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "5c4964ec071639dae28c7874694d7f04a25eb34dc8233f7453108b0b4779f4fe", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dd562bed-9be0-4308-8617-7be6acffd32f", "node_type": "1", "metadata": {}, "hash": "0c100157a2668b244a7ad3c06c33e5582393a48ae014de3a7110c86e8bed76e5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1560, "end_char_idx": 3919, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
mirrors.cloud.aliyuncs.com/pypi/simple/
    Requirement already satisfied: openai in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (0.28.0)
    Requirement already satisfied: tiktoken in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (0.4.0)
    Requirement already satisfied: langchain in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (0.0.281)
    Requirement already satisfied: tair in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (1.3.6)
    Requirement already satisfied: requests>=2.20 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from openai) (2.31.0)
    Requirement already satisfied: tqdm in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from openai) (4.66.1)
    Requirement already satisfied: aiohttp in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from openai) (3.8.5)
    Requirement already satisfied: regex>=2022.1.18 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from tiktoken) (2023.8.8)
    Requirement already satisfied: PyYAML>=5.3 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from langchain) (6.0.1)
    Requirement already satisfied: SQLAlchemy<3,>=1.4 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from langchain) (2.0.20)
    Requirement already satisfied: async-timeout<5.0.0,>=4.0.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from langchain) (4.0.3)
    Requirement already satisfied: dataclasses-json<0.6.0,>=0.5.7 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from langchain) (0.5.14)
    Requirement already satisfied: langsmith<0.1.0,>=0.0.21 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from langchain) (0.0.33)
    Requirement already satisfied: numexpr<3.0.0,>=2.8.4 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from langchain) (2.8.5)
    Requirement already satisfied: numpy<2,>=1 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from langchain) (1.25.2)
    Requirement already satisfied: pydantic<3,>=1 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from langchain) (1.10.12)
    Requirement already satisfied: tenacity<9.0.0,>=8.1.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from langchain) (8.2.3)
    Requirement already satisfied: redis>=4.4.


key: _node_content
string_value:
{"id_": "dd562bed-9be0-4308-8617-7be6acffd32f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f69e0113-07fc-44a9-8b99-9c24002c1d71", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c275cad15baaccbe5fe38077467806f30fbde690ef94207aa07ed32e8f11a7b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9ffb441b-608c-418e-8702-10816a20c711", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "e1f8415d9d7c067f7aebfd779057b0898817c5123d119070ee685654916f3b02", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5726a495-2957-4ae8-9d4b-3058a4266179", "node_type": "1", "metadata": {}, "hash": "774aae5ff50ceefb2589622b5089738434d3a91a8fa13b8db75a4078edda29d5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2713, "end_char_idx": 4992, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
1)
    Requirement already satisfied: SQLAlchemy<3,>=1.4 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from langchain) (2.0.20)
    Requirement already satisfied: async-timeout<5.0.0,>=4.0.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from langchain) (4.0.3)
    Requirement already satisfied: dataclasses-json<0.6.0,>=0.5.7 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from langchain) (0.5.14)
    Requirement already satisfied: langsmith<0.1.0,>=0.0.21 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from langchain) (0.0.33)
    Requirement already satisfied: numexpr<3.0.0,>=2.8.4 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from langchain) (2.8.5)
    Requirement already satisfied: numpy<2,>=1 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from langchain) (1.25.2)
    Requirement already satisfied: pydantic<3,>=1 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from langchain) (1.10.12)
    Requirement already satisfied: tenacity<9.0.0,>=8.1.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from langchain) (8.2.3)
    Requirement already satisfied: redis>=4.4.4 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from tair) (5.0.0)
    Requirement already satisfied: attrs>=17.3.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (22.1.0)
    Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (3.2.0)
    Requirement already satisfied: multidict<7.0,>=4.5 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (6.0.4)
    Requirement already satisfied: yarl<2.0,>=1.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (1.9.2)
    Requirement already satisfied: frozenlist>=1.1.1 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (1.4.0)
    Requirement already satisfied: aiosignal>=1.1.2 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (1.3.1)
    Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /root/anaconda3/envs/notebook/lib/python3.


key: _node_content
string_value:
{"id_": "5726a495-2957-4ae8-9d4b-3058a4266179", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f69e0113-07fc-44a9-8b99-9c24002c1d71", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c275cad15baaccbe5fe38077467806f30fbde690ef94207aa07ed32e8f11a7b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dd562bed-9be0-4308-8617-7be6acffd32f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "52b16762a3986d8b332caa804992bb55e39612e87e9f59237dd082be5505f842", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e5dff9fc-52b3-45a9-abfd-14c68cc857c9", "node_type": "1", "metadata": {}, "hash": "0e4607097f3a20f467b68c7a98736a145a3301bcb5bb0e2b565804eb69792075", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3831, "end_char_idx": 6075, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
10/site-packages (from langchain) (8.2.3)
    Requirement already satisfied: redis>=4.4.4 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from tair) (5.0.0)
    Requirement already satisfied: attrs>=17.3.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (22.1.0)
    Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (3.2.0)
    Requirement already satisfied: multidict<7.0,>=4.5 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (6.0.4)
    Requirement already satisfied: yarl<2.0,>=1.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (1.9.2)
    Requirement already satisfied: frozenlist>=1.1.1 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (1.4.0)
    Requirement already satisfied: aiosignal>=1.1.2 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from aiohttp->openai) (1.3.1)
    Requirement already satisfied: marshmallow<4.0.0,>=3.18.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from dataclasses-json<0.6.0,>=0.5.7->langchain) (3.20.1)
    Requirement already satisfied: typing-inspect<1,>=0.4.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from dataclasses-json<0.6.0,>=0.5.7->langchain) (0.9.0)
    Requirement already satisfied: typing-extensions>=4.2.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from pydantic<3,>=1->langchain) (4.7.1)
    Requirement already satisfied: idna<4,>=2.5 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from requests>=2.20->openai) (3.4)
    Requirement already satisfied: urllib3<3,>=1.21.1 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from requests>=2.20->openai) (2.0.4)
    Requirement already satisfied: certifi>=2017.4.17 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from requests>=2.20->openai) (2023.7.22)
    Requirement already satisfied: greenlet!=0.4.17 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from SQLAlchemy<3,>=1.4->langchain) (2.0.2)
    Requirement already satisfied: packaging>=17.


key: _node_content
string_value:
{"id_": "e5dff9fc-52b3-45a9-abfd-14c68cc857c9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f69e0113-07fc-44a9-8b99-9c24002c1d71", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c275cad15baaccbe5fe38077467806f30fbde690ef94207aa07ed32e8f11a7b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5726a495-2957-4ae8-9d4b-3058a4266179", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "01c60270dc75ab4ca0cca9d9b895f0e5632bd361ee1c816d2e2a95dd41f9a4f3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5d5e7dbc-5f06-4c7f-ab13-ad5fcda65caf", "node_type": "1", "metadata": {}, "hash": "944de6f1ce084d81b5521dcfa7f696664110bea09b0681684caa854da94f369f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4934, "end_char_idx": 7162, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0.0,>=3.18.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from dataclasses-json<0.6.0,>=0.5.7->langchain) (3.20.1)
    Requirement already satisfied: typing-inspect<1,>=0.4.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from dataclasses-json<0.6.0,>=0.5.7->langchain) (0.9.0)
    Requirement already satisfied: typing-extensions>=4.2.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from pydantic<3,>=1->langchain) (4.7.1)
    Requirement already satisfied: idna<4,>=2.5 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from requests>=2.20->openai) (3.4)
    Requirement already satisfied: urllib3<3,>=1.21.1 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from requests>=2.20->openai) (2.0.4)
    Requirement already satisfied: certifi>=2017.4.17 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from requests>=2.20->openai) (2023.7.22)
    Requirement already satisfied: greenlet!=0.4.17 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from SQLAlchemy<3,>=1.4->langchain) (2.0.2)
    Requirement already satisfied: packaging>=17.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses-json<0.6.0,>=0.5.7->langchain) (23.1)
    Requirement already satisfied: mypy-extensions>=0.3.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from typing-inspect<1,>=0.4.0->dataclasses-json<0.6.0,>=0.5.7->langchain) (1.0.0)
    [33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m[33m
    [0m

### Prepare your OpenAI API key
The OpenAI API key is used for vectorization of the documents and queries.

If you don't have an OpenAI API key, you can get one from [https://platform.openai.com/account/api-keys ).

Once you get your key, please add it by getpass.


```python
import getpass

openai_api_key = getpass.getpass("Input your OpenAI API key:")
```

    Input your OpenAI API key:········


### Prepare your Tair URL
To build the Tair connection, you need to have `TAIR_URL`.


key: _node_content
string_value:
{"id_": "5d5e7dbc-5f06-4c7f-ab13-ad5fcda65caf", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f69e0113-07fc-44a9-8b99-9c24002c1d71", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c275cad15baaccbe5fe38077467806f30fbde690ef94207aa07ed32e8f11a7b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e5dff9fc-52b3-45a9-abfd-14c68cc857c9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "98f2bf3ac559dc729f58151ca2aad7968d7a1c7980020abe4f68581730c1d55c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6666095c-d5f5-4d19-ab66-d1e6bfa1f6f9", "node_type": "1", "metadata": {}, "hash": "2adb94af7f679ce385e584fb7d448c2d39a347305df021e514e1ead7d6b64e61", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5861, "end_char_idx": 8443, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
7.22)
    Requirement already satisfied: greenlet!=0.4.17 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from SQLAlchemy<3,>=1.4->langchain) (2.0.2)
    Requirement already satisfied: packaging>=17.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses-json<0.6.0,>=0.5.7->langchain) (23.1)
    Requirement already satisfied: mypy-extensions>=0.3.0 in /root/anaconda3/envs/notebook/lib/python3.10/site-packages (from typing-inspect<1,>=0.4.0->dataclasses-json<0.6.0,>=0.5.7->langchain) (1.0.0)
    [33mWARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv[0m[33m
    [0m

### Prepare your OpenAI API key
The OpenAI API key is used for vectorization of the documents and queries.

If you don't have an OpenAI API key, you can get one from [https://platform.openai.com/account/api-keys ).

Once you get your key, please add it by getpass.


```python
import getpass

openai_api_key = getpass.getpass("Input your OpenAI API key:")
```

    Input your OpenAI API key:········


### Prepare your Tair URL
To build the Tair connection, you need to have `TAIR_URL`.


```python
# The format of url: redis://[[username]:[password]]@localhost:6379/0
TAIR_URL = getpass.getpass("Input your tair url:")
```

    Input your tair url:········


## Load data
In this section we are going to load the data containing some natural questions and answers to them. All the data will be used to create a Langchain application with Tair being the knowledge base.


```python
import wget

# All the examples come from https://ai.google.com/research/NaturalQuestions
# This is a sample of the training set that we download and extract for some
# further processing.
wget.download("https://storage.googleapis.com/dataset-natural-questions/questions.json")
wget.download("https://storage.googleapis.com/dataset-natural-questions/answers.json")
```

    100% [..............................................................................] 95372 / 95372




    'answers (2).json'




```python
import json

with open("questions.json", "r") as fp:
    questions = json.load(fp)

with open("answers.json", "r") as fp:
    answers = json.load(fp)
```


```python
print(questions[0])
```

    when is the last episode of season 8 of the walking dead



```python
print(answers[0])
```

    No . overall No. in season Title Directed by Written by Original air date U.S.


key: _node_content
string_value:
{"id_": "6666095c-d5f5-4d19-ab66-d1e6bfa1f6f9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f69e0113-07fc-44a9-8b99-9c24002c1d71", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c275cad15baaccbe5fe38077467806f30fbde690ef94207aa07ed32e8f11a7b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5d5e7dbc-5f06-4c7f-ab13-ad5fcda65caf", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "2c31d30f89dc1ab7a0e8c819ddc93b393edda0f18e69834ac96a6c1219833d1f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "029bd85f-450d-4229-b4fa-9b909bc8ab37", "node_type": "1", "metadata": {}, "hash": "e1aef99fb175200e696cead06fa28641f9c77cec4832440b5ba5b62f246933b6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6892, "end_char_idx": 10002, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Once you get your key, please add it by getpass.


```python
import getpass

openai_api_key = getpass.getpass("Input your OpenAI API key:")
```

    Input your OpenAI API key:········


### Prepare your Tair URL
To build the Tair connection, you need to have `TAIR_URL`.


```python
# The format of url: redis://[[username]:[password]]@localhost:6379/0
TAIR_URL = getpass.getpass("Input your tair url:")
```

    Input your tair url:········


## Load data
In this section we are going to load the data containing some natural questions and answers to them. All the data will be used to create a Langchain application with Tair being the knowledge base.


```python
import wget

# All the examples come from https://ai.google.com/research/NaturalQuestions
# This is a sample of the training set that we download and extract for some
# further processing.
wget.download("https://storage.googleapis.com/dataset-natural-questions/questions.json")
wget.download("https://storage.googleapis.com/dataset-natural-questions/answers.json")
```

    100% [..............................................................................] 95372 / 95372




    'answers (2).json'




```python
import json

with open("questions.json", "r") as fp:
    questions = json.load(fp)

with open("answers.json", "r") as fp:
    answers = json.load(fp)
```


```python
print(questions[0])
```

    when is the last episode of season 8 of the walking dead



```python
print(answers[0])
```

    No . overall No. in season Title Directed by Written by Original air date U.S. viewers ( millions ) 100 `` Mercy '' Greg Nicotero Scott M. Gimple October 22 , 2017 ( 2017 - 10 - 22 ) 11.44 Rick , Maggie , and Ezekiel rally their communities together to take down Negan . Gregory attempts to have the Hilltop residents side with Negan , but they all firmly stand behind Maggie . The group attacks the Sanctuary , taking down its fences and flooding the compound with walkers . With the Sanctuary defaced , everyone leaves except Gabriel , who reluctantly stays to save Gregory , but is left behind when Gregory abandons him . Surrounded by walkers , Gabriel hides in a trailer , where he is trapped inside with Negan . 101 `` The Damned '' Rosemary Rodriguez Matthew Negrete & Channing Powell October 29 , 2017 ( 2017 - 10 - 29 ) 8.92 Rick 's forces split into separate parties to attack several of the Saviors ' outposts , during which many members of the group are killed ; Eric is critically injured and rushed away by Aaron . Jesus stops Tara and Morgan from executing a group of surrendered Saviors . While clearing an outpost with Daryl , Rick is confronted and held at gunpoint by Morales , a survivor he met in the initial Atlanta camp , who is now with the Saviors . 102 `` Monsters '' Greg Nicotero Matthew Negrete & Channing Powell November 5 , 2017 ( 2017 - 11 - 05 ) 8.52 Daryl finds Morales threatening Rick and kills him ; the duo then pursue a group of Saviors who are transporting weapons to another outpost . Gregory returns to Hilltop , and after a heated argument , Maggie ultimately allows him back in the community .


key: _node_content
string_value:
{"id_": "029bd85f-450d-4229-b4fa-9b909bc8ab37", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f69e0113-07fc-44a9-8b99-9c24002c1d71", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c275cad15baaccbe5fe38077467806f30fbde690ef94207aa07ed32e8f11a7b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6666095c-d5f5-4d19-ab66-d1e6bfa1f6f9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4cc8ce0dfbbe35543bcb1ecc4b1c126799add81814e12ce67be733a9b98f5ec6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "622e8e28-3ed9-4014-814b-bb189d28deed", "node_type": "1", "metadata": {}, "hash": "d94c066c92c96e9babc4756e13ab1323a9069011fd0fbf90640191ab388b8e66", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8370, "end_char_idx": 11605, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
overall No. in season Title Directed by Written by Original air date U.S. viewers ( millions ) 100 `` Mercy '' Greg Nicotero Scott M. Gimple October 22 , 2017 ( 2017 - 10 - 22 ) 11.44 Rick , Maggie , and Ezekiel rally their communities together to take down Negan . Gregory attempts to have the Hilltop residents side with Negan , but they all firmly stand behind Maggie . The group attacks the Sanctuary , taking down its fences and flooding the compound with walkers . With the Sanctuary defaced , everyone leaves except Gabriel , who reluctantly stays to save Gregory , but is left behind when Gregory abandons him . Surrounded by walkers , Gabriel hides in a trailer , where he is trapped inside with Negan . 101 `` The Damned '' Rosemary Rodriguez Matthew Negrete & Channing Powell October 29 , 2017 ( 2017 - 10 - 29 ) 8.92 Rick 's forces split into separate parties to attack several of the Saviors ' outposts , during which many members of the group are killed ; Eric is critically injured and rushed away by Aaron . Jesus stops Tara and Morgan from executing a group of surrendered Saviors . While clearing an outpost with Daryl , Rick is confronted and held at gunpoint by Morales , a survivor he met in the initial Atlanta camp , who is now with the Saviors . 102 `` Monsters '' Greg Nicotero Matthew Negrete & Channing Powell November 5 , 2017 ( 2017 - 11 - 05 ) 8.52 Daryl finds Morales threatening Rick and kills him ; the duo then pursue a group of Saviors who are transporting weapons to another outpost . Gregory returns to Hilltop , and after a heated argument , Maggie ultimately allows him back in the community . Eric dies from his injuries , leaving Aaron distraught . Despite Tara and Morgan 's objections , Jesus leads the group of surrendered Saviors to Hilltop . Ezekiel 's group attacks another Savior compound , during which several Kingdommers are shot while protecting Ezekiel . 103 `` Some Guy '' Dan Liu David Leslie Johnson November 12 , 2017 ( 2017 - 11 - 12 ) 8.69 Ezekiel 's group is overwhelmed by the Saviors , who kill all of them except for Ezekiel himself and Jerry . Carol clears the inside of the compound , killing all but two Saviors , who almost escape but are eventually caught by Rick and Daryl . En route to the Kingdom , Ezekiel , Jerry , and Carol are surrounded by walkers , but Shiva sacrifices herself to save them . The trio returns to the Kingdom , where Ezekiel 's confidence in himself as a leader has diminished . 104 5 `` The Big Scary U '' Michael E. Satrazemis Story by : Scott M. Gimple & David Leslie Johnson & Angela Kang Teleplay by : David Leslie Johnson & Angela Kang November 19 , 2017 ( 2017 - 11 - 19 ) 7.85 After confessing their sins to each other , Gabriel and Negan manage to escape from the trailer . Simon and the other lieutenants grow suspicious of each other , knowing that Rick 's forces must have inside information . The workers in the Sanctuary become increasingly frustrated with their living conditions , and a riot nearly ensues , until Negan returns and restores order . Gabriel is locked in a cell , where Eugene discovers him sick and suffering . Meanwhile , Rick and Daryl argue over how to take out the Saviors , leading Daryl to abandon Rick .


key: _node_content
string_value:
{"id_": "622e8e28-3ed9-4014-814b-bb189d28deed", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f69e0113-07fc-44a9-8b99-9c24002c1d71", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c275cad15baaccbe5fe38077467806f30fbde690ef94207aa07ed32e8f11a7b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "029bd85f-450d-4229-b4fa-9b909bc8ab37", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "9f97d11b0ad262f77215543b199a3975fbf0f7e16df0d5a300b9a70dae90b80e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fbdcd002-7022-45b8-a838-33d4978e618f", "node_type": "1", "metadata": {}, "hash": "ebc0d829f274fb00d69d09328e9a63ab898617cb659c61d4d41cb4b5bd3312d4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9891, "end_char_idx": 12920, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Gregory returns to Hilltop , and after a heated argument , Maggie ultimately allows him back in the community . Eric dies from his injuries , leaving Aaron distraught . Despite Tara and Morgan 's objections , Jesus leads the group of surrendered Saviors to Hilltop . Ezekiel 's group attacks another Savior compound , during which several Kingdommers are shot while protecting Ezekiel . 103 `` Some Guy '' Dan Liu David Leslie Johnson November 12 , 2017 ( 2017 - 11 - 12 ) 8.69 Ezekiel 's group is overwhelmed by the Saviors , who kill all of them except for Ezekiel himself and Jerry . Carol clears the inside of the compound , killing all but two Saviors , who almost escape but are eventually caught by Rick and Daryl . En route to the Kingdom , Ezekiel , Jerry , and Carol are surrounded by walkers , but Shiva sacrifices herself to save them . The trio returns to the Kingdom , where Ezekiel 's confidence in himself as a leader has diminished . 104 5 `` The Big Scary U '' Michael E. Satrazemis Story by : Scott M. Gimple & David Leslie Johnson & Angela Kang Teleplay by : David Leslie Johnson & Angela Kang November 19 , 2017 ( 2017 - 11 - 19 ) 7.85 After confessing their sins to each other , Gabriel and Negan manage to escape from the trailer . Simon and the other lieutenants grow suspicious of each other , knowing that Rick 's forces must have inside information . The workers in the Sanctuary become increasingly frustrated with their living conditions , and a riot nearly ensues , until Negan returns and restores order . Gabriel is locked in a cell , where Eugene discovers him sick and suffering . Meanwhile , Rick and Daryl argue over how to take out the Saviors , leading Daryl to abandon Rick . 105 6 `` The King , the Widow , and Rick '' John Polson Angela Kang & Corey Reed November 26 , 2017 ( 2017 - 11 - 26 ) 8.28 Rick visits Jadis in hopes of convincing her to turn against Negan ; Jadis refuses , and locks Rick in a shipping container . Carl encounters Siddiq in the woods and recruits him to Alexandria . Daryl and Tara plot to deviate from Rick 's plans by destroying the Sanctuary . Ezekiel isolates himself at the Kingdom , where Carol tries to encourage him to be the leader his people need . Maggie has the group of captured Saviors placed in a holding area and forces Gregory to join them as punishment for betraying Hilltop . 106 7 `` Time for After '' Larry Teng Matthew Negrete & Corey Reed December 3 , 2017 ( 2017 - 12 - 03 ) 7.47 After learning of Dwight 's association with Rick 's group , Eugene affirms his loyalty to Negan and outlines a plan to get rid of the walkers surrounding the Sanctuary . With help from Morgan and Tara , Daryl drives a truck through the Sanctuary 's walls , flooding its interior with walkers , killing many Saviors . Rick finally convinces Jadis and the Scavengers to align with him , and they plan to force the Saviors to surrender . However , when they arrive at the Sanctuary , Rick is horrified to see the breached walls and no sign of the walker herd .


key: _node_content
string_value:
{"id_": "fbdcd002-7022-45b8-a838-33d4978e618f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f69e0113-07fc-44a9-8b99-9c24002c1d71", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c275cad15baaccbe5fe38077467806f30fbde690ef94207aa07ed32e8f11a7b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "622e8e28-3ed9-4014-814b-bb189d28deed", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "72547a030aa04d79a7bba41dfb37a395a94dfd1240469ceb88888dba83fc2397", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "28780302-f83a-4295-862e-4c1b1c1f4ffc", "node_type": "1", "metadata": {}, "hash": "424b6095d7fc27109986391a79e9512227ac6068048015d827aa8fde766526e8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11269, "end_char_idx": 14223, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The workers in the Sanctuary become increasingly frustrated with their living conditions , and a riot nearly ensues , until Negan returns and restores order . Gabriel is locked in a cell , where Eugene discovers him sick and suffering . Meanwhile , Rick and Daryl argue over how to take out the Saviors , leading Daryl to abandon Rick . 105 6 `` The King , the Widow , and Rick '' John Polson Angela Kang & Corey Reed November 26 , 2017 ( 2017 - 11 - 26 ) 8.28 Rick visits Jadis in hopes of convincing her to turn against Negan ; Jadis refuses , and locks Rick in a shipping container . Carl encounters Siddiq in the woods and recruits him to Alexandria . Daryl and Tara plot to deviate from Rick 's plans by destroying the Sanctuary . Ezekiel isolates himself at the Kingdom , where Carol tries to encourage him to be the leader his people need . Maggie has the group of captured Saviors placed in a holding area and forces Gregory to join them as punishment for betraying Hilltop . 106 7 `` Time for After '' Larry Teng Matthew Negrete & Corey Reed December 3 , 2017 ( 2017 - 12 - 03 ) 7.47 After learning of Dwight 's association with Rick 's group , Eugene affirms his loyalty to Negan and outlines a plan to get rid of the walkers surrounding the Sanctuary . With help from Morgan and Tara , Daryl drives a truck through the Sanctuary 's walls , flooding its interior with walkers , killing many Saviors . Rick finally convinces Jadis and the Scavengers to align with him , and they plan to force the Saviors to surrender . However , when they arrive at the Sanctuary , Rick is horrified to see the breached walls and no sign of the walker herd . 107 8 `` How It 's Gotta Be '' Michael E. Satrazemis David Leslie Johnson & Angela Kang December 10 , 2017 ( 2017 - 12 - 10 ) 7.89 Eugene 's plan allows the Saviors to escape , and separately , the Saviors waylay the Alexandria , Hilltop , and Kingdom forces . The Scavengers abandon Rick , after which he returns to Alexandria . Ezekiel ensures that the Kingdom residents are able to escape before locking himself in the community with the Saviors . Eugene aids Gabriel and Doctor Carson in escaping the Sanctuary in order to ease his conscience . Negan attacks Alexandria , but Carl devises a plan to allow the Alexandria residents to escape into the sewers . Carl reveals he was bitten by a walker while escorting Siddiq to Alexandria . 108 9 `` Honor '' Greg Nicotero Matthew Negrete & Channing Powell February 25 , 2018 ( 2018 - 02 - 25 ) 8.28 After the Saviors leave Alexandria , the survivors make for the Hilltop while Rick and Michonne stay behind to say their final goodbyes to a dying Carl , who pleads with Rick to build a better future alongside the Saviors before killing himself . In the Kingdom , Morgan and Carol launch a rescue mission for Ezekiel . Although they are successful and retake the Kingdom , the Saviors ' lieutenant Gavin is killed by Benjamin 's vengeful brother Henry .


key: _node_content
string_value:
{"id_": "28780302-f83a-4295-862e-4c1b1c1f4ffc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f69e0113-07fc-44a9-8b99-9c24002c1d71", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c275cad15baaccbe5fe38077467806f30fbde690ef94207aa07ed32e8f11a7b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fbdcd002-7022-45b8-a838-33d4978e618f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "608ea04d45f8e8bdaeef5bdccb72fcef4da65883f4db18295ae6d9932485c408", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a50a7212-c2f9-4da4-9f6d-96aece902736", "node_type": "1", "metadata": {}, "hash": "894ad37ee349a4ecf6ef930307658cc4bfe7b78a7daaa96867242b64c4755b8f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12533, "end_char_idx": 15464, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
With help from Morgan and Tara , Daryl drives a truck through the Sanctuary 's walls , flooding its interior with walkers , killing many Saviors . Rick finally convinces Jadis and the Scavengers to align with him , and they plan to force the Saviors to surrender . However , when they arrive at the Sanctuary , Rick is horrified to see the breached walls and no sign of the walker herd . 107 8 `` How It 's Gotta Be '' Michael E. Satrazemis David Leslie Johnson & Angela Kang December 10 , 2017 ( 2017 - 12 - 10 ) 7.89 Eugene 's plan allows the Saviors to escape , and separately , the Saviors waylay the Alexandria , Hilltop , and Kingdom forces . The Scavengers abandon Rick , after which he returns to Alexandria . Ezekiel ensures that the Kingdom residents are able to escape before locking himself in the community with the Saviors . Eugene aids Gabriel and Doctor Carson in escaping the Sanctuary in order to ease his conscience . Negan attacks Alexandria , but Carl devises a plan to allow the Alexandria residents to escape into the sewers . Carl reveals he was bitten by a walker while escorting Siddiq to Alexandria . 108 9 `` Honor '' Greg Nicotero Matthew Negrete & Channing Powell February 25 , 2018 ( 2018 - 02 - 25 ) 8.28 After the Saviors leave Alexandria , the survivors make for the Hilltop while Rick and Michonne stay behind to say their final goodbyes to a dying Carl , who pleads with Rick to build a better future alongside the Saviors before killing himself . In the Kingdom , Morgan and Carol launch a rescue mission for Ezekiel . Although they are successful and retake the Kingdom , the Saviors ' lieutenant Gavin is killed by Benjamin 's vengeful brother Henry . 109 10 `` The Lost and the Plunderers '' TBA TBA March 4 , 2018 ( 2018 - 03 - 04 ) TBD 110 11 `` Dead or Alive Or '' TBA TBA March 11 , 2018 ( 2018 - 03 - 11 ) TBD 111 12 `` The Key '' TBA TBA March 18 , 2018 ( 2018 - 03 - 18 ) TBD


## Chain definition

Langchain is already integrated with Tair and performs all the indexing for given list of documents. In our case we are going to store the set of answers we have.


```python
from langchain.vectorstores import Tair
from langchain.embeddings import OpenAIEmbeddings
from langchain import VectorDBQA, OpenAI

embeddings = OpenAIEmbeddings(openai_api_key=openai_api_key)
doc_store = Tair.from_texts(
    texts=answers, embedding=embeddings, tair_url=TAIR_URL,
)
```

At this stage all the possible answers are already stored in Tair, so we can define the whole QA chain.


```python
llm = OpenAI(openai_api_key=openai_api_key)
qa = VectorDBQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    vectorstore=doc_store,
    return_source_documents=False,
)
```

    /root/anaconda3/envs/notebook/lib/python3.10/site-packages/langchain/chains/retrieval_qa/base.py:251: UserWarning: `VectorDBQA` is deprecated - please use `from langchain.chains import RetrievalQA`
      warnings.warn(


key: _node_content
string_value:
{"id_": "a50a7212-c2f9-4da4-9f6d-96aece902736", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f69e0113-07fc-44a9-8b99-9c24002c1d71", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c275cad15baaccbe5fe38077467806f30fbde690ef94207aa07ed32e8f11a7b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "28780302-f83a-4295-862e-4c1b1c1f4ffc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "44c16f95a17aacf3df7241a54da7cb313f29296074fbd0e651b62cfec4c828b7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9ed7cd59-011f-4094-bb53-1f2ea6964b59", "node_type": "1", "metadata": {}, "hash": "0890d8fdd6e1241e2b670c4a7df45b8631a43ae62be4828755bcf4fd0fc8cfe3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14089, "end_char_idx": 15909, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Although they are successful and retake the Kingdom , the Saviors ' lieutenant Gavin is killed by Benjamin 's vengeful brother Henry . 109 10 `` The Lost and the Plunderers '' TBA TBA March 4 , 2018 ( 2018 - 03 - 04 ) TBD 110 11 `` Dead or Alive Or '' TBA TBA March 11 , 2018 ( 2018 - 03 - 11 ) TBD 111 12 `` The Key '' TBA TBA March 18 , 2018 ( 2018 - 03 - 18 ) TBD


## Chain definition

Langchain is already integrated with Tair and performs all the indexing for given list of documents. In our case we are going to store the set of answers we have.


```python
from langchain.vectorstores import Tair
from langchain.embeddings import OpenAIEmbeddings
from langchain import VectorDBQA, OpenAI

embeddings = OpenAIEmbeddings(openai_api_key=openai_api_key)
doc_store = Tair.from_texts(
    texts=answers, embedding=embeddings, tair_url=TAIR_URL,
)
```

At this stage all the possible answers are already stored in Tair, so we can define the whole QA chain.


```python
llm = OpenAI(openai_api_key=openai_api_key)
qa = VectorDBQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    vectorstore=doc_store,
    return_source_documents=False,
)
```

    /root/anaconda3/envs/notebook/lib/python3.10/site-packages/langchain/chains/retrieval_qa/base.py:251: UserWarning: `VectorDBQA` is deprecated - please use `from langchain.chains import RetrievalQA`
      warnings.warn(


## Search data

Once the data is put into Tair we can start asking some questions. A question will be automatically vectorized by OpenAI model, and the created vector will be used to find some possibly matching answers in Tair. Once retrieved, the most similar answers will be incorporated into the prompt sent to OpenAI Large Language Model.



```python
import random

random.seed(52)
selected_questions = random.choices(questions, k=5)
```


key: _node_content
string_value:
{"id_": "9ed7cd59-011f-4094-bb53-1f2ea6964b59", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f69e0113-07fc-44a9-8b99-9c24002c1d71", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c275cad15baaccbe5fe38077467806f30fbde690ef94207aa07ed32e8f11a7b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a50a7212-c2f9-4da4-9f6d-96aece902736", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "7b9bab8abecd1d1649697888fb3a730289e21c284b17996d2e5c56502df28973", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "01ebc294-3ff2-401e-964c-3ff17c7a20a5", "node_type": "1", "metadata": {}, "hash": "6a2ed9dc173bf305747376b3a351038ee1543cfedd734814de4450bac83a3787", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14458, "end_char_idx": 17120, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Chain definition

Langchain is already integrated with Tair and performs all the indexing for given list of documents. In our case we are going to store the set of answers we have.


```python
from langchain.vectorstores import Tair
from langchain.embeddings import OpenAIEmbeddings
from langchain import VectorDBQA, OpenAI

embeddings = OpenAIEmbeddings(openai_api_key=openai_api_key)
doc_store = Tair.from_texts(
    texts=answers, embedding=embeddings, tair_url=TAIR_URL,
)
```

At this stage all the possible answers are already stored in Tair, so we can define the whole QA chain.


```python
llm = OpenAI(openai_api_key=openai_api_key)
qa = VectorDBQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    vectorstore=doc_store,
    return_source_documents=False,
)
```

    /root/anaconda3/envs/notebook/lib/python3.10/site-packages/langchain/chains/retrieval_qa/base.py:251: UserWarning: `VectorDBQA` is deprecated - please use `from langchain.chains import RetrievalQA`
      warnings.warn(


## Search data

Once the data is put into Tair we can start asking some questions. A question will be automatically vectorized by OpenAI model, and the created vector will be used to find some possibly matching answers in Tair. Once retrieved, the most similar answers will be incorporated into the prompt sent to OpenAI Large Language Model.



```python
import random

random.seed(52)
selected_questions = random.choices(questions, k=5)
```


```python
import time
for question in selected_questions:
    print(">", question)
    print(qa.run(question), end="\n\n")
    # wait 20seconds because of the rate limit
    time.sleep(20)
```

    > where do frankenstein and the monster first meet
     Frankenstein and the monster first meet in the mountains.
    
    > who are the actors in fast and furious
     The actors in Fast & Furious are Vin Diesel ( Dominic Toretto ), Paul Walker ( Brian O'Conner ), Michelle Rodriguez ( Letty Ortiz ), Jordana Brewster ( Mia Toretto ), Tyrese Gibson ( Roman Pearce ), Ludacris ( Tej Parker ), Lucas Black ( Sean Boswell ), Sung Kang ( Han Lue ), Gal Gadot ( Gisele Yashar ), and Dwayne Johnson ( Luke Hobbs ).
    
    > properties of red black tree in data structure
     The properties of a red-black tree in data structure are that each node is either red or black, the root is black, if a node is red then both its children must be black, and every path from a given node to any of its descendant NIL nodes contains the same number of black nodes.
    
    > who designed the national coat of arms of south africa
     Iaan Bekker
    
    > caravaggio's death of the virgin pamela askew
     I don't know.


key: _node_content
string_value:
{"id_": "01ebc294-3ff2-401e-964c-3ff17c7a20a5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f69e0113-07fc-44a9-8b99-9c24002c1d71", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c275cad15baaccbe5fe38077467806f30fbde690ef94207aa07ed32e8f11a7b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9ed7cd59-011f-4094-bb53-1f2ea6964b59", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "9a3ca631c9c5494a266156fd68dfcf0f8f84f69a3116a451fe16f8b062aaab07", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d02b47a0-ac02-4914-a8e2-ee4c45bc07f5", "node_type": "1", "metadata": {}, "hash": "a4097d7fe51257a78f2fa074102ab57a3c50ae89aac7c47cf9103da6f7558847", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15467, "end_char_idx": 18737, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Search data

Once the data is put into Tair we can start asking some questions. A question will be automatically vectorized by OpenAI model, and the created vector will be used to find some possibly matching answers in Tair. Once retrieved, the most similar answers will be incorporated into the prompt sent to OpenAI Large Language Model.



```python
import random

random.seed(52)
selected_questions = random.choices(questions, k=5)
```


```python
import time
for question in selected_questions:
    print(">", question)
    print(qa.run(question), end="\n\n")
    # wait 20seconds because of the rate limit
    time.sleep(20)
```

    > where do frankenstein and the monster first meet
     Frankenstein and the monster first meet in the mountains.
    
    > who are the actors in fast and furious
     The actors in Fast & Furious are Vin Diesel ( Dominic Toretto ), Paul Walker ( Brian O'Conner ), Michelle Rodriguez ( Letty Ortiz ), Jordana Brewster ( Mia Toretto ), Tyrese Gibson ( Roman Pearce ), Ludacris ( Tej Parker ), Lucas Black ( Sean Boswell ), Sung Kang ( Han Lue ), Gal Gadot ( Gisele Yashar ), and Dwayne Johnson ( Luke Hobbs ).
    
    > properties of red black tree in data structure
     The properties of a red-black tree in data structure are that each node is either red or black, the root is black, if a node is red then both its children must be black, and every path from a given node to any of its descendant NIL nodes contains the same number of black nodes.
    
    > who designed the national coat of arms of south africa
     Iaan Bekker
    
    > caravaggio's death of the virgin pamela askew
     I don't know.
    


### Custom prompt templates

The `stuff` chain type in Langchain uses a specific prompt with question and context documents incorporated. This is what the default prompt looks like:

```text
Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.
{context}
Question: {question}
Helpful Answer:
```

We can, however, provide our prompt template and change the behaviour of the OpenAI LLM, while still using the `stuff` chain type. It is important to keep `{context}` and `{question}` as placeholders.

#### Experimenting with custom prompts

We can try using a different prompt template, so the model:
1. Responds with a single-sentence answer if it knows it.
2. Suggests a random song title if it doesn't know the answer to our question.


```python
from langchain.prompts import PromptTemplate
custom_prompt = """
Use the following pieces of context to answer the question at the end. Please provide
a short single-sentence summary answer only. If you don't know the answer or if it's
not present in given context, don't try to make up an answer, but suggest me a random
unrelated song title I could listen to.
Context: {context}
Question: {question}
Helpful Answer:
"""

custom_prompt_template = PromptTemplate(
    template=custom_prompt, input_variables=["context", "question"]
)
```


```python
custom_qa = VectorDBQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    vectorstore=doc_store,
    return_source_documents=False,
    chain_type_kwargs={"prompt": custom_prompt_template},
)
```


key: _node_content
string_value:
{"id_": "d02b47a0-ac02-4914-a8e2-ee4c45bc07f5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f69e0113-07fc-44a9-8b99-9c24002c1d71", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c275cad15baaccbe5fe38077467806f30fbde690ef94207aa07ed32e8f11a7b6", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "01ebc294-3ff2-401e-964c-3ff17c7a20a5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/tair/QA_with_Langchain_Tair_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Tair and OpenAI", "tokens": 5196, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "0d6879f06409772f1a94532376ba4c0a3467c59452f597125a383a3c9047e923", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17128, "end_char_idx": 19882, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Custom prompt templates

The `stuff` chain type in Langchain uses a specific prompt with question and context documents incorporated. This is what the default prompt looks like:

```text
Use the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.
{context}
Question: {question}
Helpful Answer:
```

We can, however, provide our prompt template and change the behaviour of the OpenAI LLM, while still using the `stuff` chain type. It is important to keep `{context}` and `{question}` as placeholders.

#### Experimenting with custom prompts

We can try using a different prompt template, so the model:
1. Responds with a single-sentence answer if it knows it.
2. Suggests a random song title if it doesn't know the answer to our question.


```python
from langchain.prompts import PromptTemplate
custom_prompt = """
Use the following pieces of context to answer the question at the end. Please provide
a short single-sentence summary answer only. If you don't know the answer or if it's
not present in given context, don't try to make up an answer, but suggest me a random
unrelated song title I could listen to.
Context: {context}
Question: {question}
Helpful Answer:
"""

custom_prompt_template = PromptTemplate(
    template=custom_prompt, input_variables=["context", "question"]
)
```


```python
custom_qa = VectorDBQA.from_chain_type(
    llm=llm,
    chain_type="stuff",
    vectorstore=doc_store,
    return_source_documents=False,
    chain_type_kwargs={"prompt": custom_prompt_template},
)
```


```python
random.seed(41)
for question in random.choices(questions, k=5):
    print(">", question)
    print(custom_qa.run(question), end="\n\n")
    # wait 20seconds because of the rate limit
    time.sleep(20)
```

    > what was uncle jesse's original last name on full house
    Uncle Jesse's original last name on Full House was Cochran.
    
    > when did the volcano erupt in indonesia 2018
    The given context does not mention any volcanic eruption in Indonesia in 2018. Suggested song title: "The Heat Is On" by Glenn Frey.
    
    > what does a dualist way of thinking mean
    Dualism means the belief that there is a distinction between the mind and the body, and that the mind is a non-extended, non-physical substance.
    
    > the first civil service commission in india was set up on the basis of recommendation of
    The first Civil Service Commission in India was not set up on the basis of the recommendation of the Election Commission of India's Model Code of Conduct.
    
    > how old do you have to be to get a tattoo in utah
    You must be at least 18 years old to get a tattoo in Utah.
    



```python

```


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb


key: _node_content
string_value:
{"id_": "7023f9b3-7036-49fb-8a98-88e3b7c61b83", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a38a92ce-0325-4fe1-a524-916bccee8482", "node_type": "1", "metadata": {}, "hash": "85bbd20ac6d4674e6e5c8fa079a7d47157aed9a3937729352a6e129bcd5de8a9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2638, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Retrieval augmented generation using Elasticsearch and OpenAI

[![Open In Colab](https://colab.research.google.com/assets/colab-badge.svg)](openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb)


This notebook demonstrates how to: 
- Index the OpenAI Wikipedia vector dataset into Elasticsearch 
- Embed a question with the OpenAI [`embeddings`](https://platform.openai.com/docs/api-reference/embeddings) endpoint
- Perform semantic search on the Elasticsearch index using the encoded question
- Send the top search results to the OpenAI [Chat Completions](https://platform.openai.com/docs/guides/gpt/chat-completions-api) API endpoint for retrieval augmented generation (RAG)

ℹ️ If you've already worked through our semantic search notebook, you can skip ahead to the final step!

## Install packages and import modules 


```python
# install packages

!python3 -m pip install -qU openai pandas wget elasticsearch

# import modules

from getpass import getpass
from elasticsearch import Elasticsearch, helpers
import wget
import zipfile
import pandas as pd
import json
import openai
```

## Connect to Elasticsearch

ℹ️ We're using an Elastic Cloud deployment of Elasticsearch for this notebook.
If you don't already have an Elastic deployment, you can sign up for a free [Elastic Cloud trial](https://cloud.elastic.co/registration?utm_source=github&utm_content=openai-cookbook).

To connect to Elasticsearch, you need to create a client instance with the Cloud ID and password for your deployment.

Find the Cloud ID for your deployment by going to https://cloud.elastic.co/deployments and selecting your deployment.


```python
CLOUD_ID = getpass("Elastic deployment Cloud ID")
CLOUD_PASSWORD = getpass("Elastic deployment Password")
client = Elasticsearch(
  cloud_id = CLOUD_ID,
  basic_auth=("elastic", CLOUD_PASSWORD) # Alternatively use `api_key` instead of `basic_auth`
)

# Test connection to Elasticsearch
print(client.info())
```

    {'name': 'instance-0000000001', 'cluster_name': '29ef9817e13142f5ba0ea7b29c2a86e2', 'cluster_uuid': 'absjWgQvRw63IlwWKisN8w', 'version': {'number': '8.9.1', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'a813d015ef1826148d9d389bd1c0d781c6e349f0', 'build_date': '2023-08-10T05:02:32.517455352Z', 'build_snapshot': False, 'lucene_version': '9.7.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'}


## Download the dataset 

In this step we download the OpenAI Wikipedia embeddings dataset, and extract the zip file.


key: _node_content
string_value:
{"id_": "a38a92ce-0325-4fe1-a524-916bccee8482", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7023f9b3-7036-49fb-8a98-88e3b7c61b83", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "1c46f99490b679543032d25f06e4b224e162df0996ba9927ca1d30ac7e6fc732", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8d8bf103-bce8-41ac-8532-691d75453c52", "node_type": "1", "metadata": {}, "hash": "31c3dcf3bbea8bddaaddbd3810fffb5f6c399d0603a0fa490f0dcf81a977d41d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1695, "end_char_idx": 4535, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
CLOUD_ID = getpass("Elastic deployment Cloud ID")
CLOUD_PASSWORD = getpass("Elastic deployment Password")
client = Elasticsearch(
  cloud_id = CLOUD_ID,
  basic_auth=("elastic", CLOUD_PASSWORD) # Alternatively use `api_key` instead of `basic_auth`
)

# Test connection to Elasticsearch
print(client.info())
```

    {'name': 'instance-0000000001', 'cluster_name': '29ef9817e13142f5ba0ea7b29c2a86e2', 'cluster_uuid': 'absjWgQvRw63IlwWKisN8w', 'version': {'number': '8.9.1', 'build_flavor': 'default', 'build_type': 'docker', 'build_hash': 'a813d015ef1826148d9d389bd1c0d781c6e349f0', 'build_date': '2023-08-10T05:02:32.517455352Z', 'build_snapshot': False, 'lucene_version': '9.7.0', 'minimum_wire_compatibility_version': '7.17.0', 'minimum_index_compatibility_version': '7.0.0'}, 'tagline': 'You Know, for Search'}


## Download the dataset 

In this step we download the OpenAI Wikipedia embeddings dataset, and extract the zip file.


```python
embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'
wget.download(embeddings_url)

with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip",
"r") as zip_ref:
    zip_ref.extractall("data")
```

##  Read CSV file into a Pandas DataFrame.

Next we use the Pandas library to read the unzipped CSV file into a DataFrame. This step makes it easier to index the data into Elasticsearch in bulk.


```python

wikipedia_dataframe = pd.read_csv("data/vector_database_wikipedia_articles_embedded.csv")
```

## Create index with mapping

Now we need to create an Elasticsearch index with the necessary mappings. This will enable us to index the data into Elasticsearch.

We use the `dense_vector` field type for the `title_vector` and  `content_vector` fields. This is a special field type that allows us to store dense vectors in Elasticsearch.

Later, we'll need to target the `dense_vector` field for kNN search.



```python
index_mapping= {
    "properties": {
      "title_vector": {
          "type": "dense_vector",
          "dims": 1536,
          "index": "true",
          "similarity": "cosine"
      },
      "content_vector": {
          "type": "dense_vector",
          "dims": 1536,
          "index": "true",
          "similarity": "cosine"
      },
      "text": {"type": "text"},
      "title": {"type": "text"},
      "url": { "type": "keyword"},
      "vector_id": {"type": "long"}
      
    }
}

client.indices.create(index="wikipedia_vector_index", mappings=index_mapping)
```

## Index data into Elasticsearch 

The following function generates the required bulk actions that can be passed to Elasticsearch's Bulk API, so we can index multiple documents efficiently in a single request.

For each row in the DataFrame, the function yields a dictionary representing a single document to be indexed.


key: _node_content
string_value:
{"id_": "8d8bf103-bce8-41ac-8532-691d75453c52", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a38a92ce-0325-4fe1-a524-916bccee8482", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "08c83cc82f7581c0ebdf4a4c913d2077cb2c44e9632154ed25f9f5380521a335", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a6154756-3b16-4dd3-b0ba-c1d7671ec092", "node_type": "1", "metadata": {}, "hash": "1136b25c0f517150b6d76193f73b46366aa209a4b6ba0d3d563e65713f8e4a63", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3112, "end_char_idx": 5714, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python

wikipedia_dataframe = pd.read_csv("data/vector_database_wikipedia_articles_embedded.csv")
```

## Create index with mapping

Now we need to create an Elasticsearch index with the necessary mappings. This will enable us to index the data into Elasticsearch.

We use the `dense_vector` field type for the `title_vector` and  `content_vector` fields. This is a special field type that allows us to store dense vectors in Elasticsearch.

Later, we'll need to target the `dense_vector` field for kNN search.



```python
index_mapping= {
    "properties": {
      "title_vector": {
          "type": "dense_vector",
          "dims": 1536,
          "index": "true",
          "similarity": "cosine"
      },
      "content_vector": {
          "type": "dense_vector",
          "dims": 1536,
          "index": "true",
          "similarity": "cosine"
      },
      "text": {"type": "text"},
      "title": {"type": "text"},
      "url": { "type": "keyword"},
      "vector_id": {"type": "long"}
      
    }
}

client.indices.create(index="wikipedia_vector_index", mappings=index_mapping)
```

## Index data into Elasticsearch 

The following function generates the required bulk actions that can be passed to Elasticsearch's Bulk API, so we can index multiple documents efficiently in a single request.

For each row in the DataFrame, the function yields a dictionary representing a single document to be indexed. 


```python
def dataframe_to_bulk_actions(df):
    for index, row in df.iterrows():
        yield {
            "_index": 'wikipedia_vector_index',
            "_id": row['id'],
            "_source": {
                'url' : row["url"],
                'title' : row["title"],
                'text' : row["text"],
                'title_vector' : json.loads(row["title_vector"]),
                'content_vector' : json.loads(row["content_vector"]),
                'vector_id' : row["vector_id"]
            }
        }
```

As the dataframe is large, we will index data in batches of `100`. We index the data into Elasticsearch using the Python client's [helpers](https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/client-helpers.html#bulk-helpers) for the bulk API.


```python
start = 0
end = len(wikipedia_dataframe)
batch_size = 100
for batch_start in range(start, end, batch_size):
    batch_end = min(batch_start + batch_size, end)
    batch_dataframe = wikipedia_dataframe.iloc[batch_start:batch_end]
    actions = dataframe_to_bulk_actions(batch_dataframe)
    helpers.bulk(client, actions)
```

Let's test the index with a simple match query.


key: _node_content
string_value:
{"id_": "a6154756-3b16-4dd3-b0ba-c1d7671ec092", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8d8bf103-bce8-41ac-8532-691d75453c52", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "ee91fe4688489c47fe895a00315723346ad73c8197b122386306f514cd0c3951", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a710b49c-ffba-4221-a10d-0534dbac4fad", "node_type": "1", "metadata": {}, "hash": "0e706173ca3ca409ab690a01c492ebc1af9c506ab83eaeb32af24135becb4f17", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4539, "end_char_idx": 7528, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def dataframe_to_bulk_actions(df):
    for index, row in df.iterrows():
        yield {
            "_index": 'wikipedia_vector_index',
            "_id": row['id'],
            "_source": {
                'url' : row["url"],
                'title' : row["title"],
                'text' : row["text"],
                'title_vector' : json.loads(row["title_vector"]),
                'content_vector' : json.loads(row["content_vector"]),
                'vector_id' : row["vector_id"]
            }
        }
```

As the dataframe is large, we will index data in batches of `100`. We index the data into Elasticsearch using the Python client's [helpers](https://www.elastic.co/guide/en/elasticsearch/client/python-api/current/client-helpers.html#bulk-helpers) for the bulk API.


```python
start = 0
end = len(wikipedia_dataframe)
batch_size = 100
for batch_start in range(start, end, batch_size):
    batch_end = min(batch_start + batch_size, end)
    batch_dataframe = wikipedia_dataframe.iloc[batch_start:batch_end]
    actions = dataframe_to_bulk_actions(batch_dataframe)
    helpers.bulk(client, actions)
```

Let's test the index with a simple match query.


```python
print(client.search(index="wikipedia_vector_index", body={
    "_source": {
        "excludes": ["title_vector", "content_vector"]
    },
    "query": {
        "match": {
            "text": {
                "query": "Hummingbird"
            }
        }
    }
}))
```

    {'took': 10, 'timed_out': False, '_shards': {'total': 1, 'successful': 1, 'skipped': 0, 'failed': 0}, 'hits': {'total': {'value': 4, 'relation': 'eq'}, 'max_score': 14.917897, 'hits': [{'_index': 'wikipedia_vector_index', '_id': '34227', '_score': 14.917897, '_source': {'url': 'https://simple.wikipedia.org/wiki/Hummingbird', 'title': 'Hummingbird', 'text': "Hummingbirds are small birds of the family Trochilidae.\n\nThey are among the smallest of birds: most species measure 7.5–13\xa0cm (3–5\xa0in). The smallest living bird species is the 2–5\xa0cm Bee Hummingbird. They can hover in mid-air by rapidly flapping their wings 12–80 times per second (depending on the species). They are also the only group of birds able to fly backwards. Their rapid wing beats do actually hum. They can fly at speeds over 15\xa0m/s (54\xa0km/h, 34\xa0mi/h).\n\nEating habits and pollination \nHummingbirds help flowers to pollinate, though most insects are best known for doing so. The hummingbird enjoys nectar, like the butterfly and other flower-loving insects, such as bees.\n\nHummingbirds do not have a good sense of smell; instead, they are attracted to color, especially the color red. Unlike the butterfly, the hummingbird hovers over the flower as it drinks nectar from it, like a moth. When it does so, it flaps its wings very quickly to stay in one place, which makes it look like a blur and also beats so fast it makes a humming sound. A hummingbird sometimes puts its whole head into the flower to drink the nectar properly.


key: _node_content
string_value:
{"id_": "a710b49c-ffba-4221-a10d-0534dbac4fad", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a6154756-3b16-4dd3-b0ba-c1d7671ec092", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "6f676238bb22f53b4806cf677d33f75c8b613f5f5d6505d2d9646d2c736a29b1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6fa8a90a-43b5-49f8-8d28-6027d977d119", "node_type": "1", "metadata": {}, "hash": "955a9ce0c51deed8620c945917c6226193f9afa3418da050b4646f7d7fa5527b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6507, "end_char_idx": 9560, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The smallest living bird species is the 2–5\xa0cm Bee Hummingbird. They can hover in mid-air by rapidly flapping their wings 12–80 times per second (depending on the species). They are also the only group of birds able to fly backwards. Their rapid wing beats do actually hum. They can fly at speeds over 15\xa0m/s (54\xa0km/h, 34\xa0mi/h).\n\nEating habits and pollination \nHummingbirds help flowers to pollinate, though most insects are best known for doing so. The hummingbird enjoys nectar, like the butterfly and other flower-loving insects, such as bees.\n\nHummingbirds do not have a good sense of smell; instead, they are attracted to color, especially the color red. Unlike the butterfly, the hummingbird hovers over the flower as it drinks nectar from it, like a moth. When it does so, it flaps its wings very quickly to stay in one place, which makes it look like a blur and also beats so fast it makes a humming sound. A hummingbird sometimes puts its whole head into the flower to drink the nectar properly. When it takes its head back out, its head is covered with yellow pollen, so that when it moves to another flower, it can pollinate. Or sometimes it may pollinate with its beak.\n\nLike bees, hummingbirds can assess the amount of sugar in the nectar they eat. They reject flowers whose nectar has less than 10% sugar. Nectar is a poor source of nutrients, so hummingbirds meet their needs for protein, amino acids, vitamins, minerals, etc. by preying on insects and spiders.\n\nFeeding apparatus \nMost hummingbirds have bills that are long and straight or nearly so, but in some species the bill shape is adapted for specialized feeding. Thornbills have short, sharp bills adapted for feeding from flowers with short corollas and piercing the bases of longer ones. The Sicklebills' extremely decurved bills are adapted to extracting nectar from the curved corollas of flowers in the family Gesneriaceae. The bill of the Fiery-tailed Awlbill has an upturned tip, as in the Avocets. The male Tooth-billed Hummingbird has barracuda-like spikes at the tip of its long, straight bill.\n\nThe two halves of a hummingbird's bill have a pronounced overlap, with the lower half (mandible) fitting tightly inside the upper half (maxilla). When hummingbirds feed on nectar, the bill is usually only opened slightly, allowing the tongue to dart out into the nectar.\n\nLike the similar nectar-feeding sunbirds and unlike other birds, hummingbirds drink by using grooved or trough-like tongues which they can stick out a long way.\nHummingbirds do not spend all day flying, as the energy cost would be prohibitive; the majority of their activity consists simply of sitting or perching. Hummingbirds feed in many small meals, consuming many small invertebrates and up to twelve times their own body weight in nectar each day. They spend an average of 10–15% of their time feeding and 75–80% sitting and digesting.\n\nCo-evolution with flowers\n\nSince hummingbirds are specialized nectar-eaters, they are tied to the bird-flowers they feed upon.


key: _node_content
string_value:
{"id_": "6fa8a90a-43b5-49f8-8d28-6027d977d119", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a710b49c-ffba-4221-a10d-0534dbac4fad", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "ccfefafb53a100b780c9cd5aaafacf1aac18c3618757e0fe72722376e975a6a2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fb3b1bec-c046-46bc-b5bf-8a8c153f5bc3", "node_type": "1", "metadata": {}, "hash": "7d34a5cf04c2e708f27695542d65716e663adc8ea99304519daed92f7acf85a8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7968, "end_char_idx": 10909, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
by preying on insects and spiders.\n\nFeeding apparatus \nMost hummingbirds have bills that are long and straight or nearly so, but in some species the bill shape is adapted for specialized feeding. Thornbills have short, sharp bills adapted for feeding from flowers with short corollas and piercing the bases of longer ones. The Sicklebills' extremely decurved bills are adapted to extracting nectar from the curved corollas of flowers in the family Gesneriaceae. The bill of the Fiery-tailed Awlbill has an upturned tip, as in the Avocets. The male Tooth-billed Hummingbird has barracuda-like spikes at the tip of its long, straight bill.\n\nThe two halves of a hummingbird's bill have a pronounced overlap, with the lower half (mandible) fitting tightly inside the upper half (maxilla). When hummingbirds feed on nectar, the bill is usually only opened slightly, allowing the tongue to dart out into the nectar.\n\nLike the similar nectar-feeding sunbirds and unlike other birds, hummingbirds drink by using grooved or trough-like tongues which they can stick out a long way.\nHummingbirds do not spend all day flying, as the energy cost would be prohibitive; the majority of their activity consists simply of sitting or perching. Hummingbirds feed in many small meals, consuming many small invertebrates and up to twelve times their own body weight in nectar each day. They spend an average of 10–15% of their time feeding and 75–80% sitting and digesting.\n\nCo-evolution with flowers\n\nSince hummingbirds are specialized nectar-eaters, they are tied to the bird-flowers they feed upon. Some species, especially those with unusual bill shapes such as the Sword-billed Hummingbird and the sicklebills, are co-evolved with a small number of flower species.\n\nMany plants pollinated by hummingbirds produce flowers in shades of red, orange, and bright pink, though the birds will take nectar from flowers of many colors. Hummingbirds can see wavelengths into the near-ultraviolet. However, their flowers do not reflect these wavelengths as many insect-pollinated flowers do. The narrow color spectrum may make hummingbird-pollinated flowers inconspicuous to insects, thereby reducing nectar robbing by insects. Hummingbird-pollinated flowers also produce relatively weak nectar (averaging 25% sugars w/w) containing high concentrations of sucrose, whereas insect-pollinated flowers typically produce more concentrated nectars dominated by fructose and glucose.\n\nTaxonomy \nHummingbirds have traditionally been a part of the bird order Apodiformes. This order includes the hummingbirds, the swifts and the tree swifts. The Sibley-Ahlquist taxonomy of birds, based on DNA studies done in the 1970s and 1980s, changed the classification of hummingbirds. Instead of being in the same order as the swifts, the hummingbirds were made an order, the Trochiliformes. Their previous order, Apodiformes was changed to the superorder Apodimorphae.


key: _node_content
string_value:
{"id_": "fb3b1bec-c046-46bc-b5bf-8a8c153f5bc3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6fa8a90a-43b5-49f8-8d28-6027d977d119", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "9b7425989e35e43c90837f9199fc9de6ec30d84a45a018188673c4bc40eef11e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f5574a34-e58b-40f0-a9e0-b915cbc65446", "node_type": "1", "metadata": {}, "hash": "6e48e53a858e28fb475bfa0f70ad1a5468b51d8b8001d092960fcf8f6f47db98", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9202, "end_char_idx": 11894, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Hummingbirds feed in many small meals, consuming many small invertebrates and up to twelve times their own body weight in nectar each day. They spend an average of 10–15% of their time feeding and 75–80% sitting and digesting.\n\nCo-evolution with flowers\n\nSince hummingbirds are specialized nectar-eaters, they are tied to the bird-flowers they feed upon. Some species, especially those with unusual bill shapes such as the Sword-billed Hummingbird and the sicklebills, are co-evolved with a small number of flower species.\n\nMany plants pollinated by hummingbirds produce flowers in shades of red, orange, and bright pink, though the birds will take nectar from flowers of many colors. Hummingbirds can see wavelengths into the near-ultraviolet. However, their flowers do not reflect these wavelengths as many insect-pollinated flowers do. The narrow color spectrum may make hummingbird-pollinated flowers inconspicuous to insects, thereby reducing nectar robbing by insects. Hummingbird-pollinated flowers also produce relatively weak nectar (averaging 25% sugars w/w) containing high concentrations of sucrose, whereas insect-pollinated flowers typically produce more concentrated nectars dominated by fructose and glucose.\n\nTaxonomy \nHummingbirds have traditionally been a part of the bird order Apodiformes. This order includes the hummingbirds, the swifts and the tree swifts. The Sibley-Ahlquist taxonomy of birds, based on DNA studies done in the 1970s and 1980s, changed the classification of hummingbirds. Instead of being in the same order as the swifts, the hummingbirds were made an order, the Trochiliformes. Their previous order, Apodiformes was changed to the superorder Apodimorphae. This superorder contains the three families of birds which were in it when it was an order.\n\nReferences", 'vector_id': 10024}}, {'_index': 'wikipedia_vector_index', '_id': '84773', '_score': 10.951234, '_source': {'url': 'https://simple.wikipedia.org/wiki/Inagua', 'title': 'Inagua', 'text': "Inagua is the southernmost district of the Bahamas.  It is the islands of Great Inagua and Little Inagua.\n\nGreat Inagua is the third largest island in the Bahamas at 596 square miles (1544\xa0km²) and lies about 55 miles (90\xa0km) from the eastern tip of Cuba. The island is about 55 × 19 miles (90 × 30\xa0km) in extent, the highest point being 108\xa0ft (33 m) on East Hill. It encloses several lakes, most notably the 12-mile long Lake Windsor (also called Lake Rosa) which occupies nearly ¼ of the interior. The population of Great Inagua is 969 (2000 census).\n\nThe island's capital and only harbour is Matthew Town.\n\nThere is a large bird sanctuary in the centre of the island.


key: _node_content
string_value:
{"id_": "f5574a34-e58b-40f0-a9e0-b915cbc65446", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fb3b1bec-c046-46bc-b5bf-8a8c153f5bc3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "ecbce75d02fcca3ef387a299f49634dccc3e9725e8e8d91c5c9a3c56d9b20888", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "496627d7-8f6a-46b2-840c-50201feac5ba", "node_type": "1", "metadata": {}, "hash": "3e975620fb2050ded3e23972ae210155e3f74e86ac7cb405cc063bd9477c2fb8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10522, "end_char_idx": 13359, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This order includes the hummingbirds, the swifts and the tree swifts. The Sibley-Ahlquist taxonomy of birds, based on DNA studies done in the 1970s and 1980s, changed the classification of hummingbirds. Instead of being in the same order as the swifts, the hummingbirds were made an order, the Trochiliformes. Their previous order, Apodiformes was changed to the superorder Apodimorphae. This superorder contains the three families of birds which were in it when it was an order.\n\nReferences", 'vector_id': 10024}}, {'_index': 'wikipedia_vector_index', '_id': '84773', '_score': 10.951234, '_source': {'url': 'https://simple.wikipedia.org/wiki/Inagua', 'title': 'Inagua', 'text': "Inagua is the southernmost district of the Bahamas.  It is the islands of Great Inagua and Little Inagua.\n\nGreat Inagua is the third largest island in the Bahamas at 596 square miles (1544\xa0km²) and lies about 55 miles (90\xa0km) from the eastern tip of Cuba. The island is about 55 × 19 miles (90 × 30\xa0km) in extent, the highest point being 108\xa0ft (33 m) on East Hill. It encloses several lakes, most notably the 12-mile long Lake Windsor (also called Lake Rosa) which occupies nearly ¼ of the interior. The population of Great Inagua is 969 (2000 census).\n\nThe island's capital and only harbour is Matthew Town.\n\nThere is a large bird sanctuary in the centre of the island.  There are more than 80,000 West Indian Flamingoes and many other exotic birds such as the native Bahama Parrot, the Bahama woodstar hummingbird, Bahama pintails, Brown pelicans, Tri-colored herons, Snowy egrets, Reddish egrets, Stripe-headed tanangers, Cormorants, Roseate spoonbills, American kestrels, and Burrowing owls.\n\nDistricts of the Bahamas\nIslands of the Bahamas\n1999 establishments in the Bahamas", 'vector_id': 22383}}, {'_index': 'wikipedia_vector_index', '_id': '3707', '_score': 1.1967773, '_source': {'url': 'https://simple.wikipedia.org/wiki/Bird', 'title': 'Bird', 'text': 'Birds (Aves) are a group of animals with backbones which evolved from dinosaurs. Technically speaking, they are dinosaurs. \n\nBirds are endothermic. The heat loss from their bodies is slowed down by their feathers. \nModern birds are toothless: they have beaked jaws. They lay hard-shelled eggs. They have a high metabolic rate, a four-chambered heart and a strong yet lightweight skeleton.\n\nBirds live all over the world. They range in size from the 5 cm (2 in) bee hummingbird to the 2.70 m (9 ft) ostrich. They are the tetrapods with the most living species: about ten thousand. More than half of these are passerines, sometimes known as perching birds.\n\nBirds are the closest living relatives of the Crocodilia. This is because they are the two main survivors of a once huge group called the Archosaurs. \n\nModern birds are not descended from Archaeopteryx.


key: _node_content
string_value:
{"id_": "496627d7-8f6a-46b2-840c-50201feac5ba", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f5574a34-e58b-40f0-a9e0-b915cbc65446", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "c5fca8e2971a5c8dbe9ff7438d42b4b759e4d931f44a33dd31c5a822c4ac4fc7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0114807d-6bc2-4eb0-aff7-e5fe9bd3b42a", "node_type": "1", "metadata": {}, "hash": "ab69c4204450f48c1bb5a2fccc13a8b0bddc0afdc487996143dce1f2682bf60b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11896, "end_char_idx": 14994, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
There are more than 80,000 West Indian Flamingoes and many other exotic birds such as the native Bahama Parrot, the Bahama woodstar hummingbird, Bahama pintails, Brown pelicans, Tri-colored herons, Snowy egrets, Reddish egrets, Stripe-headed tanangers, Cormorants, Roseate spoonbills, American kestrels, and Burrowing owls.\n\nDistricts of the Bahamas\nIslands of the Bahamas\n1999 establishments in the Bahamas", 'vector_id': 22383}}, {'_index': 'wikipedia_vector_index', '_id': '3707', '_score': 1.1967773, '_source': {'url': 'https://simple.wikipedia.org/wiki/Bird', 'title': 'Bird', 'text': 'Birds (Aves) are a group of animals with backbones which evolved from dinosaurs. Technically speaking, they are dinosaurs. \n\nBirds are endothermic. The heat loss from their bodies is slowed down by their feathers. \nModern birds are toothless: they have beaked jaws. They lay hard-shelled eggs. They have a high metabolic rate, a four-chambered heart and a strong yet lightweight skeleton.\n\nBirds live all over the world. They range in size from the 5 cm (2 in) bee hummingbird to the 2.70 m (9 ft) ostrich. They are the tetrapods with the most living species: about ten thousand. More than half of these are passerines, sometimes known as perching birds.\n\nBirds are the closest living relatives of the Crocodilia. This is because they are the two main survivors of a once huge group called the Archosaurs. \n\nModern birds are not descended from Archaeopteryx. According to DNA evidence, modern birds (Neornithes) evolved in the long Upper Cretaceous period. More recent estimates showed that  modern birds originated early in the Upper Cretaceous.\n\nPrimitive bird-like dinosaurs are in the broader group Avialae. They have been found back to the mid-Jurassic period, around 170 million years ago. Many of these early "stem-birds", such as Anchiornis, were not yet capable of fully powered flight. Many had primitive characteristics like teeth in their jaws and long bony tails.p274\n\nThe Cretaceous–Palaeogene extinction event 66 million years ago killed off all the non-avian dinosaur lines. Birds, especially those in the southern continents, survived this event and then migrated to other parts of the world. Diversification occurred around the Cretaceous–Palaeogene extinction event. \n\nBirds have wings which are more or less developed depending on the species. The only known groups without wings are the extinct moa and elephant birds. Wings, which evolved from forelimbs, gave birds the ability to fly. Later, many groups evolved with reduced wings, such as ratites, penguins and many island species of birds. The digestive and respiratory systems of birds are also adapted for flight. Some bird species in aquatic environments, particularly seabirds and some waterbirds, have evolved as good swimmers.\n\nIn general, birds are effective, and inherit their behaviour almost entirely. The key elements of their life are inherited. It was a great discovery that birds never learn to fly. \nSo it is quite wrong to say, when a chick waves its wings in the nest "It\'s learning to fly".


key: _node_content
string_value:
{"id_": "0114807d-6bc2-4eb0-aff7-e5fe9bd3b42a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "496627d7-8f6a-46b2-840c-50201feac5ba", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "06b5db7a791ab5e0feb83e67ff89ce9434bd43a21e41f20de30dea2bc601fe96", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e5640aca-dce5-426b-b300-18bec7fa7e5c", "node_type": "1", "metadata": {}, "hash": "9aea84059c02862693de4eb49d6b6b888921cc673133d536c5b97228c22ec7f8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13305, "end_char_idx": 16701, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
\n\nModern birds are not descended from Archaeopteryx. According to DNA evidence, modern birds (Neornithes) evolved in the long Upper Cretaceous period. More recent estimates showed that  modern birds originated early in the Upper Cretaceous.\n\nPrimitive bird-like dinosaurs are in the broader group Avialae. They have been found back to the mid-Jurassic period, around 170 million years ago. Many of these early "stem-birds", such as Anchiornis, were not yet capable of fully powered flight. Many had primitive characteristics like teeth in their jaws and long bony tails.p274\n\nThe Cretaceous–Palaeogene extinction event 66 million years ago killed off all the non-avian dinosaur lines. Birds, especially those in the southern continents, survived this event and then migrated to other parts of the world. Diversification occurred around the Cretaceous–Palaeogene extinction event. \n\nBirds have wings which are more or less developed depending on the species. The only known groups without wings are the extinct moa and elephant birds. Wings, which evolved from forelimbs, gave birds the ability to fly. Later, many groups evolved with reduced wings, such as ratites, penguins and many island species of birds. The digestive and respiratory systems of birds are also adapted for flight. Some bird species in aquatic environments, particularly seabirds and some waterbirds, have evolved as good swimmers.\n\nIn general, birds are effective, and inherit their behaviour almost entirely. The key elements of their life are inherited. It was a great discovery that birds never learn to fly. \nSo it is quite wrong to say, when a chick waves its wings in the nest "It\'s learning to fly". What the chick is doing is exercising its muscles. They develop the ability to fly automatically (assuming they are species that do fly). And if they are species which migrate, that behaviour is also inherited. Many species migrate over great distances each year. Other main features of their life may be inherited, though they can and do learn. Birds have good memories which they use, for example, when they search for food.\n\nSeveral bird species make and use tools. Some social species pass on some knowledge across generations, a form of culture. Birds are social. They communicate with visual signals, calls and bird songs. Most of their social behaviours are inherited, such as cooperative breeding and hunting, flocking and mobbing of predators.\n\nMost bird species are socially monogamous, usually for one breeding season at a time, sometimes for years, but rarely for life. Other species are polygynous (one male with many females) or, rarely, polyandrous (one female with many males). Birds produce offspring by laying eggs which are fertilised by sexual reproduction. They are often laid in a nest and incubated by the parents. Most birds have an extended period of parental care after hatching. Some birds, such as hens, lay eggs even when not fertilised, though unfertilised eggs do not produce offspring.\n\nMany species of birds are eaten by humans. Domesticated and undomesticated birds are sources of eggs, meat, and feathers.  In English, domesticated birds are often called poultry, undomesticated birds are called game. Songbirds, parrots and other species are popular as pets. Guano, which is bird manure, is harvested for use as a fertiliser. Birds figure throughout human culture.


key: _node_content
string_value:
{"id_": "e5640aca-dce5-426b-b300-18bec7fa7e5c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0114807d-6bc2-4eb0-aff7-e5fe9bd3b42a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f7252bd2bd1bca4e213597f4fc67987a2e17d38473fbc819170103f25a0274e6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "45e7d917-5bf4-4f9e-9983-993938291359", "node_type": "1", "metadata": {}, "hash": "4ff555ebb801278dd8d3231db2b6933ee708a27c6a90d9a3ba404c5ceb460fff", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14995, "end_char_idx": 18286, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
What the chick is doing is exercising its muscles. They develop the ability to fly automatically (assuming they are species that do fly). And if they are species which migrate, that behaviour is also inherited. Many species migrate over great distances each year. Other main features of their life may be inherited, though they can and do learn. Birds have good memories which they use, for example, when they search for food.\n\nSeveral bird species make and use tools. Some social species pass on some knowledge across generations, a form of culture. Birds are social. They communicate with visual signals, calls and bird songs. Most of their social behaviours are inherited, such as cooperative breeding and hunting, flocking and mobbing of predators.\n\nMost bird species are socially monogamous, usually for one breeding season at a time, sometimes for years, but rarely for life. Other species are polygynous (one male with many females) or, rarely, polyandrous (one female with many males). Birds produce offspring by laying eggs which are fertilised by sexual reproduction. They are often laid in a nest and incubated by the parents. Most birds have an extended period of parental care after hatching. Some birds, such as hens, lay eggs even when not fertilised, though unfertilised eggs do not produce offspring.\n\nMany species of birds are eaten by humans. Domesticated and undomesticated birds are sources of eggs, meat, and feathers.  In English, domesticated birds are often called poultry, undomesticated birds are called game. Songbirds, parrots and other species are popular as pets. Guano, which is bird manure, is harvested for use as a fertiliser. Birds figure throughout human culture. About 120–130 species have become extinct due to human activity since the 17th century and hundreds more before then. Human activity threatens about 1,200 bird species with extinction, though efforts are underway to protect them. Recreational bird-watching is an important part of the ecotourism industry.\n\nBird colours \n\nBirds come in a huge range of colours. These colours can be useful to a bird in two ways. Camouflage colours help to hide the bird, and bright colours identify the bird to others of the same species. Often the male is brightly coloured while the female is camouflaged. The logic is as follows: the female carries the "precious package" of developing eggs. The male has to defend a territory, and the function of his colour and song is to let others know that "this place is occupied".\n\nBird camouflage \n\nMany birds are brown, green or grey. These colours make a bird harder to be seen: they camouflage the bird. Brown is the most common colour. Brown birds include: sparrows, emus, thrushes, larks, eagles and falcons and the female birds of many species such as: wrens, ducks, blackbirds and peafowls. When a brown bird is in long grass or among tree trunks or rocks, it is camouflaged. Birds that live in long grass often have brown feathers streaked with black which looks like shadows. A bittern is almost invisible in long reeds because its camouflage is helped by its posture (beak and head pointed upwards). Other birds, including starlings and mynas, are quite dark in colour, but are flecked with little spots that look like raindrops on leaves.


key: _node_content
string_value:
{"id_": "45e7d917-5bf4-4f9e-9983-993938291359", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e5640aca-dce5-426b-b300-18bec7fa7e5c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "2e6e1d8ffcdd32c8b198d63c6b213b0c2e2a8cd9001b2b699f2bcf507a7f552a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6ca790a6-08e9-4400-96c9-43c00b9a5b68", "node_type": "1", "metadata": {}, "hash": "337e29a804c5cc9d98cec310af26dabaafc096b2953dd915214f4baa56a95661", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16663, "end_char_idx": 19827, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Birds figure throughout human culture. About 120–130 species have become extinct due to human activity since the 17th century and hundreds more before then. Human activity threatens about 1,200 bird species with extinction, though efforts are underway to protect them. Recreational bird-watching is an important part of the ecotourism industry.\n\nBird colours \n\nBirds come in a huge range of colours. These colours can be useful to a bird in two ways. Camouflage colours help to hide the bird, and bright colours identify the bird to others of the same species. Often the male is brightly coloured while the female is camouflaged. The logic is as follows: the female carries the "precious package" of developing eggs. The male has to defend a territory, and the function of his colour and song is to let others know that "this place is occupied".\n\nBird camouflage \n\nMany birds are brown, green or grey. These colours make a bird harder to be seen: they camouflage the bird. Brown is the most common colour. Brown birds include: sparrows, emus, thrushes, larks, eagles and falcons and the female birds of many species such as: wrens, ducks, blackbirds and peafowls. When a brown bird is in long grass or among tree trunks or rocks, it is camouflaged. Birds that live in long grass often have brown feathers streaked with black which looks like shadows. A bittern is almost invisible in long reeds because its camouflage is helped by its posture (beak and head pointed upwards). Other birds, including starlings and mynas, are quite dark in colour, but are flecked with little spots that look like raindrops on leaves. Bird may also camouflage their nests.\n\nMany birds from hot countries are green or have some green feathers, particularly parrots. Birds that live in green trees often have green backs, even if they have bright-coloured breasts. From the back, the birds are camouflaged. This is very useful when sitting on a nest. The bird\'s bright-coloured breast is hidden. Budgerigars are bred in different colours such as blue, white and mauve, but in the wild, they are nearly all green and yellow. Even though they fly very well, they normally spend a lot of time on the ground, eating grass seeds. Their yellow and black striped back helps to hide them in the shadows made by long dry grass, while their green breasts are a similar colour to the leaves of gum trees.\n\nGrey birds include most pigeons and doves, cranes, storks and herons. Grey birds are often rock-living birds like pigeons or birds that sit on dead tree trunks looking like a broken branch. Water birds like herons often have a pale grey colour which makes it harder for a fish to notice that the bird is standing, looking down for something to catch. Water birds, no matter what colour they are on top, are often white underneath, so that when a fish looks up, the bird looks like part of the sky.\n\nBlack birds include crows, ravens and male blackbirds. Some birds that are dark colours spend quite a lot of time on the ground, hopping around in the shadows under bushes. Among these birds are the male blackbird and the satin bowerbird which is not black but very dark blue.


key: _node_content
string_value:
{"id_": "6ca790a6-08e9-4400-96c9-43c00b9a5b68", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "45e7d917-5bf4-4f9e-9983-993938291359", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "becf73eacb34007317a2db2ab0b7cf5a11b7a6f8f5e72e609c0322901939422d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2f87efc0-103b-44b4-b2fb-98b15bf078c3", "node_type": "1", "metadata": {}, "hash": "5dfa12be929b945639a374052707ba49e87460935305eee7fa8f9c08665d642b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18147, "end_char_idx": 21387, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Other birds, including starlings and mynas, are quite dark in colour, but are flecked with little spots that look like raindrops on leaves. Bird may also camouflage their nests.\n\nMany birds from hot countries are green or have some green feathers, particularly parrots. Birds that live in green trees often have green backs, even if they have bright-coloured breasts. From the back, the birds are camouflaged. This is very useful when sitting on a nest. The bird\'s bright-coloured breast is hidden. Budgerigars are bred in different colours such as blue, white and mauve, but in the wild, they are nearly all green and yellow. Even though they fly very well, they normally spend a lot of time on the ground, eating grass seeds. Their yellow and black striped back helps to hide them in the shadows made by long dry grass, while their green breasts are a similar colour to the leaves of gum trees.\n\nGrey birds include most pigeons and doves, cranes, storks and herons. Grey birds are often rock-living birds like pigeons or birds that sit on dead tree trunks looking like a broken branch. Water birds like herons often have a pale grey colour which makes it harder for a fish to notice that the bird is standing, looking down for something to catch. Water birds, no matter what colour they are on top, are often white underneath, so that when a fish looks up, the bird looks like part of the sky.\n\nBlack birds include crows, ravens and male blackbirds. Some birds that are dark colours spend quite a lot of time on the ground, hopping around in the shadows under bushes. Among these birds are the male blackbird and the satin bowerbird which is not black but very dark blue. Crows and ravens often perch high on bare trees in the winter, where their black shape against the sky looks like the dark bare branches.\n\nNoticeable colours \n\nMany birds are not camouflaged, but stand out with vivid colours. They are usually male birds whose females are dull and camouflaged. The function of the colours is two-fold. First, the colours help them get mates, and second, the colours identify them to other males of the same species. Many birds are territorial, especially in the nesting season. They give out territory sounds and are easily seen. This lets other males know they will defend their territory. It sends out a "look elsewhere" signal to their competitors.\n\nSome birds are famous for their colour and are named for it, such as the bluebird, the azure kingfisher, the golden pheasant, the scarlet macaw, the violet wren and the robin.\n\nMany other birds are very brightly coloured, in countless combinations. Some of the most colourful birds are quite common, like pheasants, peacocks, domestic fowl and parrots. Colourful small birds include blue tits, the goldfinches, hummingbirds, fairy wrens and bee eaters (which are also called rainbow birds). Some birds, like those of the bird of paradise in Papua New Guinea have such beautiful feathers that they have been hunted for them.\n\nThe peafowl is the best example of a display of colour to attract a mate. Also the male domestic fowl and junglefowl have long shiny feathers above his tail and also long neck feathers that may be a different colour to his wings and body.


key: _node_content
string_value:
{"id_": "2f87efc0-103b-44b4-b2fb-98b15bf078c3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6ca790a6-08e9-4400-96c9-43c00b9a5b68", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d7051733277b1c9305f55defc494e15d4c44664e47a89c126c1b59a6b59ebb94", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8cabfb27-774f-4c28-9ce7-2f014e387db3", "node_type": "1", "metadata": {}, "hash": "341d7bdfcb58f7550480c1a6e979b41759a9e5687700fff611133c27cad35836", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19724, "end_char_idx": 22985, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Among these birds are the male blackbird and the satin bowerbird which is not black but very dark blue. Crows and ravens often perch high on bare trees in the winter, where their black shape against the sky looks like the dark bare branches.\n\nNoticeable colours \n\nMany birds are not camouflaged, but stand out with vivid colours. They are usually male birds whose females are dull and camouflaged. The function of the colours is two-fold. First, the colours help them get mates, and second, the colours identify them to other males of the same species. Many birds are territorial, especially in the nesting season. They give out territory sounds and are easily seen. This lets other males know they will defend their territory. It sends out a "look elsewhere" signal to their competitors.\n\nSome birds are famous for their colour and are named for it, such as the bluebird, the azure kingfisher, the golden pheasant, the scarlet macaw, the violet wren and the robin.\n\nMany other birds are very brightly coloured, in countless combinations. Some of the most colourful birds are quite common, like pheasants, peacocks, domestic fowl and parrots. Colourful small birds include blue tits, the goldfinches, hummingbirds, fairy wrens and bee eaters (which are also called rainbow birds). Some birds, like those of the bird of paradise in Papua New Guinea have such beautiful feathers that they have been hunted for them.\n\nThe peafowl is the best example of a display of colour to attract a mate. Also the male domestic fowl and junglefowl have long shiny feathers above his tail and also long neck feathers that may be a different colour to his wings and body. There are only a very few types of birds (like the eclectus parrot) where the female is more colourful than the male.\n\n\'\'Pied birds\'\' are black and white. Black and white birds include magpies, pied geese, pelicans and Australian magpies (which are not really magpies at all). Pied birds often have brightly coloured beaks and legs of yellow or red. The silver pheasant, with its long white tail striped with fine bars of black, has a brightly coloured face.\n\nFlight \nMost birds can fly, and if they do, then the ability is inherited, not learnt. They fly by pushing through the air with their wings. The curved surfaces of the wings cause air currents (wind) which lift the bird. Flapping keeps the air current moving to create lift and also moves the bird forward.\n\nSome birds can glide on air currents without flapping. Many birds use this method when they are about to land. Some birds can also hover in the air. This method is used by birds of prey such as falcons that are looking for something to eat. Seagulls are also good at hovering, particularly if there is a strong breeze. The most expert hovering birds are tiny hummingbirds which can beat their wings both backwards and forwards and can stay quite still in the air while they dip their long beaks into flowers to feed on the sweet nectar.\n\nTypes of flight \nDifferent types of birds have different needs. Their wings have evolved to suit their lifestyle. Large birds of prey, such as eagles, spend a lot of time soaring on the wind. They have wings that are large and broad. The main flight feathers are long and wide.


key: _node_content
string_value:
{"id_": "8cabfb27-774f-4c28-9ce7-2f014e387db3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2f87efc0-103b-44b4-b2fb-98b15bf078c3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "6d579b6ad913cb326a63f011c02eadc1a117b369d5965073486e73f35e44f30b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9b33a0e8-1a00-4d5e-8ad4-d4077d1a9d0d", "node_type": "1", "metadata": {}, "hash": "dfc87a44f1f5be6bd9e8d9d7beafde4cfad28080f5746a71a3577180d2924be4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21388, "end_char_idx": 24503, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
There are only a very few types of birds (like the eclectus parrot) where the female is more colourful than the male.\n\n\'\'Pied birds\'\' are black and white. Black and white birds include magpies, pied geese, pelicans and Australian magpies (which are not really magpies at all). Pied birds often have brightly coloured beaks and legs of yellow or red. The silver pheasant, with its long white tail striped with fine bars of black, has a brightly coloured face.\n\nFlight \nMost birds can fly, and if they do, then the ability is inherited, not learnt. They fly by pushing through the air with their wings. The curved surfaces of the wings cause air currents (wind) which lift the bird. Flapping keeps the air current moving to create lift and also moves the bird forward.\n\nSome birds can glide on air currents without flapping. Many birds use this method when they are about to land. Some birds can also hover in the air. This method is used by birds of prey such as falcons that are looking for something to eat. Seagulls are also good at hovering, particularly if there is a strong breeze. The most expert hovering birds are tiny hummingbirds which can beat their wings both backwards and forwards and can stay quite still in the air while they dip their long beaks into flowers to feed on the sweet nectar.\n\nTypes of flight \nDifferent types of birds have different needs. Their wings have evolved to suit their lifestyle. Large birds of prey, such as eagles, spend a lot of time soaring on the wind. They have wings that are large and broad. The main flight feathers are long and wide. They help the eagle to stay on rising air currents without using much energy, while the eagle looks at the ground below, to find the next meal. When the eagle sees some small creature move, it can close its wings and fall from the sky like a missile, opening its great wings again to slow down as it comes to land. The world\'s largest eagle, the Philippine eagle has a wingspan of about 2 m (6.7\xa0ft) wide.\n\nBirds that live in grassland areas or open forests and feed on fruit, insects and reptiles often spend a lot of time flying short journeys looking for food and water. They have wings that are shaped in a similar way to eagles, but rounder and not as good for soaring. These include many Australian birds like cockatoos.\n\nBirds such as geese that migrate from one country to another fly very long distances. Their wings are big and strong, because the birds are large. They stock up on food for the long flight. Migrating water birds usually form family groups of 1230 birds. They fly very high, making use of long streams of air that blow from north to south in different seasons. They are well organised, often flying in a V pattern. The geese at the back do not have to flap so hard; they are pulled on by the wind of the ones at the front. Every so often, they change the leader so that the front bird, who does most work and sets the pace, can have a rest. Geese and swans are the highest-flying birds, reaching 8,000 metres or more when on migration. Geese often honk loudly while they are flying.


key: _node_content
string_value:
{"id_": "9b33a0e8-1a00-4d5e-8ad4-d4077d1a9d0d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8cabfb27-774f-4c28-9ce7-2f014e387db3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "68cff1e57d089636d4ed3882c1b86af2c8254718a455a445a9222b4e85d373f2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "da63bdbc-f257-4090-89a1-6b40bdc848df", "node_type": "1", "metadata": {}, "hash": "2ef27cef2b7e94f25639be5d390f3855e597d32aed4da08299317d18f328f500", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 22900, "end_char_idx": 26177, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
They have wings that are large and broad. The main flight feathers are long and wide. They help the eagle to stay on rising air currents without using much energy, while the eagle looks at the ground below, to find the next meal. When the eagle sees some small creature move, it can close its wings and fall from the sky like a missile, opening its great wings again to slow down as it comes to land. The world\'s largest eagle, the Philippine eagle has a wingspan of about 2 m (6.7\xa0ft) wide.\n\nBirds that live in grassland areas or open forests and feed on fruit, insects and reptiles often spend a lot of time flying short journeys looking for food and water. They have wings that are shaped in a similar way to eagles, but rounder and not as good for soaring. These include many Australian birds like cockatoos.\n\nBirds such as geese that migrate from one country to another fly very long distances. Their wings are big and strong, because the birds are large. They stock up on food for the long flight. Migrating water birds usually form family groups of 1230 birds. They fly very high, making use of long streams of air that blow from north to south in different seasons. They are well organised, often flying in a V pattern. The geese at the back do not have to flap so hard; they are pulled on by the wind of the ones at the front. Every so often, they change the leader so that the front bird, who does most work and sets the pace, can have a rest. Geese and swans are the highest-flying birds, reaching 8,000 metres or more when on migration. Geese often honk loudly while they are flying. It is thought that they do this to support the leader and help the young ones.\n\nBirds that fly very quickly, such as swifts and swallows, have long narrow pointed wings. These birds need great speed because they eat insects, catching most of them while they are flying. These birds also migrate. They often collect in huge flocks of thousands of birds that move together like a whirling cloud.\n\nBirds that live in bushes and branches have triangular wings that help the bird change direction. Many forest birds are expert at getting up speed by flapping and then gliding steadily among the trees, tilting to avoid things as they go. Members of the kingfisher family are expert at this type of flying.\n\nBirds such as owls that hunt at night have wings with soft rounded feathers so that they do not flap loudly. Birds that are awake at night are called nocturnal birds. Birds that are awake during the day are diurnal.\n\nWandering albatross might spend several years without coming to land. They can sleep while gliding. Arctic terns nest every one to three years.\n\nFlocks \nFlocks of birds can be very highly organised in a way that takes care of all the flock members. Studies of small flocking birds like tree sparrows show that they clearly communicate with each other, as sometimes thousands of birds may fly in close formation and spiral patterns without colliding (or flying into each other).\n\nTwo common behaviours in flocking birds are guarding and reconnaissance. When a flock of birds is feeding it is common for one bird to perch on a high place to keep guard over the flock. In the same way, when a flock is asleep, often, one bird will remain awake.


key: _node_content
string_value:
{"id_": "da63bdbc-f257-4090-89a1-6b40bdc848df", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9b33a0e8-1a00-4d5e-8ad4-d4077d1a9d0d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "2468870b2f7bd9e990c62a09bdaca3c20488524e924b8cda78be9f0cd6cd1923", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2d170763-9c06-4374-af0f-752adbe0a2b3", "node_type": "1", "metadata": {}, "hash": "b1b3bea047463d7ada3029bcfe50115b7bdafadc0e188410f219629e87b386e2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24457, "end_char_idx": 27803, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Geese often honk loudly while they are flying. It is thought that they do this to support the leader and help the young ones.\n\nBirds that fly very quickly, such as swifts and swallows, have long narrow pointed wings. These birds need great speed because they eat insects, catching most of them while they are flying. These birds also migrate. They often collect in huge flocks of thousands of birds that move together like a whirling cloud.\n\nBirds that live in bushes and branches have triangular wings that help the bird change direction. Many forest birds are expert at getting up speed by flapping and then gliding steadily among the trees, tilting to avoid things as they go. Members of the kingfisher family are expert at this type of flying.\n\nBirds such as owls that hunt at night have wings with soft rounded feathers so that they do not flap loudly. Birds that are awake at night are called nocturnal birds. Birds that are awake during the day are diurnal.\n\nWandering albatross might spend several years without coming to land. They can sleep while gliding. Arctic terns nest every one to three years.\n\nFlocks \nFlocks of birds can be very highly organised in a way that takes care of all the flock members. Studies of small flocking birds like tree sparrows show that they clearly communicate with each other, as sometimes thousands of birds may fly in close formation and spiral patterns without colliding (or flying into each other).\n\nTwo common behaviours in flocking birds are guarding and reconnaissance. When a flock of birds is feeding it is common for one bird to perch on a high place to keep guard over the flock. In the same way, when a flock is asleep, often, one bird will remain awake. It is also common for large flocks to send one or two birds ahead of them when they are flying to a new area. The look-out birds can spy the lie of the land to find food, water and good places to perch. Mixed feeding flocks occur, and can help to spot predators.\n\nFlightless birds \nSome birds do not fly. Flightlessness in birds has evolved many times.\nThese include running birds like ostriches and emus and ocean-living birds, the large penguin family. Birds on islands have usually lost the power of flight. This is to their advantage because birds with the power of flight can be blown off their island during a storm. The same ability which got them to the island may later take them away in a storm.\n\nOstriches and emus do not need to fly because although they feed and nest on the ground, their great size and their speed is their protection. Some other ground-feeding birds have not been so lucky. Some birds such as the dodo and the kiwi were ground-feeding birds that lived in safety on islands where there was nothing dangerous to eat them. They lost the power of flight. Kiwis are endangered because European settlement to New Zealand brought animals like cats, dogs and rats which kill kiwis and eat their eggs. However, kiwis and also the rare New Zealand ground parrot have survived. In the case of dodos, they were fat and disgusting in taste. All the same, they were killed and eaten by sailors until there was none left. Other flightless birds which have disappeared are the great auk and the moa.\n\nPenguins are a very successful group of birds. They are a clade. They spend half their time on land.


key: _node_content
string_value:
{"id_": "2d170763-9c06-4374-af0f-752adbe0a2b3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "da63bdbc-f257-4090-89a1-6b40bdc848df", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f2a6eef8f27a43efa4ccdd98ce123229717ad743125276435adf38af3c4597bd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a08ff3aa-2b0f-4194-bea8-e4209ac59af3", "node_type": "1", "metadata": {}, "hash": "8529cf73feb049214eb38862c5ae7da80db3e8197613da167de7ac6d4837bfc9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 26102, "end_char_idx": 29324, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In the same way, when a flock is asleep, often, one bird will remain awake. It is also common for large flocks to send one or two birds ahead of them when they are flying to a new area. The look-out birds can spy the lie of the land to find food, water and good places to perch. Mixed feeding flocks occur, and can help to spot predators.\n\nFlightless birds \nSome birds do not fly. Flightlessness in birds has evolved many times.\nThese include running birds like ostriches and emus and ocean-living birds, the large penguin family. Birds on islands have usually lost the power of flight. This is to their advantage because birds with the power of flight can be blown off their island during a storm. The same ability which got them to the island may later take them away in a storm.\n\nOstriches and emus do not need to fly because although they feed and nest on the ground, their great size and their speed is their protection. Some other ground-feeding birds have not been so lucky. Some birds such as the dodo and the kiwi were ground-feeding birds that lived in safety on islands where there was nothing dangerous to eat them. They lost the power of flight. Kiwis are endangered because European settlement to New Zealand brought animals like cats, dogs and rats which kill kiwis and eat their eggs. However, kiwis and also the rare New Zealand ground parrot have survived. In the case of dodos, they were fat and disgusting in taste. All the same, they were killed and eaten by sailors until there was none left. Other flightless birds which have disappeared are the great auk and the moa.\n\nPenguins are a very successful group of birds. They are a clade. They spend half their time on land. Their wings are adapted to life in the sea and have become flippers which let them in swim fast. They catch fish at sea, where they are in danger from seals.\n\nDigestion \nModern birds do not have teeth, and many swallow their prey whole. Nevertheless, they must break up food before it is digested. First of all, along their throat (oesophagus) they have a crop. This stores food items before digestion. That way a bird can eat several items, and then fly off to a quiet spot to digest them. \n\nTheir stomach comes next, with two very different parts. One part is like a straight hollow rod (the proventriculus) which secretes mild hydrochloric acid and an enzyme to break down protein. The other part of the stomach is the gizzard. This is muscular, and grinds up the contents. In herbivorous birds the gizzard contains some gastroliths (small stones or pieces of grit). Bones of fish will mostly be dissolved by the stomach acid. The partly digested and ground-up food now goes to the intestine, where digestion is completed, and most contents are absorbed. Anything indigestible, for example remains of feathers, is regurgitated via the mouth, not the cloaca.\n\nThe system is effective, and carnivorous birds can swallow quite large prey. A blue heron can swallow a fish as large as a carp successfully. Raptors eat by holding the prey down with a foot, and tearing it apart with their beak.\n\nReproduction\n\nMating \nAlthough birds are warm-blooded creatures like mammals, they do not give birth to live young.


key: _node_content
string_value:
{"id_": "a08ff3aa-2b0f-4194-bea8-e4209ac59af3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2d170763-9c06-4374-af0f-752adbe0a2b3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "62740ca8600203e47a8405d335133b167b3189b393559780ce7543a4f8ea6404", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "87cb2388-49a5-49a1-a426-38f83764791b", "node_type": "1", "metadata": {}, "hash": "685bb762849be688bae280050ee651e16c2e247c33bd44b6f745238dbd1657c9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27750, "end_char_idx": 30973, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
They are a clade. They spend half their time on land. Their wings are adapted to life in the sea and have become flippers which let them in swim fast. They catch fish at sea, where they are in danger from seals.\n\nDigestion \nModern birds do not have teeth, and many swallow their prey whole. Nevertheless, they must break up food before it is digested. First of all, along their throat (oesophagus) they have a crop. This stores food items before digestion. That way a bird can eat several items, and then fly off to a quiet spot to digest them. \n\nTheir stomach comes next, with two very different parts. One part is like a straight hollow rod (the proventriculus) which secretes mild hydrochloric acid and an enzyme to break down protein. The other part of the stomach is the gizzard. This is muscular, and grinds up the contents. In herbivorous birds the gizzard contains some gastroliths (small stones or pieces of grit). Bones of fish will mostly be dissolved by the stomach acid. The partly digested and ground-up food now goes to the intestine, where digestion is completed, and most contents are absorbed. Anything indigestible, for example remains of feathers, is regurgitated via the mouth, not the cloaca.\n\nThe system is effective, and carnivorous birds can swallow quite large prey. A blue heron can swallow a fish as large as a carp successfully. Raptors eat by holding the prey down with a foot, and tearing it apart with their beak.\n\nReproduction\n\nMating \nAlthough birds are warm-blooded creatures like mammals, they do not give birth to live young. They lay eggs as reptiles do, but the shell of a bird\'s egg is hard. The baby bird grows inside the egg, and after a few weeks hatches (breaks out of the egg).\n\nBirds in cold climates usually have a breeding season once a year in the spring. Migratory birds can have two springs and two mating seasons in a year. \n\nNinety-five per cent of bird species are socially monogamous. These birds pair for at least the length of the breeding season. In some cases this arrangement lasts until the death of one of the pair. Monogamy clearly helps if females need males\' help to raise a brood successfully. It has other practical advantages: the nest is never left without defence. Birds are generally small, and they have many potential enemies.\n\nSome birds mate for life, like married couples. These birds include pigeons, geese, and cranes. Other birds look for new partners each year. For birds that choose new mates, part of the breeding season is display. The male bird will do all sorts of things to attract females. These include singing, dancing, showing off the feathers and building a beautiful nest. Some male birds have splendid feathers for attracting females. The most famous is the peacock who can spread the feathers above his tail into a huge fan. \n\nOther mating systems do occur in some species. Polygyny, polyandry, polygamy, polygynandry, and promiscuity do happen. Polygamous breeding systems arise when females are able to raise broods without the help of males. Some species may use more than one system depending on the circumstances.\n\nNesting \nOnce the birds have found partners, they find a suitable place to lay eggs.


key: _node_content
string_value:
{"id_": "87cb2388-49a5-49a1-a426-38f83764791b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a08ff3aa-2b0f-4194-bea8-e4209ac59af3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "46ba24b8f684912043f5e85ffde7e46f99ce224dc4caebfe40895fe845798da7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4d8b38c0-b2d8-4334-a0be-8d4e285d5b76", "node_type": "1", "metadata": {}, "hash": "62dda77043a1625b780522ae4ad660f62d4e7f76d8b896c73126ee172c6573a9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29325, "end_char_idx": 32504, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
They lay eggs as reptiles do, but the shell of a bird\'s egg is hard. The baby bird grows inside the egg, and after a few weeks hatches (breaks out of the egg).\n\nBirds in cold climates usually have a breeding season once a year in the spring. Migratory birds can have two springs and two mating seasons in a year. \n\nNinety-five per cent of bird species are socially monogamous. These birds pair for at least the length of the breeding season. In some cases this arrangement lasts until the death of one of the pair. Monogamy clearly helps if females need males\' help to raise a brood successfully. It has other practical advantages: the nest is never left without defence. Birds are generally small, and they have many potential enemies.\n\nSome birds mate for life, like married couples. These birds include pigeons, geese, and cranes. Other birds look for new partners each year. For birds that choose new mates, part of the breeding season is display. The male bird will do all sorts of things to attract females. These include singing, dancing, showing off the feathers and building a beautiful nest. Some male birds have splendid feathers for attracting females. The most famous is the peacock who can spread the feathers above his tail into a huge fan. \n\nOther mating systems do occur in some species. Polygyny, polyandry, polygamy, polygynandry, and promiscuity do happen. Polygamous breeding systems arise when females are able to raise broods without the help of males. Some species may use more than one system depending on the circumstances.\n\nNesting \nOnce the birds have found partners, they find a suitable place to lay eggs. The idea of what is a suitable place differs between species, but most build bird nests. The bird is driven by a hormone (estradiol E2) to prepare a place for the eggs to hatch. Birds\' nests may be up a tree, in a cliff or on the ground according to species. When filled with eggs they are almost always guarded by one of the pair. In fact it is virtually impossible for the eggs to survive if one of the parents dies.\n\nRobins will make a beautiful little round nest of woven grass and carefully line it with feathers, bits of fluff and other soft things. Swallows like to nest near other swallows. They make nests from little blobs of clay, often on a beam near the roof of a building where it is well sheltered. Many birds like a hollow tree to nest in. Eagle\'s nests are often just piles of dead wood on the top of the tallest tree or mountain. Scrub turkeys scratch together a huge pile of leaves that may be 10 metres across. Guillemots lay their eggs on rock shelves with no nest at all. Their eggs are shaped so that they roll around in circles and do not fall off cliffs. A cuckoo does not make its own nest. It lays its egg in the nest of another bird and leaves it for them to care for. The cuckoo eggs are camouflaged to look like the host\'s eggs.\n\nWhen the nest has been prepared, the birds mate so that the eggs are fertilised and the chicks will start growing. Unlike mammals, birds (and reptiles) only have one opening as the exit hole for body fluids, and for reproduction. The opening is called the cloaca.


key: _node_content
string_value:
{"id_": "4d8b38c0-b2d8-4334-a0be-8d4e285d5b76", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "87cb2388-49a5-49a1-a426-38f83764791b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "7b45111faab8e273b27d0b172c2c309455cd57c1e3d06f14bb57ffa352c218bb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "29fd268a-ca2f-424e-902c-347ebf3d340b", "node_type": "1", "metadata": {}, "hash": "886b166e3bdc8cde7806d2e71f274e1c943849e68e4b39ca53804ba63836464e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30974, "end_char_idx": 34118, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The idea of what is a suitable place differs between species, but most build bird nests. The bird is driven by a hormone (estradiol E2) to prepare a place for the eggs to hatch. Birds\' nests may be up a tree, in a cliff or on the ground according to species. When filled with eggs they are almost always guarded by one of the pair. In fact it is virtually impossible for the eggs to survive if one of the parents dies.\n\nRobins will make a beautiful little round nest of woven grass and carefully line it with feathers, bits of fluff and other soft things. Swallows like to nest near other swallows. They make nests from little blobs of clay, often on a beam near the roof of a building where it is well sheltered. Many birds like a hollow tree to nest in. Eagle\'s nests are often just piles of dead wood on the top of the tallest tree or mountain. Scrub turkeys scratch together a huge pile of leaves that may be 10 metres across. Guillemots lay their eggs on rock shelves with no nest at all. Their eggs are shaped so that they roll around in circles and do not fall off cliffs. A cuckoo does not make its own nest. It lays its egg in the nest of another bird and leaves it for them to care for. The cuckoo eggs are camouflaged to look like the host\'s eggs.\n\nWhen the nest has been prepared, the birds mate so that the eggs are fertilised and the chicks will start growing. Unlike mammals, birds (and reptiles) only have one opening as the exit hole for body fluids, and for reproduction. The opening is called the cloaca. A female bird, called a hen, has two ovaries, of which the left one usually produces eggs.\n\nMost male birds have no sex organs that can be seen. But inside the male are two testes which produce sperm which is stored in the cloaca. Birds mate by rubbing their cloacas together, although with some birds, particularly large water birds, the male has a sort of a penis inside the cloaca.\n\nHatching \nOnce the hen has mated, she produces fertile eggs which have chicks growing inside them. She lays the eggs in the nest. There might be just one egg or a number of them, called a clutch. Emus might lay as many as fifteen huge dark green eggs in a clutch. After the eggs are laid, they are incubated, or kept warm so the chicks form inside. Most birds stay together for the whole nesting season, and one advantage is that the work is shared. Many birds take turns sitting on the eggs, so that each adult can feed.\n\nThis is not always the case. With emus, the male does all the sitting and all the baby-minding. With emperor penguins it is also the male that cares for the egg. There is only one egg, which he keeps on his feet and under his feathers, standing in a big group of males without feeding until the chick is hatched. While the eggs are hatching, the females are at sea, feeding, so that they can care for the chicks when they return.\n\nSome birds put the eggs inside or on top of the mound of leaves and twigs. The mound acts like a compost heap. The decomposition of the rotting leaves causes the temperature to rise. This is heat released by the chemical action of bacterial and fungal respiration.


key: _node_content
string_value:
{"id_": "29fd268a-ca2f-424e-902c-347ebf3d340b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4d8b38c0-b2d8-4334-a0be-8d4e285d5b76", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "68cc3f8524db320f97d50bc4743d7b16d4f9d56e1a3e3bb42d468347a69643d3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0396c9ad-253c-4fac-a3d5-6f915cbf6628", "node_type": "1", "metadata": {}, "hash": "efa3949b1cc86c94d190b2dc3095fac371f8fe4357c252bbc23ef5c684f4e5b4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 32471, "end_char_idx": 35659, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The opening is called the cloaca. A female bird, called a hen, has two ovaries, of which the left one usually produces eggs.\n\nMost male birds have no sex organs that can be seen. But inside the male are two testes which produce sperm which is stored in the cloaca. Birds mate by rubbing their cloacas together, although with some birds, particularly large water birds, the male has a sort of a penis inside the cloaca.\n\nHatching \nOnce the hen has mated, she produces fertile eggs which have chicks growing inside them. She lays the eggs in the nest. There might be just one egg or a number of them, called a clutch. Emus might lay as many as fifteen huge dark green eggs in a clutch. After the eggs are laid, they are incubated, or kept warm so the chicks form inside. Most birds stay together for the whole nesting season, and one advantage is that the work is shared. Many birds take turns sitting on the eggs, so that each adult can feed.\n\nThis is not always the case. With emus, the male does all the sitting and all the baby-minding. With emperor penguins it is also the male that cares for the egg. There is only one egg, which he keeps on his feet and under his feathers, standing in a big group of males without feeding until the chick is hatched. While the eggs are hatching, the females are at sea, feeding, so that they can care for the chicks when they return.\n\nSome birds put the eggs inside or on top of the mound of leaves and twigs. The mound acts like a compost heap. The decomposition of the rotting leaves causes the temperature to rise. This is heat released by the chemical action of bacterial and fungal respiration. It is the same reaction as that which keeps mammals and birds at a high temperature. The parents leave the mound. When the chicks hatch, they are able to feed themselves.\n\nMany small birds take 2–4 weeks to hatch eggs. Albatrosses take 80 days. During this time the female loses a lot of her body weight.\n\nThe quickest hatching time is for the cuckoo. Some types of cuckoos take only 10 days. This means that when they hatch in the nest of their \'\'foster parents\'\', the eggs that the parents have laid are not yet ready. Newborn cuckoos are naked, blind and ugly, but they are strong. They get under any eggs that are in the nest and throw them out before they hatch. That means that the cuckoo has the whole care of both parents. Baby cuckoos grow fast and often get bigger than the parents who feed them.\n\nWhen baby birds hatch, in most types of birds, they are fed by both parents, and sometimes by older aunties as well. Their mouths are open all the time and are often very brightly coloured, which acts as a releaser\'\', a trigger which stimulates the parent to feed them. For birds that eat grain and fruit, the parents eat and partly digest the food for the babies. It is then vomited carefully into the baby\'s mouth.\n  \n\n Families \nMany birds, particularly those that mate for life, are very sociable and keep together in a family group which might be anything from 4 or 6 adult birds and their young to a very large flock.\n\nAs chicks grow they change the fluffy down that covers them as babies for real feathers.


key: _node_content
string_value:
{"id_": "0396c9ad-253c-4fac-a3d5-6f915cbf6628", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "29fd268a-ca2f-424e-902c-347ebf3d340b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "9b4d3df1852385bf6c1fc2b9f97cd8adaec92a0df7787200fabc58c4e0cdbcd4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "628e2692-e11b-43ff-95dd-8d7945e7bc73", "node_type": "1", "metadata": {}, "hash": "98a8d0bd232fc1ecdecf8f9b20f83ae046fa1645f13e1749018ffe0d0a686b39", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 33965, "end_char_idx": 37016, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The decomposition of the rotting leaves causes the temperature to rise. This is heat released by the chemical action of bacterial and fungal respiration. It is the same reaction as that which keeps mammals and birds at a high temperature. The parents leave the mound. When the chicks hatch, they are able to feed themselves.\n\nMany small birds take 2–4 weeks to hatch eggs. Albatrosses take 80 days. During this time the female loses a lot of her body weight.\n\nThe quickest hatching time is for the cuckoo. Some types of cuckoos take only 10 days. This means that when they hatch in the nest of their \'\'foster parents\'\', the eggs that the parents have laid are not yet ready. Newborn cuckoos are naked, blind and ugly, but they are strong. They get under any eggs that are in the nest and throw them out before they hatch. That means that the cuckoo has the whole care of both parents. Baby cuckoos grow fast and often get bigger than the parents who feed them.\n\nWhen baby birds hatch, in most types of birds, they are fed by both parents, and sometimes by older aunties as well. Their mouths are open all the time and are often very brightly coloured, which acts as a releaser\'\', a trigger which stimulates the parent to feed them. For birds that eat grain and fruit, the parents eat and partly digest the food for the babies. It is then vomited carefully into the baby\'s mouth.\n  \n\n Families \nMany birds, particularly those that mate for life, are very sociable and keep together in a family group which might be anything from 4 or 6 adult birds and their young to a very large flock.\n\nAs chicks grow they change the fluffy down that covers them as babies for real feathers. At this stage they are called fledglings. Other family members may help care for fledgling chicks, feeding them and protecting them from attack while parents are feeding. When the fledglings have their new feathers, they come out of the nest to learn to fly. In some types of birds, like pigeons, the parents watch over this and as the young ones get stronger, will give them flying lessons, teaching them how to glide, how to fly in spirals and how to land like an expert.\n\n Communication \nMost birds are social animals, at least part of the time. They communicate to each other using sounds and displays.\n\nAlmost all birds make sounds to communicate. The types of noises that vary greatly. Some birds can sing, and  they are called songbirds or passerines. Examples are robins, larks, canaries, thrushes, nightingales. Corvids are passerines, but they do not sing. Birds that are not songbirds include: pigeons, seagulls, eagles, owls and ducks. Parrots are not songbirds, even though they can be taught to sing human songs.\n\n Songbirds \nAll birds make noises (\'\'bird vocalisation\'\'), but not all sing. Songbirds are passerines, many of which have beautiful melodic songs. Songs have different functions. Danger cries are different from territorial songs and mating calls are a third type. Fledgling may also have different calls from adults.


key: _node_content
string_value:
{"id_": "628e2692-e11b-43ff-95dd-8d7945e7bc73", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0396c9ad-253c-4fac-a3d5-6f915cbf6628", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "ededb905190e4b653ccd5e5e829f51051876e33735720c31c8b56f1c1364206e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a86a4cb5-e95d-4714-8355-e59ea13680dc", "node_type": "1", "metadata": {}, "hash": "e483fdd78711de407af626ad88a8d9a9d2d52b3e8fc56a66bcd3f9f2753388e2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 35660, "end_char_idx": 38910, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
At this stage they are called fledglings. Other family members may help care for fledgling chicks, feeding them and protecting them from attack while parents are feeding. When the fledglings have their new feathers, they come out of the nest to learn to fly. In some types of birds, like pigeons, the parents watch over this and as the young ones get stronger, will give them flying lessons, teaching them how to glide, how to fly in spirals and how to land like an expert.\n\n Communication \nMost birds are social animals, at least part of the time. They communicate to each other using sounds and displays.\n\nAlmost all birds make sounds to communicate. The types of noises that vary greatly. Some birds can sing, and  they are called songbirds or passerines. Examples are robins, larks, canaries, thrushes, nightingales. Corvids are passerines, but they do not sing. Birds that are not songbirds include: pigeons, seagulls, eagles, owls and ducks. Parrots are not songbirds, even though they can be taught to sing human songs.\n\n Songbirds \nAll birds make noises (\'\'bird vocalisation\'\'), but not all sing. Songbirds are passerines, many of which have beautiful melodic songs. Songs have different functions. Danger cries are different from territorial songs and mating calls are a third type. Fledgling may also have different calls from adults. Recognition calls for partners are quite common.\n\nAs to where the song comes from, there are three kinds of species:\nThose where the song is mainly inherited, and the bird always sings the same song in the same situations. The capacity is inherited, and only details are learnt from its neighbours.\nThose where the song is partly inherited, but the bird tunes it in by copying others. In this case the slight differences between the calls of different birds may be used by partners for identification.\nThose where the song is entirely learnt, and the bird often copies sounds from its environment. Only the capacity to sing is inherited.\n\nMost singing birds that are kept as pets, like canaries, have several tunes and some variations.\n\nThe same species of bird will sing different songs in different regions. A good example of this is the currawong. This is an Australia bird which is like a black and white crow. In the autumn, families get together in large flocks and do a lot of singing. Currawongs from some areas sing much more complex songs than others. Generally, currawongs from the Blue Mountains are the finest singers. The song of the currawong can be sung as a solo, but is often performed as a choir. One bird will take the lead and sing "Warble-warble-warble-warble!" All the other birds will join in and sing "Wooooooo!". When all the birds know the song, the choir will sing the "Warble" part and the soloist will sing the "Woo!". The song changes from year to year and from place to place.\n\n Lorenz\'s studies \nThe Austrian naturalist Konrad Lorenz studied the way in which birds communicate, or talk to each other. He found that each type of bird had a number of sounds which they made automatically, when ever they felt a certain way. Every sound had an action that went with it. So, if the bird was frightened, it acted frightened and made a frightened sound.


key: _node_content
string_value:
{"id_": "a86a4cb5-e95d-4714-8355-e59ea13680dc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "628e2692-e11b-43ff-95dd-8d7945e7bc73", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "53bbd2a0745a13dd9e9669607928e6ba9a0948045f53fd8c1493abfcf3d33200", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "94b3d20c-de2b-4bf4-9cd9-f3ae0bcf49b3", "node_type": "1", "metadata": {}, "hash": "c1c439af84bc74a22e2af60430de694f76da2134686502d7b3c2177ed554936e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 37243, "end_char_idx": 40472, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The capacity is inherited, and only details are learnt from its neighbours.\nThose where the song is partly inherited, but the bird tunes it in by copying others. In this case the slight differences between the calls of different birds may be used by partners for identification.\nThose where the song is entirely learnt, and the bird often copies sounds from its environment. Only the capacity to sing is inherited.\n\nMost singing birds that are kept as pets, like canaries, have several tunes and some variations.\n\nThe same species of bird will sing different songs in different regions. A good example of this is the currawong. This is an Australia bird which is like a black and white crow. In the autumn, families get together in large flocks and do a lot of singing. Currawongs from some areas sing much more complex songs than others. Generally, currawongs from the Blue Mountains are the finest singers. The song of the currawong can be sung as a solo, but is often performed as a choir. One bird will take the lead and sing "Warble-warble-warble-warble!" All the other birds will join in and sing "Wooooooo!". When all the birds know the song, the choir will sing the "Warble" part and the soloist will sing the "Woo!". The song changes from year to year and from place to place.\n\n Lorenz\'s studies \nThe Austrian naturalist Konrad Lorenz studied the way in which birds communicate, or talk to each other. He found that each type of bird had a number of sounds which they made automatically, when ever they felt a certain way. Every sound had an action that went with it. So, if the bird was frightened, it acted frightened and made a frightened sound. This told the other birds around it that something frightening was happening.\n\nIf a flock of birds were flying over a field, they would be calling "Fly! Fly!" But a hungry bird, seeing something good to eat down below might start calling "Food! Food!" If other birds were also hungry, they would make the same call until more birds were calling "Food! Food!" than "Fly! Fly!". At this point, the mind of the flock would be changed. Some of the birds would start to yell "Fly downwards! Fly downwards!" as they sank from the sky, until the whole flock was all noisily calling the same thing.\n\nThese communication sounds are often short hard sounds like: chirps, squeaks, squawks and twitters. Sometimes the calls are longer and more musical. They include the "Rookety-coo" sound of a pigeon and the "Cockadoodledoo!" of a rooster. The bird cannot change these sounds. They always make them in the same way. The bird is locked into making each sound every time a particular idea comes into its head. The connection between how they feel and how they call is innate: they are born with it. Some calls in some species are learnt. Then, it is the tendency to learn which is inherited.\n\n The Jackdaw of Altenberg \nKonrad Lorenz noticed that when birds sing, they often use a lot of their regular calls as part of the song. Lorenz had a flock of jackdaws which were scattered during World War II. One day, an old bird returned. For many months she sat on the chimney singing her song, but in the song she kept making the call which Lorenz knew meant "Come home!


key: _node_content
string_value:
{"id_": "94b3d20c-de2b-4bf4-9cd9-f3ae0bcf49b3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a86a4cb5-e95d-4714-8355-e59ea13680dc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f5b54e9638dd838f02c0439f49df364f851b0a8760883cb13c3c2d08dda9cbb0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "574a6c47-4496-4227-80bb-ab2088dea25f", "node_type": "1", "metadata": {}, "hash": "64fa5a9ecee0fd06bfaeb44649803337cc5aed0a9a5ef1efeaba03b21702dbe7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 38911, "end_char_idx": 41861, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This told the other birds around it that something frightening was happening.\n\nIf a flock of birds were flying over a field, they would be calling "Fly! Fly!" But a hungry bird, seeing something good to eat down below might start calling "Food! Food!" If other birds were also hungry, they would make the same call until more birds were calling "Food! Food!" than "Fly! Fly!". At this point, the mind of the flock would be changed. Some of the birds would start to yell "Fly downwards! Fly downwards!" as they sank from the sky, until the whole flock was all noisily calling the same thing.\n\nThese communication sounds are often short hard sounds like: chirps, squeaks, squawks and twitters. Sometimes the calls are longer and more musical. They include the "Rookety-coo" sound of a pigeon and the "Cockadoodledoo!" of a rooster. The bird cannot change these sounds. They always make them in the same way. The bird is locked into making each sound every time a particular idea comes into its head. The connection between how they feel and how they call is innate: they are born with it. Some calls in some species are learnt. Then, it is the tendency to learn which is inherited.\n\n The Jackdaw of Altenberg \nKonrad Lorenz noticed that when birds sing, they often use a lot of their regular calls as part of the song. Lorenz had a flock of jackdaws which were scattered during World War II. One day, an old bird returned. For many months she sat on the chimney singing her song, but in the song she kept making the call which Lorenz knew meant "Come home! Come home!" One day, to the great surprise of Lorenz, a male bird flew from a passing flock and joined her on the chimney. Lorenz was sure that it was her long-lost "husband" who had found his way home at last.\n\n Evolution and taxonomy \n\nPalaeontologists have found some exceptional places (lagerstätten) where fossils of early birds are found. The preservation is so good that on the best examples impressions of their feathers can be seen, and sometimes even the remains of meals they have eaten. From these remains we know that birds evolved from small carnivorous dinosaurs (theropods) in the Jurassic period. They radiated into a huge variety in the Lower Cretaceous. At the same time, their direct competitors, the pterosaurs, dwindled in numbers and variety, and became extinct at the end of the Mesozoic.\n\nBirds are classified by taxonomists as \'Aves\' (Avialae). Birds are the only living descendants of dinosaurs (strictly speaking, they are dinosaurs). Birds and Crocodilia are the only living members of the once-dominant Archosaur reptiles.\n\n Definition \nThe class Aves is was defined (1990) as all the descendants of the most recent common ancestor of modern birds and Archaeopteryx lithographica. But Archaeopteryx is almost certainly not the ancestor of modern birds. The transition to flight happened a number of times. The researchers offered four definitions.


key: _node_content
string_value:
{"id_": "574a6c47-4496-4227-80bb-ab2088dea25f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ad23dba4-0813-4673-ac29-e4d1fa914e18", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "810673c71d58abecc211996ab14c132c2ff569b3017d12e4f721eb99d77704d5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "94b3d20c-de2b-4bf4-9cd9-f3ae0bcf49b3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/elasticsearch/elasticsearch-retrieval-augmented-generation.ipynb", "title": "Retrieval augmented generation using Elasticsearch and OpenAI", "tokens": 28563, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "4604483649ce472a991f40ab41082aa2ec892915b52cb700dd097a5846d770bf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ea133859-f44e-42ae-967b-ce5a23650978", "node_type": "1", "metadata": {}, "hash": "9a7c48fdf27b03ea078866c1db294cce76a84210a4d91d9b1d68fe17bac14ce2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 40235, "end_char_idx": 43253, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


