# embedding_metadata

key: chroma:document
string_value:
```python
# Use OpenAI get_embeddings batch requests to speed up embedding creation
def embeddings_batch_request(documents: pd.DataFrame):
    records = documents.to_dict("records")
    print("Records to process: ", len(records))
    product_vectors = []
    docs = []
    batchsize = 1000

    for idx,doc in enumerate(records,start=1):
        # create byte vectors
        docs.append(doc["product_text"])
        if idx % batchsize == 0:
            product_vectors += get_embeddings(docs, EMBEDDING_MODEL)
            docs.clear()
            print("Vectors processed ", len(product_vectors), end='\r')
    product_vectors += get_embeddings(docs, EMBEDDING_MODEL)
    print("Vectors processed ", len(product_vectors), end='\r')
    return product_vectors

```


```python
def index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):
    product_vectors = embeddings_batch_request(documents)
    records = documents.to_dict("records")
    batchsize = 500

    # Use Redis pipelines to batch calls and save on round trip network communication
    pipe = client.pipeline()
    for idx,doc in enumerate(records,start=1):
        key = f"{prefix}:{str(doc['product_id'])}"

        # create byte vectors
        text_embedding = np.array((product_vectors[idx-1]), dtype=np.float32).tobytes()

        # replace list of floats with byte vectors
        doc["product_vector"] = text_embedding

        pipe.hset(key, mapping = doc)
        if idx % batchsize == 0:
            pipe.execute()
    pipe.execute()

```


```python
%%time
index_documents(redis_client, PREFIX, df)
print(f"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}")

```

    Records to process:  1978
    Loaded 1978 documents in Redis search index with name: product_embeddings
    CPU times: user 619 ms, sys: 78.9 ms, total: 698 ms
    Wall time: 3.34 s


## Simple Vector Search Queries with OpenAI Query Embeddings

Now that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database.


key: _node_content
string_value:
{"id_": "60b44ce2-1c72-4725-897e-3d98515ef59a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "1a1e9c86a0c7e0dd7950d0417fc95d0b24cfef8382d201a54d7fa0cebf30ede5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "adff5a9b-fb91-434b-873b-2ae793037e55", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "32dd92a228b6de464e11d9ec2e4a9f6109f93c3f83f535e16d97447abb5e6423", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2fb68a1d-092c-48dd-abd6-de94d5ce12e7", "node_type": "1", "metadata": {}, "hash": "427afe109f8013c071a1ccf83d173d235e01dba5add7f8ea8e5834734dbe578c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14571, "end_char_idx": 17385, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):
    product_vectors = embeddings_batch_request(documents)
    records = documents.to_dict("records")
    batchsize = 500

    # Use Redis pipelines to batch calls and save on round trip network communication
    pipe = client.pipeline()
    for idx,doc in enumerate(records,start=1):
        key = f"{prefix}:{str(doc['product_id'])}"

        # create byte vectors
        text_embedding = np.array((product_vectors[idx-1]), dtype=np.float32).tobytes()

        # replace list of floats with byte vectors
        doc["product_vector"] = text_embedding

        pipe.hset(key, mapping = doc)
        if idx % batchsize == 0:
            pipe.execute()
    pipe.execute()

```


```python
%%time
index_documents(redis_client, PREFIX, df)
print(f"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}")

```

    Records to process:  1978
    Loaded 1978 documents in Redis search index with name: product_embeddings
    CPU times: user 619 ms, sys: 78.9 ms, total: 698 ms
    Wall time: 3.34 s


## Simple Vector Search Queries with OpenAI Query Embeddings

Now that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database.


```python
def search_redis(
    redis_client: redis.Redis,
    user_query: str,
    index_name: str = "product_embeddings",
    vector_field: str = "product_vector",
    return_fields: list = ["productDisplayName", "masterCategory", "gender", "season", "year", "vector_score"],
    hybrid_fields = "*",
    k: int = 20,
    print_results: bool = True,
) -> List[dict]:

    # Use OpenAI to create embedding vector from user query
    embedded_query = openai.Embedding.create(input=user_query,
                                            model="text-embedding-3-small",
                                            )["data"][0]['embedding']

    # Prepare the Query
    base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'
    query = (
        Query(base_query)
         .return_fields(*return_fields)
         .sort_by("vector_score")
         .paging(0, k)
         .dialect(2)
    )
    params_dict = {"vector": np.array(embedded_query).astype(dtype=np.float32).tobytes()}

    # perform vector search
    results = redis_client.ft(index_name).search(query, params_dict)
    if print_results:
        for i, product in enumerate(results.docs):
            score = 1 - float(product.vector_score)
            print(f"{i}. {product.productDisplayName} (Score: {round(score ,3) })")
    return results.docs

```


key: _node_content
string_value:
{"id_": "2fb68a1d-092c-48dd-abd6-de94d5ce12e7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "1a1e9c86a0c7e0dd7950d0417fc95d0b24cfef8382d201a54d7fa0cebf30ede5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "60b44ce2-1c72-4725-897e-3d98515ef59a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "66cb3dec06f3fbd8ab69b2cc02977a133900246a0ad9cb87bd926e954a445c89", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8fa7a490-2893-4f55-8511-9e71139a8f36", "node_type": "1", "metadata": {}, "hash": "e98b9963d8996e68db99fea6328ba8f9af5d69fb5bd4690d91718a7e655aae6c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15702, "end_char_idx": 18333, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Simple Vector Search Queries with OpenAI Query Embeddings

Now that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database.


```python
def search_redis(
    redis_client: redis.Redis,
    user_query: str,
    index_name: str = "product_embeddings",
    vector_field: str = "product_vector",
    return_fields: list = ["productDisplayName", "masterCategory", "gender", "season", "year", "vector_score"],
    hybrid_fields = "*",
    k: int = 20,
    print_results: bool = True,
) -> List[dict]:

    # Use OpenAI to create embedding vector from user query
    embedded_query = openai.Embedding.create(input=user_query,
                                            model="text-embedding-3-small",
                                            )["data"][0]['embedding']

    # Prepare the Query
    base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'
    query = (
        Query(base_query)
         .return_fields(*return_fields)
         .sort_by("vector_score")
         .paging(0, k)
         .dialect(2)
    )
    params_dict = {"vector": np.array(embedded_query).astype(dtype=np.float32).tobytes()}

    # perform vector search
    results = redis_client.ft(index_name).search(query, params_dict)
    if print_results:
        for i, product in enumerate(results.docs):
            score = 1 - float(product.vector_score)
            print(f"{i}. {product.productDisplayName} (Score: {round(score ,3) })")
    return results.docs

```


```python
# Execute a simple vector search in Redis
results = search_redis(redis_client, 'man blue jeans', k=10)

```

    0. John Players Men Blue Jeans (Score: 0.791)
    1. Lee Men Tino Blue Jeans (Score: 0.775)
    2. Peter England Men Party Blue Jeans (Score: 0.763)
    3. Lee Men Blue Chicago Fit Jeans (Score: 0.761)
    4. Lee Men Blue Chicago Fit Jeans (Score: 0.761)
    5. French Connection Men Blue Jeans (Score: 0.74)
    6. Locomotive Men Washed Blue Jeans (Score: 0.739)
    7. Locomotive Men Washed Blue Jeans (Score: 0.739)
    8. Do U Speak Green Men Blue Shorts (Score: 0.736)
    9. Palm Tree Kids Boy Washed Blue Jeans (Score: 0.732)


## Hybrid Queries with Redis

The previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the example below, we will combine vector search with full text search.


key: _node_content
string_value:
{"id_": "8fa7a490-2893-4f55-8511-9e71139a8f36", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "1a1e9c86a0c7e0dd7950d0417fc95d0b24cfef8382d201a54d7fa0cebf30ede5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2fb68a1d-092c-48dd-abd6-de94d5ce12e7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c9ccb5648f159e81637ff10710e65ea328103ee6bafd0a40f4463f1d498936dd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5a6bfd67-140b-4124-ad93-2f1325e5d550", "node_type": "1", "metadata": {}, "hash": "78139ef8cfd93f539797b1e3f18d032888c1a0f0ac35a4e06eea55ce811870b3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17388, "end_char_idx": 20144, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Execute a simple vector search in Redis
results = search_redis(redis_client, 'man blue jeans', k=10)

```

    0. John Players Men Blue Jeans (Score: 0.791)
    1. Lee Men Tino Blue Jeans (Score: 0.775)
    2. Peter England Men Party Blue Jeans (Score: 0.763)
    3. Lee Men Blue Chicago Fit Jeans (Score: 0.761)
    4. Lee Men Blue Chicago Fit Jeans (Score: 0.761)
    5. French Connection Men Blue Jeans (Score: 0.74)
    6. Locomotive Men Washed Blue Jeans (Score: 0.739)
    7. Locomotive Men Washed Blue Jeans (Score: 0.739)
    8. Do U Speak Green Men Blue Shorts (Score: 0.736)
    9. Palm Tree Kids Boy Washed Blue Jeans (Score: 0.732)


## Hybrid Queries with Redis

The previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the example below, we will combine vector search with full text search.


```python
# improve search quality by adding hybrid query for "man blue jeans" in the product vector combined with a phrase search for "blue jeans"
results = search_redis(redis_client,
                       "man blue jeans",
                       vector_field="product_vector",
                       k=10,
                       hybrid_fields='@productDisplayName:"blue jeans"'
                       )

```

    0. John Players Men Blue Jeans (Score: 0.791)
    1. Lee Men Tino Blue Jeans (Score: 0.775)
    2. Peter England Men Party Blue Jeans (Score: 0.763)
    3. French Connection Men Blue Jeans (Score: 0.74)
    4. Locomotive Men Washed Blue Jeans (Score: 0.739)
    5. Locomotive Men Washed Blue Jeans (Score: 0.739)
    6. Palm Tree Kids Boy Washed Blue Jeans (Score: 0.732)
    7. Denizen Women Blue Jeans (Score: 0.725)
    8. Jealous 21 Women Washed Blue Jeans (Score: 0.713)
    9. Jealous 21 Women Washed Blue Jeans (Score: 0.713)



```python
# hybrid query for shirt in the product vector and only include results with the phrase "slim fit" in the title
results = search_redis(redis_client,
                       "shirt",
                       vector_field="product_vector",
                       k=10,
                       hybrid_fields='@productDisplayName:"slim fit"'
                       )

```

    0. Basics Men White Slim Fit Striped Shirt (Score: 0.633)
    1. ADIDAS Men's Slim Fit White T-shirt (Score: 0.628)
    2. Basics Men Blue Slim Fit Checked Shirt (Score: 0.627)
    3. Basics Men Blue Slim Fit Checked Shirt (Score: 0.627)
    4. Basics Men Red Slim Fit Checked Shirt (Score: 0.623)
    5. Basics Men Navy Slim Fit Checked Shirt (Score: 0.613)
    6. Lee Rinse Navy Blue Slim Fit Jeans (Score: 0.558)
    7. Tokyo Talkies Women Navy Slim Fit Jeans (Score: 0.552)


key: _node_content
string_value:
{"id_": "5a6bfd67-140b-4124-ad93-2f1325e5d550", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "1a1e9c86a0c7e0dd7950d0417fc95d0b24cfef8382d201a54d7fa0cebf30ede5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8fa7a490-2893-4f55-8511-9e71139a8f36", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "32b3a6b01e1a416b139c815169249bd53c8046cf35ca1960b8c28815b116a079", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b5d24f36-075a-422f-9331-b9ae16990241", "node_type": "1", "metadata": {}, "hash": "6542bcbadd9014a90bbf608532b3c81be147d74bc42ba28447573347a75ab055", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19288, "end_char_idx": 21168, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# hybrid query for shirt in the product vector and only include results with the phrase "slim fit" in the title
results = search_redis(redis_client,
                       "shirt",
                       vector_field="product_vector",
                       k=10,
                       hybrid_fields='@productDisplayName:"slim fit"'
                       )

```

    0. Basics Men White Slim Fit Striped Shirt (Score: 0.633)
    1. ADIDAS Men's Slim Fit White T-shirt (Score: 0.628)
    2. Basics Men Blue Slim Fit Checked Shirt (Score: 0.627)
    3. Basics Men Blue Slim Fit Checked Shirt (Score: 0.627)
    4. Basics Men Red Slim Fit Checked Shirt (Score: 0.623)
    5. Basics Men Navy Slim Fit Checked Shirt (Score: 0.613)
    6. Lee Rinse Navy Blue Slim Fit Jeans (Score: 0.558)
    7. Tokyo Talkies Women Navy Slim Fit Jeans (Score: 0.552)



```python
# hybrid query for watch in the product vector and only include results with the tag "Accessories" in the masterCategory field
results = search_redis(redis_client,
                       "watch",
                       vector_field="product_vector",
                       k=10,
                       hybrid_fields='@masterCategory:{Accessories}'
                       )

```

    0. Titan Women Gold Watch (Score: 0.544)
    1. Being Human Men Grey Dial Blue Strap Watch (Score: 0.544)
    2. Police Men Black Dial Watch PL12170JSB (Score: 0.544)
    3. Titan Men Black Watch (Score: 0.543)
    4. Police Men Black Dial Chronograph Watch PL12777JS-02M (Score: 0.542)
    5. CASIO Youth Series Digital Men Black Small Dial Digital Watch W-210-1CVDF I065 (Score: 0.542)
    6. Titan Women Silver Watch (Score: 0.542)
    7. Police Men Black Dial Watch PL12778MSU-61 (Score: 0.541)
    8. Titan Raga Women Gold Watch (Score: 0.539)
    9. ADIDAS Original Men Black Dial Chronograph Watch ADH2641 (Score: 0.539)


key: _node_content
string_value:
{"id_": "b5d24f36-075a-422f-9331-b9ae16990241", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "1a1e9c86a0c7e0dd7950d0417fc95d0b24cfef8382d201a54d7fa0cebf30ede5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5a6bfd67-140b-4124-ad93-2f1325e5d550", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "e1034bcd49914cd2cd6dc0ba2368dd905c06667d1d4c103c632a32cf5ea02afd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8ad14b5d-180c-4397-9d03-c26f8430e4f1", "node_type": "1", "metadata": {}, "hash": "02816a573c7830234a50742354c3079de7e0a8d731e89b6d8555aa2988d7a767", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20148, "end_char_idx": 22057, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# hybrid query for watch in the product vector and only include results with the tag "Accessories" in the masterCategory field
results = search_redis(redis_client,
                       "watch",
                       vector_field="product_vector",
                       k=10,
                       hybrid_fields='@masterCategory:{Accessories}'
                       )

```

    0. Titan Women Gold Watch (Score: 0.544)
    1. Being Human Men Grey Dial Blue Strap Watch (Score: 0.544)
    2. Police Men Black Dial Watch PL12170JSB (Score: 0.544)
    3. Titan Men Black Watch (Score: 0.543)
    4. Police Men Black Dial Chronograph Watch PL12777JS-02M (Score: 0.542)
    5. CASIO Youth Series Digital Men Black Small Dial Digital Watch W-210-1CVDF I065 (Score: 0.542)
    6. Titan Women Silver Watch (Score: 0.542)
    7. Police Men Black Dial Watch PL12778MSU-61 (Score: 0.541)
    8. Titan Raga Women Gold Watch (Score: 0.539)
    9. ADIDAS Original Men Black Dial Chronograph Watch ADH2641 (Score: 0.539)



```python
# hybrid query for sandals in the product vector and only include results within the 2011-2012 year range
results = search_redis(redis_client,
                       "sandals",
                       vector_field="product_vector",
                       k=10,
                       hybrid_fields='@year:[2011 2012]'
                       )

```

    0. Enroute Teens Orange Sandals (Score: 0.701)
    1. Fila Men Camper Brown Sandals (Score: 0.692)
    2. Clarks Men Black Leather Closed Sandals (Score: 0.691)
    3. Coolers Men Black Sandals (Score: 0.69)
    4. Coolers Men Black Sandals (Score: 0.69)
    5. Enroute Teens Brown Sandals (Score: 0.69)
    6. Crocs Dora Boots Pink Sandals (Score: 0.69)
    7. Enroute Men Leather Black Sandals (Score: 0.685)
    8. ADIDAS Men Navy Blue Benton Sandals (Score: 0.684)
    9. Coolers Men Black Sports Sandals (Score: 0.684)


key: _node_content
string_value:
{"id_": "8ad14b5d-180c-4397-9d03-c26f8430e4f1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "60d79eba-f7df-4e3d-a0e3-5be420e13a39", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "1a1e9c86a0c7e0dd7950d0417fc95d0b24cfef8382d201a54d7fa0cebf30ede5", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b5d24f36-075a-422f-9331-b9ae16990241", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "5a2fc793bc4c0f3659196dbf2cfb0fc342d3549a89e89b356952899ffa72b85a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21172, "end_char_idx": 23790, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# hybrid query for sandals in the product vector and only include results within the 2011-2012 year range
results = search_redis(redis_client,
                       "sandals",
                       vector_field="product_vector",
                       k=10,
                       hybrid_fields='@year:[2011 2012]'
                       )

```

    0. Enroute Teens Orange Sandals (Score: 0.701)
    1. Fila Men Camper Brown Sandals (Score: 0.692)
    2. Clarks Men Black Leather Closed Sandals (Score: 0.691)
    3. Coolers Men Black Sandals (Score: 0.69)
    4. Coolers Men Black Sandals (Score: 0.69)
    5. Enroute Teens Brown Sandals (Score: 0.69)
    6. Crocs Dora Boots Pink Sandals (Score: 0.69)
    7. Enroute Men Leather Black Sandals (Score: 0.685)
    8. ADIDAS Men Navy Blue Benton Sandals (Score: 0.684)
    9. Coolers Men Black Sports Sandals (Score: 0.684)



```python
# hybrid query for sandals in the product vector and only include results within the 2011-2012 year range from the summer season
results = search_redis(redis_client,
                       "blue sandals",
                       vector_field="product_vector",
                       k=10,
                       hybrid_fields='(@year:[2011 2012] @season:{Summer})'
                       )

```

    0. ADIDAS Men Navy Blue Benton Sandals (Score: 0.691)
    1. Enroute Teens Brown Sandals (Score: 0.681)
    2. ADIDAS Women's Adi Groove Blue Flip Flop (Score: 0.672)
    3. Enroute Women Turquoise Blue Flats (Score: 0.671)
    4. Red Tape Men Black Sandals (Score: 0.67)
    5. Enroute Teens Orange Sandals (Score: 0.661)
    6. Vans Men Blue Era Scilla Plaid Shoes (Score: 0.658)
    7. FILA Men Aruba Navy Blue Sandal (Score: 0.657)
    8. Quiksilver Men Blue Flip Flops (Score: 0.656)
    9. Reebok Men Navy Twist Sandals (Score: 0.656)



```python
# hybrid query for a brown belt filtering results by a year (NUMERIC) with a specific article types (TAG) and with a brand name (TEXT)
results = search_redis(redis_client,
                       "brown belt",
                       vector_field="product_vector",
                       k=10,
                       hybrid_fields='(@year:[2012 2012] @articleType:{Shirts | Belts} @productDisplayName:"Wrangler")'
                       )

```

    0. Wrangler Men Leather Brown Belt (Score: 0.67)
    1. Wrangler Women Black Belt (Score: 0.639)
    2. Wrangler Men Green Striped Shirt (Score: 0.575)
    3. Wrangler Men Purple Striped Shirt (Score: 0.549)
    4. Wrangler Men Griffith White Shirt (Score: 0.543)
    5. Wrangler Women Stella Green Shirt (Score: 0.542)


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb


key: _node_content
string_value:
{"id_": "845cb823-0152-473f-97fd-547a4c7c0398", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73e09a74-6398-44e1-955c-76b6a0839554", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "9eec624cd3ebfae27701b1260b8fc26541da89d72f061e152021165c691d897c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "39ff86fd-e87f-4d32-8876-bed104b3a574", "node_type": "1", "metadata": {}, "hash": "9577e2f6f7b9db4edb4ac6f9c29af58f8407f3420a92bfa98d400fc6e6a1e7eb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3175, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Redis as a Context Store with OpenAI Chat
This notebook demonstrates how to use Redis as high-speed context memory with ChatGPT.

## Prerequisites
* Redis instance with the Redis Search and Redis JSON modules
* Redis-py client lib
* OpenAI Python client lib
* OpenAI API key

## Installation
Install Python modules necessary for the examples.


```python
! pip install redis openai python-dotenv openai[datalib]
```

## OpenAI API Key
Create a .env file and add your OpenAI key to it


```python
OPENAI_API_KEY=your_key
```

## OpenAI Setup
Key load + helper function for chat completion


```python
import openai
import os
from dotenv import load_dotenv

load_dotenv()
openai.api_key = os.getenv("OPENAI_API_KEY")

def get_completion(prompt, model="gpt-3.5-turbo"):
    messages = [{"role": "user", "content": prompt}]
    response = openai.ChatCompletion.create(
        model=model,
        messages=messages,
        temperature=0, 
    )
    return response.choices[0].message["content"]
```

## Experiment - Chat Completion on a Topic outside of the Model's Knowledge Cutoff Date
Gpt-3.5-turbo was trained on data up to Sep 2021.  Let's ask it a question about something that is beyond that date.  In this case, the FTX/Sam Bankman-Fried scandal.


```python
prompt = "Is Sam Bankman-Fried's company, FTX, considered a well-managed company?"
response = get_completion(prompt)
print(response)
```

## Incomplete Information
An unfortunate behavior of these AI systems is the system will provide a confident-sounding response - even when the system is not confident with its result.  One way to mitigate this is prompt re-engineering, as seen below.


```python
prompt ="Is Sam Bankman-Fried's company, FTX, considered a well-managed company?  If you don't know for certain, say unknown."
response = get_completion(prompt)
print(response)
```

## Additional Context
Another way to combat incomplete information is to give the system more information such that it can make intelligent decisions vs guessing.  We'll use Redis as the source for that additional context.  We'll pull in business news articles from after the GPT knowledge cut-off date such that the system will have a better understanding of how FTX was actually managed. 

## Start the Redis Stack Docker container


```python
! docker compose up -d
```

## Connect Redis client


```python
from redis import from_url

REDIS_URL = 'redis://localhost:6379'
client = from_url(REDIS_URL)
client.ping()
```




    True



## Create Index
[FT.CREATE](https://redis.io/commands/ft.create/)


```python
from redis.commands.search.field import TextField, VectorField
from redis.commands.search.indexDefinition import IndexDefinition, IndexType

schema = [ VectorField('$.vector', 
            "FLAT", 
            {   "TYPE": 'FLOAT32', 
                "DIM": 1536, 
                "DISTANCE_METRIC": "COSINE"
            },  as_name='vector' ),
            TextField('$.content', as_name='content')
        ]
idx_def = IndexDefinition(index_type=IndexType.JSON, prefix=['doc:'])
try: 
    client.ft('idx').dropindex()
except:
    pass
client.ft('idx').create_index(schema, definition=idx_def)
```




    b'OK'


key: _node_content
string_value:
{"id_": "39ff86fd-e87f-4d32-8876-bed104b3a574", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73e09a74-6398-44e1-955c-76b6a0839554", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "9eec624cd3ebfae27701b1260b8fc26541da89d72f061e152021165c691d897c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "845cb823-0152-473f-97fd-547a4c7c0398", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "7094249ebfbcb0436b8f10a3d8010d9f21ffb426d6191d7805d1c6e8f51a129a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4ab80487-d602-41e0-9afd-63ece9967ac0", "node_type": "1", "metadata": {}, "hash": "6f8b77ba77826d25bb17bac139a8a74e96a27b2ea3b399b86bb720d397d6930f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1658, "end_char_idx": 4564, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
prompt ="Is Sam Bankman-Fried's company, FTX, considered a well-managed company?  If you don't know for certain, say unknown."
response = get_completion(prompt)
print(response)
```

## Additional Context
Another way to combat incomplete information is to give the system more information such that it can make intelligent decisions vs guessing.  We'll use Redis as the source for that additional context.  We'll pull in business news articles from after the GPT knowledge cut-off date such that the system will have a better understanding of how FTX was actually managed. 

## Start the Redis Stack Docker container


```python
! docker compose up -d
```

## Connect Redis client


```python
from redis import from_url

REDIS_URL = 'redis://localhost:6379'
client = from_url(REDIS_URL)
client.ping()
```




    True



## Create Index
[FT.CREATE](https://redis.io/commands/ft.create/)


```python
from redis.commands.search.field import TextField, VectorField
from redis.commands.search.indexDefinition import IndexDefinition, IndexType

schema = [ VectorField('$.vector', 
            "FLAT", 
            {   "TYPE": 'FLOAT32', 
                "DIM": 1536, 
                "DISTANCE_METRIC": "COSINE"
            },  as_name='vector' ),
            TextField('$.content', as_name='content')
        ]
idx_def = IndexDefinition(index_type=IndexType.JSON, prefix=['doc:'])
try: 
    client.ft('idx').dropindex()
except:
    pass
client.ft('idx').create_index(schema, definition=idx_def)
```




    b'OK'



## Load Data Files into Redis as JSON Objects with Text and Vector Fields
[Redis JSON](https://redis.io/docs/stack/json/)


```python
import os
import openai

directory = './assets/'
model='text-embedding-3-small'
i = 1
for file in os.listdir(directory):
    with open(os.path.join(directory, file)) as f:
        content = f.read()
        vector = openai.Embedding.create(input = [content], model = model)['data'][0]['embedding']
        client.json().set(f'doc:{i}', '$', {'content': content, 'vector': vector})
    i += 1
```

## Embed the Question and Perform VSS to find the most relevant document
[KNN Search](https://redis.io/docs/stack/search/reference/vectors/#knn-search)


```python
from redis.commands.search.query import Query
import numpy as np

vec = np.array(openai.Embedding.create(input = [prompt], model = model)['data'][0]['embedding'], dtype=np.float32).tobytes()
q = Query('*=>[KNN 1 @vector $query_vec AS vector_score]')\
    .sort_by('vector_score')\
    .return_fields('content')\
    .dialect(2)    
params = {"query_vec": vec}

context = client.ft('idx').search(q, query_params=params).docs[0].content
print(context)
```

    Embattled Crypto Exchange FTX Files for Bankruptcy
    
    Nov. 11, 2022
    On Monday, Sam Bankman-Fried, the chief executive of the cryptocurrency exchange FTX, took to Twitter to reassure his customers: “FTX is fine,” he wrote.


key: _node_content
string_value:
{"id_": "4ab80487-d602-41e0-9afd-63ece9967ac0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73e09a74-6398-44e1-955c-76b6a0839554", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "9eec624cd3ebfae27701b1260b8fc26541da89d72f061e152021165c691d897c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "39ff86fd-e87f-4d32-8876-bed104b3a574", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "e869331b83b94b7193f8230745a040de1f8612142621a37e9b3972dd504280e9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9fec01b2-2394-48f7-904e-b6743777d905", "node_type": "1", "metadata": {}, "hash": "a70e33542e45a8ab605868decd6c71edd5f94f50b9e50309a56f0b591bcebcbd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3170, "end_char_idx": 6457, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
b'OK'



## Load Data Files into Redis as JSON Objects with Text and Vector Fields
[Redis JSON](https://redis.io/docs/stack/json/)


```python
import os
import openai

directory = './assets/'
model='text-embedding-3-small'
i = 1
for file in os.listdir(directory):
    with open(os.path.join(directory, file)) as f:
        content = f.read()
        vector = openai.Embedding.create(input = [content], model = model)['data'][0]['embedding']
        client.json().set(f'doc:{i}', '$', {'content': content, 'vector': vector})
    i += 1
```

## Embed the Question and Perform VSS to find the most relevant document
[KNN Search](https://redis.io/docs/stack/search/reference/vectors/#knn-search)


```python
from redis.commands.search.query import Query
import numpy as np

vec = np.array(openai.Embedding.create(input = [prompt], model = model)['data'][0]['embedding'], dtype=np.float32).tobytes()
q = Query('*=>[KNN 1 @vector $query_vec AS vector_score]')\
    .sort_by('vector_score')\
    .return_fields('content')\
    .dialect(2)    
params = {"query_vec": vec}

context = client.ft('idx').search(q, query_params=params).docs[0].content
print(context)
```

    Embattled Crypto Exchange FTX Files for Bankruptcy
    
    Nov. 11, 2022
    On Monday, Sam Bankman-Fried, the chief executive of the cryptocurrency exchange FTX, took to Twitter to reassure his customers: “FTX is fine,” he wrote. “Assets are fine.”
    
    On Friday, FTX announced that it was filing for bankruptcy, capping an extraordinary week of corporate drama that has upended crypto markets, sent shock waves through an industry struggling to gain mainstream credibility and sparked government investigations that could lead to more damaging revelations or even criminal charges.
    
    In a statement on Twitter, the company said that Mr. Bankman-Fried had resigned, with John J. Ray III, a corporate turnaround specialist, taking over as chief executive.
    
    The speed of FTX’s downfall has left crypto insiders stunned. Just days ago, Mr. Bankman-Fried was considered one of the smartest leaders in the crypto industry, an influential figure in Washington who was lobbying to shape regulations. And FTX was widely viewed as one of the most stable and responsible companies in the freewheeling, loosely regulated crypto industry.
    
    “Here we are, with one of the richest people in the world, his net worth dropping to zero, his business dropping to zero,” said Jared Ellias, a bankruptcy professor at Harvard Law School. “The velocity of this failure is just unbelievable.”
    
    Now, the bankruptcy has set up a rush among investors and customers to salvage funds from what remains of FTX. A surge of customers tried to withdraw funds from the platform this week, and the company couldn’t meet the demand. The exchange owes as much as $8 billion, according to people familiar with its finances.
    
    FTX’s collapse has destabilized the crypto industry, which was already reeling from a crash in the spring that drained $1 trillion from the market. The prices of the leading cryptocurrencies, Bitcoin and Ether, have plummeted. The crypto lender BlockFi, which was closely entangled with FTX, announced on Thursday that it was suspending operations as a result of FTX’s collapse.
    
    Mr.


key: _node_content
string_value:
{"id_": "9fec01b2-2394-48f7-904e-b6743777d905", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73e09a74-6398-44e1-955c-76b6a0839554", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "9eec624cd3ebfae27701b1260b8fc26541da89d72f061e152021165c691d897c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4ab80487-d602-41e0-9afd-63ece9967ac0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "d8822cdbf4806eacde0372165e2ff29e9de4a2b1b7405a1c7ba98512dcc6a9eb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "219880c5-75ed-4cee-bd07-807f22773840", "node_type": "1", "metadata": {}, "hash": "b2153be666f5197b61503fa31fcdd4b832fd34f27f3c69be0b1bde62eaba416f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4565, "end_char_idx": 8177, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
“Assets are fine.”
    
    On Friday, FTX announced that it was filing for bankruptcy, capping an extraordinary week of corporate drama that has upended crypto markets, sent shock waves through an industry struggling to gain mainstream credibility and sparked government investigations that could lead to more damaging revelations or even criminal charges.
    
    In a statement on Twitter, the company said that Mr. Bankman-Fried had resigned, with John J. Ray III, a corporate turnaround specialist, taking over as chief executive.
    
    The speed of FTX’s downfall has left crypto insiders stunned. Just days ago, Mr. Bankman-Fried was considered one of the smartest leaders in the crypto industry, an influential figure in Washington who was lobbying to shape regulations. And FTX was widely viewed as one of the most stable and responsible companies in the freewheeling, loosely regulated crypto industry.
    
    “Here we are, with one of the richest people in the world, his net worth dropping to zero, his business dropping to zero,” said Jared Ellias, a bankruptcy professor at Harvard Law School. “The velocity of this failure is just unbelievable.”
    
    Now, the bankruptcy has set up a rush among investors and customers to salvage funds from what remains of FTX. A surge of customers tried to withdraw funds from the platform this week, and the company couldn’t meet the demand. The exchange owes as much as $8 billion, according to people familiar with its finances.
    
    FTX’s collapse has destabilized the crypto industry, which was already reeling from a crash in the spring that drained $1 trillion from the market. The prices of the leading cryptocurrencies, Bitcoin and Ether, have plummeted. The crypto lender BlockFi, which was closely entangled with FTX, announced on Thursday that it was suspending operations as a result of FTX’s collapse.
    
    Mr. Bankman-Fried was backed by some of the highest-profile venture capital investors in Silicon Valley, including Sequoia Capital and Lightspeed Venture Partners. Some of those investors, facing questions about how closely they scrutinized FTX before they put money into it, have said that their nine-figure investments in the crypto exchange are now essentially worthless.
    
    The company’s demise has also set off a reckoning over risky practices that have become pervasive in crypto, an industry that was founded partly as a corrective to the type of dangerous financial engineering that caused the 2008 economic crisis.
    
    “I’m really sorry, again, that we ended up here,” Mr. Bankman-Fried said on Twitter on Friday. “Hopefully this can bring some amount of transparency, trust, and governance.”
    
    The bankruptcy filing marks the start of what will probably be months or even years of legal fallout, as lawyers try to work out whether the exchange can ever continue to operate in some form and customers demand compensation. FTX is already the target of investigations by the Securities and Exchange Commission and the Justice Department, with investigators focused on whether the company improperly used customer funds to prop up Alameda Research, a trading firm that Mr. Bankman-Fried also founded.
    
    The bankruptcy filing included FTX, its U.S. arm and Alameda. According to a bare-bones legal filing in U.S. Bankruptcy Court in Delaware, FTX has assets valued between $10 billion and $50 billion, with the size of its liabilities in the same range. The company has more than 100,000 creditors, the filing said.
    
    The bankruptcy is a stunning fall from grace for the 30-year-old Mr.


key: _node_content
string_value:
{"id_": "219880c5-75ed-4cee-bd07-807f22773840", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73e09a74-6398-44e1-955c-76b6a0839554", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "9eec624cd3ebfae27701b1260b8fc26541da89d72f061e152021165c691d897c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9fec01b2-2394-48f7-904e-b6743777d905", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "b8ad5c5bcf7bcabedd32ee64167098d9dec7b44d42fab23c5a17be6d2749b229", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8a08a232-fbfc-4cfd-9cfd-a06cbe48e204", "node_type": "1", "metadata": {}, "hash": "12e28056c6f8a916c4f0b7be06f8de47e4a3bcb67333d3d611275c27ff046d25", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6293, "end_char_idx": 9656, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The crypto lender BlockFi, which was closely entangled with FTX, announced on Thursday that it was suspending operations as a result of FTX’s collapse.
    
    Mr. Bankman-Fried was backed by some of the highest-profile venture capital investors in Silicon Valley, including Sequoia Capital and Lightspeed Venture Partners. Some of those investors, facing questions about how closely they scrutinized FTX before they put money into it, have said that their nine-figure investments in the crypto exchange are now essentially worthless.
    
    The company’s demise has also set off a reckoning over risky practices that have become pervasive in crypto, an industry that was founded partly as a corrective to the type of dangerous financial engineering that caused the 2008 economic crisis.
    
    “I’m really sorry, again, that we ended up here,” Mr. Bankman-Fried said on Twitter on Friday. “Hopefully this can bring some amount of transparency, trust, and governance.”
    
    The bankruptcy filing marks the start of what will probably be months or even years of legal fallout, as lawyers try to work out whether the exchange can ever continue to operate in some form and customers demand compensation. FTX is already the target of investigations by the Securities and Exchange Commission and the Justice Department, with investigators focused on whether the company improperly used customer funds to prop up Alameda Research, a trading firm that Mr. Bankman-Fried also founded.
    
    The bankruptcy filing included FTX, its U.S. arm and Alameda. According to a bare-bones legal filing in U.S. Bankruptcy Court in Delaware, FTX has assets valued between $10 billion and $50 billion, with the size of its liabilities in the same range. The company has more than 100,000 creditors, the filing said.
    
    The bankruptcy is a stunning fall from grace for the 30-year-old Mr. Bankman-Fried, who cultivated a reputation as a boy genius with a host of endearing quirks, including a habit of sleeping on a beanbag at the office. At one point, he was one of the richest people in the industry, with an estimated fortune of $24 billion. He hobnobbed with actors, professional athletes and former world leaders.
    
    Mr. Bankman-Fried’s crypto empire had an elaborate structure. The bankruptcy filing lists more than 130 corporate entities affiliated with FTX and Alameda. But as of June, FTX had only about 300 employees, a point of pride for Mr. Bankman-Fried, who said he had resisted calls from venture investors to hire more staff.
    
    “We told them additional employees added too quickly were net negative,” Mr. Bankman-Fried said on Twitter in June. “They could take it or leave it.”
    
    Unusually for a major start-up, none of FTX’s investors had seats on the board, which instead consisted of Mr. Bankman-Fried, another FTX executive and a lawyer in Antigua and Barbuda.
    
    FTX and Alameda were based in the Bahamas, where Mr. Bankman-Fried and a small circle of top executives called most of the shots and lived together in a luxury resort. Officially, Alameda was run by Caroline Ellison, a former trader for the hedge fund Jane Street, but Mr. Bankman-Fried was heavily involved, contributing to the decision-making on big trades, according to a person familiar with the matter.
    
    In addition to Mr. Bankman-Fried and Ms.


key: _node_content
string_value:
{"id_": "8a08a232-fbfc-4cfd-9cfd-a06cbe48e204", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73e09a74-6398-44e1-955c-76b6a0839554", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "9eec624cd3ebfae27701b1260b8fc26541da89d72f061e152021165c691d897c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "219880c5-75ed-4cee-bd07-807f22773840", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "1322f9f4969fe5bfcbdb6f71769ca418e476f5ff61ffc39c370db6a52f0fc178", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fc55b6a9-2edf-4992-bad9-095a27dda8dd", "node_type": "1", "metadata": {}, "hash": "f0785849910c8ffbd3a0e71ac91ea9380511878168d243920912b8a520b60862", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8038, "end_char_idx": 11334, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The company has more than 100,000 creditors, the filing said.
    
    The bankruptcy is a stunning fall from grace for the 30-year-old Mr. Bankman-Fried, who cultivated a reputation as a boy genius with a host of endearing quirks, including a habit of sleeping on a beanbag at the office. At one point, he was one of the richest people in the industry, with an estimated fortune of $24 billion. He hobnobbed with actors, professional athletes and former world leaders.
    
    Mr. Bankman-Fried’s crypto empire had an elaborate structure. The bankruptcy filing lists more than 130 corporate entities affiliated with FTX and Alameda. But as of June, FTX had only about 300 employees, a point of pride for Mr. Bankman-Fried, who said he had resisted calls from venture investors to hire more staff.
    
    “We told them additional employees added too quickly were net negative,” Mr. Bankman-Fried said on Twitter in June. “They could take it or leave it.”
    
    Unusually for a major start-up, none of FTX’s investors had seats on the board, which instead consisted of Mr. Bankman-Fried, another FTX executive and a lawyer in Antigua and Barbuda.
    
    FTX and Alameda were based in the Bahamas, where Mr. Bankman-Fried and a small circle of top executives called most of the shots and lived together in a luxury resort. Officially, Alameda was run by Caroline Ellison, a former trader for the hedge fund Jane Street, but Mr. Bankman-Fried was heavily involved, contributing to the decision-making on big trades, according to a person familiar with the matter.
    
    In addition to Mr. Bankman-Fried and Ms. Ellison, the circle of executives running FTX included Nishad Singh, FTX’s director of engineering, and Gary Wang, the chief technology officer. Few others had visibility into how the company was run: When the firm collapsed this week, lower-ranking employees were left confused and blindsided, according to people familiar with the matter. Mr. Singh and Ms. Ellison did not respond to requests for comment; Mr. Wang could not immediately be reached.
    
    As a crypto exchange, FTX provided a marketplace for customers to buy, sell and store a wide range of digital currencies. Most of its revenue stemmed from a risky type of trade — in which crypto investors borrowed money to make huge bets on the future prices of cryptocurrencies — that remains illegal in the United States. But Mr. Bankman-Fried also ran a smaller U.S. affiliate that offered more basic trading options.
    
    Mr. Bankman-Fried’s problems started over the weekend, when the chief executive of Binance, the largest crypto exchange, suggested publicly that FTX might be on shaky financial footing. A rush of customers tried to withdraw their crypto holdings from the platform, and FTX was unable to meet the demand.
    
    On Tuesday, Mr. Bankman-Fried said he had struck a deal to sell FTX to Binance. But after reviewing the company’s financial documents, Binance’s chief executive, Changpeng Zhao, pulled out of the agreement, leaving Mr. Bankman-Fried with limited options.
    
    In calls with investors and messages to employees this week, he apologized repeatedly and stressed that he was working hard to raise money and resolve the situation. But the hole was ultimately too big to fill.


key: _node_content
string_value:
{"id_": "fc55b6a9-2edf-4992-bad9-095a27dda8dd", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73e09a74-6398-44e1-955c-76b6a0839554", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "9eec624cd3ebfae27701b1260b8fc26541da89d72f061e152021165c691d897c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8a08a232-fbfc-4cfd-9cfd-a06cbe48e204", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "0e859605259b640dfbd69c91e876fa8cc3e2e7b0126cd8d807e72c3427333292", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a81fc8c1-bb84-4d46-9dd0-5186c244d765", "node_type": "1", "metadata": {}, "hash": "9ca998b635bf1b5ffd80e15f3d78bca5560b3ef1b0c58578bf8f7095e46337a9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9616, "end_char_idx": 13136, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In addition to Mr. Bankman-Fried and Ms. Ellison, the circle of executives running FTX included Nishad Singh, FTX’s director of engineering, and Gary Wang, the chief technology officer. Few others had visibility into how the company was run: When the firm collapsed this week, lower-ranking employees were left confused and blindsided, according to people familiar with the matter. Mr. Singh and Ms. Ellison did not respond to requests for comment; Mr. Wang could not immediately be reached.
    
    As a crypto exchange, FTX provided a marketplace for customers to buy, sell and store a wide range of digital currencies. Most of its revenue stemmed from a risky type of trade — in which crypto investors borrowed money to make huge bets on the future prices of cryptocurrencies — that remains illegal in the United States. But Mr. Bankman-Fried also ran a smaller U.S. affiliate that offered more basic trading options.
    
    Mr. Bankman-Fried’s problems started over the weekend, when the chief executive of Binance, the largest crypto exchange, suggested publicly that FTX might be on shaky financial footing. A rush of customers tried to withdraw their crypto holdings from the platform, and FTX was unable to meet the demand.
    
    On Tuesday, Mr. Bankman-Fried said he had struck a deal to sell FTX to Binance. But after reviewing the company’s financial documents, Binance’s chief executive, Changpeng Zhao, pulled out of the agreement, leaving Mr. Bankman-Fried with limited options.
    
    In calls with investors and messages to employees this week, he apologized repeatedly and stressed that he was working hard to raise money and resolve the situation. But the hole was ultimately too big to fill.
    
    FTX’s bankruptcy is the latest — and by far the biggest — in a series of bankruptcies that have shaken the crypto world this year. After a market crash in the spring, two crypto lending companies, Celsius Network and Voyager Digital, filed for bankruptcy, kicking off months of legal maneuvering over how their remaining assets should be divided. In an ironic twist, FTX had recently won an auction to buy Voyager’s remaining assets.
    
    As it enters its own bankruptcy process, FTX will be led by Mr. Ray, who has ample experience managing distressed situations. He helped manage Enron after the collapse of its business in an accounting fraud scandal in 2001. And he helped liquidate the trust of the subprime mortgage company ResCap after its 2012 bankruptcy.
    
    The bankruptcy proceedings may be only the beginning of Mr. Bankman-Fried’s legal troubles. Federal investigators are examining the relationship between FTX and Alameda, and customers are likely to file lawsuits.
    
    Mr. Bankman-Fried’s old allies have quickly abandoned him. On Thursday night, the team running the FTX Future Fund, a charitable group that Mr. Bankman-Fried bankrolled, announced that they were resigning.
    
    “We were shocked and immensely saddened to learn of the recent events at FTX,” they wrote in a statement. “We have fundamental questions about the legitimacy and integrity of the business operations that were funding the FTX Foundation and the Future Fund.”
    
    Not long ago, Mr. Bankman-Fried was performing a comedy routine onstage at a conference with Anthony Scaramucci, the former White House communications director and a business partner of FTX.
    
    “I’m disappointed,” Mr. Scaramucci said in an interview on CNBC on Friday. “Duped, I guess, is the right word.”


key: _node_content
string_value:
{"id_": "a81fc8c1-bb84-4d46-9dd0-5186c244d765", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73e09a74-6398-44e1-955c-76b6a0839554", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "9eec624cd3ebfae27701b1260b8fc26541da89d72f061e152021165c691d897c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fc55b6a9-2edf-4992-bad9-095a27dda8dd", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "24ca6164e71f071a7a47e5ec8b01515360edbffd85fa2b87d552948f3bb38530", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11290, "end_char_idx": 14107, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
But the hole was ultimately too big to fill.
    
    FTX’s bankruptcy is the latest — and by far the biggest — in a series of bankruptcies that have shaken the crypto world this year. After a market crash in the spring, two crypto lending companies, Celsius Network and Voyager Digital, filed for bankruptcy, kicking off months of legal maneuvering over how their remaining assets should be divided. In an ironic twist, FTX had recently won an auction to buy Voyager’s remaining assets.
    
    As it enters its own bankruptcy process, FTX will be led by Mr. Ray, who has ample experience managing distressed situations. He helped manage Enron after the collapse of its business in an accounting fraud scandal in 2001. And he helped liquidate the trust of the subprime mortgage company ResCap after its 2012 bankruptcy.
    
    The bankruptcy proceedings may be only the beginning of Mr. Bankman-Fried’s legal troubles. Federal investigators are examining the relationship between FTX and Alameda, and customers are likely to file lawsuits.
    
    Mr. Bankman-Fried’s old allies have quickly abandoned him. On Thursday night, the team running the FTX Future Fund, a charitable group that Mr. Bankman-Fried bankrolled, announced that they were resigning.
    
    “We were shocked and immensely saddened to learn of the recent events at FTX,” they wrote in a statement. “We have fundamental questions about the legitimacy and integrity of the business operations that were funding the FTX Foundation and the Future Fund.”
    
    Not long ago, Mr. Bankman-Fried was performing a comedy routine onstage at a conference with Anthony Scaramucci, the former White House communications director and a business partner of FTX.
    
    “I’m disappointed,” Mr. Scaramucci said in an interview on CNBC on Friday. “Duped, I guess, is the right word.”
    


## Repeat the Question to OpenAI with context
Now that we have relevant context, add that to the prompt to OpenAI and get a very different response.


```python
prompt = f"""
Using the information delimited by triple backticks, answer this question: Is Sam Bankman-Fried's company, FTX, considered a well-managed company?

Context: ```{context}```
"""

response = get_completion(prompt)
print(response)
```

    No, Sam Bankman-Fried's company FTX is not considered a well-managed company as it has filed for bankruptcy and owes as much as $8 billion to its creditors. The collapse of FTX has destabilized the crypto industry, and the company is already the target of investigations by the Securities and Exchange Commission and the Justice Department. FTX was widely viewed as one of the most stable and responsible companies in the freewheeling, loosely regulated crypto industry, but its risky practices have become pervasive in crypto, leading to a reckoning.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb


key: _node_content
string_value:
{"id_": "f5f916bf-98fb-4b91-a510-5046868f4bfb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "59ff14ffeee6b7e47ae59ae97b4b178f7ca83621d6168206707ca95cbea61f95", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6e838e90-90ae-4998-8315-298334c945b3", "node_type": "1", "metadata": {}, "hash": "8d2d0ca97243dde3ea8c9044ae310d031613be0ed046441b15d692edd9f715e9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 709, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Redis Vectors as JSON with OpenAI
This notebook expands on the other Redis OpenAI-cookbook examples with examples of how to use JSON with vectors.
[Storing Vectors in JSON](https://redis.io/docs/stack/search/reference/vectors/#storing-vectors-in-json)

## Prerequisites
* Redis instance with the Redis Search and Redis JSON modules
* Redis-py client lib
* OpenAI API key

## Installation
Install Python modules necessary for the examples.


```python
! pip install redis openai python-dotenv openai[datalib]
```

## OpenAI API Key
Create a .env file and add your OpenAI key to it


```python
OPENAI_API_KEY=your_key
```

## Create Text Vectors
Create embeddings (array of floats) of the news excerpts below.


key: _node_content
string_value:
{"id_": "6e838e90-90ae-4998-8315-298334c945b3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "59ff14ffeee6b7e47ae59ae97b4b178f7ca83621d6168206707ca95cbea61f95", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f5f916bf-98fb-4b91-a510-5046868f4bfb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "348ddf8544ecf5da8aa39d931d3b7b382d73f867aed87216bdf5582a88a7244c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6109e86a-35a8-47e8-9247-316235354563", "node_type": "1", "metadata": {}, "hash": "b52d340c57370ee932a3f11059eb95a1125a10f80eaf12ffe964ec2d6c67af04", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3176, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Redis Vectors as JSON with OpenAI
This notebook expands on the other Redis OpenAI-cookbook examples with examples of how to use JSON with vectors.
[Storing Vectors in JSON](https://redis.io/docs/stack/search/reference/vectors/#storing-vectors-in-json)

## Prerequisites
* Redis instance with the Redis Search and Redis JSON modules
* Redis-py client lib
* OpenAI API key

## Installation
Install Python modules necessary for the examples.


```python
! pip install redis openai python-dotenv openai[datalib]
```

## OpenAI API Key
Create a .env file and add your OpenAI key to it


```python
OPENAI_API_KEY=your_key
```

## Create Text Vectors
Create embeddings (array of floats) of the news excerpts below.


```python
import openai
import os
from dotenv import load_dotenv

load_dotenv()
openai.api_key = os.getenv("OPENAI_API_KEY")

def get_vector(text, model="text-embedding-3-small"):
    text = text.replace("\n", " ")
    return openai.Embedding.create(input = [text], model = model)['data'][0]['embedding']

text_1 = """Japan narrowly escapes recession

Japan's economy teetered on the brink of a technical recession in the three months to September, figures show.

Revised figures indicated growth of just 0.1% - and a similar-sized contraction in the previous quarter. On an annual basis, the data suggests annual growth of just 0.2%, suggesting a much more hesitant recovery than had previously been thought. A common technical definition of a recession is two successive quarters of negative growth.
The government was keen to play down the worrying implications of the data. "I maintain the view that Japan's economy remains in a minor adjustment phase in an upward climb, and we will monitor developments carefully," said economy minister Heizo Takenaka. But in the face of the strengthening yen making exports less competitive and indications of weakening economic conditions ahead, observers were less sanguine. "It's painting a picture of a recovery... much patchier than previously thought," said Paul Sheard, economist at Lehman Brothers in Tokyo. Improvements in the job market apparently have yet to feed through to domestic demand, with private consumption up just 0.2% in the third quarter.
"""

text_2 = """Dibaba breaks 5,000m world record

Ethiopia's Tirunesh Dibaba set a new world record in winning the women's 5,000m at the Boston Indoor Games.

Dibaba won in 14 minutes 32.93 seconds to erase the previous world indoor mark of 14:39.29 set by another Ethiopian, Berhane Adera, in Stuttgart last year. But compatriot Kenenisa Bekele's record hopes were dashed when he miscounted his laps in the men's 3,000m and staged his sprint finish a lap too soon. Ireland's Alistair Cragg won in 7:39.89 as Bekele battled to second in 7:41.42. "I didn't want to sit back and get out-kicked," said Cragg. "So I kept on the pace. The plan was to go with 500m to go no matter what, but when Bekele made the mistake that was it. The race was mine." Sweden's Carolina Kluft, the Olympic heptathlon champion, and Slovenia's Jolanda Ceplak had winning performances, too. Kluft took the long jump at 6.63m, while Ceplak easily won the women's 800m in 2:01.52. 
"""


key: _node_content
string_value:
{"id_": "6109e86a-35a8-47e8-9247-316235354563", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "59ff14ffeee6b7e47ae59ae97b4b178f7ca83621d6168206707ca95cbea61f95", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6e838e90-90ae-4998-8315-298334c945b3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "adf90edc5a7283a25aa69c87a8dc7f0186fde47d3c6b6530d3f5bcc4d047327f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e642379a-83fd-41a5-9252-fecf05eb91a4", "node_type": "1", "metadata": {}, "hash": "8aa3fa81465b5cd8bfb62fd6cb659f9a29efc7516604b196434a77fe9e194f48", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3179, "end_char_idx": 6356, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
text_3 = """Google's toolbar sparks concern

Search engine firm Google has released a trial tool which is concerning some net users because it directs people to pre-selected commercial websites.

The AutoLink feature comes with Google's latest toolbar and provides links in a webpage to Amazon.com if it finds a book's ISBN number on the site. It also links to Google's map service, if there is an address, or to car firm Carfax, if there is a licence plate. Google said the feature, available only in the US, "adds useful links". But some users are concerned that Google's dominant position in the search engine market place could mean it would be giving a competitive edge to firms like Amazon.

AutoLink works by creating a link to a website based on information contained in a webpage - even if there is no link specified and whether or not the publisher of the page has given permission.

If a user clicks the AutoLink feature in the Google toolbar then a webpage with a book's unique ISBN number would link directly to Amazon's website. It could mean online libraries that list ISBN book numbers find they are directing users to Amazon.com whether they like it or not. Websites which have paid for advertising on their pages may also be directing people to rival services. Dan Gillmor, founder of Grassroots Media, which supports citizen-based media, said the tool was a "bad idea, and an unfortunate move by a company that is looking to continue its hypergrowth". In a statement Google said the feature was still only in beta, ie trial, stage and that the company welcomed feedback from users. It said: "The user can choose never to click on the AutoLink button, and web pages she views will never be modified. "In addition, the user can choose to disable the AutoLink feature entirely at any time."

The new tool has been compared to the Smart Tags feature from Microsoft by some users. It was widely criticised by net users and later dropped by Microsoft after concerns over trademark use were raised. Smart Tags allowed Microsoft to link any word on a web page to another site chosen by the company. Google said none of the companies which received AutoLinks had paid for the service. Some users said AutoLink would only be fair if websites had to sign up to allow the feature to work on their pages or if they received revenue for any "click through" to a commercial site. Cory Doctorow, European outreach coordinator for digital civil liberties group Electronic Fronter Foundation, said that Google should not be penalised for its market dominance. "Of course Google should be allowed to direct people to whatever proxies it chooses. "But as an end user I would want to know - 'Can I choose to use this service?, 'How much is Google being paid?', 'Can I substitute my own companies for the ones chosen by Google?'." Mr Doctorow said the only objection would be if users were forced into using AutoLink or "tricked into using the service".
"""

doc_1 = {"content": text_1, "vector": get_vector(text_1)}
doc_2 = {"content": text_2, "vector": get_vector(text_2)}
doc_3 = {"content": text_3, "vector": get_vector(text_3)}

```

## Start the Redis Stack Docker container


key: _node_content
string_value:
{"id_": "e642379a-83fd-41a5-9252-fecf05eb91a4", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "59ff14ffeee6b7e47ae59ae97b4b178f7ca83621d6168206707ca95cbea61f95", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6109e86a-35a8-47e8-9247-316235354563", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "8b21fde02b0bd82bbfe5baf341271a51a0e4519f36a6245b1628383fc826b519", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7cf8afae-6c9b-4741-92ea-ffccdaf522d3", "node_type": "1", "metadata": {}, "hash": "b9286c8669040e5a17b1e8e84fda5b717dfcc7b4eb9245ee5613eb8a88da4958", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6359, "end_char_idx": 8920, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
! docker compose up -d
```

    [1A[1B[0G[?25l[+] Running 0/0
     ⠿ Container redisjson-redis-1  Starting                                   [34m0.1s [0m
    [?25h[1A[1A[0G[?25l[+] Running 0/1
     ⠿ Container redisjson-redis-1  Starting                                   [34m0.2s [0m
    [?25h[1A[1A[0G[?25l[+] Running 0/1
     ⠿ Container redisjson-redis-1  Starting                                   [34m0.3s [0m
    [?25h[1A[1A[0G[?25l[+] Running 0/1
     ⠿ Container redisjson-redis-1  Starting                                   [34m0.4s [0m
    [?25h[1A[1A[0G[?25l[34m[+] Running 1/1[0m
     [32m✔[0m Container redisjson-redis-1  [32mStarted[0m                                    [34m0.4s [0m
    [?25h

## Connect Redis client


```python
from redis import from_url

REDIS_URL = 'redis://localhost:6379'
client = from_url(REDIS_URL)
client.ping()
```




    True



## Create Index
[FT.CREATE](https://redis.io/commands/ft.create/)


```python
from redis.commands.search.field import TextField, VectorField
from redis.commands.search.indexDefinition import IndexDefinition, IndexType

schema = [ VectorField('$.vector', 
            "FLAT", 
            {   "TYPE": 'FLOAT32', 
                "DIM": len(doc_1['vector']), 
                "DISTANCE_METRIC": "COSINE"
            },  as_name='vector' ),
            TextField('$.content', as_name='content')
        ]
idx_def = IndexDefinition(index_type=IndexType.JSON, prefix=['doc:'])
try: 
    client.ft('idx').dropindex()
except:
    pass
client.ft('idx').create_index(schema, definition=idx_def)
```




    b'OK'



## Load Data into Redis as JSON objects
[Redis JSON](https://redis.io/docs/stack/json/)


```python
client.json().set('doc:1', '$', doc_1)
client.json().set('doc:2', '$', doc_2)
client.json().set('doc:3', '$', doc_3)
```




    True



# Semantic Search
Given a sports-related article, search Redis via Vector Similarity Search (VSS) for similar articles.  
[KNN Search](https://redis.io/docs/stack/search/reference/vectors/#knn-search)


```python
from redis.commands.search.query import Query
import numpy as np

text_4 = """Radcliffe yet to answer GB call

Paula Radcliffe has been granted extra time to decide whether to compete in the World Cross-Country Championships.

The 31-year-old is concerned the event, which starts on 19 March in France, could upset her preparations for the London Marathon on 17 April. "There is no question that Paula would be a huge asset to the GB team," said Zara Hyde Peters of UK Athletics.


key: _node_content
string_value:
{"id_": "7cf8afae-6c9b-4741-92ea-ffccdaf522d3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "59ff14ffeee6b7e47ae59ae97b4b178f7ca83621d6168206707ca95cbea61f95", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e642379a-83fd-41a5-9252-fecf05eb91a4", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "fd5fd94403a097a4a4fafc9b86338ce408558cdf1bd78e12479b0815f0b47e1a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d6b59611-6bca-4882-813f-6d77b2851e80", "node_type": "1", "metadata": {}, "hash": "431278fcc6f7763266c02451d5e8c6b681ddd24f0951f25d768679f2244321cb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7352, "end_char_idx": 10431, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from redis.commands.search.field import TextField, VectorField
from redis.commands.search.indexDefinition import IndexDefinition, IndexType

schema = [ VectorField('$.vector', 
            "FLAT", 
            {   "TYPE": 'FLOAT32', 
                "DIM": len(doc_1['vector']), 
                "DISTANCE_METRIC": "COSINE"
            },  as_name='vector' ),
            TextField('$.content', as_name='content')
        ]
idx_def = IndexDefinition(index_type=IndexType.JSON, prefix=['doc:'])
try: 
    client.ft('idx').dropindex()
except:
    pass
client.ft('idx').create_index(schema, definition=idx_def)
```




    b'OK'



## Load Data into Redis as JSON objects
[Redis JSON](https://redis.io/docs/stack/json/)


```python
client.json().set('doc:1', '$', doc_1)
client.json().set('doc:2', '$', doc_2)
client.json().set('doc:3', '$', doc_3)
```




    True



# Semantic Search
Given a sports-related article, search Redis via Vector Similarity Search (VSS) for similar articles.  
[KNN Search](https://redis.io/docs/stack/search/reference/vectors/#knn-search)


```python
from redis.commands.search.query import Query
import numpy as np

text_4 = """Radcliffe yet to answer GB call

Paula Radcliffe has been granted extra time to decide whether to compete in the World Cross-Country Championships.

The 31-year-old is concerned the event, which starts on 19 March in France, could upset her preparations for the London Marathon on 17 April. "There is no question that Paula would be a huge asset to the GB team," said Zara Hyde Peters of UK Athletics. "But she is working out whether she can accommodate the worlds without too much compromise in her marathon training." Radcliffe must make a decision by Tuesday - the deadline for team nominations. British team member Hayley Yelling said the team would understand if Radcliffe opted out of the event. "It would be fantastic to have Paula in the team," said the European cross-country champion. "But you have to remember that athletics is basically an individual sport and anything achieved for the team is a bonus. "She is not messing us around. We all understand the problem." Radcliffe was world cross-country champion in 2001 and 2002 but missed last year's event because of injury. In her absence, the GB team won bronze in Brussels.
"""

vec = np.array(get_vector(text_4), dtype=np.float32).tobytes()
q = Query('*=>[KNN 3 @vector $query_vec AS vector_score]')\
    .sort_by('vector_score')\
    .return_fields('vector_score', 'content')\
    .dialect(2)    
params = {"query_vec": vec}

results = client.ft('idx').search(q, query_params=params)
for doc in results.docs:
    print(f"distance:{round(float(doc['vector_score']),3)} content:{doc['content']}\n")

```

    distance:0.188 content:Dibaba breaks 5,000m world record
    
    Ethiopia's Tirunesh Dibaba set a new world record in winning the women's 5,000m at the Boston Indoor Games.
    
    Dibaba won in 14 minutes 32.93 seconds to erase the previous world indoor mark of 14:39.29 set by another Ethiopian, Berhane Adera, in Stuttgart last year.


key: _node_content
string_value:
{"id_": "d6b59611-6bca-4882-813f-6d77b2851e80", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "59ff14ffeee6b7e47ae59ae97b4b178f7ca83621d6168206707ca95cbea61f95", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7cf8afae-6c9b-4741-92ea-ffccdaf522d3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "838cb0ce7446c5b956afbbce3f40a0bdaa122004cc68f3fd09234decbdb42e4d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2db36b52-60f6-47ca-873a-6624a100765f", "node_type": "1", "metadata": {}, "hash": "5559c858b990357d3a95ded10a53e69b4f160ff1ee6e0a09f415b29c5a2f59b6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8810, "end_char_idx": 11869, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"There is no question that Paula would be a huge asset to the GB team," said Zara Hyde Peters of UK Athletics. "But she is working out whether she can accommodate the worlds without too much compromise in her marathon training." Radcliffe must make a decision by Tuesday - the deadline for team nominations. British team member Hayley Yelling said the team would understand if Radcliffe opted out of the event. "It would be fantastic to have Paula in the team," said the European cross-country champion. "But you have to remember that athletics is basically an individual sport and anything achieved for the team is a bonus. "She is not messing us around. We all understand the problem." Radcliffe was world cross-country champion in 2001 and 2002 but missed last year's event because of injury. In her absence, the GB team won bronze in Brussels.
"""

vec = np.array(get_vector(text_4), dtype=np.float32).tobytes()
q = Query('*=>[KNN 3 @vector $query_vec AS vector_score]')\
    .sort_by('vector_score')\
    .return_fields('vector_score', 'content')\
    .dialect(2)    
params = {"query_vec": vec}

results = client.ft('idx').search(q, query_params=params)
for doc in results.docs:
    print(f"distance:{round(float(doc['vector_score']),3)} content:{doc['content']}\n")

```

    distance:0.188 content:Dibaba breaks 5,000m world record
    
    Ethiopia's Tirunesh Dibaba set a new world record in winning the women's 5,000m at the Boston Indoor Games.
    
    Dibaba won in 14 minutes 32.93 seconds to erase the previous world indoor mark of 14:39.29 set by another Ethiopian, Berhane Adera, in Stuttgart last year. But compatriot Kenenisa Bekele's record hopes were dashed when he miscounted his laps in the men's 3,000m and staged his sprint finish a lap too soon. Ireland's Alistair Cragg won in 7:39.89 as Bekele battled to second in 7:41.42. "I didn't want to sit back and get out-kicked," said Cragg. "So I kept on the pace. The plan was to go with 500m to go no matter what, but when Bekele made the mistake that was it. The race was mine." Sweden's Carolina Kluft, the Olympic heptathlon champion, and Slovenia's Jolanda Ceplak had winning performances, too. Kluft took the long jump at 6.63m, while Ceplak easily won the women's 800m in 2:01.52. 
    
    
    distance:0.268 content:Japan narrowly escapes recession
    
    Japan's economy teetered on the brink of a technical recession in the three months to September, figures show.
    
    Revised figures indicated growth of just 0.1% - and a similar-sized contraction in the previous quarter. On an annual basis, the data suggests annual growth of just 0.2%, suggesting a much more hesitant recovery than had previously been thought. A common technical definition of a recession is two successive quarters of negative growth.
    The government was keen to play down the worrying implications of the data. "I maintain the view that Japan's economy remains in a minor adjustment phase in an upward climb, and we will monitor developments carefully," said economy minister Heizo Takenaka.


key: _node_content
string_value:
{"id_": "2db36b52-60f6-47ca-873a-6624a100765f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "59ff14ffeee6b7e47ae59ae97b4b178f7ca83621d6168206707ca95cbea61f95", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d6b59611-6bca-4882-813f-6d77b2851e80", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "dcbcdcb4481ee40ea00855430d647843db9b260f4724f951b28556b6d67ec0fa", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ea1f9bf7-f397-40c5-a97b-7cbc76ec027f", "node_type": "1", "metadata": {}, "hash": "f5fd67aa2975f011b683e2f53bde92842adbc5e18bdc32524915cf8a9b43883f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10432, "end_char_idx": 13842, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
But compatriot Kenenisa Bekele's record hopes were dashed when he miscounted his laps in the men's 3,000m and staged his sprint finish a lap too soon. Ireland's Alistair Cragg won in 7:39.89 as Bekele battled to second in 7:41.42. "I didn't want to sit back and get out-kicked," said Cragg. "So I kept on the pace. The plan was to go with 500m to go no matter what, but when Bekele made the mistake that was it. The race was mine." Sweden's Carolina Kluft, the Olympic heptathlon champion, and Slovenia's Jolanda Ceplak had winning performances, too. Kluft took the long jump at 6.63m, while Ceplak easily won the women's 800m in 2:01.52. 
    
    
    distance:0.268 content:Japan narrowly escapes recession
    
    Japan's economy teetered on the brink of a technical recession in the three months to September, figures show.
    
    Revised figures indicated growth of just 0.1% - and a similar-sized contraction in the previous quarter. On an annual basis, the data suggests annual growth of just 0.2%, suggesting a much more hesitant recovery than had previously been thought. A common technical definition of a recession is two successive quarters of negative growth.
    The government was keen to play down the worrying implications of the data. "I maintain the view that Japan's economy remains in a minor adjustment phase in an upward climb, and we will monitor developments carefully," said economy minister Heizo Takenaka. But in the face of the strengthening yen making exports less competitive and indications of weakening economic conditions ahead, observers were less sanguine. "It's painting a picture of a recovery... much patchier than previously thought," said Paul Sheard, economist at Lehman Brothers in Tokyo. Improvements in the job market apparently have yet to feed through to domestic demand, with private consumption up just 0.2% in the third quarter.
    
    
    distance:0.287 content:Google's toolbar sparks concern
    
    Search engine firm Google has released a trial tool which is concerning some net users because it directs people to pre-selected commercial websites.
    
    The AutoLink feature comes with Google's latest toolbar and provides links in a webpage to Amazon.com if it finds a book's ISBN number on the site. It also links to Google's map service, if there is an address, or to car firm Carfax, if there is a licence plate. Google said the feature, available only in the US, "adds useful links". But some users are concerned that Google's dominant position in the search engine market place could mean it would be giving a competitive edge to firms like Amazon.
    
    AutoLink works by creating a link to a website based on information contained in a webpage - even if there is no link specified and whether or not the publisher of the page has given permission.
    
    If a user clicks the AutoLink feature in the Google toolbar then a webpage with a book's unique ISBN number would link directly to Amazon's website. It could mean online libraries that list ISBN book numbers find they are directing users to Amazon.com whether they like it or not. Websites which have paid for advertising on their pages may also be directing people to rival services. Dan Gillmor, founder of Grassroots Media, which supports citizen-based media, said the tool was a "bad idea, and an unfortunate move by a company that is looking to continue its hypergrowth".


key: _node_content
string_value:
{"id_": "ea1f9bf7-f397-40c5-a97b-7cbc76ec027f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "59ff14ffeee6b7e47ae59ae97b4b178f7ca83621d6168206707ca95cbea61f95", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2db36b52-60f6-47ca-873a-6624a100765f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "f0f217db404b988a5caae17123c30d0b46bfbbd58b7d9061cec715a7dd97505d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cc901719-d812-4094-9ee1-d4f00cf91877", "node_type": "1", "metadata": {}, "hash": "59d57b77dacb9f73dc95a23572e4f5d54c8d60d40435da540189d34bac7c85e3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12029, "end_char_idx": 15330, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"It's painting a picture of a recovery... much patchier than previously thought," said Paul Sheard, economist at Lehman Brothers in Tokyo. Improvements in the job market apparently have yet to feed through to domestic demand, with private consumption up just 0.2% in the third quarter.
    
    
    distance:0.287 content:Google's toolbar sparks concern
    
    Search engine firm Google has released a trial tool which is concerning some net users because it directs people to pre-selected commercial websites.
    
    The AutoLink feature comes with Google's latest toolbar and provides links in a webpage to Amazon.com if it finds a book's ISBN number on the site. It also links to Google's map service, if there is an address, or to car firm Carfax, if there is a licence plate. Google said the feature, available only in the US, "adds useful links". But some users are concerned that Google's dominant position in the search engine market place could mean it would be giving a competitive edge to firms like Amazon.
    
    AutoLink works by creating a link to a website based on information contained in a webpage - even if there is no link specified and whether or not the publisher of the page has given permission.
    
    If a user clicks the AutoLink feature in the Google toolbar then a webpage with a book's unique ISBN number would link directly to Amazon's website. It could mean online libraries that list ISBN book numbers find they are directing users to Amazon.com whether they like it or not. Websites which have paid for advertising on their pages may also be directing people to rival services. Dan Gillmor, founder of Grassroots Media, which supports citizen-based media, said the tool was a "bad idea, and an unfortunate move by a company that is looking to continue its hypergrowth". In a statement Google said the feature was still only in beta, ie trial, stage and that the company welcomed feedback from users. It said: "The user can choose never to click on the AutoLink button, and web pages she views will never be modified. "In addition, the user can choose to disable the AutoLink feature entirely at any time."
    
    The new tool has been compared to the Smart Tags feature from Microsoft by some users. It was widely criticised by net users and later dropped by Microsoft after concerns over trademark use were raised. Smart Tags allowed Microsoft to link any word on a web page to another site chosen by the company. Google said none of the companies which received AutoLinks had paid for the service. Some users said AutoLink would only be fair if websites had to sign up to allow the feature to work on their pages or if they received revenue for any "click through" to a commercial site. Cory Doctorow, European outreach coordinator for digital civil liberties group Electronic Fronter Foundation, said that Google should not be penalised for its market dominance. "Of course Google should be allowed to direct people to whatever proxies it chooses. "But as an end user I would want to know - 'Can I choose to use this service?, 'How much is Google being paid?', 'Can I substitute my own companies for the ones chosen by Google?'." Mr Doctorow said the only objection would be if users were forced into using AutoLink or "tricked into using the service".


key: _node_content
string_value:
{"id_": "cc901719-d812-4094-9ee1-d4f00cf91877", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "59ff14ffeee6b7e47ae59ae97b4b178f7ca83621d6168206707ca95cbea61f95", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ea1f9bf7-f397-40c5-a97b-7cbc76ec027f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "2a8c63230564c84ce9bba3498279342e634228264ea1b4bbec7ad7d7ddbd9cec", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "823a3fe2-651b-4df9-bf5a-35ae33f36a2b", "node_type": "1", "metadata": {}, "hash": "68b8a1c2f77389e0808b1e31504826bc9502b2ddd3dda8171adee4c77a9d1d76", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13547, "end_char_idx": 15715, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Websites which have paid for advertising on their pages may also be directing people to rival services. Dan Gillmor, founder of Grassroots Media, which supports citizen-based media, said the tool was a "bad idea, and an unfortunate move by a company that is looking to continue its hypergrowth". In a statement Google said the feature was still only in beta, ie trial, stage and that the company welcomed feedback from users. It said: "The user can choose never to click on the AutoLink button, and web pages she views will never be modified. "In addition, the user can choose to disable the AutoLink feature entirely at any time."
    
    The new tool has been compared to the Smart Tags feature from Microsoft by some users. It was widely criticised by net users and later dropped by Microsoft after concerns over trademark use were raised. Smart Tags allowed Microsoft to link any word on a web page to another site chosen by the company. Google said none of the companies which received AutoLinks had paid for the service. Some users said AutoLink would only be fair if websites had to sign up to allow the feature to work on their pages or if they received revenue for any "click through" to a commercial site. Cory Doctorow, European outreach coordinator for digital civil liberties group Electronic Fronter Foundation, said that Google should not be penalised for its market dominance. "Of course Google should be allowed to direct people to whatever proxies it chooses. "But as an end user I would want to know - 'Can I choose to use this service?, 'How much is Google being paid?', 'Can I substitute my own companies for the ones chosen by Google?'." Mr Doctorow said the only objection would be if users were forced into using AutoLink or "tricked into using the service".
    
    


## Hybrid Search
Use a combination of full text search and VSS to find a matching article.  For this scenario, we filter on a full text search of the term 'recession' and then find the KNN articles.  In this case, business-related.  Reminder document #1 was about a recession in Japan.
[Hybrid Queries](https://redis.io/docs/stack/search/reference/vectors/#hybrid-queries)


key: _node_content
string_value:
{"id_": "823a3fe2-651b-4df9-bf5a-35ae33f36a2b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "08e115f5-0f6b-41c0-8aa1-a201735de5bc", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "59ff14ffeee6b7e47ae59ae97b4b178f7ca83621d6168206707ca95cbea61f95", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cc901719-d812-4094-9ee1-d4f00cf91877", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "b038f2b907af95711e8dbf4ebb517fd9ebe1f43c1d2b53621e4585483a232389", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13973, "end_char_idx": 18915, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
It said: "The user can choose never to click on the AutoLink button, and web pages she views will never be modified. "In addition, the user can choose to disable the AutoLink feature entirely at any time."
    
    The new tool has been compared to the Smart Tags feature from Microsoft by some users. It was widely criticised by net users and later dropped by Microsoft after concerns over trademark use were raised. Smart Tags allowed Microsoft to link any word on a web page to another site chosen by the company. Google said none of the companies which received AutoLinks had paid for the service. Some users said AutoLink would only be fair if websites had to sign up to allow the feature to work on their pages or if they received revenue for any "click through" to a commercial site. Cory Doctorow, European outreach coordinator for digital civil liberties group Electronic Fronter Foundation, said that Google should not be penalised for its market dominance. "Of course Google should be allowed to direct people to whatever proxies it chooses. "But as an end user I would want to know - 'Can I choose to use this service?, 'How much is Google being paid?', 'Can I substitute my own companies for the ones chosen by Google?'." Mr Doctorow said the only objection would be if users were forced into using AutoLink or "tricked into using the service".
    
    


## Hybrid Search
Use a combination of full text search and VSS to find a matching article.  For this scenario, we filter on a full text search of the term 'recession' and then find the KNN articles.  In this case, business-related.  Reminder document #1 was about a recession in Japan.
[Hybrid Queries](https://redis.io/docs/stack/search/reference/vectors/#hybrid-queries)


```python
text_5 = """Ethiopia's crop production up 24%

Ethiopia produced 14.27 million tonnes of crops in 2004, 24% higher than in 2003 and 21% more than the average of the past five years, a report says.

In 2003, crop production totalled 11.49 million tonnes, the joint report from the Food and Agriculture Organisation and the World Food Programme said. Good rains, increased use of fertilizers and improved seeds contributed to the rise in production. Nevertheless, 2.2 million Ethiopians will still need emergency assistance.

The report calculated emergency food requirements for 2005 to be 387,500 tonnes. On top of that, 89,000 tonnes of fortified blended food and vegetable oil for "targeted supplementary food distributions for a survival programme for children under five and pregnant and lactating women" will be needed.

In eastern and southern Ethiopia, a prolonged drought has killed crops and drained wells. Last year, a total of 965,000 tonnes of food assistance was needed to help seven million Ethiopians. The Food and Agriculture Organisation (FAO) recommend that the food assistance is bought locally. "Local purchase of cereals for food assistance programmes is recommended as far as possible, so as to assist domestic markets and farmers," said Henri Josserand, chief of FAO's Global Information and Early Warning System. Agriculture is the main economic activity in Ethiopia, representing 45% of gross domestic product. About 80% of Ethiopians depend directly or indirectly on agriculture.
"""

vec = np.array(get_vector(text_5), dtype=np.float32).tobytes()
q = Query('@content:recession => [KNN 3 @vector $query_vec AS vector_score]')\
    .sort_by('vector_score')\
    .return_fields('vector_score', 'content')\
    .dialect(2)    
params = {"query_vec": vec}

results = client.ft('idx').search(q, query_params=params)
for doc in results.docs:
    print(f"distance:{round(float(doc['vector_score']),3)} content:{doc['content']}\n")
```

    distance:0.241 content:Japan narrowly escapes recession
    
    Japan's economy teetered on the brink of a technical recession in the three months to September, figures show.
    
    Revised figures indicated growth of just 0.1% - and a similar-sized contraction in the previous quarter. On an annual basis, the data suggests annual growth of just 0.2%, suggesting a much more hesitant recovery than had previously been thought. A common technical definition of a recession is two successive quarters of negative growth.
    The government was keen to play down the worrying implications of the data. "I maintain the view that Japan's economy remains in a minor adjustment phase in an upward climb, and we will monitor developments carefully," said economy minister Heizo Takenaka. But in the face of the strengthening yen making exports less competitive and indications of weakening economic conditions ahead, observers were less sanguine. "It's painting a picture of a recovery... much patchier than previously thought," said Paul Sheard, economist at Lehman Brothers in Tokyo. Improvements in the job market apparently have yet to feed through to domestic demand, with private consumption up just 0.2% in the third quarter.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb


key: _node_content
string_value:
{"id_": "1842d8e7-8a06-4e77-af14-2c55ce25d92b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2bc858dc-ca27-4f5f-9212-33eb88bed6bb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4d05c5c7f2f4fadc6bfa392d8e54521303b098b379ac26589d42375d374c1ffa", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f423d2b9-7012-473b-800a-4c7e77ddf29a", "node_type": "1", "metadata": {}, "hash": "2a91c59f48d3a558669bf051ea0cccd23a42c6f47c8e1eb52f8a859f4aaef26f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3594, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Using Typesense for Embeddings Search

This notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.

### What is a Vector Database

A vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.

### Why use a Vector Database

Vector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.


### Demo Flow
The demo flow is:
- **Setup**: Import packages and set any required variables
- **Load data**: Load a dataset and embed it using OpenAI embeddings
- **Typesense**
    - *Setup*: Set up the Typesense Python client. For more details go [here](https://typesense.org/docs/0.24.0/api/)
    - *Index Data*: We'll create a collection and index it for both __titles__ and __content__.
    - *Search Data*: Run a few example queries with various goals in mind.

Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.

## Setup

Import the required libraries and set the embedding model that we'd like to use.


```python
# We'll need to install the Typesense client
!pip install typesense

#Install wget to pull zip file
!pip install wget
```


```python
import openai

from typing import List, Iterator
import pandas as pd
import numpy as np
import os
import wget
from ast import literal_eval

# Typesense's client library for Python
import typesense

# I've set this to our new embeddings model, this can be changed to the embedding model of your choice
EMBEDDING_MODEL = "text-embedding-3-small"

# Ignore unclosed SSL socket warnings - optional in case you get these errors
import warnings

warnings.filterwarnings(action="ignore", message="unclosed", category=ResourceWarning)
warnings.filterwarnings("ignore", category=DeprecationWarning) 
```

## Load data

In this section we'll load embedded data that we've prepared previous to this session.


```python
embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```


```python
import zipfile
with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip","r") as zip_ref:
    zip_ref.extractall("../data")
```


```python
article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')
```


key: _node_content
string_value:
{"id_": "f423d2b9-7012-473b-800a-4c7e77ddf29a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2bc858dc-ca27-4f5f-9212-33eb88bed6bb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4d05c5c7f2f4fadc6bfa392d8e54521303b098b379ac26589d42375d374c1ffa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1842d8e7-8a06-4e77-af14-2c55ce25d92b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c9db8ce78a76cbdb199f5d75ccdb1f70ff903be1c98caa1bb8086ac97eafb125", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ca12d2e1-bed7-4211-a611-5f88d6981afe", "node_type": "1", "metadata": {}, "hash": "55d75d734c6c465942a4bbe99c9e2550ca1dde90d5af183d906f3fc23ae149c3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2296, "end_char_idx": 4644, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# We'll need to install the Typesense client
!pip install typesense

#Install wget to pull zip file
!pip install wget
```


```python
import openai

from typing import List, Iterator
import pandas as pd
import numpy as np
import os
import wget
from ast import literal_eval

# Typesense's client library for Python
import typesense

# I've set this to our new embeddings model, this can be changed to the embedding model of your choice
EMBEDDING_MODEL = "text-embedding-3-small"

# Ignore unclosed SSL socket warnings - optional in case you get these errors
import warnings

warnings.filterwarnings(action="ignore", message="unclosed", category=ResourceWarning)
warnings.filterwarnings("ignore", category=DeprecationWarning) 
```

## Load data

In this section we'll load embedded data that we've prepared previous to this session.


```python
embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```


```python
import zipfile
with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip","r") as zip_ref:
    zip_ref.extractall("../data")
```


```python
article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')
```


```python
article_df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>url</th>
      <th>title</th>
      <th>text</th>
      <th>title_vector</th>
      <th>content_vector</th>
      <th>vector_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>https://simple.wikipedia.org/wiki/April</td>
      <td>April</td>
      <td>April is the fourth month of the year in the J...</td>
      <td>[0.001009464613161981, -0.020700545981526375, ...</td>
      <td>[-0.011253940872848034, -0.013491976074874401,...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>https://simple.wikipedia.org/wiki/August</td>
      <td>August</td>
      <td>August (Aug.)


key: _node_content
string_value:
{"id_": "ca12d2e1-bed7-4211-a611-5f88d6981afe", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2bc858dc-ca27-4f5f-9212-33eb88bed6bb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4d05c5c7f2f4fadc6bfa392d8e54521303b098b379ac26589d42375d374c1ffa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f423d2b9-7012-473b-800a-4c7e77ddf29a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "93359a813944fe8ab2b2a5830ed879e30891152eac81e2636a36bc34b5a0d280", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2c80ce7f-1c4c-4094-becb-385283f724c6", "node_type": "1", "metadata": {}, "hash": "0eb9f1fbcad7c87e0ae1cc80d55f22c888212d66f0901dbcd88725f848899c3c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3497, "end_char_idx": 5675, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: _node_content
string_value:
{"id_": "2c80ce7f-1c4c-4094-becb-385283f724c6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2bc858dc-ca27-4f5f-9212-33eb88bed6bb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4d05c5c7f2f4fadc6bfa392d8e54521303b098b379ac26589d42375d374c1ffa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ca12d2e1-bed7-4211-a611-5f88d6981afe", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "fb9c864aac39166d421671947475e435aea0945061c07fa2e8152eddffe8e64d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "16f3a5d3-79b7-455b-8814-1142df3ba34d", "node_type": "1", "metadata": {}, "hash": "3f7b4d59d71a4aeb15f37ff473c5aaf54e96b32c0390919255591521f059a058", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4645, "end_char_idx": 6843, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: _node_content
string_value:
{"id_": "16f3a5d3-79b7-455b-8814-1142df3ba34d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2bc858dc-ca27-4f5f-9212-33eb88bed6bb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4d05c5c7f2f4fadc6bfa392d8e54521303b098b379ac26589d42375d374c1ffa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2c80ce7f-1c4c-4094-becb-385283f724c6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "64d84a49e2f13f7f267cb3e70af9d4e8aee0e5c80b952d27fd973cbaaa6afdaf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8ea7ed92-278e-4a9f-9aee-fcc35050f547", "node_type": "1", "metadata": {}, "hash": "084fafb325520eb33296139dcc48c0723a8178600ea83911ab4b1b154ceeaa1f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5676, "end_char_idx": 8300, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Air is a...</td>
      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>
      <td>[0.021524671465158463, 0.018522677943110466, -...</td>
      <td>4</td>
    </tr>
  </tbody>
</table>
</div>




```python
# Read vectors from strings back into a list
article_df['title_vector'] = article_df.title_vector.apply(literal_eval)
article_df['content_vector'] = article_df.content_vector.apply(literal_eval)

# Set vector_id to be a string
article_df['vector_id'] = article_df['vector_id'].apply(str)
```


```python
article_df.info(show_counts=True)
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 25000 entries, 0 to 24999
    Data columns (total 7 columns):
     #   Column          Non-Null Count  Dtype 
    ---  ------          --------------  ----- 
     0   id              25000 non-null  int64 
     1   url             25000 non-null  object
     2   title           25000 non-null  object
     3   text            25000 non-null  object
     4   title_vector    25000 non-null  object
     5   content_vector  25000 non-null  object
     6   vector_id       25000 non-null  object
    dtypes: int64(1), object(6)
    memory usage: 1.3+ MB


## Typesense

The next vector store we'll look at is [Typesense](https://typesense.org/), which is an open source, in-memory search engine, that you can either self-host or run on [Typesense Cloud](https://cloud.typesense.org).

Typesense focuses on performance by storing the entire index in RAM (with a backup on disk) and also focuses on providing an out-of-the-box developer experience by simplifying available options and setting good defaults. It also lets you combine attribute-based filtering together with vector queries.

For this example, we will set up a local docker-based Typesense server, index our vectors in Typesense and then do some nearest-neighbor search queries. If you use Typesense Cloud, you can skip the docker setup part and just obtain the hostname and API keys from your cluster dashboard.

### Setup

To run Typesense locally, you'll need [Docker](https://www.docker.com/). Following the instructions contained in the Typesense documentation [here](https://typesense.org/docs/guide/install-typesense.html#docker-compose), we created an example docker-compose.yml file in this repo saved at [./typesense/docker-compose.yml](./typesense/docker-compose.yml).

After starting Docker, you can start Typesense locally by navigating to the `examples/vector_databases/typesense/` directory and running `docker-compose up -d`.

The default API key is set to `xyz` in the Docker compose file, and the default Typesense port to `8108`.


key: _node_content
string_value:
{"id_": "8ea7ed92-278e-4a9f-9aee-fcc35050f547", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2bc858dc-ca27-4f5f-9212-33eb88bed6bb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4d05c5c7f2f4fadc6bfa392d8e54521303b098b379ac26589d42375d374c1ffa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "16f3a5d3-79b7-455b-8814-1142df3ba34d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "a2094d5ecf336e6be2c4d4aa071264aa68a624e2aa14c47d152a9496d42c1181", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e1ac8ba4-924e-4d22-9e51-6c372297093d", "node_type": "1", "metadata": {}, "hash": "80fd9de12e06d8a76d5f1cc38bf576e9a5fb1604235eef258f985d538f3ec540", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6846, "end_char_idx": 8938, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Typesense

The next vector store we'll look at is [Typesense](https://typesense.org/), which is an open source, in-memory search engine, that you can either self-host or run on [Typesense Cloud](https://cloud.typesense.org).

Typesense focuses on performance by storing the entire index in RAM (with a backup on disk) and also focuses on providing an out-of-the-box developer experience by simplifying available options and setting good defaults. It also lets you combine attribute-based filtering together with vector queries.

For this example, we will set up a local docker-based Typesense server, index our vectors in Typesense and then do some nearest-neighbor search queries. If you use Typesense Cloud, you can skip the docker setup part and just obtain the hostname and API keys from your cluster dashboard.

### Setup

To run Typesense locally, you'll need [Docker](https://www.docker.com/). Following the instructions contained in the Typesense documentation [here](https://typesense.org/docs/guide/install-typesense.html#docker-compose), we created an example docker-compose.yml file in this repo saved at [./typesense/docker-compose.yml](./typesense/docker-compose.yml).

After starting Docker, you can start Typesense locally by navigating to the `examples/vector_databases/typesense/` directory and running `docker-compose up -d`.

The default API key is set to `xyz` in the Docker compose file, and the default Typesense port to `8108`.


```python
import typesense

typesense_client = \
    typesense.Client({
        "nodes": [{
            "host": "localhost",  # For Typesense Cloud use xxx.a1.typesense.net
            "port": "8108",       # For Typesense Cloud use 443
            "protocol": "http"    # For Typesense Cloud use https
          }],
          "api_key": "xyz",
          "connection_timeout_seconds": 60
        })
```

### Index data

To index vectors in Typesense, we'll first create a Collection (which is a collection of Documents) and turn on vector indexing for a particular field. You can even store multiple vector fields in a single document.


key: _node_content
string_value:
{"id_": "e1ac8ba4-924e-4d22-9e51-6c372297093d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2bc858dc-ca27-4f5f-9212-33eb88bed6bb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4d05c5c7f2f4fadc6bfa392d8e54521303b098b379ac26589d42375d374c1ffa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8ea7ed92-278e-4a9f-9aee-fcc35050f547", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "d884a3fafd04c95bbc808f1d0bff31de0676db23c5727d2fd227d93d61d342fe", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "29d31499-ddbb-424e-98af-9a7bb3c53b4f", "node_type": "1", "metadata": {}, "hash": "0440dbd67a67af32b5a21cba9b4f772e822c871315ff337767aee466213ca7bd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import typesense

typesense_client = \
    typesense.Client({
        "nodes": [{
            "host": "localhost",  # For Typesense Cloud use xxx.a1.typesense.net
            "port": "8108",       # For Typesense Cloud use 443
            "protocol": "http"    # For Typesense Cloud use https
          }],
          "api_key": "xyz",
          "connection_timeout_seconds": 60
        })
```

### Index data

To index vectors in Typesense, we'll first create a Collection (which is a collection of Documents) and turn on vector indexing for a particular field. You can even store multiple vector fields in a single document.


```python
# Delete existing collections if they already exist
try:
    typesense_client.collections['wikipedia_articles'].delete()
except Exception as e:
    pass

# Create a new collection

schema = {
    "name": "wikipedia_articles",
    "fields": [
        {
            "name": "content_vector",
            "type": "float[]",
            "num_dim": len(article_df['content_vector'][0])
        },
        {
            "name": "title_vector",
            "type": "float[]",
            "num_dim": len(article_df['title_vector'][0])
        }
    ]
}

create_response = typesense_client.collections.create(schema)
print(create_response)

print("Created new collection wikipedia-articles")
```

    {'created_at': 1687165065, 'default_sorting_field': '', 'enable_nested_fields': False, 'fields': [{'facet': False, 'index': True, 'infix': False, 'locale': '', 'name': 'content_vector', 'num_dim': 1536, 'optional': False, 'sort': False, 'type': 'float[]'}, {'facet': False, 'index': True, 'infix': False, 'locale': '', 'name': 'title_vector', 'num_dim': 1536, 'optional': False, 'sort': False, 'type': 'float[]'}], 'name': 'wikipedia_articles', 'num_documents': 0, 'symbols_to_index': [], 'token_separators': []}
    Created new collection wikipedia-articles



```python
# Upsert the vector data into the collection we just created
#
# Note: This can take a few minutes, especially if your on an M1 and running docker in an emulated mode

print("Indexing vectors in Typesense...")

document_counter = 0
documents_batch = []

for k,v in article_df.iterrows():
    # Create a document with the vector data

    # Notice how you can add any fields that you haven't added to the schema to the document.
    # These will be stored on disk and returned when the document is a hit.
    # This is useful to store attributes required for display purposes.

    document = {
        "title_vector": v["title_vector"],
        "content_vector": v["content_vector"],
        "title": v["title"],
        "content": v["text"],
    }
    documents_batch.append(document)
    document_counter = document_counter + 1

    # Upsert a batch of 100 documents
    if document_counter % 100 == 0 or document_counter == len(article_df):
        response = typesense_client.collections['wikipedia_articles'].documents.import_(documents_batch)
        # print(response)

        documents_batch = []
        print(f"Processed {document_counter} / {len(article_df)} ")

print(f"Imported ({len(article_df)}) articles.")
```

    Indexing vectors in Typesense.
    Processed 100 /


key: _node_content
string_value:
{"id_": "29d31499-ddbb-424e-98af-9a7bb3c53b4f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2bc858dc-ca27-4f5f-9212-33eb88bed6bb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4d05c5c7f2f4fadc6bfa392d8e54521303b098b379ac26589d42375d374c1ffa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e1ac8ba4-924e-4d22-9e51-6c372297093d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "5e139b6e1d1b0631f2042cb9b059200850639f6a20277c6ae88a83e049c87b68", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bc65b5b2-619a-4ca2-a64c-6e441f54d680", "node_type": "1", "metadata": {}, "hash": "e740111a14d7ed1b2bcdfee2f41000d2cc098ccacef9758058b5b4c84f1a129e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Upsert the vector data into the collection we just created
#
# Note: This can take a few minutes, especially if your on an M1 and running docker in an emulated mode

print("Indexing vectors in Typesense...")

document_counter = 0
documents_batch = []

for k,v in article_df.iterrows():
    # Create a document with the vector data

    # Notice how you can add any fields that you haven't added to the schema to the document.
    # These will be stored on disk and returned when the document is a hit.
    # This is useful to store attributes required for display purposes.

    document = {
        "title_vector": v["title_vector"],
        "content_vector": v["content_vector"],
        "title": v["title"],
        "content": v["text"],
    }
    documents_batch.append(document)
    document_counter = document_counter + 1

    # Upsert a batch of 100 documents
    if document_counter % 100 == 0 or document_counter == len(article_df):
        response = typesense_client.collections['wikipedia_articles'].documents.import_(documents_batch)
        # print(response)

        documents_batch = []
        print(f"Processed {document_counter} / {len(article_df)} ")

print(f"Imported ({len(article_df)}) articles.")
```

    Indexing vectors in Typesense.
    Processed 100 / 25000 
    Processed 200 / 25000 
    Processed 300 / 25000 
    Processed 400 / 25000 
    Processed 500 / 25000 
    Processed 600 / 25000 
    Processed 700 / 25000 
    Processed 800 / 25000 
    Processed 900 / 25000 
    Processed 1000 / 25000 
    Processed 1100 / 25000 
    Processed 1200 / 25000 
    Processed 1300 / 25000 
    Processed 1400 / 25000 
    Processed 1500 / 25000 
    Processed 1600 / 25000 
    Processed 1700 / 25000 
    Processed 1800 / 25000 
    Processed 1900 / 25000 
    Processed 2000 / 25000 
    Processed 2100 / 25000 
    Processed 2200 / 25000 
    Processed 2300 / 25000 
    Processed 2400 / 25000 
    Processed 2500 / 25000 
    Processed 2600 / 25000 
    Processed 2700 / 25000 
    Processed 2800 / 25000 
    Processed 2900 / 25000 
    Processed 3000 / 25000 
    Processed 3100 / 25000 
    Processed 3200 / 25000 
    Processed 3300 / 25000 
    Processed 3400 / 25000 
    Processed 3500 / 25000 
    Processed 3600 / 25000 
    Processed 3700 / 25000 
    Processed 3800


key: _node_content
string_value:
{"id_": "bc65b5b2-619a-4ca2-a64c-6e441f54d680", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2bc858dc-ca27-4f5f-9212-33eb88bed6bb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4d05c5c7f2f4fadc6bfa392d8e54521303b098b379ac26589d42375d374c1ffa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "29d31499-ddbb-424e-98af-9a7bb3c53b4f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "9ce17cbe20915c185c16082e026e37190b668d45a4604195d43d459df7561922", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f8adcd55-933d-4425-95cb-3f32262e3810", "node_type": "1", "metadata": {}, "hash": "a58bfc04366523ea0a6a10ba35571f4d56ffa547d9fd796e19d8980dfb948ee3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11662, "end_char_idx": 13317, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
25000 
    Processed 800 / 25000 
    Processed 900 / 25000 
    Processed 1000 / 25000 
    Processed 1100 / 25000 
    Processed 1200 / 25000 
    Processed 1300 / 25000 
    Processed 1400 / 25000 
    Processed 1500 / 25000 
    Processed 1600 / 25000 
    Processed 1700 / 25000 
    Processed 1800 / 25000 
    Processed 1900 / 25000 
    Processed 2000 / 25000 
    Processed 2100 / 25000 
    Processed 2200 / 25000 
    Processed 2300 / 25000 
    Processed 2400 / 25000 
    Processed 2500 / 25000 
    Processed 2600 / 25000 
    Processed 2700 / 25000 
    Processed 2800 / 25000 
    Processed 2900 / 25000 
    Processed 3000 / 25000 
    Processed 3100 / 25000 
    Processed 3200 / 25000 
    Processed 3300 / 25000 
    Processed 3400 / 25000 
    Processed 3500 / 25000 
    Processed 3600 / 25000 
    Processed 3700 / 25000 
    Processed 3800 / 25000 
    Processed 3900 / 25000 
    Processed 4000 / 25000 
    Processed 4100 / 25000 
    Processed 4200 / 25000 
    Processed 4300 / 25000 
    Processed 4400 / 25000 
    Processed 4500 / 25000 
    Processed 4600 / 25000 
    Processed 4700 / 25000 
    Processed 4800 / 25000 
    Processed 4900 / 25000 
    Processed 5000 / 25000 
    Processed 5100 / 25000 
    Processed 5200 / 25000 
    Processed 5300 / 25000 
    Processed 5400 / 25000 
    Processed 5500 / 25000 
    Processed 5600 / 25000 
    Processed 5700 / 25000 
    Processed 5800 / 25000 
    Processed 5900 / 25000 
    Processed 6000 / 25000 
    Processed 6100 / 25000 
    Processed 6200 / 25000 
    Processed 6300 / 25000 
    Processed 6400 / 25000 
    Processed 6500 / 25000 
    Processed 6600 / 25000


key: _node_content
string_value:
{"id_": "f8adcd55-933d-4425-95cb-3f32262e3810", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2bc858dc-ca27-4f5f-9212-33eb88bed6bb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4d05c5c7f2f4fadc6bfa392d8e54521303b098b379ac26589d42375d374c1ffa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bc65b5b2-619a-4ca2-a64c-6e441f54d680", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "43b5f699263b0307e8663a3454748069fb6f663757800a2e6b5332587205b084", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1b557643-5c91-470d-87de-5db474e959ff", "node_type": "1", "metadata": {}, "hash": "df9720db75276e6165c6e3b2dc655e31019e4ae142c7fab10afb43eebf320983", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12455, "end_char_idx": 14101, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Processed 3600 / 25000 
    Processed 3700 / 25000 
    Processed 3800 / 25000 
    Processed 3900 / 25000 
    Processed 4000 / 25000 
    Processed 4100 / 25000 
    Processed 4200 / 25000 
    Processed 4300 / 25000 
    Processed 4400 / 25000 
    Processed 4500 / 25000 
    Processed 4600 / 25000 
    Processed 4700 / 25000 
    Processed 4800 / 25000 
    Processed 4900 / 25000 
    Processed 5000 / 25000 
    Processed 5100 / 25000 
    Processed 5200 / 25000 
    Processed 5300 / 25000 
    Processed 5400 / 25000 
    Processed 5500 / 25000 
    Processed 5600 / 25000 
    Processed 5700 / 25000 
    Processed 5800 / 25000 
    Processed 5900 / 25000 
    Processed 6000 / 25000 
    Processed 6100 / 25000 
    Processed 6200 / 25000 
    Processed 6300 / 25000 
    Processed 6400 / 25000 
    Processed 6500 / 25000 
    Processed 6600 / 25000 
    Processed 6700 / 25000 
    Processed 6800 / 25000 
    Processed 6900 / 25000 
    Processed 7000 / 25000 
    Processed 7100 / 25000 
    Processed 7200 / 25000 
    Processed 7300 / 25000 
    Processed 7400 / 25000 
    Processed 7500 / 25000 
    Processed 7600 / 25000 
    Processed 7700 / 25000 
    Processed 7800 / 25000 
    Processed 7900 / 25000 
    Processed 8000 / 25000 
    Processed 8100 / 25000 
    Processed 8200 / 25000 
    Processed 8300 / 25000 
    Processed 8400 / 25000 
    Processed 8500 / 25000 
    Processed 8600 / 25000 
    Processed 8700 / 25000 
    Processed 8800 / 25000 
    Processed 8900 / 25000 
    Processed 9000 / 25000 
    Processed 9100 / 25000 
    Processed 9200 / 25000 
    Processed 9300 / 25000 
    Processed 9400 / 25000


key: _node_content
string_value:
{"id_": "1b557643-5c91-470d-87de-5db474e959ff", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2bc858dc-ca27-4f5f-9212-33eb88bed6bb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4d05c5c7f2f4fadc6bfa392d8e54521303b098b379ac26589d42375d374c1ffa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f8adcd55-933d-4425-95cb-3f32262e3810", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "bbea561a3816d32d95f9ff55f62780c315c46d69f83b5f347506f114f3b4921a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "550a202d-b588-4b24-be6c-6ad0decf16dd", "node_type": "1", "metadata": {}, "hash": "7ec2973f64034579cebccea6270625b62f239a0041b894b76a87b39fc847afec", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13254, "end_char_idx": 14931, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
/ 25000 
    Processed 6500 / 25000 
    Processed 6600 / 25000 
    Processed 6700 / 25000 
    Processed 6800 / 25000 
    Processed 6900 / 25000 
    Processed 7000 / 25000 
    Processed 7100 / 25000 
    Processed 7200 / 25000 
    Processed 7300 / 25000 
    Processed 7400 / 25000 
    Processed 7500 / 25000 
    Processed 7600 / 25000 
    Processed 7700 / 25000 
    Processed 7800 / 25000 
    Processed 7900 / 25000 
    Processed 8000 / 25000 
    Processed 8100 / 25000 
    Processed 8200 / 25000 
    Processed 8300 / 25000 
    Processed 8400 / 25000 
    Processed 8500 / 25000 
    Processed 8600 / 25000 
    Processed 8700 / 25000 
    Processed 8800 / 25000 
    Processed 8900 / 25000 
    Processed 9000 / 25000 
    Processed 9100 / 25000 
    Processed 9200 / 25000 
    Processed 9300 / 25000 
    Processed 9400 / 25000 
    Processed 9500 / 25000 
    Processed 9600 / 25000 
    Processed 9700 / 25000 
    Processed 9800 / 25000 
    Processed 9900 / 25000 
    Processed 10000 / 25000 
    Processed 10100 / 25000 
    Processed 10200 / 25000 
    Processed 10300 / 25000 
    Processed 10400 / 25000 
    Processed 10500 / 25000 
    Processed 10600 / 25000 
    Processed 10700 / 25000 
    Processed 10800 / 25000 
    Processed 10900 / 25000 
    Processed 11000 / 25000 
    Processed 11100 / 25000 
    Processed 11200 / 25000 
    Processed 11300 / 25000 
    Processed 11400 / 25000 
    Processed 11500 / 25000 
    Processed 11600 / 25000 
    Processed 11700 / 25000 
    Processed 11800 / 25000 
    Processed 11900 / 25000 
    Processed 12000 / 25000 
    Processed 12100 / 25000 
    Processed 12200 / 25000 
    Processed 12300 /


key: _node_content
string_value:
{"id_": "550a202d-b588-4b24-be6c-6ad0decf16dd", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2bc858dc-ca27-4f5f-9212-33eb88bed6bb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4d05c5c7f2f4fadc6bfa392d8e54521303b098b379ac26589d42375d374c1ffa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1b557643-5c91-470d-87de-5db474e959ff", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "f9105d4c97adb471b431e74fcf20d561a795074bc2f449a7a256e9dcce98abba", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "55200db9-7afa-4c7a-9594-1959109a04ae", "node_type": "1", "metadata": {}, "hash": "c9c38900b3c582612849178156fa59e7f64e37aee204b59154ccea47d86e8556", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14051, "end_char_idx": 15749, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Processed 9300 / 25000 
    Processed 9400 / 25000 
    Processed 9500 / 25000 
    Processed 9600 / 25000 
    Processed 9700 / 25000 
    Processed 9800 / 25000 
    Processed 9900 / 25000 
    Processed 10000 / 25000 
    Processed 10100 / 25000 
    Processed 10200 / 25000 
    Processed 10300 / 25000 
    Processed 10400 / 25000 
    Processed 10500 / 25000 
    Processed 10600 / 25000 
    Processed 10700 / 25000 
    Processed 10800 / 25000 
    Processed 10900 / 25000 
    Processed 11000 / 25000 
    Processed 11100 / 25000 
    Processed 11200 / 25000 
    Processed 11300 / 25000 
    Processed 11400 / 25000 
    Processed 11500 / 25000 
    Processed 11600 / 25000 
    Processed 11700 / 25000 
    Processed 11800 / 25000 
    Processed 11900 / 25000 
    Processed 12000 / 25000 
    Processed 12100 / 25000 
    Processed 12200 / 25000 
    Processed 12300 / 25000 
    Processed 12400 / 25000 
    Processed 12500 / 25000 
    Processed 12600 / 25000 
    Processed 12700 / 25000 
    Processed 12800 / 25000 
    Processed 12900 / 25000 
    Processed 13000 / 25000 
    Processed 13100 / 25000 
    Processed 13200 / 25000 
    Processed 13300 / 25000 
    Processed 13400 / 25000 
    Processed 13500 / 25000 
    Processed 13600 / 25000 
    Processed 13700 / 25000 
    Processed 13800 / 25000 
    Processed 13900 / 25000 
    Processed 14000 / 25000 
    Processed 14100 / 25000 
    Processed 14200 / 25000 
    Processed 14300 / 25000 
    Processed 14400 / 25000 
    Processed 14500 / 25000 
    Processed 14600 / 25000 
    Processed 14700 / 25000 
    Processed 14800 / 25000 
    Processed 14900 / 25000 
    Processed 15000 / 25000 
    Processed 15100 / 25000


key: _node_content
string_value:
{"id_": "55200db9-7afa-4c7a-9594-1959109a04ae", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2bc858dc-ca27-4f5f-9212-33eb88bed6bb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4d05c5c7f2f4fadc6bfa392d8e54521303b098b379ac26589d42375d374c1ffa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "550a202d-b588-4b24-be6c-6ad0decf16dd", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "f246c362cb6bfa49bafef649be5cc6c050204173576c7293c2b4266891917b43", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "209c8707-0a3d-40ea-92f3-09fa3be36202", "node_type": "1", "metadata": {}, "hash": "545d1e91400799db133de10c1f6050cfdbfb7962dfa8ca4090dd406116618b5d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14856, "end_char_idx": 16561, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Processed 12100 / 25000 
    Processed 12200 / 25000 
    Processed 12300 / 25000 
    Processed 12400 / 25000 
    Processed 12500 / 25000 
    Processed 12600 / 25000 
    Processed 12700 / 25000 
    Processed 12800 / 25000 
    Processed 12900 / 25000 
    Processed 13000 / 25000 
    Processed 13100 / 25000 
    Processed 13200 / 25000 
    Processed 13300 / 25000 
    Processed 13400 / 25000 
    Processed 13500 / 25000 
    Processed 13600 / 25000 
    Processed 13700 / 25000 
    Processed 13800 / 25000 
    Processed 13900 / 25000 
    Processed 14000 / 25000 
    Processed 14100 / 25000 
    Processed 14200 / 25000 
    Processed 14300 / 25000 
    Processed 14400 / 25000 
    Processed 14500 / 25000 
    Processed 14600 / 25000 
    Processed 14700 / 25000 
    Processed 14800 / 25000 
    Processed 14900 / 25000 
    Processed 15000 / 25000 
    Processed 15100 / 25000 
    Processed 15200 / 25000 
    Processed 15300 / 25000 
    Processed 15400 / 25000 
    Processed 15500 / 25000 
    Processed 15600 / 25000 
    Processed 15700 / 25000 
    Processed 15800 / 25000 
    Processed 15900 / 25000 
    Processed 16000 / 25000 
    Processed 16100 / 25000 
    Processed 16200 / 25000 
    Processed 16300 / 25000 
    Processed 16400 / 25000 
    Processed 16500 / 25000 
    Processed 16600 / 25000 
    Processed 16700 / 25000 
    Processed 16800 / 25000 
    Processed 16900 / 25000 
    Processed 17000 / 25000 
    Processed 17100 / 25000 
    Processed 17200 / 25000 
    Processed 17300 / 25000 
    Processed 17400 / 25000 
    Processed 17500 / 25000 
    Processed 17600 / 25000 
    Processed 17700 / 25000 
    Processed 17800 / 25000 
    Processed 17900 / 25000


key: _node_content
string_value:
{"id_": "209c8707-0a3d-40ea-92f3-09fa3be36202", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2bc858dc-ca27-4f5f-9212-33eb88bed6bb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4d05c5c7f2f4fadc6bfa392d8e54521303b098b379ac26589d42375d374c1ffa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "55200db9-7afa-4c7a-9594-1959109a04ae", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "67d8c6d3e45c17b76b3c12fec017a25c6c1e795f9a3ac8f475761da8c48c5585", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ef0d1fb3-b1e7-4bff-b645-41d79e4021f7", "node_type": "1", "metadata": {}, "hash": "509bdeeb6a8ae6bbb811ab366799b3bfbe244e264bc29cbc96b3ac71a7e07667", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15684, "end_char_idx": 17396, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
/ 25000 
    Processed 15000 / 25000 
    Processed 15100 / 25000 
    Processed 15200 / 25000 
    Processed 15300 / 25000 
    Processed 15400 / 25000 
    Processed 15500 / 25000 
    Processed 15600 / 25000 
    Processed 15700 / 25000 
    Processed 15800 / 25000 
    Processed 15900 / 25000 
    Processed 16000 / 25000 
    Processed 16100 / 25000 
    Processed 16200 / 25000 
    Processed 16300 / 25000 
    Processed 16400 / 25000 
    Processed 16500 / 25000 
    Processed 16600 / 25000 
    Processed 16700 / 25000 
    Processed 16800 / 25000 
    Processed 16900 / 25000 
    Processed 17000 / 25000 
    Processed 17100 / 25000 
    Processed 17200 / 25000 
    Processed 17300 / 25000 
    Processed 17400 / 25000 
    Processed 17500 / 25000 
    Processed 17600 / 25000 
    Processed 17700 / 25000 
    Processed 17800 / 25000 
    Processed 17900 / 25000 
    Processed 18000 / 25000 
    Processed 18100 / 25000 
    Processed 18200 / 25000 
    Processed 18300 / 25000 
    Processed 18400 / 25000 
    Processed 18500 / 25000 
    Processed 18600 / 25000 
    Processed 18700 / 25000 
    Processed 18800 / 25000 
    Processed 18900 / 25000 
    Processed 19000 / 25000 
    Processed 19100 / 25000 
    Processed 19200 / 25000 
    Processed 19300 / 25000 
    Processed 19400 / 25000 
    Processed 19500 / 25000 
    Processed 19600 / 25000 
    Processed 19700 / 25000 
    Processed 19800 / 25000 
    Processed 19900 / 25000 
    Processed 20000 / 25000 
    Processed 20100 / 25000 
    Processed 20200 / 25000 
    Processed 20300 / 25000 
    Processed 20400 / 25000 
    Processed 20500 / 25000 
    Processed 20600 / 25000 
    Processed 20700 / 25000 
    Processed 20800 /


key: _node_content
string_value:
{"id_": "ef0d1fb3-b1e7-4bff-b645-41d79e4021f7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2bc858dc-ca27-4f5f-9212-33eb88bed6bb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4d05c5c7f2f4fadc6bfa392d8e54521303b098b379ac26589d42375d374c1ffa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "209c8707-0a3d-40ea-92f3-09fa3be36202", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "ab1fd97f49bca783790b625e0b48f2e10bbc071312072c4b2404d4ce618f874c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2116115a-633c-4cfd-9797-248b5ee61b45", "node_type": "1", "metadata": {}, "hash": "4525b48a91c24c452758aa3fac7af83d08bd6a505c16513411d85cff30147fcb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16509, "end_char_idx": 18214, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Processed 17800 / 25000 
    Processed 17900 / 25000 
    Processed 18000 / 25000 
    Processed 18100 / 25000 
    Processed 18200 / 25000 
    Processed 18300 / 25000 
    Processed 18400 / 25000 
    Processed 18500 / 25000 
    Processed 18600 / 25000 
    Processed 18700 / 25000 
    Processed 18800 / 25000 
    Processed 18900 / 25000 
    Processed 19000 / 25000 
    Processed 19100 / 25000 
    Processed 19200 / 25000 
    Processed 19300 / 25000 
    Processed 19400 / 25000 
    Processed 19500 / 25000 
    Processed 19600 / 25000 
    Processed 19700 / 25000 
    Processed 19800 / 25000 
    Processed 19900 / 25000 
    Processed 20000 / 25000 
    Processed 20100 / 25000 
    Processed 20200 / 25000 
    Processed 20300 / 25000 
    Processed 20400 / 25000 
    Processed 20500 / 25000 
    Processed 20600 / 25000 
    Processed 20700 / 25000 
    Processed 20800 / 25000 
    Processed 20900 / 25000 
    Processed 21000 / 25000 
    Processed 21100 / 25000 
    Processed 21200 / 25000 
    Processed 21300 / 25000 
    Processed 21400 / 25000 
    Processed 21500 / 25000 
    Processed 21600 / 25000 
    Processed 21700 / 25000 
    Processed 21800 / 25000 
    Processed 21900 / 25000 
    Processed 22000 / 25000 
    Processed 22100 / 25000 
    Processed 22200 / 25000 
    Processed 22300 / 25000 
    Processed 22400 / 25000 
    Processed 22500 / 25000 
    Processed 22600 / 25000 
    Processed 22700 / 25000 
    Processed 22800 / 25000 
    Processed 22900 / 25000 
    Processed 23000 / 25000 
    Processed 23100 / 25000 
    Processed 23200 / 25000 
    Processed 23300 / 25000 
    Processed 23400 / 25000 
    Processed 23500 / 25000 
    Processed 23600 / 25000


key: _node_content
string_value:
{"id_": "2116115a-633c-4cfd-9797-248b5ee61b45", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2bc858dc-ca27-4f5f-9212-33eb88bed6bb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4d05c5c7f2f4fadc6bfa392d8e54521303b098b379ac26589d42375d374c1ffa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ef0d1fb3-b1e7-4bff-b645-41d79e4021f7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "494af7a51db8f7fd14d68f0c4bd6479e11a8be3122ed042fb5fe502e882fdec9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f62b038f-216e-4591-9178-35324e8a5178", "node_type": "1", "metadata": {}, "hash": "a176813de6b675a350625d89711fa62b709a4be57050b2eb9023f7249b513b66", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17321, "end_char_idx": 19043, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Processed 20600 / 25000 
    Processed 20700 / 25000 
    Processed 20800 / 25000 
    Processed 20900 / 25000 
    Processed 21000 / 25000 
    Processed 21100 / 25000 
    Processed 21200 / 25000 
    Processed 21300 / 25000 
    Processed 21400 / 25000 
    Processed 21500 / 25000 
    Processed 21600 / 25000 
    Processed 21700 / 25000 
    Processed 21800 / 25000 
    Processed 21900 / 25000 
    Processed 22000 / 25000 
    Processed 22100 / 25000 
    Processed 22200 / 25000 
    Processed 22300 / 25000 
    Processed 22400 / 25000 
    Processed 22500 / 25000 
    Processed 22600 / 25000 
    Processed 22700 / 25000 
    Processed 22800 / 25000 
    Processed 22900 / 25000 
    Processed 23000 / 25000 
    Processed 23100 / 25000 
    Processed 23200 / 25000 
    Processed 23300 / 25000 
    Processed 23400 / 25000 
    Processed 23500 / 25000 
    Processed 23600 / 25000 
    Processed 23700 / 25000 
    Processed 23800 / 25000 
    Processed 23900 / 25000 
    Processed 24000 / 25000 
    Processed 24100 / 25000 
    Processed 24200 / 25000 
    Processed 24300 / 25000 
    Processed 24400 / 25000 
    Processed 24500 / 25000 
    Processed 24600 / 25000 
    Processed 24700 / 25000 
    Processed 24800 / 25000 
    Processed 24900 / 25000 
    Processed 25000 / 25000 
    Imported (25000) articles.



```python
# Check the number of documents imported

collection = typesense_client.collections['wikipedia_articles'].retrieve()
print(f'Collection has {collection["num_documents"]} documents')
```

    Collection has 25000 documents


### Search Data

Now that we've imported the vectors into Typesense, we can do a nearest neighbor search on the `title_vector` or `content_vector` field.


key: _node_content
string_value:
{"id_": "f62b038f-216e-4591-9178-35324e8a5178", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2bc858dc-ca27-4f5f-9212-33eb88bed6bb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4d05c5c7f2f4fadc6bfa392d8e54521303b098b379ac26589d42375d374c1ffa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2116115a-633c-4cfd-9797-248b5ee61b45", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "a9f4f6ebaf2eaf2e07e044b35aa4465f6fb44897e60863151ccffb4b3473aefc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2eb5b3f2-239e-4c9f-bb0b-05a2d75b5acd", "node_type": "1", "metadata": {}, "hash": "f01b0cb75b86adc6763a042d104eda364466e791c8d4ac51908a45ebc5cea659", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17940, "end_char_idx": 19698, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
22700 / 25000 
    Processed 22800 / 25000 
    Processed 22900 / 25000 
    Processed 23000 / 25000 
    Processed 23100 / 25000 
    Processed 23200 / 25000 
    Processed 23300 / 25000 
    Processed 23400 / 25000 
    Processed 23500 / 25000 
    Processed 23600 / 25000 
    Processed 23700 / 25000 
    Processed 23800 / 25000 
    Processed 23900 / 25000 
    Processed 24000 / 25000 
    Processed 24100 / 25000 
    Processed 24200 / 25000 
    Processed 24300 / 25000 
    Processed 24400 / 25000 
    Processed 24500 / 25000 
    Processed 24600 / 25000 
    Processed 24700 / 25000 
    Processed 24800 / 25000 
    Processed 24900 / 25000 
    Processed 25000 / 25000 
    Imported (25000) articles.



```python
# Check the number of documents imported

collection = typesense_client.collections['wikipedia_articles'].retrieve()
print(f'Collection has {collection["num_documents"]} documents')
```

    Collection has 25000 documents


### Search Data

Now that we've imported the vectors into Typesense, we can do a nearest neighbor search on the `title_vector` or `content_vector` field.


```python
def query_typesense(query, field='title', top_k=20):

    # Creates embedding vector from user query
    openai.api_key = os.getenv("OPENAI_API_KEY", "sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")
    embedded_query = openai.Embedding.create(
        input=query,
        model=EMBEDDING_MODEL,
    )['data'][0]['embedding']

    typesense_results = typesense_client.multi_search.perform({
        "searches": [{
            "q": "*",
            "collection": "wikipedia_articles",
            "vector_query": f"{field}_vector:([{','.join(str(v) for v in embedded_query)}], k:{top_k})"
        }]
    }, {})

    return typesense_results
```


key: _node_content
string_value:
{"id_": "2eb5b3f2-239e-4c9f-bb0b-05a2d75b5acd", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2bc858dc-ca27-4f5f-9212-33eb88bed6bb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4d05c5c7f2f4fadc6bfa392d8e54521303b098b379ac26589d42375d374c1ffa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f62b038f-216e-4591-9178-35324e8a5178", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "2e8dea233b66b2e06b97675d4f9caede84a7989394ec6674721eba609c86fdc3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0e5765e7-47ba-4d05-a191-1b44b9218b37", "node_type": "1", "metadata": {}, "hash": "214a5e4f55e039124db871c7a2355222d0b9e74e281662c658e97e9d8b6f1424", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18294, "end_char_idx": 20525, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
/ 25000 
    Processed 24000 / 25000 
    Processed 24100 / 25000 
    Processed 24200 / 25000 
    Processed 24300 / 25000 
    Processed 24400 / 25000 
    Processed 24500 / 25000 
    Processed 24600 / 25000 
    Processed 24700 / 25000 
    Processed 24800 / 25000 
    Processed 24900 / 25000 
    Processed 25000 / 25000 
    Imported (25000) articles.



```python
# Check the number of documents imported

collection = typesense_client.collections['wikipedia_articles'].retrieve()
print(f'Collection has {collection["num_documents"]} documents')
```

    Collection has 25000 documents


### Search Data

Now that we've imported the vectors into Typesense, we can do a nearest neighbor search on the `title_vector` or `content_vector` field.


```python
def query_typesense(query, field='title', top_k=20):

    # Creates embedding vector from user query
    openai.api_key = os.getenv("OPENAI_API_KEY", "sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx")
    embedded_query = openai.Embedding.create(
        input=query,
        model=EMBEDDING_MODEL,
    )['data'][0]['embedding']

    typesense_results = typesense_client.multi_search.perform({
        "searches": [{
            "q": "*",
            "collection": "wikipedia_articles",
            "vector_query": f"{field}_vector:([{','.join(str(v) for v in embedded_query)}], k:{top_k})"
        }]
    }, {})

    return typesense_results
```


```python
query_results = query_typesense('modern art in Europe', 'title')

for i, hit in enumerate(query_results['results'][0]['hits']):
    document = hit["document"]
    vector_distance = hit["vector_distance"]
    print(f'{i + 1}. {document["title"]} (Distance: {vector_distance})')
```

    1. Museum of Modern Art (Distance: 0.12482291460037231)
    2. Western Europe (Distance: 0.13255876302719116)
    3. Renaissance art (Distance: 0.13584274053573608)
    4. Pop art (Distance: 0.1396539807319641)
    5. Northern Europe (Distance: 0.14534103870391846)
    6. Hellenistic art (Distance: 0.1472070813179016)
    7. Modernist literature (Distance: 0.15296930074691772)
    8. Art film (Distance: 0.1567266583442688)
    9. Central Europe (Distance: 0.15741699934005737)
    10. European (Distance: 0.1585891842842102)


key: _node_content
string_value:
{"id_": "0e5765e7-47ba-4d05-a191-1b44b9218b37", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "2bc858dc-ca27-4f5f-9212-33eb88bed6bb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4d05c5c7f2f4fadc6bfa392d8e54521303b098b379ac26589d42375d374c1ffa", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2eb5b3f2-239e-4c9f-bb0b-05a2d75b5acd", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "f09c09cdb1f34127ec00d07ca675e0cf32756959a3fd297ce7f982bedd7054b8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19701, "end_char_idx": 21632, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
query_results = query_typesense('modern art in Europe', 'title')

for i, hit in enumerate(query_results['results'][0]['hits']):
    document = hit["document"]
    vector_distance = hit["vector_distance"]
    print(f'{i + 1}. {document["title"]} (Distance: {vector_distance})')
```

    1. Museum of Modern Art (Distance: 0.12482291460037231)
    2. Western Europe (Distance: 0.13255876302719116)
    3. Renaissance art (Distance: 0.13584274053573608)
    4. Pop art (Distance: 0.1396539807319641)
    5. Northern Europe (Distance: 0.14534103870391846)
    6. Hellenistic art (Distance: 0.1472070813179016)
    7. Modernist literature (Distance: 0.15296930074691772)
    8. Art film (Distance: 0.1567266583442688)
    9. Central Europe (Distance: 0.15741699934005737)
    10. European (Distance: 0.1585891842842102)



```python
query_results = query_typesense('Famous battles in Scottish history', 'content')

for i, hit in enumerate(query_results['results'][0]['hits']):
    document = hit["document"]
    vector_distance = hit["vector_distance"]
    print(f'{i + 1}. {document["title"]} (Distance: {vector_distance})')
```

    1. Battle of Bannockburn (Distance: 0.1306111216545105)
    2. Wars of Scottish Independence (Distance: 0.1384994387626648)
    3. 1651 (Distance: 0.14744246006011963)
    4. First War of Scottish Independence (Distance: 0.15033596754074097)
    5. Robert I of Scotland (Distance: 0.15376019477844238)
    6. 841 (Distance: 0.15609073638916016)
    7. 1716 (Distance: 0.15615153312683105)
    8. 1314 (Distance: 0.16280347108840942)
    9. 1263 (Distance: 0.16361045837402344)
    10. William Wallace (Distance: 0.16464537382125854)


Thanks for following along, you're now equipped to set up your own vector databases and use embeddings to do all kinds of cool things - enjoy! For more complex use cases please continue to work through other cookbook examples in this repo.


```python

```


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/README.ipynb


key: _node_content
string_value:
{"id_": "d18071d9-7db6-45d1-a340-8b25352f7c09", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/README.ipynb", "title": "Typesense", "tokens": 509, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "af0dafdb-85e3-4244-97a8-c56b51f8de95", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/README.ipynb", "title": "Typesense", "tokens": 509, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "6d248d81a8e4ea465a85b980a90aa9afd8b8300e480e74154f4ea49ad809f0d0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2378, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Typesense

Typesense is an open source, in-memory search engine, that you can either [self-host](https://typesense.org/docs/guide/install-typesense.html#option-2-local-machine-self-hosting) or run on [Typesense Cloud](https://cloud.typesense.org/).

## Why Typesense?

Typesense focuses on performance by storing the entire index in RAM (with a backup on disk) and also focuses on providing an out-of-the-box developer experience by simplifying available options and setting good defaults. 

It also lets you combine attribute-based filtering together with vector queries, to fetch the most relevant documents.

### Other features

Besides vector storage and search, Typesense also offers the following features:

- Typo Tolerance: Handles typographical errors elegantly, out-of-the-box.
- Tunable Ranking: Easy to tailor your search results to perfection.
- Sorting: Dynamically sort results based on a particular field at query time (helpful for features like "Sort by Price (asc)").
- Faceting & Filtering: Drill down and refine results.
- Grouping & Distinct: Group similar results together to show more variety.
- Federated Search: Search across multiple collections (indices) in a single HTTP request.
- Scoped API Keys: Generate API keys that only allow access to certain records, for multi-tenant applications.
- Synonyms: Define words as equivalents of each other, so searching for a word will also return results for the synonyms defined.
- Curation & Merchandizing: Boost particular records to a fixed position in the search results, to feature them.
- Raft-based Clustering: Set up a distributed cluster that is highly available.
- Seamless Version Upgrades: As new versions of Typesense come out, upgrading is as simple as swapping out the binary and restarting Typesense.
- No Runtime Dependencies: Typesense is a single binary that you can run locally or in production with a single command.

## How To

- To learn more about how to use Typesense with OpenAI embeddings, see the notebook here for an example: [examples/vector_databases/Using_vector_databases_for_embeddings_search.ipynb](/examples/vector_databases/Using_vector_databases_for_embeddings_search.ipynb)
- To learn more about Typesense's vector search feature, read the docs here: [https://typesense.org/docs/0.24.1/api/vector-search.html](https://typesense.org/docs/0.24.1/api/vector-search.html).


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb


key: _node_content
string_value:
{"id_": "07f4d4db-0aab-4259-a73e-abebd25b3319", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73739acd-3f10-4a5b-8082-8796612ce31b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "843e023154df0d5db2d5a9597b807f14244742f0cf0485a03aad985bf7001ef7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "60d77fe3-92af-448d-86bb-18a939a233b6", "node_type": "1", "metadata": {}, "hash": "b817823958446adb3e1a30a3aa0d120d661aa758da39a3f0de871d6dbcf57a41", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3428, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Azure AI Search as a vector database for OpenAI embeddings

This notebook provides step by step instuctions on using Azure AI Search (f.k.a Azure Cognitive Search) as a vector database with OpenAI embeddings. Azure AI Search is a cloud search service that gives developers infrastructure, APIs, and tools for building a rich search experience over private, heterogeneous content in web, mobile, and enterprise applications.

## Prerequistites:
For the purposes of this exercise you must have the following:
- [Azure AI Search Service](https://learn.microsoft.com/azure/search/)
- [OpenAI Key](https://platform.openai.com/account/api-keys) or [Azure OpenAI credentials](https://learn.microsoft.com/azure/cognitive-services/openai/)


```python
! pip install wget
! pip install azure-search-documents 
! pip install azure-identity
! pip install openai
```

## Import required libraries


```python
import json  
import wget
import pandas as pd
import zipfile
from openai import AzureOpenAI
from azure.identity import DefaultAzureCredential, get_bearer_token_provider
from azure.core.credentials import AzureKeyCredential  
from azure.search.documents import SearchClient, SearchIndexingBufferedSender  
from azure.search.documents.indexes import SearchIndexClient  
from azure.search.documents.models import (
    QueryAnswerType,
    QueryCaptionType,
    QueryType,
    VectorizedQuery,
)
from azure.search.documents.indexes.models import (
    HnswAlgorithmConfiguration,
    HnswParameters,
    SearchField,
    SearchableField,
    SearchFieldDataType,
    SearchIndex,
    SemanticConfiguration,
    SemanticField,
    SemanticPrioritizedFields,
    SemanticSearch,
    SimpleField,
    VectorSearch,
    VectorSearchAlgorithmKind,
    VectorSearchAlgorithmMetric,
    VectorSearchProfile,
)

```

## Configure OpenAI settings

This section guides you through setting up authentication for Azure OpenAI, allowing you to securely interact with the service using either Azure Active Directory (AAD) or an API key. Before proceeding, ensure you have your Azure OpenAI endpoint and credentials ready. For detailed instructions on setting up AAD with Azure OpenAI, refer to the [official documentation](https://learn.microsoft.com/azure/ai-services/openai/how-to/managed-identity).



```python
endpoint: str = "YOUR_AZURE_OPENAI_ENDPOINT"
api_key: str = "YOUR_AZURE_OPENAI_KEY"
api_version: str = "2023-05-15"
deployment = "YOUR_AZURE_OPENAI_DEPLOYMENT_NAME"
credential = DefaultAzureCredential()
token_provider = get_bearer_token_provider(
    credential, "https://cognitiveservices.azure.com/.default"
)

# Set this flag to True if you are using Azure Active Directory
use_aad_for_aoai = True 

if use_aad_for_aoai:
    # Use Azure Active Directory (AAD) authentication
    client = AzureOpenAI(
        azure_endpoint=endpoint,
        api_version=api_version,
        azure_ad_token_provider=token_provider,
    )
else:
    # Use API key authentication
    client = AzureOpenAI(
        api_key=api_key,
        api_version=api_version,
        azure_endpoint=endpoint,
    )
```

## Configure Azure AI Search Vector Store settings
This section explains how to set up the Azure AI Search client for integrating with the Vector Store feature. You can locate your Azure AI Search service details in the Azure Portal or programmatically via the [Search Management SDK](https://learn.microsoft.com/rest/api/searchmanagement/).


key: _node_content
string_value:
{"id_": "60d77fe3-92af-448d-86bb-18a939a233b6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73739acd-3f10-4a5b-8082-8796612ce31b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "843e023154df0d5db2d5a9597b807f14244742f0cf0485a03aad985bf7001ef7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "07f4d4db-0aab-4259-a73e-abebd25b3319", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "d727091d2b836b7cb81be86f0d21dceb9845e695c6b5947c9dbcf6bbe7b964db", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "16ff56fc-f232-45b5-ba77-0d2d30640286", "node_type": "1", "metadata": {}, "hash": "a9196c8859376eee01a2bfb36455618ef0e4678c2323a7d483b62efc6426c836", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2286, "end_char_idx": 4949, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
endpoint: str = "YOUR_AZURE_OPENAI_ENDPOINT"
api_key: str = "YOUR_AZURE_OPENAI_KEY"
api_version: str = "2023-05-15"
deployment = "YOUR_AZURE_OPENAI_DEPLOYMENT_NAME"
credential = DefaultAzureCredential()
token_provider = get_bearer_token_provider(
    credential, "https://cognitiveservices.azure.com/.default"
)

# Set this flag to True if you are using Azure Active Directory
use_aad_for_aoai = True 

if use_aad_for_aoai:
    # Use Azure Active Directory (AAD) authentication
    client = AzureOpenAI(
        azure_endpoint=endpoint,
        api_version=api_version,
        azure_ad_token_provider=token_provider,
    )
else:
    # Use API key authentication
    client = AzureOpenAI(
        api_key=api_key,
        api_version=api_version,
        azure_endpoint=endpoint,
    )
```

## Configure Azure AI Search Vector Store settings
This section explains how to set up the Azure AI Search client for integrating with the Vector Store feature. You can locate your Azure AI Search service details in the Azure Portal or programmatically via the [Search Management SDK](https://learn.microsoft.com/rest/api/searchmanagement/).



```python
# Configuration
search_service_endpoint: str = "YOUR_AZURE_SEARCH_ENDPOINT"
search_service_api_key: str = "YOUR_AZURE_SEARCH_ADMIN_KEY"
index_name: str = "azure-ai-search-openai-cookbook-demo"

# Set this flag to True if you are using Azure Active Directory
use_aad_for_search = True  

if use_aad_for_search:
    # Use Azure Active Directory (AAD) authentication
    credential = DefaultAzureCredential()
else:
    # Use API key authentication
    credential = AzureKeyCredential(search_service_api_key)

# Initialize the SearchClient with the selected authentication method
search_client = SearchClient(
    endpoint=search_service_endpoint, index_name=index_name, credential=credential
)
```

## Load data



```python
embeddings_url = "https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip"

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```




    'vector_database_wikipedia_articles_embedded.zip'




```python
with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip", "r") as zip_ref:
    zip_ref.extractall("../../data")
```


```python
article_df = pd.read_csv("../../data/vector_database_wikipedia_articles_embedded.csv")

# Read vectors from strings back into a list using json.loads
article_df["title_vector"] = article_df.title_vector.apply(json.loads)
article_df["content_vector"] = article_df.content_vector.apply(json.loads)
article_df["vector_id"] = article_df["vector_id"].apply(str)
article_df.head()
```


key: _node_content
string_value:
{"id_": "16ff56fc-f232-45b5-ba77-0d2d30640286", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73739acd-3f10-4a5b-8082-8796612ce31b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "843e023154df0d5db2d5a9597b807f14244742f0cf0485a03aad985bf7001ef7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "60d77fe3-92af-448d-86bb-18a939a233b6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "25b7479c437cb0ea97a6403735649ee2e5a0bcf4cae0998e2d322d877907d567", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0ca2640c-5bd8-4a70-876f-ba8d994d435b", "node_type": "1", "metadata": {}, "hash": "8cd0a9c9c8bed21afa4c25e5abe679fb75283551f60ad360179fa48e2e466a3e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3432, "end_char_idx": 5965, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Configuration
search_service_endpoint: str = "YOUR_AZURE_SEARCH_ENDPOINT"
search_service_api_key: str = "YOUR_AZURE_SEARCH_ADMIN_KEY"
index_name: str = "azure-ai-search-openai-cookbook-demo"

# Set this flag to True if you are using Azure Active Directory
use_aad_for_search = True  

if use_aad_for_search:
    # Use Azure Active Directory (AAD) authentication
    credential = DefaultAzureCredential()
else:
    # Use API key authentication
    credential = AzureKeyCredential(search_service_api_key)

# Initialize the SearchClient with the selected authentication method
search_client = SearchClient(
    endpoint=search_service_endpoint, index_name=index_name, credential=credential
)
```

## Load data



```python
embeddings_url = "https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip"

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```




    'vector_database_wikipedia_articles_embedded.zip'




```python
with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip", "r") as zip_ref:
    zip_ref.extractall("../../data")
```


```python
article_df = pd.read_csv("../../data/vector_database_wikipedia_articles_embedded.csv")

# Read vectors from strings back into a list using json.loads
article_df["title_vector"] = article_df.title_vector.apply(json.loads)
article_df["content_vector"] = article_df.content_vector.apply(json.loads)
article_df["vector_id"] = article_df["vector_id"].apply(str)
article_df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>url</th>
      <th>title</th>
      <th>text</th>
      <th>title_vector</th>
      <th>content_vector</th>
      <th>vector_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>https://simple.wikipedia.org/wiki/April</td>
      <td>April</td>
      <td>April is the fourth month of the year in the J...</td>
      <td>[0.001009464613161981, -0.020700545981526375, ...</td>
      <td>[-0.011253940872848034, -0.013491976074874401,...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>https://simple.wikipedia.org/wiki/August</td>
      <td>August</td>
      <td>August (Aug.)


key: _node_content
string_value:
{"id_": "0ca2640c-5bd8-4a70-876f-ba8d994d435b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73739acd-3f10-4a5b-8082-8796612ce31b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "843e023154df0d5db2d5a9597b807f14244742f0cf0485a03aad985bf7001ef7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "16ff56fc-f232-45b5-ba77-0d2d30640286", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "07a0a4dfc475f99018b20f80392c5ee61176f7c585e47107a2ff89413a1ec441", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d11e6c3f-e229-4d33-b29c-0888c89dde49", "node_type": "1", "metadata": {}, "hash": "d4235367260c1a0275a057bb0bbf3eefa97b9b7dc3c0634b8fbec9670ab8f4de", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4954, "end_char_idx": 6996, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: _node_content
string_value:
{"id_": "d11e6c3f-e229-4d33-b29c-0888c89dde49", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73739acd-3f10-4a5b-8082-8796612ce31b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "843e023154df0d5db2d5a9597b807f14244742f0cf0485a03aad985bf7001ef7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0ca2640c-5bd8-4a70-876f-ba8d994d435b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "1b4f0f91ed54ae756ec631176c6d3dfae584c9fc8c6149f3c4673eed149cc5de", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7873ccf9-571c-4a90-85c4-9e395bbdc9ab", "node_type": "1", "metadata": {}, "hash": "a51cc74e1e1a0eda1ed3d3ed049f385ed1e816d85f75d70300952e5de30d28f1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5966, "end_char_idx": 7656, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
is the eighth month of the year ...</td>
      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>
      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>
      <td>1</td>
    </tr>
    <tr>
      <th>2</th>
      <td>6</td>
      <td>https://simple.wikipedia.org/wiki/Art</td>
      <td>Art</td>
      <td>Art is a creative activity that expresses imag...</td>
      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>
      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>
      <td>2</td>
    </tr>
    <tr>
      <th>3</th>
      <td>8</td>
      <td>https://simple.wikipedia.org/wiki/A</td>
      <td>A</td>
      <td>A or a is the first letter of the English alph...</td>
      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>
      <td>[0.024894846603274345, -0.022186409682035446, ...</td>
      <td>3</td>
    </tr>
    <tr>
      <th>4</th>
      <td>9</td>
      <td>https://simple.wikipedia.org/wiki/Air</td>
      <td>Air</td>
      <td>Air refers to the Earth's atmosphere. Air is a...</td>
      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>
      <td>[0.021524671465158463, 0.018522677943110466, -...</td>
      <td>4</td>
    </tr>
  </tbody>
</table>
</div>



##  Create an index
This code snippet demonstrates how to define and create a search index using the `SearchIndexClient` from the Azure AI Search Python SDK. The index incorporates both vector search and semantic ranker capabilities. For more details, visit our documentation on how to [Create a Vector Index](https://learn.microsoft.com/azure/search/vector-search-how-to-create-index?.tabs=config-2023-11-01%2Crest-2023-11-01%2Cpush%2Cportal-check-index)


key: _node_content
string_value:
{"id_": "7873ccf9-571c-4a90-85c4-9e395bbdc9ab", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73739acd-3f10-4a5b-8082-8796612ce31b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "843e023154df0d5db2d5a9597b807f14244742f0cf0485a03aad985bf7001ef7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d11e6c3f-e229-4d33-b29c-0888c89dde49", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "2b846ce4a97d0cf8ecfbfef82e9278296374efa901dc5d21b5f0655dada5851b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "523be423-bb0f-40fb-a15c-d48bc45cc7f5", "node_type": "1", "metadata": {}, "hash": "ce12792f3674b96822509b0107a8c99b7aa015b4808bf0de4ebfc23c55112dab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6997, "end_char_idx": 10470, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Air is a...</td>
      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>
      <td>[0.021524671465158463, 0.018522677943110466, -...</td>
      <td>4</td>
    </tr>
  </tbody>
</table>
</div>



##  Create an index
This code snippet demonstrates how to define and create a search index using the `SearchIndexClient` from the Azure AI Search Python SDK. The index incorporates both vector search and semantic ranker capabilities. For more details, visit our documentation on how to [Create a Vector Index](https://learn.microsoft.com/azure/search/vector-search-how-to-create-index?.tabs=config-2023-11-01%2Crest-2023-11-01%2Cpush%2Cportal-check-index)


```python
# Initialize the SearchIndexClient
index_client = SearchIndexClient(
    endpoint=search_service_endpoint, credential=credential
)

# Define the fields for the index
fields = [
    SimpleField(name="id", type=SearchFieldDataType.String),
    SimpleField(name="vector_id", type=SearchFieldDataType.String, key=True),
    SimpleField(name="url", type=SearchFieldDataType.String),
    SearchableField(name="title", type=SearchFieldDataType.String),
    SearchableField(name="text", type=SearchFieldDataType.String),
    SearchField(
        name="title_vector",
        type=SearchFieldDataType.Collection(SearchFieldDataType.Single),
        vector_search_dimensions=1536,
        vector_search_profile_name="my-vector-config",
    ),
    SearchField(
        name="content_vector",
        type=SearchFieldDataType.Collection(SearchFieldDataType.Single),
        vector_search_dimensions=1536,
        vector_search_profile_name="my-vector-config",
    ),
]

# Configure the vector search configuration
vector_search = VectorSearch(
    algorithms=[
        HnswAlgorithmConfiguration(
            name="my-hnsw",
            kind=VectorSearchAlgorithmKind.HNSW,
            parameters=HnswParameters(
                m=4,
                ef_construction=400,
                ef_search=500,
                metric=VectorSearchAlgorithmMetric.COSINE,
            ),
        )
    ],
    profiles=[
        VectorSearchProfile(
            name="my-vector-config",
            algorithm_configuration_name="my-hnsw",
        )
    ],
)

# Configure the semantic search configuration
semantic_search = SemanticSearch(
    configurations=[
        SemanticConfiguration(
            name="my-semantic-config",
            prioritized_fields=SemanticPrioritizedFields(
                title_field=SemanticField(field_name="title"),
                keywords_fields=[SemanticField(field_name="url")],
                content_fields=[SemanticField(field_name="text")],
            ),
        )
    ]
)

# Create the search index with the vector search and semantic search configurations
index = SearchIndex(
    name=index_name,
    fields=fields,
    vector_search=vector_search,
    semantic_search=semantic_search,
)

# Create or update the index
result = index_client.create_or_update_index(index)
print(f"{result.name} created")
```

    azure-ai-search-openai-cookbook-demo created


## Uploading Data to Azure AI Search Index

The following code snippet outlines the process of uploading a batch of documents—specifically, Wikipedia articles with pre-computed embeddings—from a pandas DataFrame to an Azure AI Search index. For a detailed guide on data import strategies and best practices, refer to  [Data Import in Azure AI Search](https://learn.microsoft.com/azure/search/search-what-is-data-import).


key: _node_content
string_value:
{"id_": "523be423-bb0f-40fb-a15c-d48bc45cc7f5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73739acd-3f10-4a5b-8082-8796612ce31b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "843e023154df0d5db2d5a9597b807f14244742f0cf0485a03aad985bf7001ef7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7873ccf9-571c-4a90-85c4-9e395bbdc9ab", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "edfb23e230850f89f1c37b73d8bb20600d3ff461bb7a97fc63e2a628aa5f1b89", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4e190dfc-8797-4a0a-9b13-fbdabfb306b4", "node_type": "1", "metadata": {}, "hash": "9cb511fe0a0b3f02dbf8d7e68cba010bc0c634c7a5c8c9b9f666f38155cb5622", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10050, "end_char_idx": 13371, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Uploading Data to Azure AI Search Index

The following code snippet outlines the process of uploading a batch of documents—specifically, Wikipedia articles with pre-computed embeddings—from a pandas DataFrame to an Azure AI Search index. For a detailed guide on data import strategies and best practices, refer to  [Data Import in Azure AI Search](https://learn.microsoft.com/azure/search/search-what-is-data-import).



```python
from azure.core.exceptions import HttpResponseError

# Convert the 'id' and 'vector_id' columns to string so one of them can serve as our key field
article_df["id"] = article_df["id"].astype(str)
article_df["vector_id"] = article_df["vector_id"].astype(str)
# Convert the DataFrame to a list of dictionaries
documents = article_df.to_dict(orient="records")

# Create a SearchIndexingBufferedSender
batch_client = SearchIndexingBufferedSender(
    search_service_endpoint, index_name, credential
)

try:
    # Add upload actions for all documents in a single call
    batch_client.upload_documents(documents=documents)

    # Manually flush to send any remaining documents in the buffer
    batch_client.flush()
except HttpResponseError as e:
    print(f"An error occurred: {e}")
finally:
    # Clean up resources
    batch_client.close()

print(f"Uploaded {len(documents)} documents in total")
```

    Uploaded 25000 documents in total


If your dataset didn't already contain pre-computed embeddings, you can create embeddings by using the below function using the `openai` python library. You'll also notice the same function and model are being used to generate query embeddings for performing vector searches.


```python
# Example function to generate document embedding
def generate_embeddings(text, model):
    # Generate embeddings for the provided text using the specified model
    embeddings_response = client.embeddings.create(model=model, input=text)
    # Extract the embedding data from the response
    embedding = embeddings_response.data[0].embedding
    return embedding


first_document_content = documents[0]["text"]
print(f"Content: {first_document_content[:100]}")

content_vector = generate_embeddings(first_document_content, deployment)
print("Content vector generated")
```

    Content: April is the fourth month of the year in the Julian and Gregorian calendars, and comes between March
    Content vector generated


## Perform a vector similarity search


```python
# Pure Vector Search
query = "modern art in Europe"
  
search_client = SearchClient(search_service_endpoint, index_name, credential)  
vector_query = VectorizedQuery(vector=generate_embeddings(query, deployment), k_nearest_neighbors=3, fields="content_vector")
  
results = search_client.search(  
    search_text=None,  
    vector_queries= [vector_query], 
    select=["title", "text", "url"] 
)
  
for result in results:  
    print(f"Title: {result['title']}")  
    print(f"Score: {result['@search.score']}")  
    print(f"URL: {result['url']}\n")  
```

    Title: Documenta
    Score: 0.8599451
    URL: https://simple.wikipedia.org/wiki/Documenta
    
    Title: Museum of Modern Art
    Score: 0.85260946
    URL: https://simple.wikipedia.org/wiki/Museum%20of%20Modern%20Art
    
    Title: Expressionism
    Score: 0.852354
    URL: https://simple.wikipedia.org/wiki/Expressionism


key: _node_content
string_value:
{"id_": "4e190dfc-8797-4a0a-9b13-fbdabfb306b4", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73739acd-3f10-4a5b-8082-8796612ce31b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "843e023154df0d5db2d5a9597b807f14244742f0cf0485a03aad985bf7001ef7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "523be423-bb0f-40fb-a15c-d48bc45cc7f5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "f352944368c34c3f52233d693b6a01e34c3571c64b4aebd509b0a5e53d1ebcd3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1f71ddee-7c4c-4be6-a978-92ced38e4e20", "node_type": "1", "metadata": {}, "hash": "bbb1dc309c39ee2dae8ac5640e358fddb5885afe9e676a6bff92031cab38570d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11701, "end_char_idx": 14778, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Example function to generate document embedding
def generate_embeddings(text, model):
    # Generate embeddings for the provided text using the specified model
    embeddings_response = client.embeddings.create(model=model, input=text)
    # Extract the embedding data from the response
    embedding = embeddings_response.data[0].embedding
    return embedding


first_document_content = documents[0]["text"]
print(f"Content: {first_document_content[:100]}")

content_vector = generate_embeddings(first_document_content, deployment)
print("Content vector generated")
```

    Content: April is the fourth month of the year in the Julian and Gregorian calendars, and comes between March
    Content vector generated


## Perform a vector similarity search


```python
# Pure Vector Search
query = "modern art in Europe"
  
search_client = SearchClient(search_service_endpoint, index_name, credential)  
vector_query = VectorizedQuery(vector=generate_embeddings(query, deployment), k_nearest_neighbors=3, fields="content_vector")
  
results = search_client.search(  
    search_text=None,  
    vector_queries= [vector_query], 
    select=["title", "text", "url"] 
)
  
for result in results:  
    print(f"Title: {result['title']}")  
    print(f"Score: {result['@search.score']}")  
    print(f"URL: {result['url']}\n")  
```

    Title: Documenta
    Score: 0.8599451
    URL: https://simple.wikipedia.org/wiki/Documenta
    
    Title: Museum of Modern Art
    Score: 0.85260946
    URL: https://simple.wikipedia.org/wiki/Museum%20of%20Modern%20Art
    
    Title: Expressionism
    Score: 0.852354
    URL: https://simple.wikipedia.org/wiki/Expressionism
    


## Perform a Hybrid Search
Hybrid search combines the capabilities of traditional keyword-based search with vector-based similarity search to provide more relevant and contextual results. This approach is particularly useful when dealing with complex queries that benefit from understanding the semantic meaning behind the text.

The provided code snippet demonstrates how to execute a hybrid search query:


```python
# Hybrid Search
query = "Famous battles in Scottish history"  
  
search_client = SearchClient(search_service_endpoint, index_name, credential)  
vector_query = VectorizedQuery(vector=generate_embeddings(query, deployment), k_nearest_neighbors=3, fields="content_vector")
  
results = search_client.search(  
    search_text=query,  
    vector_queries= [vector_query], 
    select=["title", "text", "url"],
    top=3
)
  
for result in results:  
    print(f"Title: {result['title']}")  
    print(f"Score: {result['@search.score']}")  
    print(f"URL: {result['url']}\n")  
```

    Title: Wars of Scottish Independence
    Score: 0.03306011110544205
    URL: https://simple.wikipedia.org/wiki/Wars%20of%20Scottish%20Independence
    
    Title: Battle of Bannockburn
    Score: 0.022253260016441345
    URL: https://simple.wikipedia.org/wiki/Battle%20of%20Bannockburn
    
    Title: Scottish
    Score: 0.016393441706895828
    URL: https://simple.wikipedia.org/wiki/Scottish


key: _node_content
string_value:
{"id_": "1f71ddee-7c4c-4be6-a978-92ced38e4e20", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73739acd-3f10-4a5b-8082-8796612ce31b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "843e023154df0d5db2d5a9597b807f14244742f0cf0485a03aad985bf7001ef7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4e190dfc-8797-4a0a-9b13-fbdabfb306b4", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "4860d191fbbde98318091cae5b0358d6eb3b7004ff7bd1b66e3e6e1f6e0c9a5d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d50ac499-8459-476e-a7fc-927b1a1406e2", "node_type": "1", "metadata": {}, "hash": "35286841c342fb1f74eddd0c4edcbab0ee1b65805bc4297f96ef8a15bb0e7d6b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13379, "end_char_idx": 15097, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Perform a Hybrid Search
Hybrid search combines the capabilities of traditional keyword-based search with vector-based similarity search to provide more relevant and contextual results. This approach is particularly useful when dealing with complex queries that benefit from understanding the semantic meaning behind the text.

The provided code snippet demonstrates how to execute a hybrid search query:


```python
# Hybrid Search
query = "Famous battles in Scottish history"  
  
search_client = SearchClient(search_service_endpoint, index_name, credential)  
vector_query = VectorizedQuery(vector=generate_embeddings(query, deployment), k_nearest_neighbors=3, fields="content_vector")
  
results = search_client.search(  
    search_text=query,  
    vector_queries= [vector_query], 
    select=["title", "text", "url"],
    top=3
)
  
for result in results:  
    print(f"Title: {result['title']}")  
    print(f"Score: {result['@search.score']}")  
    print(f"URL: {result['url']}\n")  
```

    Title: Wars of Scottish Independence
    Score: 0.03306011110544205
    URL: https://simple.wikipedia.org/wiki/Wars%20of%20Scottish%20Independence
    
    Title: Battle of Bannockburn
    Score: 0.022253260016441345
    URL: https://simple.wikipedia.org/wiki/Battle%20of%20Bannockburn
    
    Title: Scottish
    Score: 0.016393441706895828
    URL: https://simple.wikipedia.org/wiki/Scottish
    


## Perform a Hybrid Search with Reranking (powered by Bing)
[Semantic ranker](https://learn.microsoft.com/azure/search/semantic-search-overview) measurably improves search relevance by using language understanding to rerank search results. Additionally, you can get extractive captions, answers, and highlights.


key: _node_content
string_value:
{"id_": "d50ac499-8459-476e-a7fc-927b1a1406e2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "73739acd-3f10-4a5b-8082-8796612ce31b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "843e023154df0d5db2d5a9597b807f14244742f0cf0485a03aad985bf7001ef7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1f71ddee-7c4c-4be6-a978-92ced38e4e20", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "f769a01e6ba965db6b068ba7303cb51a9ee72b792efc9c3f93daed55fb53a9ea", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13788, "end_char_idx": 18220, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Hybrid Search
query = "Famous battles in Scottish history"  
  
search_client = SearchClient(search_service_endpoint, index_name, credential)  
vector_query = VectorizedQuery(vector=generate_embeddings(query, deployment), k_nearest_neighbors=3, fields="content_vector")
  
results = search_client.search(  
    search_text=query,  
    vector_queries= [vector_query], 
    select=["title", "text", "url"],
    top=3
)
  
for result in results:  
    print(f"Title: {result['title']}")  
    print(f"Score: {result['@search.score']}")  
    print(f"URL: {result['url']}\n")  
```

    Title: Wars of Scottish Independence
    Score: 0.03306011110544205
    URL: https://simple.wikipedia.org/wiki/Wars%20of%20Scottish%20Independence
    
    Title: Battle of Bannockburn
    Score: 0.022253260016441345
    URL: https://simple.wikipedia.org/wiki/Battle%20of%20Bannockburn
    
    Title: Scottish
    Score: 0.016393441706895828
    URL: https://simple.wikipedia.org/wiki/Scottish
    


## Perform a Hybrid Search with Reranking (powered by Bing)
[Semantic ranker](https://learn.microsoft.com/azure/search/semantic-search-overview) measurably improves search relevance by using language understanding to rerank search results. Additionally, you can get extractive captions, answers, and highlights. 


```python
# Semantic Hybrid Search
query = "What were the key technological advancements during the Industrial Revolution?"

search_client = SearchClient(search_service_endpoint, index_name, credential)
vector_query = VectorizedQuery(
    vector=generate_embeddings(query, deployment),
    k_nearest_neighbors=3,
    fields="content_vector",
)

results = search_client.search(
    search_text=query,
    vector_queries=[vector_query],
    select=["title", "text", "url"],
    query_type=QueryType.SEMANTIC,
    semantic_configuration_name="my-semantic-config",
    query_caption=QueryCaptionType.EXTRACTIVE,
    query_answer=QueryAnswerType.EXTRACTIVE,
    top=3,
)

semantic_answers = results.get_answers()
for answer in semantic_answers:
    if answer.highlights:
        print(f"Semantic Answer: {answer.highlights}")
    else:
        print(f"Semantic Answer: {answer.text}")
    print(f"Semantic Answer Score: {answer.score}\n")

for result in results:
    print(f"Title: {result['title']}")
    print(f"Reranker Score: {result['@search.reranker_score']}")
    print(f"URL: {result['url']}")
    captions = result["@search.captions"]
    if captions:
        caption = captions[0]
        if caption.highlights:
            print(f"Caption: {caption.highlights}\n")
        else:
            print(f"Caption: {caption.text}\n")
```

    Semantic Answer: Advancements  During the industrial revolution, new technology brought many changes. For example:<em>   Canals</em> were built to allow heavy goods to be moved easily where they were needed. The steam engine became the main source of power. It replaced horses and human labor. Cheap iron and steel became mass-produced.
    Semantic Answer Score: 0.90478515625
    
    Title: Industrial Revolution
    Reranker Score: 3.408700942993164
    URL: https://simple.wikipedia.org/wiki/Industrial%20Revolution
    Caption: Advancements  During the industrial revolution, new technology brought many changes. For example:   Canals were built to allow heavy goods to be moved easily where they were needed. The steam engine became the main source of power. It replaced horses and human labor. Cheap iron and steel became mass-produced.
    
    Title: Printing
    Reranker Score: 1.603400707244873
    URL: https://simple.wikipedia.org/wiki/Printing
    Caption: Machines to speed printing, cheaper paper, automatic stitching and binding all arrived in the 19th century during the industrial revolution. What had once been done by a few men by hand was now done by limited companies on huge machines. The result was much lower prices, and a much wider readership.
    
    Title: Industrialisation
    Reranker Score: 1.3238357305526733
    URL: https://simple.wikipedia.org/wiki/Industrialisation
    Caption: <em>Industrialisation</em> (or<em> industrialization)</em> is a process that happens in countries when they start to use machines to do work that was once done by people.<em> Industrialisation changes</em> the things people do.<em> Industrialisation</em> caused towns to grow larger. Many people left farming to take higher paid jobs in factories in towns.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb


key: _node_content
string_value:
{"id_": "78d2fa8a-f82f-4407-b918-017c216d8e31", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cba42c1d-7388-446c-9b78-fa0b85e42e53", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "1d4b6c92197c0b267327aac1e1d4efdecaee38bdc95189ddf7a242480afa6fb7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7816fdb3-600d-43f5-af8a-8735d4dc2c03", "node_type": "1", "metadata": {}, "hash": "357aabe7ee275efe4192419d0085587472be4045ce60ac2fb8c1b9bd971909d2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3468, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Using PolarDB-PG as a vector database for OpenAI embeddings

This notebook guides you step by step on using PolarDB-PG as a vector database for OpenAI embeddings.

This notebook presents an end-to-end process of:
1. Using precomputed embeddings created by OpenAI API.
2. Storing the embeddings in a cloud instance of PolarDB-PG.
3. Converting raw text query to an embedding with OpenAI API.
4. Using PolarDB-PG to perform the nearest neighbour search in the created collection.

### What is PolarDB-PG

[PolarDB-PG](https://www.alibabacloud.com/help/en/polardb/latest/what-is-polardb-2) is a high-performance vector database that adopts a read-write separation architecture. It is a cloud-native database managed by Alibaba Cloud, 100% compatible with PostgreSQL, and highly compatible with Oracle syntax. It supports processing massive vector data storage and queries, and greatly improves the efficiency of vector calculations through optimization of underlying execution algorithms, providing users with fast, elastic, high-performance, massive storage, and secure and reliable vector database services. Additionally, PolarDB-PG also supports multi-dimensional and multi-modal spatiotemporal information engines and geographic information engines.At the same time, PolarDB-PG is equipped with complete OLAP functionality and service level agreements, which has been recognized and used by many users;







### Deployment options

- Using [PolarDB-PG Cloud Vector Database](https://www.alibabacloud.com/product/polardb-for-postgresql). [Click here](https://www.alibabacloud.com/product/polardb-for-postgresql?spm=a3c0i.147400.6791778070.243.9f204881g5cjpP) to fast deploy it.

## Prerequisites

For the purposes of this exercise we need to prepare a couple of things:

1. PolarDB-PG cloud server instance.
2. The 'psycopg2' library to interact with the vector database. Any other postgresql client library is ok.
3. An [OpenAI API key](https://beta.openai.com/account/api-keys).

We might validate if the server was launched successfully by running a simple curl command:

### Install requirements

This notebook obviously requires the `openai` and `psycopg2` packages, but there are also some other additional libraries we will use. The following command installs them all:



```python
! pip install openai psycopg2 pandas wget
```

Prepare your OpenAI API key
The OpenAI API key is used for vectorization of the documents and queries.

If you don't have an OpenAI API key, you can get one from https://beta.openai.com/account/api-keys.

Once you get your key, please add it to your environment variables as OPENAI_API_KEY.

If you have any doubts about setting the API key through environment variables, please refer to [Best Practices for API Key Safety](https://help.openai.com/en/articles/5112595-best-practices-for-api-key-safety).


```python
# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.

if os.getenv("OPENAI_API_KEY") is not None:
    print("OPENAI_API_KEY is ready")
else:
    print("OPENAI_API_KEY environment variable not found")
```

    OPENAI_API_KEY is ready


## Connect to PolarDB
First add it to your environment variables. or you can just change the "psycopg2.connect" parameters below

Connecting to a running instance of PolarDB server is easy with the official Python library:


key: _node_content
string_value:
{"id_": "7816fdb3-600d-43f5-af8a-8735d4dc2c03", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cba42c1d-7388-446c-9b78-fa0b85e42e53", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "1d4b6c92197c0b267327aac1e1d4efdecaee38bdc95189ddf7a242480afa6fb7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "78d2fa8a-f82f-4407-b918-017c216d8e31", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "c3906db3c2fa988a36dc98395533d167a280fd3f001ff591dbe712b9c31fcec1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "47e50d0f-1549-41db-a450-eec1a727ee0a", "node_type": "1", "metadata": {}, "hash": "eb8d4ce2a9e537423ed98f7afabb29a18a9add9fe31825a69ca22d9ffcd99ead", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2284, "end_char_idx": 5584, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
! pip install openai psycopg2 pandas wget
```

Prepare your OpenAI API key
The OpenAI API key is used for vectorization of the documents and queries.

If you don't have an OpenAI API key, you can get one from https://beta.openai.com/account/api-keys.

Once you get your key, please add it to your environment variables as OPENAI_API_KEY.

If you have any doubts about setting the API key through environment variables, please refer to [Best Practices for API Key Safety](https://help.openai.com/en/articles/5112595-best-practices-for-api-key-safety).


```python
# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.

if os.getenv("OPENAI_API_KEY") is not None:
    print("OPENAI_API_KEY is ready")
else:
    print("OPENAI_API_KEY environment variable not found")
```

    OPENAI_API_KEY is ready


## Connect to PolarDB
First add it to your environment variables. or you can just change the "psycopg2.connect" parameters below

Connecting to a running instance of PolarDB server is easy with the official Python library:


```python
import os
import psycopg2

# Note. alternatively you can set a temporary env variable like this:
# os.environ["PGHOST"] = "your_host"
# os.environ["PGPORT"] "5432"),
# os.environ["PGDATABASE"] "postgres"),
# os.environ["PGUSER"] "user"),
# os.environ["PGPASSWORD"] "password"),

connection = psycopg2.connect(
    host=os.environ.get("PGHOST", "localhost"),
    port=os.environ.get("PGPORT", "5432"),
    database=os.environ.get("PGDATABASE", "postgres"),
    user=os.environ.get("PGUSER", "user"),
    password=os.environ.get("PGPASSWORD", "password")
)

# Create a new cursor object
cursor = connection.cursor()
```

We can test the connection by running any available method:


```python
# Execute a simple query to test the connection
cursor.execute("SELECT 1;")
result = cursor.fetchone()

# Check the query result
if result == (1,):
    print("Connection successful!")
else:
    print("Connection failed.")
```

    Connection successful!



```python
import wget

embeddings_url = "https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip"

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```




    'vector_database_wikipedia_articles_embedded.zip'



The downloaded file has to be then extracted:


```python
import zipfile
import os
import re
import tempfile

current_directory = os.getcwd()
zip_file_path = os.path.join(current_directory, "vector_database_wikipedia_articles_embedded.zip")
output_directory = os.path.join(current_directory, "../../data")

with zipfile.ZipFile(zip_file_path, "r") as zip_ref:
    zip_ref.extractall(output_directory)


# check the csv file exist
file_name = "vector_database_wikipedia_articles_embedded.csv"
data_directory = os.path.join(current_directory, "../../data")
file_path = os.path.join(data_directory, file_name)


if os.path.exists(file_path):
    print(f"The file {file_name} exists in the data directory.")
else:
    print(f"The file {file_name} does not exist in the data directory.")
```

    The file vector_database_wikipedia_articles_embedded.csv exists in the data directory.


key: _node_content
string_value:
{"id_": "47e50d0f-1549-41db-a450-eec1a727ee0a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cba42c1d-7388-446c-9b78-fa0b85e42e53", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "1d4b6c92197c0b267327aac1e1d4efdecaee38bdc95189ddf7a242480afa6fb7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7816fdb3-600d-43f5-af8a-8735d4dc2c03", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "ebf3efc54e50ec4b7b69e9523d6b41deb3abf93c245058f4c3ad37e0615fae5c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "59ed3a7e-f5ac-49e6-93af-2a5a25c6dfc8", "node_type": "1", "metadata": {}, "hash": "6e4f36f0512d624a799fb7e9d5169aa3dbb52592b6ca5380a7f650bec67db57f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4162, "end_char_idx": 6832, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Execute a simple query to test the connection
cursor.execute("SELECT 1;")
result = cursor.fetchone()

# Check the query result
if result == (1,):
    print("Connection successful!")
else:
    print("Connection failed.")
```

    Connection successful!



```python
import wget

embeddings_url = "https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip"

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```




    'vector_database_wikipedia_articles_embedded.zip'



The downloaded file has to be then extracted:


```python
import zipfile
import os
import re
import tempfile

current_directory = os.getcwd()
zip_file_path = os.path.join(current_directory, "vector_database_wikipedia_articles_embedded.zip")
output_directory = os.path.join(current_directory, "../../data")

with zipfile.ZipFile(zip_file_path, "r") as zip_ref:
    zip_ref.extractall(output_directory)


# check the csv file exist
file_name = "vector_database_wikipedia_articles_embedded.csv"
data_directory = os.path.join(current_directory, "../../data")
file_path = os.path.join(data_directory, file_name)


if os.path.exists(file_path):
    print(f"The file {file_name} exists in the data directory.")
else:
    print(f"The file {file_name} does not exist in the data directory.")
```

    The file vector_database_wikipedia_articles_embedded.csv exists in the data directory.


## Index data

PolarDB stores data in __relation__ where each object is described by at least one vector. Our relation will be called **articles** and each object will be described by both **title** and **content** vectors. 

We will start with creating a relation and create a vector index on both **title** and **content**, and then we will fill it with our precomputed embeddings.


```python
create_table_sql = '''
CREATE TABLE IF NOT EXISTS public.articles (
    id INTEGER NOT NULL,
    url TEXT,
    title TEXT,
    content TEXT,
    title_vector vector(1536),
    content_vector vector(1536),
    vector_id INTEGER
);

ALTER TABLE public.articles ADD PRIMARY KEY (id);
'''

# SQL statement for creating indexes
create_indexes_sql = '''
CREATE INDEX ON public.articles USING ivfflat (content_vector) WITH (lists = 1000);

CREATE INDEX ON public.articles USING ivfflat (title_vector) WITH (lists = 1000);
'''

# Execute the SQL statements
cursor.execute(create_table_sql)
cursor.execute(create_indexes_sql)

# Commit the changes
connection.commit()
```

## Load data

In this section we are going to load the data prepared previous to this session, so you don't have to recompute the embeddings of Wikipedia articles with your own credits.


key: _node_content
string_value:
{"id_": "59ed3a7e-f5ac-49e6-93af-2a5a25c6dfc8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cba42c1d-7388-446c-9b78-fa0b85e42e53", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "1d4b6c92197c0b267327aac1e1d4efdecaee38bdc95189ddf7a242480afa6fb7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "47e50d0f-1549-41db-a450-eec1a727ee0a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "e593726021c53e45b918283185838096ae0ea73401a275b3681b9373402be8fc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "62e5f2e7-c5d2-4789-bd60-5a888486e329", "node_type": "1", "metadata": {}, "hash": "f706c861f786060476c68eb7e51f1a6fb1b683ec10e3695acce7fef2f191d7b1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5109, "end_char_idx": 8230, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# check the csv file exist
file_name = "vector_database_wikipedia_articles_embedded.csv"
data_directory = os.path.join(current_directory, "../../data")
file_path = os.path.join(data_directory, file_name)


if os.path.exists(file_path):
    print(f"The file {file_name} exists in the data directory.")
else:
    print(f"The file {file_name} does not exist in the data directory.")
```

    The file vector_database_wikipedia_articles_embedded.csv exists in the data directory.


## Index data

PolarDB stores data in __relation__ where each object is described by at least one vector. Our relation will be called **articles** and each object will be described by both **title** and **content** vectors. 

We will start with creating a relation and create a vector index on both **title** and **content**, and then we will fill it with our precomputed embeddings.


```python
create_table_sql = '''
CREATE TABLE IF NOT EXISTS public.articles (
    id INTEGER NOT NULL,
    url TEXT,
    title TEXT,
    content TEXT,
    title_vector vector(1536),
    content_vector vector(1536),
    vector_id INTEGER
);

ALTER TABLE public.articles ADD PRIMARY KEY (id);
'''

# SQL statement for creating indexes
create_indexes_sql = '''
CREATE INDEX ON public.articles USING ivfflat (content_vector) WITH (lists = 1000);

CREATE INDEX ON public.articles USING ivfflat (title_vector) WITH (lists = 1000);
'''

# Execute the SQL statements
cursor.execute(create_table_sql)
cursor.execute(create_indexes_sql)

# Commit the changes
connection.commit()
```

## Load data

In this section we are going to load the data prepared previous to this session, so you don't have to recompute the embeddings of Wikipedia articles with your own credits.


```python
import io

# Path to your local CSV file
csv_file_path = '../../data/vector_database_wikipedia_articles_embedded.csv'

# Define a generator function to process the file line by line
def process_file(file_path):
    with open(file_path, 'r') as file:
        for line in file:
            yield line

# Create a StringIO object to store the modified lines
modified_lines = io.StringIO(''.join(list(process_file(csv_file_path))))

# Create the COPY command for the copy_expert method
copy_command = '''
COPY public.articles (id, url, title, content, title_vector, content_vector, vector_id)
FROM STDIN WITH (FORMAT CSV, HEADER true, DELIMITER ',');
'''

# Execute the COPY command using the copy_expert method
cursor.copy_expert(copy_command, modified_lines)

# Commit the changes
connection.commit()
```


```python
# Check the collection size to make sure all the points have been stored
count_sql = """select count(*) from public.articles;"""
cursor.execute(count_sql)
result = cursor.fetchone()
print(f"Count:{result[0]}")
```

    Count:25000


## Search data

Once the data is put into Qdrant we will start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search. Since the precomputed embeddings were created with `text-embedding-3-small` OpenAI model we also have to use it during search.


key: _node_content
string_value:
{"id_": "62e5f2e7-c5d2-4789-bd60-5a888486e329", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cba42c1d-7388-446c-9b78-fa0b85e42e53", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "1d4b6c92197c0b267327aac1e1d4efdecaee38bdc95189ddf7a242480afa6fb7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "59ed3a7e-f5ac-49e6-93af-2a5a25c6dfc8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "a0fe43b920ac375661e363f3fe35d87cb929a26bedfcc7c52b6b4d4c15d79428", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ee2949a1-c30f-4834-8f61-f2ae8db4315c", "node_type": "1", "metadata": {}, "hash": "bc9b4d11117499d2f634b6dcc04e14259d7684f9aec7f2138c743cce142eccd4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6835, "end_char_idx": 9025, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import io

# Path to your local CSV file
csv_file_path = '../../data/vector_database_wikipedia_articles_embedded.csv'

# Define a generator function to process the file line by line
def process_file(file_path):
    with open(file_path, 'r') as file:
        for line in file:
            yield line

# Create a StringIO object to store the modified lines
modified_lines = io.StringIO(''.join(list(process_file(csv_file_path))))

# Create the COPY command for the copy_expert method
copy_command = '''
COPY public.articles (id, url, title, content, title_vector, content_vector, vector_id)
FROM STDIN WITH (FORMAT CSV, HEADER true, DELIMITER ',');
'''

# Execute the COPY command using the copy_expert method
cursor.copy_expert(copy_command, modified_lines)

# Commit the changes
connection.commit()
```


```python
# Check the collection size to make sure all the points have been stored
count_sql = """select count(*) from public.articles;"""
cursor.execute(count_sql)
result = cursor.fetchone()
print(f"Count:{result[0]}")
```

    Count:25000


## Search data

Once the data is put into Qdrant we will start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search. Since the precomputed embeddings were created with `text-embedding-3-small` OpenAI model we also have to use it during search.


```python
def query_polardb(query, collection_name, vector_name="title_vector", top_k=20):

    # Creates embedding vector from user query
    embedded_query = openai.Embedding.create(
        input=query,
        model="text-embedding-3-small",
    )["data"][0]["embedding"]

    # Convert the embedded_query to PostgreSQL compatible format
    embedded_query_pg = "[" + ",".join(map(str, embedded_query)) + "]"

    # Create SQL query
    query_sql = f"""
    SELECT id, url, title, l2_distance({vector_name},'{embedded_query_pg}'::VECTOR(1536)) AS similarity
    FROM {collection_name}
    ORDER BY {vector_name} <-> '{embedded_query_pg}'::VECTOR(1536)
    LIMIT {top_k};
    """
    # Execute the query
    cursor.execute(query_sql)
    results = cursor.fetchall()

    return results
```


key: _node_content
string_value:
{"id_": "ee2949a1-c30f-4834-8f61-f2ae8db4315c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cba42c1d-7388-446c-9b78-fa0b85e42e53", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "1d4b6c92197c0b267327aac1e1d4efdecaee38bdc95189ddf7a242480afa6fb7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "62e5f2e7-c5d2-4789-bd60-5a888486e329", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "0458e02150b0c5cca90a71c6b380382ada7717fa2e8fc8970521ab79c8d2c277", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e0fbb6b0-95d2-47e7-a73b-0f170bdca57d", "node_type": "1", "metadata": {}, "hash": "3e415c5c34d49f9e4ff77a6eb2a8d4d18805c54291a7e2021f3e34fee063d0e7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7650, "end_char_idx": 9938, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Check the collection size to make sure all the points have been stored
count_sql = """select count(*) from public.articles;"""
cursor.execute(count_sql)
result = cursor.fetchone()
print(f"Count:{result[0]}")
```

    Count:25000


## Search data

Once the data is put into Qdrant we will start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search. Since the precomputed embeddings were created with `text-embedding-3-small` OpenAI model we also have to use it during search.


```python
def query_polardb(query, collection_name, vector_name="title_vector", top_k=20):

    # Creates embedding vector from user query
    embedded_query = openai.Embedding.create(
        input=query,
        model="text-embedding-3-small",
    )["data"][0]["embedding"]

    # Convert the embedded_query to PostgreSQL compatible format
    embedded_query_pg = "[" + ",".join(map(str, embedded_query)) + "]"

    # Create SQL query
    query_sql = f"""
    SELECT id, url, title, l2_distance({vector_name},'{embedded_query_pg}'::VECTOR(1536)) AS similarity
    FROM {collection_name}
    ORDER BY {vector_name} <-> '{embedded_query_pg}'::VECTOR(1536)
    LIMIT {top_k};
    """
    # Execute the query
    cursor.execute(query_sql)
    results = cursor.fetchall()

    return results
```


```python
import openai

query_results = query_polardb("modern art in Europe", "Articles")
for i, result in enumerate(query_results):
    print(f"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})")
```

    1. Museum of Modern Art (Score: 0.5)
    2. Western Europe (Score: 0.485)
    3. Renaissance art (Score: 0.479)
    4. Pop art (Score: 0.472)
    5. Northern Europe (Score: 0.461)
    6. Hellenistic art (Score: 0.457)
    7. Modernist literature (Score: 0.447)
    8. Art film (Score: 0.44)
    9. Central Europe (Score: 0.439)
    10. European (Score: 0.437)
    11. Art (Score: 0.437)
    12. Byzantine art (Score: 0.436)
    13. Postmodernism (Score: 0.434)
    14. Eastern Europe (Score: 0.433)
    15. Europe (Score: 0.433)
    16. Cubism (Score: 0.432)
    17. Impressionism (Score: 0.432)
    18. Bauhaus (Score: 0.431)
    19. Surrealism (Score: 0.429)
    20. Expressionism (Score: 0.429)


key: _node_content
string_value:
{"id_": "e0fbb6b0-95d2-47e7-a73b-0f170bdca57d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cba42c1d-7388-446c-9b78-fa0b85e42e53", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "1d4b6c92197c0b267327aac1e1d4efdecaee38bdc95189ddf7a242480afa6fb7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ee2949a1-c30f-4834-8f61-f2ae8db4315c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "82fa57aadb1bba4a0dd3998fe800c9ba7a6d7a7079c16794e24ad51b472edde6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9028, "end_char_idx": 10891, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import openai

query_results = query_polardb("modern art in Europe", "Articles")
for i, result in enumerate(query_results):
    print(f"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})")
```

    1. Museum of Modern Art (Score: 0.5)
    2. Western Europe (Score: 0.485)
    3. Renaissance art (Score: 0.479)
    4. Pop art (Score: 0.472)
    5. Northern Europe (Score: 0.461)
    6. Hellenistic art (Score: 0.457)
    7. Modernist literature (Score: 0.447)
    8. Art film (Score: 0.44)
    9. Central Europe (Score: 0.439)
    10. European (Score: 0.437)
    11. Art (Score: 0.437)
    12. Byzantine art (Score: 0.436)
    13. Postmodernism (Score: 0.434)
    14. Eastern Europe (Score: 0.433)
    15. Europe (Score: 0.433)
    16. Cubism (Score: 0.432)
    17. Impressionism (Score: 0.432)
    18. Bauhaus (Score: 0.431)
    19. Surrealism (Score: 0.429)
    20. Expressionism (Score: 0.429)



```python
# This time we'll query using content vector
query_results = query_polardb("Famous battles in Scottish history", "Articles", "content_vector")
for i, result in enumerate(query_results):
    print(f"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})")
```

    1. Battle of Bannockburn (Score: 0.489)
    2. Wars of Scottish Independence (Score: 0.474)
    3. 1651 (Score: 0.457)
    4. First War of Scottish Independence (Score: 0.452)
    5. Robert I of Scotland (Score: 0.445)
    6. 841 (Score: 0.441)
    7. 1716 (Score: 0.441)
    8. 1314 (Score: 0.429)
    9. 1263 (Score: 0.428)
    10. William Wallace (Score: 0.426)
    11. Stirling (Score: 0.419)
    12. 1306 (Score: 0.419)
    13. 1746 (Score: 0.418)
    14. 1040s (Score: 0.414)
    15. 1106 (Score: 0.412)
    16. 1304 (Score: 0.411)
    17. David II of Scotland (Score: 0.408)
    18. Braveheart (Score: 0.407)
    19. 1124 (Score: 0.406)
    20. July 27 (Score: 0.405)


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/README.ipynb


key: _node_content
string_value:
{"id_": "17807595-68b4-4577-8e9a-60fe85e90cd4", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/README.ipynb", "title": "MongoDB Atlas Vector Search", "tokens": 157, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d794f53a-dd7f-4397-945c-8ac8f3e96099", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/README.ipynb", "title": "MongoDB Atlas Vector Search", "tokens": 157, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "aec52227cc362cbe03db294981e4d1ef35e09b67f68d06c75013fc697f2b1d85", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 843, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# MongoDB Atlas Vector Search


[Atlas Vector Search](https://www.mongodb.com/products/platform/atlas-vector-search) is a fully managed service that simplifies the process of effectively indexing high-dimensional vector data within MongoDB and being able to perform fast vector similarity searches. With Atlas Vector Search, you can use MongoDB as a standalone vector database for a new project or augment your existing MongoDB collections with vector search functionality. With Atlas Vector Search, you can use the powerful capabilities of vector search in any major public cloud (AWS, Azure, GCP) and achieve massive scalability and data security out of the box while being enterprise-ready with provisions like FedRamp, SoC2 compliance.

Documentation - [link](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-overview/)


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb


key: _node_content
string_value:
{"id_": "aedadf9d-ef76-4c52-b1de-6f0a62fcc2f6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c9d10ca-b404-49d2-ba00-c23405a6bcf2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "35a027f721ba6f28e842a4430ab564d5367d3841a944ae1c08a1cc70c1fd6aab", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a5d0b6d8-93aa-436f-b295-f2058318d53d", "node_type": "1", "metadata": {}, "hash": "bf834f704d62afb2fe7e52d066cf8100ddb46d3a676d5263c89b50c754e9b754", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1876, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This notebook demonstrates how to  build a semantic search application  using OpenAI and [MongoDB Atlas vector search](https://www.mongodb.com/products/platform/atlas-vector-search)


```python
!pip install pymongo openai
```

    Collecting pymongo
      Downloading pymongo-4.6.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (677 kB)
    [2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m677.1/677.1 kB[0m [31m10.3 MB/s[0m eta [36m0:00:00[0m
    [?25hCollecting openai
      Downloading openai-1.3.3-py3-none-any.whl (220 kB)
    [2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m220.3/220.3 kB[0m [31m24.4 MB/s[0m eta [36m0:00:00[0m
    [?25hCollecting dnspython<3.0.0,>=1.16.0 (from pymongo)
      Downloading dnspython-2.4.2-py3-none-any.whl (300 kB)
    [2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m300.4/300.4 kB[0m [31m29.0 MB/s[0m eta [36m0:00:00[0m
    [?25hRequirement already satisfied: anyio<4,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from openai) (3.7.1)
    Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai) (1.7.0)
    Collecting httpx<1,>=0.23.0 (from openai)
      Downloading httpx-0.25.1-py3-none-any.whl (75 kB)
    [2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m75.0/75.0 kB[0m [31m9.8 MB/s[0m eta [36m0:00:00[0m
    [?25hRequirement already satisfied: pydantic<3,>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from openai) (1.10.13)
    Requirement already satisfied: tqdm>4 in /usr/local/lib/python3.10/dist-packages (from openai) (4.66.1)
    Requirement already satisfied: typing-extensions<5,>=4.5 in /usr/local/lib/python3.10/dist-packages (from openai) (4.5.0)
    Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/dist-packages (from anyio<4,>=3.5.0->openai) (3.


key: _node_content
string_value:
{"id_": "a5d0b6d8-93aa-436f-b295-f2058318d53d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c9d10ca-b404-49d2-ba00-c23405a6bcf2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "35a027f721ba6f28e842a4430ab564d5367d3841a944ae1c08a1cc70c1fd6aab", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aedadf9d-ef76-4c52-b1de-6f0a62fcc2f6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "cee05cc6f1e4ad9e3c94c43cff2e070690e366dc07146ea4e9bb8c5c7be1ccef", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2c2b4670-f771-47c0-bb22-9b1522259fee", "node_type": "1", "metadata": {}, "hash": "ffbaea1a2dffc3cf0494ba5f6589dc9bf4e8e0d451fd7b014fe17c0bc4dd9d0c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 882, "end_char_idx": 2734, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
4/300.4 kB[0m [31m29.0 MB/s[0m eta [36m0:00:00[0m
    [?25hRequirement already satisfied: anyio<4,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from openai) (3.7.1)
    Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai) (1.7.0)
    Collecting httpx<1,>=0.23.0 (from openai)
      Downloading httpx-0.25.1-py3-none-any.whl (75 kB)
    [2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m75.0/75.0 kB[0m [31m9.8 MB/s[0m eta [36m0:00:00[0m
    [?25hRequirement already satisfied: pydantic<3,>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from openai) (1.10.13)
    Requirement already satisfied: tqdm>4 in /usr/local/lib/python3.10/dist-packages (from openai) (4.66.1)
    Requirement already satisfied: typing-extensions<5,>=4.5 in /usr/local/lib/python3.10/dist-packages (from openai) (4.5.0)
    Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/dist-packages (from anyio<4,>=3.5.0->openai) (3.4)
    Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.10/dist-packages (from anyio<4,>=3.5.0->openai) (1.3.0)
    Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<4,>=3.5.0->openai) (1.1.3)
    Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->openai) (2023.7.22)
    Collecting httpcore (from httpx<1,>=0.23.0->openai)
      Downloading httpcore-1.0.2-py3-none-any.whl (76 kB)
    [2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m76.9/76.9 kB[0m [31m7.9 MB/s[0m eta [36m0:00:00[0m
    [?25hCollecting h11<0.15,>=0.13 (from httpcore->httpx<1,>=0.23.0->openai)
      Downloading h11-0.14.0-py3-none-any.whl (58 kB)
    [2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m58.3/58.3 kB[0m [31m6.


key: _node_content
string_value:
{"id_": "2c2b4670-f771-47c0-bb22-9b1522259fee", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c9d10ca-b404-49d2-ba00-c23405a6bcf2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "35a027f721ba6f28e842a4430ab564d5367d3841a944ae1c08a1cc70c1fd6aab", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a5d0b6d8-93aa-436f-b295-f2058318d53d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "59003f271aec8c32f04ff23db2533d30727cfc959f56d2ff11cb32997d7c9b2e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e59eadab-d8e9-4888-87d7-3a28b3c14fa7", "node_type": "1", "metadata": {}, "hash": "94e807b7219fe8c59ad1f9b8cae8ef0cb093a36f7270ee06453b7131a0139df3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1797, "end_char_idx": 3281, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
8 in /usr/local/lib/python3.10/dist-packages (from anyio<4,>=3.5.0->openai) (3.4)
    Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.10/dist-packages (from anyio<4,>=3.5.0->openai) (1.3.0)
    Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<4,>=3.5.0->openai) (1.1.3)
    Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->openai) (2023.7.22)
    Collecting httpcore (from httpx<1,>=0.23.0->openai)
      Downloading httpcore-1.0.2-py3-none-any.whl (76 kB)
    [2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m76.9/76.9 kB[0m [31m7.9 MB/s[0m eta [36m0:00:00[0m
    [?25hCollecting h11<0.15,>=0.13 (from httpcore->httpx<1,>=0.23.0->openai)
      Downloading h11-0.14.0-py3-none-any.whl (58 kB)
    [2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m58.3/58.3 kB[0m [31m6.8 MB/s[0m eta [36m0:00:00[0m
    [?25hInstalling collected packages: h11, dnspython, pymongo, httpcore, httpx, openai
    [31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    llmx 0.0.15a0 requires cohere, which is not installed.
    llmx 0.0.15a0 requires tiktoken, which is not installed.[0m[31m
    [0mSuccessfully installed dnspython-2.4.2 h11-0.14.0 httpcore-1.0.2 httpx-0.25.1 openai-1.3.3 pymongo-4.6.0


key: _node_content
string_value:
{"id_": "e59eadab-d8e9-4888-87d7-3a28b3c14fa7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c9d10ca-b404-49d2-ba00-c23405a6bcf2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "35a027f721ba6f28e842a4430ab564d5367d3841a944ae1c08a1cc70c1fd6aab", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2c2b4670-f771-47c0-bb22-9b1522259fee", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "ab16b2cefe7e8af2e821cf912f2c6edddece80e0eaf47e70b149f83c1605fd95", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f72e2525-f9bc-4dac-9dda-a563e69d2533", "node_type": "1", "metadata": {}, "hash": "02278226c2c323bbf32f73259683f52f9b87c1351ac311f682742132edcd9d89", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2356, "end_char_idx": 4821, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0.2-py3-none-any.whl (76 kB)
    [2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m76.9/76.9 kB[0m [31m7.9 MB/s[0m eta [36m0:00:00[0m
    [?25hCollecting h11<0.15,>=0.13 (from httpcore->httpx<1,>=0.23.0->openai)
      Downloading h11-0.14.0-py3-none-any.whl (58 kB)
    [2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m58.3/58.3 kB[0m [31m6.8 MB/s[0m eta [36m0:00:00[0m
    [?25hInstalling collected packages: h11, dnspython, pymongo, httpcore, httpx, openai
    [31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.
    llmx 0.0.15a0 requires cohere, which is not installed.
    llmx 0.0.15a0 requires tiktoken, which is not installed.[0m[31m
    [0mSuccessfully installed dnspython-2.4.2 h11-0.14.0 httpcore-1.0.2 httpx-0.25.1 openai-1.3.3 pymongo-4.6.0


# Step 1: Setup the environment

There are 2 pre-requisites for this:

1.   **MongoDB Atlas cluster**: To create a forever free MongoDB Atlas cluster, first, you need to create a MongoDB Atlas account if you don't already have one. Visit the [MongoDB Atlas website](https://www.mongodb.com/atlas/database) and click on “Register.” Visit the [MongoDB Atlas](https://account.mongodb.com/account/login) dashboard and set up your cluster. In order to take advantage of the `$vectorSearch` operator in an aggregation pipeline, you need to run MongoDB Atlas 6.0.11 or higher. This tutorial can be built using a free cluster. When you’re setting up your deployment, you’ll be prompted to set up a database user and rules for your network connection. Please ensure you save your username and password somewhere safe and have the correct IP address rules in place so your cluster can connect properly. If you need more help getting started, check out our [tutorial on MongoDB Atlas](https://www.mongodb.com/basics/mongodb-atlas-tutorial).

2. **OpenAI API key**   To create your OpenAI key, you'll need to create an account. Once you have that, visit the [OpenAI platform](https://platform.openai.com/). Click on your profile icon in the top right of the screen to get the dropdown menu and select “View API keys”.



```python
import getpass

MONGODB_ATLAS_CLUSTER_URI = getpass.getpass("MongoDB Atlas Cluster URI:")
OPENAI_API_KEY = getpass.getpass("OpenAI API Key:")

```

    MongoDB Atlas Cluster URI:··········
    OpenAI API Key:··········


key: _node_content
string_value:
{"id_": "f72e2525-f9bc-4dac-9dda-a563e69d2533", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c9d10ca-b404-49d2-ba00-c23405a6bcf2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "35a027f721ba6f28e842a4430ab564d5367d3841a944ae1c08a1cc70c1fd6aab", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e59eadab-d8e9-4888-87d7-3a28b3c14fa7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "0fbdc8a6cbab79a7d916c74af058cd76491ccfd528046416cd75f7d6bbf102d2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "acbfb4a1-cb11-49de-afd7-b6c77c02adb1", "node_type": "1", "metadata": {}, "hash": "571a5955e7443e30e668be4a9be947c18d75d86dee8f72d6952262232532546f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3284, "end_char_idx": 6448, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Step 1: Setup the environment

There are 2 pre-requisites for this:

1.   **MongoDB Atlas cluster**: To create a forever free MongoDB Atlas cluster, first, you need to create a MongoDB Atlas account if you don't already have one. Visit the [MongoDB Atlas website](https://www.mongodb.com/atlas/database) and click on “Register.” Visit the [MongoDB Atlas](https://account.mongodb.com/account/login) dashboard and set up your cluster. In order to take advantage of the `$vectorSearch` operator in an aggregation pipeline, you need to run MongoDB Atlas 6.0.11 or higher. This tutorial can be built using a free cluster. When you’re setting up your deployment, you’ll be prompted to set up a database user and rules for your network connection. Please ensure you save your username and password somewhere safe and have the correct IP address rules in place so your cluster can connect properly. If you need more help getting started, check out our [tutorial on MongoDB Atlas](https://www.mongodb.com/basics/mongodb-atlas-tutorial).

2. **OpenAI API key**   To create your OpenAI key, you'll need to create an account. Once you have that, visit the [OpenAI platform](https://platform.openai.com/). Click on your profile icon in the top right of the screen to get the dropdown menu and select “View API keys”.



```python
import getpass

MONGODB_ATLAS_CLUSTER_URI = getpass.getpass("MongoDB Atlas Cluster URI:")
OPENAI_API_KEY = getpass.getpass("OpenAI API Key:")

```

    MongoDB Atlas Cluster URI:··········
    OpenAI API Key:··········


Note: After executing the step above you will be prompted to enter the credentials.

For this tutorial, we will be using the
[MongoDB sample dataset](https://www.mongodb.com/docs/atlas/sample-data/). Load the sample dataset using the Atlas UI. We'll be using the “sample_mflix” database, which contains a “movies” collection where each document contains fields like title, plot, genres, cast, directors, etc.



```python
import openai
import pymongo

client = pymongo.MongoClient(MONGODB_ATLAS_CLUSTER_URI)
db = client.sample_mflix
collection = db.movies

openai.api_key = OPENAI_API_KEY
```


```python
ATLAS_VECTOR_SEARCH_INDEX_NAME = "default"
EMBEDDING_FIELD_NAME = "embedding_openai_nov19_23"
```

# Step 2: Setup embeddings generation function


```python
model = "text-embedding-3-small"
def generate_embedding(text: str) -> list[float]:
    return openai.embeddings.create(input = [text], model=model).data[0].embedding

```

# Step 3: Create and store embeddings

Each document in the sample dataset sample_mflix.movies corresponds to a movie; we will execute an operation to create a vector embedding for the data in the "plot" field and store it in the database. Creating vector embeddings using OpenAI embeddings endpoint is necessary for performing a similarity search based on intent.


```python
from pymongo import ReplaceOne

# Update the collection with the embeddings
requests = []

for doc in collection.find({'plot':{"$exists": True}}).limit(500):
  doc[EMBEDDING_FIELD_NAME] = generate_embedding(doc['plot'])
  requests.append(ReplaceOne({'_id': doc['_id']}, doc))

collection.bulk_write(requests)
```


key: _node_content
string_value:
{"id_": "acbfb4a1-cb11-49de-afd7-b6c77c02adb1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c9d10ca-b404-49d2-ba00-c23405a6bcf2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "35a027f721ba6f28e842a4430ab564d5367d3841a944ae1c08a1cc70c1fd6aab", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f72e2525-f9bc-4dac-9dda-a563e69d2533", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "3053abf07b55af7489ff0c4c71c6856afed11e61a5a041a855f42e0ea143938e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "27559451-944b-4aaa-ba4e-81fc5e1869f5", "node_type": "1", "metadata": {}, "hash": "3a34d797ad92aaceb42b14a8efcfb4685b5ab57ac3a23603b92b063227964f51", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4824, "end_char_idx": 7459, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Note: After executing the step above you will be prompted to enter the credentials.

For this tutorial, we will be using the
[MongoDB sample dataset](https://www.mongodb.com/docs/atlas/sample-data/). Load the sample dataset using the Atlas UI. We'll be using the “sample_mflix” database, which contains a “movies” collection where each document contains fields like title, plot, genres, cast, directors, etc.



```python
import openai
import pymongo

client = pymongo.MongoClient(MONGODB_ATLAS_CLUSTER_URI)
db = client.sample_mflix
collection = db.movies

openai.api_key = OPENAI_API_KEY
```


```python
ATLAS_VECTOR_SEARCH_INDEX_NAME = "default"
EMBEDDING_FIELD_NAME = "embedding_openai_nov19_23"
```

# Step 2: Setup embeddings generation function


```python
model = "text-embedding-3-small"
def generate_embedding(text: str) -> list[float]:
    return openai.embeddings.create(input = [text], model=model).data[0].embedding

```

# Step 3: Create and store embeddings

Each document in the sample dataset sample_mflix.movies corresponds to a movie; we will execute an operation to create a vector embedding for the data in the "plot" field and store it in the database. Creating vector embeddings using OpenAI embeddings endpoint is necessary for performing a similarity search based on intent.


```python
from pymongo import ReplaceOne

# Update the collection with the embeddings
requests = []

for doc in collection.find({'plot':{"$exists": True}}).limit(500):
  doc[EMBEDDING_FIELD_NAME] = generate_embedding(doc['plot'])
  requests.append(ReplaceOne({'_id': doc['_id']}, doc))

collection.bulk_write(requests)
```




    BulkWriteResult({'writeErrors': [], 'writeConcernErrors': [], 'nInserted': 0, 'nUpserted': 0, 'nMatched': 50, 'nModified': 50, 'nRemoved': 0, 'upserted': []}, acknowledged=True)



After executing the above, the documents in "movies" collection will contain an additional field of "embedding", as defined by the `EMBEDDDING_FIELD_NAME` variable, apart from already existing fields like title, plot, genres, cast, directors, etc.

Note: We are restricting this to just 500 documents in the interest of time. If you want to do this over the entire dataset of 23,000+ documents in our sample_mflix database, it will take a little while. Alternatively, you can use the [sample_mflix.embedded_movies collection](https://www.mongodb.com/docs/atlas/sample-data/sample-mflix/#sample_mflix.embedded_movies) which includes a pre-populated `plot_embedding` field that contains embeddings created using OpenAI's `text-embedding-3-small` embedding model that you can use with the Atlas Search vector search feature.


key: _node_content
string_value:
{"id_": "27559451-944b-4aaa-ba4e-81fc5e1869f5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c9d10ca-b404-49d2-ba00-c23405a6bcf2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "35a027f721ba6f28e842a4430ab564d5367d3841a944ae1c08a1cc70c1fd6aab", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "acbfb4a1-cb11-49de-afd7-b6c77c02adb1", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "cc4251b3057dbfb1892b8d2a8061cefa94f8b41c56986791d7bd20ca534dacd2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9afdd4f6-be8a-4ecd-bf38-3922d0539bf0", "node_type": "1", "metadata": {}, "hash": "4e17cf03984c26731caab4e18a9b46fef2c7c804125d480a06f5b764ad35110a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6126, "end_char_idx": 9246, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from pymongo import ReplaceOne

# Update the collection with the embeddings
requests = []

for doc in collection.find({'plot':{"$exists": True}}).limit(500):
  doc[EMBEDDING_FIELD_NAME] = generate_embedding(doc['plot'])
  requests.append(ReplaceOne({'_id': doc['_id']}, doc))

collection.bulk_write(requests)
```




    BulkWriteResult({'writeErrors': [], 'writeConcernErrors': [], 'nInserted': 0, 'nUpserted': 0, 'nMatched': 50, 'nModified': 50, 'nRemoved': 0, 'upserted': []}, acknowledged=True)



After executing the above, the documents in "movies" collection will contain an additional field of "embedding", as defined by the `EMBEDDDING_FIELD_NAME` variable, apart from already existing fields like title, plot, genres, cast, directors, etc.

Note: We are restricting this to just 500 documents in the interest of time. If you want to do this over the entire dataset of 23,000+ documents in our sample_mflix database, it will take a little while. Alternatively, you can use the [sample_mflix.embedded_movies collection](https://www.mongodb.com/docs/atlas/sample-data/sample-mflix/#sample_mflix.embedded_movies) which includes a pre-populated `plot_embedding` field that contains embeddings created using OpenAI's `text-embedding-3-small` embedding model that you can use with the Atlas Search vector search feature.




# Step 4: Create a vector search index

We will create Atlas Vector Search Index on this collection which will allow us to perform the Approximate KNN search, which powers the semantic search.
We will cover 2 ways to create this index - Atlas UI and using MongoDB python driver.

(Optional) [Documentation: Create a Vector Search Index ](https://www.mongodb.com/docs/atlas/atlas-search/field-types/knn-vector/)

Now head over to [Atlas UI](cloud.mongodb.com) and create an Atlas Vector Search index using the steps descibed [here](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-tutorial/#create-the-atlas-vector-search-index).  The 'dimensions' field with value 1536, corresponds to openAI text-embedding-ada002.

Use the definition given below in the JSON editor on the Atlas UI.

```
{
  "mappings": {
    "dynamic": true,
    "fields": {
      "embedding": {
        "dimensions": 1536,
        "similarity": "dotProduct",
        "type": "knnVector"
      }
    }
  }
}
```

(Optional) Alternatively, we can use [pymongo driver to create these vector search indexes programatically](https://pymongo.readthedocs.io/en/stable/api/pymongo/collection.html#pymongo.collection.Collection.create_search_index)
The python command given in the cell below will create the index (this only works for the most recent version of the Python Driver for MongoDB and MongoDB server version 7.0+ Atlas cluster).


```python
collection.create_search_index(
    {"definition":
        {"mappings": {"dynamic": True, "fields": {
            EMBEDDING_FIELD_NAME : {
                "dimensions": 1536,
                "similarity": "dotProduct",
                "type": "knnVector"
                }}}},
     "name": ATLAS_VECTOR_SEARCH_INDEX_NAME
    }
)
```




    'default'


key: _node_content
string_value:
{"id_": "9afdd4f6-be8a-4ecd-bf38-3922d0539bf0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c9d10ca-b404-49d2-ba00-c23405a6bcf2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "35a027f721ba6f28e842a4430ab564d5367d3841a944ae1c08a1cc70c1fd6aab", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "27559451-944b-4aaa-ba4e-81fc5e1869f5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "011c782e421f1adc3a757a52f32f49d58d4f005d7f9bcc5e007e45535ca31721", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8886, "end_char_idx": 10153, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
collection.create_search_index(
    {"definition":
        {"mappings": {"dynamic": True, "fields": {
            EMBEDDING_FIELD_NAME : {
                "dimensions": 1536,
                "similarity": "dotProduct",
                "type": "knnVector"
                }}}},
     "name": ATLAS_VECTOR_SEARCH_INDEX_NAME
    }
)
```




    'default'



# Step 5: Query your data

The results for the query here finds movies which have semantically similar plots to the text captured in the query string, rather than being based on the keyword search.

(Optional) [Documentation: Run Vector Search Queries](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-stage/)


```python

def query_results(query, k):
  results = collection.aggregate([
    {
        '$vectorSearch': {
            "index": ATLAS_VECTOR_SEARCH_INDEX_NAME,
            "path": EMBEDDING_FIELD_NAME,
            "queryVector": generate_embedding(query),
            "numCandidates": 50,
            "limit": 5,
        }
    }
    ])
  return results
```


```python
query="imaginary characters from outerspace at war with earthlings"
movies = query_results(query, 5)

for movie in movies:
    print(f'Movie Name: {movie["title"]},\nMovie Plot: {movie["plot"]}\n')
```


key: _node_content
string_value:
{"id_": "7f81363e-1f53-4894-84bf-d97cee7ce0ed", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/README.ipynb", "title": "Supabase Vector Database", "tokens": 485, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6cfced8e-3f60-449e-b468-9e436dd69458", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/README.ipynb", "title": "Supabase Vector Database", "tokens": 485, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "8a45d3ce9e8964ade23c6b62cc75653c34b34e6569f84c6bd2d2e105893459fd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2108, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Supabase Vector Database

[Supabase](https://supabase.com/docs) is an open-source Firebase alternative built on top of [Postgres](https://en.wikipedia.org/wiki/PostgreSQL), a production-grade SQL database.

[Supabase Vector](https://supabase.com/docs/guides/ai) is a vector toolkit built on [pgvector](https://github.com/pgvector/pgvector), a Postgres extension that allows you to store your embeddings inside the same database that holds the rest of your application data. When combined with pgvector's indexing algorithms, vector search remains [fast at large scales](https://supabase.com/blog/increase-performance-pgvector-hnsw).

Supabase adds an ecosystem of services and tools on top of Postgres that makes app development as quick as possible, including:

- [Auto-generated REST APIs](https://supabase.com/docs/guides/api)
- [Auto-generated GraphQL APIs](https://supabase.com/docs/guides/graphql)
- [Realtime APIs](https://supabase.com/docs/guides/realtime)
- [Authentication](https://supabase.com/docs/guides/auth)
- [File storage](https://supabase.com/docs/guides/storage)
- [Edge functions](https://supabase.com/docs/guides/functions)

We can use these services alongside pgvector to store and query embeddings within Postgres.

## OpenAI Cookbook Examples

Below are guides and resources that walk you through how to use OpenAI embedding models with Supabase Vector.

| Guide                                    | Description                                                |
| ---------------------------------------- | ---------------------------------------------------------- |
| [Semantic search](./semantic-search.mdx) | Store, index, and query embeddings at scale using pgvector |

## Additional resources

- [Vector columns](https://supabase.com/docs/guides/ai/vector-columns)
- [Vector indexes](https://supabase.com/docs/guides/ai/vector-indexes)
- [RAG with permissions](https://supabase.com/docs/guides/ai/rag-with-permissions)
- [Going to production](https://supabase.com/docs/guides/ai/going-to-prod)
- [Deciding on compute](https://supabase.com/docs/guides/ai/choosing-compute-addon)


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb


key: _node_content
string_value:
{"id_": "d269ab3e-b7eb-41f9-bb63-29c8460472d4", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb", "title": "Semantic search using Supabase Vector", "tokens": 2931, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fff1478c-9413-4b4e-aee4-2674d1edf367", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb", "title": "Semantic search using Supabase Vector", "tokens": 2931, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "73d7f5adc2bd965f762d24d404f52abe9ec49f9b7d25dd223b81da3db8cbb8f9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c5529901-c872-4205-ae67-91ca1dd66269", "node_type": "1", "metadata": {}, "hash": "274e5dfc41f35f0b36786cb64a0a123a242ab8e22fbc13569da7e0aa96d3a065", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3306, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Semantic search using Supabase Vector

The purpose of this guide is to demonstrate how to store OpenAI embeddings in [Supabase Vector](https://supabase.com/docs/guides/ai) (Postgres + pgvector) for the purposes of semantic search.

[Supabase](https://supabase.com/docs) is an open-source Firebase alternative built on top of [Postgres](https://en.wikipedia.org/wiki/PostgreSQL), a production-grade SQL database. Since Supabase Vector is built on [pgvector](https://github.com/pgvector/pgvector), you can store your embeddings within the same database that holds the rest of your application data. When combined with pgvector's indexing algorithms, vector search remains [fast at large scales](https://supabase.com/blog/increase-performance-pgvector-hnsw).

Supabase adds an ecosystem of services and tools to make app development as quick as possible (such as an [auto-generated REST API](https://postgrest.org/)). We'll use these services to store and query embeddings within Postgres.

This guide covers:

1. [Setting up your database](#setup-database)
2. [Creating a SQL table](#create-a-vector-table) that can store vector data
3. [Generating OpenAI embeddings](#generate-openai-embeddings) using OpenAI's JavaScript client
4. [Storing the embeddings](#store-embeddings-in-database) in your SQL table using the Supabase JavaScript client
5. [Performing semantic search](#semantic-search) over the embeddings using a Postgres function and the Supabase JavaScript client

## Setup database

First head over to https://database.new to provision your Supabase database. This will create a Postgres database on the Supabase cloud platform. Alternatively, you can follow the [local development](https://supabase.com/docs/guides/cli/getting-started) options if you prefer to run your database locally using Docker.

In the studio, jump to the [SQL editor](https://supabase.com/dashboard/project/_/sql/new) and execute the following SQL to enable pgvector:

```sql
-- Enable the pgvector extension
create extension if not exists vector;
```

> In a production application, the best practice is to use [database migrations](https://supabase.com/docs/guides/cli/local-development#database-migrations) so that all SQL operations are managed within source control. To keep things simple in this guide, we'll execute queries directly in the SQL Editor. If you are building a production app, feel free to move these into a database migration.

## Create a vector table

Next we'll create a table to store documents and embeddings. In the SQL Editor, run:

```sql
create table documents (
  id bigint primary key generated always as identity,
  content text not null,
  embedding vector (1536) not null
);
```

Since Supabase is built on Postgres, we're just using regular SQL here. You can modify this table however you like to better fit your application. If you have existing database tables, you can simply add a new `vector` column to the appropriate table.

The important piece to understand is the `vector` data type, which is a new data type that became available when we enabled the pgvector extension earlier. The size of the vector (1536 here) represents the number of dimensions in the embedding. Since we're using OpenAI's `text-embedding-3-small` model in this example, we set the vector size to 1536.


key: _node_content
string_value:
{"id_": "c5529901-c872-4205-ae67-91ca1dd66269", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb", "title": "Semantic search using Supabase Vector", "tokens": 2931, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fff1478c-9413-4b4e-aee4-2674d1edf367", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb", "title": "Semantic search using Supabase Vector", "tokens": 2931, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "73d7f5adc2bd965f762d24d404f52abe9ec49f9b7d25dd223b81da3db8cbb8f9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d269ab3e-b7eb-41f9-bb63-29c8460472d4", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb", "title": "Semantic search using Supabase Vector", "tokens": 2931, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "bbcb99d53cf9878278f7be17db754abadcff412c42f9aa6ca437087214625d22", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d6e54d88-eceb-4020-82cb-0c947848e5db", "node_type": "1", "metadata": {}, "hash": "7d979b28daafe7b9225568861afd644a52d9f7d3e724cdb07ed822d282d9efdc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1572, "end_char_idx": 4865, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This will create a Postgres database on the Supabase cloud platform. Alternatively, you can follow the [local development](https://supabase.com/docs/guides/cli/getting-started) options if you prefer to run your database locally using Docker.

In the studio, jump to the [SQL editor](https://supabase.com/dashboard/project/_/sql/new) and execute the following SQL to enable pgvector:

```sql
-- Enable the pgvector extension
create extension if not exists vector;
```

> In a production application, the best practice is to use [database migrations](https://supabase.com/docs/guides/cli/local-development#database-migrations) so that all SQL operations are managed within source control. To keep things simple in this guide, we'll execute queries directly in the SQL Editor. If you are building a production app, feel free to move these into a database migration.

## Create a vector table

Next we'll create a table to store documents and embeddings. In the SQL Editor, run:

```sql
create table documents (
  id bigint primary key generated always as identity,
  content text not null,
  embedding vector (1536) not null
);
```

Since Supabase is built on Postgres, we're just using regular SQL here. You can modify this table however you like to better fit your application. If you have existing database tables, you can simply add a new `vector` column to the appropriate table.

The important piece to understand is the `vector` data type, which is a new data type that became available when we enabled the pgvector extension earlier. The size of the vector (1536 here) represents the number of dimensions in the embedding. Since we're using OpenAI's `text-embedding-3-small` model in this example, we set the vector size to 1536.

Let's go ahead and create a vector index on this table so that future queries remain performant as the table grows:

```sql
create index on documents using hnsw (embedding vector_ip_ops);
```

This index uses the [HNSW](https://supabase.com/docs/guides/ai/vector-indexes/hnsw-indexes) algorithm to index vectors stored in the `embedding` column, and specifically when using the inner product operator (`<#>`). We'll explain more about this operator later when we implement our match function.

Let's also follow security best practices by enabling row level security on the table:

```sql
alter table documents enable row level security;
```

This will prevent unauthorized access to this table through the auto-generated REST API (more on this shortly).

## Generate OpenAI embeddings

This guide uses JavaScript to generate embeddings, but you can easily modify it to use any [language supported by OpenAI](https://platform.openai.com/docs/libraries).

If you are using JavaScript, feel free to use whichever server-side JavaScript runtime that you prefer (Node.js, Deno, Supabase Edge Functions).

If you're using Node.js, first install `openai` as a dependency:

```shell
npm install openai
```

then import it:

```js
import OpenAI from "openai";
```

If you're using Deno or Supabase Edge Functions, you can import `openai` directly from a URL:

```js
import OpenAI from "https://esm.sh/openai@4";
```

> In this example we import from https://esm.sh which is a CDN that automatically fetches the respective NPM module for you and serves it over HTTP.


key: _node_content
string_value:
{"id_": "d6e54d88-eceb-4020-82cb-0c947848e5db", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb", "title": "Semantic search using Supabase Vector", "tokens": 2931, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fff1478c-9413-4b4e-aee4-2674d1edf367", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb", "title": "Semantic search using Supabase Vector", "tokens": 2931, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "73d7f5adc2bd965f762d24d404f52abe9ec49f9b7d25dd223b81da3db8cbb8f9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c5529901-c872-4205-ae67-91ca1dd66269", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb", "title": "Semantic search using Supabase Vector", "tokens": 2931, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "b44c6973e9272a0fe54bf542f3b4839dffb6eac0d0c19b74fcf920fea394455d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "16a8421b-17c4-49b3-8018-a1b6a9bf2ccf", "node_type": "1", "metadata": {}, "hash": "ff459c51c38b88aa433a1a0d170e2e0051640b9e3066ad146bb8f12b733bf3e3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3200, "end_char_idx": 6407, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Since we're using OpenAI's `text-embedding-3-small` model in this example, we set the vector size to 1536.

Let's go ahead and create a vector index on this table so that future queries remain performant as the table grows:

```sql
create index on documents using hnsw (embedding vector_ip_ops);
```

This index uses the [HNSW](https://supabase.com/docs/guides/ai/vector-indexes/hnsw-indexes) algorithm to index vectors stored in the `embedding` column, and specifically when using the inner product operator (`<#>`). We'll explain more about this operator later when we implement our match function.

Let's also follow security best practices by enabling row level security on the table:

```sql
alter table documents enable row level security;
```

This will prevent unauthorized access to this table through the auto-generated REST API (more on this shortly).

## Generate OpenAI embeddings

This guide uses JavaScript to generate embeddings, but you can easily modify it to use any [language supported by OpenAI](https://platform.openai.com/docs/libraries).

If you are using JavaScript, feel free to use whichever server-side JavaScript runtime that you prefer (Node.js, Deno, Supabase Edge Functions).

If you're using Node.js, first install `openai` as a dependency:

```shell
npm install openai
```

then import it:

```js
import OpenAI from "openai";
```

If you're using Deno or Supabase Edge Functions, you can import `openai` directly from a URL:

```js
import OpenAI from "https://esm.sh/openai@4";
```

> In this example we import from https://esm.sh which is a CDN that automatically fetches the respective NPM module for you and serves it over HTTP.

Next we'll generate an OpenAI embedding using [`text-embedding-3-small`](https://platform.openai.com/docs/guides/embeddings/embedding-models):

```js
const openai = new OpenAI();

const input = "The cat chases the mouse";

const result = await openai.embeddings.create({
  input,
  model: "text-embedding-3-small",
});

const [{ embedding }] = result.data;
```

Remember that you will need an [OpenAI API key](https://platform.openai.com/api-keys) to interact with the OpenAI API. You can pass this as an environment variable called `OPENAI_API_KEY`, or manually set it when you instantiate your OpenAI client:

```js
const openai = new OpenAI({
  apiKey: "<openai-api-key>",
});
```

_**Remember:** Never hard-code API keys in your code. Best practice is to either store it in a `.env` file and load it using a library like [`dotenv`](https://github.com/motdotla/dotenv) or load it from an external key management system._

## Store embeddings in database

Supabase comes with an [auto-generated REST API](https://postgrest.org/) that dynamically builds REST endpoints for each of your tables. This means you don't need to establish a direct Postgres connection to your database - instead you can interact with it simply using by the REST API. This is especially useful in serverless environments that run short-lived processes where re-establishing a database connection every time can be expensive.

Supabase comes with a number of [client libraries](https://supabase.com/docs#client-libraries) to simplify interaction with the REST API.


key: _node_content
string_value:
{"id_": "16a8421b-17c4-49b3-8018-a1b6a9bf2ccf", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb", "title": "Semantic search using Supabase Vector", "tokens": 2931, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fff1478c-9413-4b4e-aee4-2674d1edf367", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb", "title": "Semantic search using Supabase Vector", "tokens": 2931, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "73d7f5adc2bd965f762d24d404f52abe9ec49f9b7d25dd223b81da3db8cbb8f9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d6e54d88-eceb-4020-82cb-0c947848e5db", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb", "title": "Semantic search using Supabase Vector", "tokens": 2931, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "12e59e36a07c193413a13a813b70802c3e26c38d603ab878d7c67f76393ca70f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6bc785f1-d31f-4605-a2c0-71e452c7226e", "node_type": "1", "metadata": {}, "hash": "b2b3d19b608d28836edc44bd2c70ef99c4ecca42d57a063ce33031a3f9f7e33e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4867, "end_char_idx": 7909, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Next we'll generate an OpenAI embedding using [`text-embedding-3-small`](https://platform.openai.com/docs/guides/embeddings/embedding-models):

```js
const openai = new OpenAI();

const input = "The cat chases the mouse";

const result = await openai.embeddings.create({
  input,
  model: "text-embedding-3-small",
});

const [{ embedding }] = result.data;
```

Remember that you will need an [OpenAI API key](https://platform.openai.com/api-keys) to interact with the OpenAI API. You can pass this as an environment variable called `OPENAI_API_KEY`, or manually set it when you instantiate your OpenAI client:

```js
const openai = new OpenAI({
  apiKey: "<openai-api-key>",
});
```

_**Remember:** Never hard-code API keys in your code. Best practice is to either store it in a `.env` file and load it using a library like [`dotenv`](https://github.com/motdotla/dotenv) or load it from an external key management system._

## Store embeddings in database

Supabase comes with an [auto-generated REST API](https://postgrest.org/) that dynamically builds REST endpoints for each of your tables. This means you don't need to establish a direct Postgres connection to your database - instead you can interact with it simply using by the REST API. This is especially useful in serverless environments that run short-lived processes where re-establishing a database connection every time can be expensive.

Supabase comes with a number of [client libraries](https://supabase.com/docs#client-libraries) to simplify interaction with the REST API. In this guide we'll use the [JavaScript client library](https://supabase.com/docs/reference/javascript), but feel free to adjust this to your preferred language.

If you're using Node.js, install `@supabase/supabase-js` as a dependency:

```shell
npm install @supabase/supabase-js
```

then import it:

```js
import { createClient } from "@supabase/supabase-js";
```

If you're using Deno or Supabase Edge Functions, you can import `@supabase/supabase-js` directly from a URL:

```js
import { createClient } from "https://esm.sh/@supabase/supabase-js@2";
```

Next we'll instantiate our Supabase client and configure it so that it points to your Supabase project. In this guide we'll store a reference to your Supabase URL and key in a `.env` file, but feel free to modify this based on how your application handles configuration.

If you are using Node.js or Deno, add your Supabase URL and service role key to a `.env` file. If you are using the cloud platform, you can find these from your Supabase dashboard [settings page](https://supabase.com/dashboard/project/_/settings/api). If you're running Supabase locally, you can find these by running `npx supabase status` in a terminal.

_.env_

```
SUPABASE_URL=<supabase-url>
SUPABASE_SERVICE_ROLE_KEY=<supabase-service-role-key>
```

If you are using Supabase Edge Functions, these environment variables are automatically injected into your function for you so you can skip the above step.

Next we'll pull these environment variables into our app.


key: _node_content
string_value:
{"id_": "6bc785f1-d31f-4605-a2c0-71e452c7226e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb", "title": "Semantic search using Supabase Vector", "tokens": 2931, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fff1478c-9413-4b4e-aee4-2674d1edf367", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb", "title": "Semantic search using Supabase Vector", "tokens": 2931, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "73d7f5adc2bd965f762d24d404f52abe9ec49f9b7d25dd223b81da3db8cbb8f9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "16a8421b-17c4-49b3-8018-a1b6a9bf2ccf", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb", "title": "Semantic search using Supabase Vector", "tokens": 2931, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "480b96a40b69b6327ec8e4affd557236fa3905a076ea6c2341e326dcdad49e82", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bb7e1eca-2ce7-42f3-904d-aedbedac6bbc", "node_type": "1", "metadata": {}, "hash": "36e8d4af2824e722835469176fdaee287ab8add1530d639a1b941a27f999b524", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6408, "end_char_idx": 9395, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In this guide we'll use the [JavaScript client library](https://supabase.com/docs/reference/javascript), but feel free to adjust this to your preferred language.

If you're using Node.js, install `@supabase/supabase-js` as a dependency:

```shell
npm install @supabase/supabase-js
```

then import it:

```js
import { createClient } from "@supabase/supabase-js";
```

If you're using Deno or Supabase Edge Functions, you can import `@supabase/supabase-js` directly from a URL:

```js
import { createClient } from "https://esm.sh/@supabase/supabase-js@2";
```

Next we'll instantiate our Supabase client and configure it so that it points to your Supabase project. In this guide we'll store a reference to your Supabase URL and key in a `.env` file, but feel free to modify this based on how your application handles configuration.

If you are using Node.js or Deno, add your Supabase URL and service role key to a `.env` file. If you are using the cloud platform, you can find these from your Supabase dashboard [settings page](https://supabase.com/dashboard/project/_/settings/api). If you're running Supabase locally, you can find these by running `npx supabase status` in a terminal.

_.env_

```
SUPABASE_URL=<supabase-url>
SUPABASE_SERVICE_ROLE_KEY=<supabase-service-role-key>
```

If you are using Supabase Edge Functions, these environment variables are automatically injected into your function for you so you can skip the above step.

Next we'll pull these environment variables into our app.

In Node.js, install the `dotenv` dependency:

```shell
npm install dotenv
```

And retrieve the environment variables from `process.env`:

```js
import { config } from "dotenv";

// Load .env file
config();

const supabaseUrl = process.env["SUPABASE_URL"];
const supabaseServiceRoleKey = process.env["SUPABASE_SERVICE_ROLE_KEY"];
```

In Deno, load the `.env` file using the `dotenv` standard library:

```js
import { load } from "https://deno.land/std@0.208.0/dotenv/mod.ts";

// Load .env file
const env = await load();

const supabaseUrl = env["SUPABASE_URL"];
const supabaseServiceRoleKey = env["SUPABASE_SERVICE_ROLE_KEY"];
```

In Supabase Edge Functions, simply load the injected environment variables directly:

```js
const supabaseUrl = Deno.env.get("SUPABASE_URL");
const supabaseServiceRoleKey = Deno.env.get("SUPABASE_SERVICE_ROLE_KEY");
```

Next let's instantiate our `supabase` client:

```js
const supabase = createClient(supabaseUrl, supabaseServiceRoleKey, {
  auth: { persistSession: false },
});
```

From here we use the `supabase` client to insert our text and embedding (generated earlier) into the database:

```js
const { error } = await supabase.from("documents").insert({
  content: input,
  embedding,
});
```

> In production, best practice would be to check the response `error` to see if there were any problems inserting the data and handle it accordingly.

## Semantic search

Finally let's perform semantic search over the embeddings in our database.


key: _node_content
string_value:
{"id_": "bb7e1eca-2ce7-42f3-904d-aedbedac6bbc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb", "title": "Semantic search using Supabase Vector", "tokens": 2931, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fff1478c-9413-4b4e-aee4-2674d1edf367", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb", "title": "Semantic search using Supabase Vector", "tokens": 2931, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "73d7f5adc2bd965f762d24d404f52abe9ec49f9b7d25dd223b81da3db8cbb8f9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6bc785f1-d31f-4605-a2c0-71e452c7226e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb", "title": "Semantic search using Supabase Vector", "tokens": 2931, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "f35f33810a189d223f7e9c21bb86fa908bfd93a43d5999567c253189c7b461c8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "53b0db58-996e-4bb6-908c-8b9765dd15a5", "node_type": "1", "metadata": {}, "hash": "25964d42f2a524bd77af5e8dabc4dd823705662b6c097df70cc3d4ae88031dbb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7852, "end_char_idx": 11139, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Next we'll pull these environment variables into our app.

In Node.js, install the `dotenv` dependency:

```shell
npm install dotenv
```

And retrieve the environment variables from `process.env`:

```js
import { config } from "dotenv";

// Load .env file
config();

const supabaseUrl = process.env["SUPABASE_URL"];
const supabaseServiceRoleKey = process.env["SUPABASE_SERVICE_ROLE_KEY"];
```

In Deno, load the `.env` file using the `dotenv` standard library:

```js
import { load } from "https://deno.land/std@0.208.0/dotenv/mod.ts";

// Load .env file
const env = await load();

const supabaseUrl = env["SUPABASE_URL"];
const supabaseServiceRoleKey = env["SUPABASE_SERVICE_ROLE_KEY"];
```

In Supabase Edge Functions, simply load the injected environment variables directly:

```js
const supabaseUrl = Deno.env.get("SUPABASE_URL");
const supabaseServiceRoleKey = Deno.env.get("SUPABASE_SERVICE_ROLE_KEY");
```

Next let's instantiate our `supabase` client:

```js
const supabase = createClient(supabaseUrl, supabaseServiceRoleKey, {
  auth: { persistSession: false },
});
```

From here we use the `supabase` client to insert our text and embedding (generated earlier) into the database:

```js
const { error } = await supabase.from("documents").insert({
  content: input,
  embedding,
});
```

> In production, best practice would be to check the response `error` to see if there were any problems inserting the data and handle it accordingly.

## Semantic search

Finally let's perform semantic search over the embeddings in our database. At this point we'll assume your `documents` table has been filled with multiple records that we can search over.

Let's create a match function in Postgres that performs the semantic search query. Execute the following in the [SQL Editor](https://supabase.com/dashboard/project/_/sql/new):

```sql
create function match_documents (
  query_embedding vector (1536),
  match_threshold float,
)
returns setof documents
language plpgsql
as $$
begin
  return query
  select *
  from documents
  where documents.embedding <#> query_embedding < -match_threshold
  order by documents.embedding <#> query_embedding;
end;
$$;
```

This function accepts a `query_embedding` which represents the embedding generated from the search query text (more on this shortly). It also accepts a `match_threshold` which specifies how similar the document embeddings have to be in order for `query_embedding` to count as a match.

Inside the function we implement the query which does two things:

- Filters the documents to only include those who's embeddings match within the above `match_threshold`. Since the `<#>` operator performs the negative inner product (versus positive inner product), we negate the similarity threshold before comparing. This means a `match_threshold` of 1 is most similar, and -1 is most dissimilar.
- Orders the documents by negative inner product (`<#>`) ascending. This allows us to retrieve documents that match closest first.

> Since OpenAI embeddings are normalized, we opted to use inner product (`<#>`) because it is slightly more performant than other operators like cosine distance (`<=>`). It is important to note though this only works because the embeddings are normalized - if they weren't, cosine distance should be used.


key: _node_content
string_value:
{"id_": "53b0db58-996e-4bb6-908c-8b9765dd15a5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb", "title": "Semantic search using Supabase Vector", "tokens": 2931, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "fff1478c-9413-4b4e-aee4-2674d1edf367", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb", "title": "Semantic search using Supabase Vector", "tokens": 2931, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "73d7f5adc2bd965f762d24d404f52abe9ec49f9b7d25dd223b81da3db8cbb8f9", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bb7e1eca-2ce7-42f3-904d-aedbedac6bbc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb", "title": "Semantic search using Supabase Vector", "tokens": 2931, "retrieve_doc": true, "source": "OpenAI_Cookbooks"}, "hash": "ba3f4829d6db0d26d4e6d24ee3314638c81a84c5a9b6e40e25c6c62dbc235025", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9301, "end_char_idx": 12701, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Semantic search

Finally let's perform semantic search over the embeddings in our database. At this point we'll assume your `documents` table has been filled with multiple records that we can search over.

Let's create a match function in Postgres that performs the semantic search query. Execute the following in the [SQL Editor](https://supabase.com/dashboard/project/_/sql/new):

```sql
create function match_documents (
  query_embedding vector (1536),
  match_threshold float,
)
returns setof documents
language plpgsql
as $$
begin
  return query
  select *
  from documents
  where documents.embedding <#> query_embedding < -match_threshold
  order by documents.embedding <#> query_embedding;
end;
$$;
```

This function accepts a `query_embedding` which represents the embedding generated from the search query text (more on this shortly). It also accepts a `match_threshold` which specifies how similar the document embeddings have to be in order for `query_embedding` to count as a match.

Inside the function we implement the query which does two things:

- Filters the documents to only include those who's embeddings match within the above `match_threshold`. Since the `<#>` operator performs the negative inner product (versus positive inner product), we negate the similarity threshold before comparing. This means a `match_threshold` of 1 is most similar, and -1 is most dissimilar.
- Orders the documents by negative inner product (`<#>`) ascending. This allows us to retrieve documents that match closest first.

> Since OpenAI embeddings are normalized, we opted to use inner product (`<#>`) because it is slightly more performant than other operators like cosine distance (`<=>`). It is important to note though this only works because the embeddings are normalized - if they weren't, cosine distance should be used.

Now we can call this function from our application using the `supabase.rpc()` method:

```js
const query = "What does the cat chase?";

// First create an embedding on the query itself
const result = await openai.embeddings.create({
  input: query,
  model: "text-embedding-3-small",
});

const [{ embedding }] = result.data;

// Then use this embedding to search for matches
const { data: documents, error: matchError } = await supabase
  .rpc("match_documents", {
    query_embedding: embedding,
    match_threshold: 0.8,
  })
  .select("content")
  .limit(5);
```

In this example, we set a match threshold to 0.8. Adjust this threshold based on what works best with your data.

Note that since `match_documents` returns a set of `documents`, we can treat this `rpc()` like a regular table query. Specifically this means we can chain additional commands to this query, like `select()` and `limit()`. Here we select just the columns we care about from the `documents` table (`content`), and we limit the number of documents returned (max 5 in this example).

At this point you have a list of documents that matched the query based on semantic relationship, ordered by most similar first.

## Next steps

You can use this example as the foundation for other semantic search techniques, like retrieval augmented generation (RAG).

For more information on OpenAI embeddings, read the [Embedding](https://platform.openai.com/docs/guides/embeddings) docs.

For more information on Supabase Vector, read the [AI & Vector](https://supabase.com/docs/guides/ai) docs.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb


key: _node_content
string_value:
{"id_": "46cc518e-757e-41d6-940d-24508b227a86", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "daeab515-6b4f-40c7-a1a5-a97592b75399", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d7cfae8881809a7e19fe70260803d9ad55217e7fdcfba2e97c643bf2b4919e8e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3554d58c-40aa-479c-b996-fc2a9297e268", "node_type": "1", "metadata": {}, "hash": "52e024ab81bc7502195e48b614ce04b5672bb626ad4eed7affbc668a75d19ef1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1346, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Semantic Search with Pinecone and OpenAI

In this guide you will learn how to use the OpenAI Embedding API to generate language embeddings, and then index those embeddings in the Pinecone vector database for fast and scalable vector search.

This is a powerful and common combination for building semantic search, question-answering, threat-detection, and other applications that rely on NLP and search over a large corpus of text data.

The basic workflow looks like this:

**Embed and index**

* Use the OpenAI Embedding API to generate vector embeddings of your documents (or any text data).
* Upload those vector embeddings into Pinecone, which can store and index millions/billions of these vector embeddings, and search through them at ultra-low latencies.

**Search**

* Pass your query text or document through the OpenAI Embedding API again.
* Take the resulting vector embedding and send it as a query to Pinecone.
* Get back semantically similar documents, even if they don't share any keywords with the query.

![Architecture overview](https://files.readme.io/6a3ea5a-pinecone-openai-overview.png)

Let's get started...

## Setup

We first need to setup our environment and retrieve API keys for OpenAI and Pinecone. Let's start with our environment, we need HuggingFace *Datasets* for our data, and the OpenAI and Pinecone clients:


key: _node_content
string_value:
{"id_": "3554d58c-40aa-479c-b996-fc2a9297e268", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "daeab515-6b4f-40c7-a1a5-a97592b75399", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d7cfae8881809a7e19fe70260803d9ad55217e7fdcfba2e97c643bf2b4919e8e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "46cc518e-757e-41d6-940d-24508b227a86", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "23372386078eb65278883f0826612652023d7087155c539b2fc35bc42387f0d3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "abea66af-48be-40f6-94a1-9fc3e1bbb6af", "node_type": "1", "metadata": {}, "hash": "84af6e0371bbb614d0ad15fcbd59ab87bd87cf598291defb7fa5c0a7bfb9762d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2705, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Semantic Search with Pinecone and OpenAI

In this guide you will learn how to use the OpenAI Embedding API to generate language embeddings, and then index those embeddings in the Pinecone vector database for fast and scalable vector search.

This is a powerful and common combination for building semantic search, question-answering, threat-detection, and other applications that rely on NLP and search over a large corpus of text data.

The basic workflow looks like this:

**Embed and index**

* Use the OpenAI Embedding API to generate vector embeddings of your documents (or any text data).
* Upload those vector embeddings into Pinecone, which can store and index millions/billions of these vector embeddings, and search through them at ultra-low latencies.

**Search**

* Pass your query text or document through the OpenAI Embedding API again.
* Take the resulting vector embedding and send it as a query to Pinecone.
* Get back semantically similar documents, even if they don't share any keywords with the query.

![Architecture overview](https://files.readme.io/6a3ea5a-pinecone-openai-overview.png)

Let's get started...

## Setup

We first need to setup our environment and retrieve API keys for OpenAI and Pinecone. Let's start with our environment, we need HuggingFace *Datasets* for our data, and the OpenAI and Pinecone clients:


```python
!pip install -qU \
    pinecone-client==3.0.2 \
    openai==1.10.0 \
    datasets==2.16.1
```

    [2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m201.4/201.4 kB[0m [31m2.0 MB/s[0m eta [36m0:00:00[0m
    [2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m225.1/225.1 kB[0m [31m12.1 MB/s[0m eta [36m0:00:00[0m
    [2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m507.1/507.1 kB[0m [31m12.4 MB/s[0m eta [36m0:00:00[0m
    [2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m75.9/75.9 kB[0m [31m4.4 MB/s[0m eta [36m0:00:00[0m
    [2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m115.3/115.3 kB[0m [31m9.7 MB/s[0m eta [36m0:00:00[0m
    [2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m134.8/134.8 kB[0m [31m7.6 MB/s[0m eta [36m0:00:00[0m
    [2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m76.9/76.9 kB[0m [31m4.7 MB/s[0m eta [36m0:00:00[0m
    [2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m58.3/58.3 kB[0m [31m3.8 MB/s[0m eta [36m0:00:00[0m
    [2K     [90m━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━[0m [32m134.8/134.8 kB[0m [31m5.2 MB/s[0m eta [36m0:00:00[0m
    [?25h[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed.


key: _node_content
string_value:
{"id_": "abea66af-48be-40f6-94a1-9fc3e1bbb6af", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "daeab515-6b4f-40c7-a1a5-a97592b75399", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d7cfae8881809a7e19fe70260803d9ad55217e7fdcfba2e97c643bf2b4919e8e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3554d58c-40aa-479c-b996-fc2a9297e268", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "bd50b43d7ba8b63ed9ed78010b8047c5738a3c37298f0ec84b3719f6d86ef8d0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c4690776-81af-45df-bb7b-c42238156948", "node_type": "1", "metadata": {}, "hash": "f5177ef366427a428ebe6231b4826ab840f4ea7d63a461e6f7c1f49b0d420b65", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2706, "end_char_idx": 4802, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
This behaviour is the source of the following dependency conflicts.
    llmx 0.0.15a0 requires cohere, which is not installed.
    llmx 0.0.15a0 requires tiktoken, which is not installed.
    tensorflow-probability 0.22.0 requires typing-extensions<4.6.0, but you have typing-extensions 4.9.0 which is incompatible.[0m[31m
    [0m

### Creating Embeddings

Then we initialize our connection to OpenAI Embeddings *and* Pinecone vector DB. Sign up for an API key over at [OpenAI](https://platform.openai.com) and [Pinecone](https://app.pinecone.io).


```python
from openai import OpenAI

client = OpenAI(
    api_key="OPENAI_API_KEY"
)  # get API key from platform.openai.com
```

We can now create embeddings with the OpenAI Ada similarity model like so:


```python
MODEL = "text-embedding-3-small"

res = client.embeddings.create(
    input=[
        "Sample document text goes here",
        "there will be several phrases in each batch"
    ], model=MODEL
)
res
```




    CreateEmbeddingResponse(data=[Embedding(embedding=[-0.0007019874756224453, 0.017813093960285187, 0.028484342619776726, -0.01655358262360096, -0.04467806592583656, -0.03371616080403328, 0.02429058402776718, -0.015460160560905933, 0.0147542804479599, -0.006034583784639835, 0.03413138538599014, -0.010325227864086628, 0.004678186494857073, -0.006508630700409412, 0.06505170464515686, 0.07252573221921921, -0.004359848331660032, 0.013335599564015865, -0.026740403845906258, 0.03684417903423309, 0.004705868195742369, 0.04675418511033058, -0.021384017542004585, 0.03440820053219795, 0.012664321810007095, -0.021896127611398697, -0.040553510189056396, -0.024913419038057327, 0.05112787336111069, -0.06244963780045509, -0.01853281632065773, -0.04124554991722107, 0.01914181001484394, -0.03044973500072956, -0.040165968239307404, 0.054698795080184937, 0.06560533493757248, 0.0040449704974889755, -0.049799155443906784, -0.04819362610578537, -0.009563985280692577, -0.012491311877965927, 0.0549202486872673, 0.009536303579807281, -0.0008953260257840157, 0.03521096706390381, 0.008242189884185791, -0.019653920084238052, -0.


key: _node_content
string_value:
{"id_": "c4690776-81af-45df-bb7b-c42238156948", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "daeab515-6b4f-40c7-a1a5-a97592b75399", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d7cfae8881809a7e19fe70260803d9ad55217e7fdcfba2e97c643bf2b4919e8e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "abea66af-48be-40f6-94a1-9fc3e1bbb6af", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f6992186632187695be7dbcf19a5acb016513a6e70d058020e87c791b00bee72", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a2c740fd-2f3f-4a11-b1ff-d24c0c3f63bf", "node_type": "1", "metadata": {}, "hash": "d030f39d2b303613b0b1123ad3c35ef340f79cf6b1de0311cb53aeaf1e1a6f72", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3936, "end_char_idx": 5619, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
-0.006034583784639835, 0.03413138538599014, -0.010325227864086628, 0.004678186494857073, -0.006508630700409412, 0.06505170464515686, 0.07252573221921921, -0.004359848331660032, 0.013335599564015865, -0.026740403845906258, 0.03684417903423309, 0.004705868195742369, 0.04675418511033058, -0.021384017542004585, 0.03440820053219795, 0.012664321810007095, -0.021896127611398697, -0.040553510189056396, -0.024913419038057327, 0.05112787336111069, -0.06244963780045509, -0.01853281632065773, -0.04124554991722107, 0.01914181001484394, -0.03044973500072956, -0.040165968239307404, 0.054698795080184937, 0.06560533493757248, 0.0040449704974889755, -0.049799155443906784, -0.04819362610578537, -0.009563985280692577, -0.012491311877965927, 0.0549202486872673, 0.009536303579807281, -0.0008953260257840157, 0.03521096706390381, 0.008242189884185791, -0.019653920084238052, -0.005422128830105066, 0.056221283972263336, 0.005581297911703587, -0.020747341215610504, -0.0012690272415056825, 0.022989550605416298, 0.03498951345682144, -0.002600338077172637, -0.024802692234516144, 0.02240823581814766, -0.00995152723044157, -0.02694801613688469, 0.009224885143339634, 0.017453234642744064, -0.01533559337258339, 0.033273257315158844, -0.015266389586031437, 0.03438051789999008, -0.045314740389585495, 0.01100342720746994, -0.020567411556839943, 0.0014826945262029767, 0.057300865650177, 0.017259463667869568, -0.004224900621920824, 0.017688527703285217, 0.010048412717878819, -0.00761243375018239, -0.05719013884663582, -0.005692024249583483, -0.031695406883955, -0.019764646887779236, 0.030505098402500153, 0.03241512551903725, -0.0039446246810257435, -0.04224208742380142, -0.0654946118593216, 0.


key: _node_content
string_value:
{"id_": "a2c740fd-2f3f-4a11-b1ff-d24c0c3f63bf", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "daeab515-6b4f-40c7-a1a5-a97592b75399", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d7cfae8881809a7e19fe70260803d9ad55217e7fdcfba2e97c643bf2b4919e8e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c4690776-81af-45df-bb7b-c42238156948", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "cbe917866c003cb6e1c75e96aa580d16c729d3f5db7e4183ddb9bcb982e01228", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "94541415-c160-4407-865c-f45e22463d42", "node_type": "1", "metadata": {}, "hash": "e08c2f96451487fb10262e512cacb0b724b5f283a3d4690460f2f184a79752e6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4754, "end_char_idx": 6443, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0.008242189884185791, -0.019653920084238052, -0.005422128830105066, 0.056221283972263336, 0.005581297911703587, -0.020747341215610504, -0.0012690272415056825, 0.022989550605416298, 0.03498951345682144, -0.002600338077172637, -0.024802692234516144, 0.02240823581814766, -0.00995152723044157, -0.02694801613688469, 0.009224885143339634, 0.017453234642744064, -0.01533559337258339, 0.033273257315158844, -0.015266389586031437, 0.03438051789999008, -0.045314740389585495, 0.01100342720746994, -0.020567411556839943, 0.0014826945262029767, 0.057300865650177, 0.017259463667869568, -0.004224900621920824, 0.017688527703285217, 0.010048412717878819, -0.00761243375018239, -0.05719013884663582, -0.005692024249583483, -0.031695406883955, -0.019764646887779236, 0.030505098402500153, 0.03241512551903725, -0.0039446246810257435, -0.04224208742380142, -0.0654946118593216, 0.01167470496147871, -0.06571606546640396, 0.04395834356546402, -0.04747390374541283, -0.02152242697775364, -0.01990305446088314, 0.004453273490071297, 0.00022145261755213141, -0.0035570827312767506, 0.02429058402776718, 0.005408287979662418, -0.062172822654247284, 0.008103781379759312, -0.020802704617381096, -0.008539766073226929, 0.06084410473704338, -0.023418614640831947, -0.02286498248577118, -0.011134914122521877, 0.015515523962676525, 0.01594458892941475, 0.028152164071798325, -0.021231770515441895, -0.0457853302359581, -0.02380615659058094, -0.06997902691364288, -0.0011297543533146381, 0.019127970561385155, 0.03828362002968788, -0.009114159271121025, -0.005788909737020731, 0.04296180605888367, -0.036899540573358536, 0.008539766073226929, -0.024027608335018158, -0.03471269831061363, -0.015709295868873596, -0.


key: _node_content
string_value:
{"id_": "94541415-c160-4407-865c-f45e22463d42", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "daeab515-6b4f-40c7-a1a5-a97592b75399", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d7cfae8881809a7e19fe70260803d9ad55217e7fdcfba2e97c643bf2b4919e8e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a2c740fd-2f3f-4a11-b1ff-d24c0c3f63bf", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "8f1cb716db1aea9e704de1981d49660df2d9dd97a3ef91d52ac8855c6cf38c66", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8b981df0-3895-4971-9c48-d2c05ba08a11", "node_type": "1", "metadata": {}, "hash": "a721314968cc4228e99299cea5e63d8183911c1726b7ed2515b98068df3ec9f4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5574, "end_char_idx": 7252, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
-0.04224208742380142, -0.0654946118593216, 0.01167470496147871, -0.06571606546640396, 0.04395834356546402, -0.04747390374541283, -0.02152242697775364, -0.01990305446088314, 0.004453273490071297, 0.00022145261755213141, -0.0035570827312767506, 0.02429058402776718, 0.005408287979662418, -0.062172822654247284, 0.008103781379759312, -0.020802704617381096, -0.008539766073226929, 0.06084410473704338, -0.023418614640831947, -0.02286498248577118, -0.011134914122521877, 0.015515523962676525, 0.01594458892941475, 0.028152164071798325, -0.021231770515441895, -0.0457853302359581, -0.02380615659058094, -0.06997902691364288, -0.0011297543533146381, 0.019127970561385155, 0.03828362002968788, -0.009114159271121025, -0.005788909737020731, 0.04296180605888367, -0.036899540573358536, 0.008539766073226929, -0.024027608335018158, -0.03471269831061363, -0.015709295868873596, -0.005726626142859459, -0.03623518347740173, -0.009197204373776913, -0.05115555599331856, 0.021868444979190826, -0.026422064751386642, -0.03244280815124512, 0.025591617450118065, 0.0505465604364872, -0.04182686284184456, 0.017826935276389122, -0.03828362002968788, -0.028332093730568886, 0.015543205663561821, -0.04005524143576622, 0.05281645059585571, -0.028456661850214005, 0.037176359444856644, 0.014574350789189339, -0.0366504080593586, 0.02651895023882389, 0.03128018230199814, -0.020041462033987045, -0.0295085608959198, -0.024152176454663277, -0.01840824820101261, -0.02697569690644741, 0.01929405890405178, -0.02413833513855934, 0.04495488107204437, -0.04628359526395798, 0.021896127611398697, 0.03634591028094292, -0.023584702983498573, 0.04617287218570709, -0.020941112190485, -0.039556972682476044, 0.


key: _node_content
string_value:
{"id_": "8b981df0-3895-4971-9c48-d2c05ba08a11", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "daeab515-6b4f-40c7-a1a5-a97592b75399", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d7cfae8881809a7e19fe70260803d9ad55217e7fdcfba2e97c643bf2b4919e8e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "94541415-c160-4407-865c-f45e22463d42", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d36dfe9c3474656c5d79743b9f358772026023ee84f6485465d702c11e8b29d9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f8aaea3e-5146-476e-aba7-9f135099b206", "node_type": "1", "metadata": {}, "hash": "c99e8a19716605a10d9fce1292a4d1ff66c81c86ca5fce1d1387dcab36cdda77", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6375, "end_char_idx": 8070, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
024027608335018158, -0.03471269831061363, -0.015709295868873596, -0.005726626142859459, -0.03623518347740173, -0.009197204373776913, -0.05115555599331856, 0.021868444979190826, -0.026422064751386642, -0.03244280815124512, 0.025591617450118065, 0.0505465604364872, -0.04182686284184456, 0.017826935276389122, -0.03828362002968788, -0.028332093730568886, 0.015543205663561821, -0.04005524143576622, 0.05281645059585571, -0.028456661850214005, 0.037176359444856644, 0.014574350789189339, -0.0366504080593586, 0.02651895023882389, 0.03128018230199814, -0.020041462033987045, -0.0295085608959198, -0.024152176454663277, -0.01840824820101261, -0.02697569690644741, 0.01929405890405178, -0.02413833513855934, 0.04495488107204437, -0.04628359526395798, 0.021896127611398697, 0.03634591028094292, -0.023584702983498573, 0.04617287218570709, -0.020941112190485, -0.039556972682476044, 0.023238683119416237, -0.012885774485766888, -0.0464496873319149, 0.012110689654946327, 0.017840776592493057, -0.016387494280934334, -0.02390304207801819, 0.014948051422834396, -0.014242171309888363, -0.06770914047956467, 0.012027645483613014, -0.01006225310266018, -0.04456733912229538, -0.012345983646810055, -0.024498196318745613, -0.021591629832983017, 0.03573691472411156, -0.013300998136401176, -0.017397871240973473, 0.020235233008861542, 0.03252585232257843, 0.010871939361095428, 0.04827667027711868, 0.012449788860976696, 0.02195149101316929, 0.024221379309892654, -0.0514046885073185, 0.010235263034701347, 0.047086361795663834, 0.02377847395837307, -0.011882317252457142, 0.003674729261547327, 0.029287109151482582, -0.02719714865088463, -0.03260889649391174, -0.021148724481463432, -0.006574374623596668, 0.


key: _node_content
string_value:
{"id_": "f8aaea3e-5146-476e-aba7-9f135099b206", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "daeab515-6b4f-40c7-a1a5-a97592b75399", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d7cfae8881809a7e19fe70260803d9ad55217e7fdcfba2e97c643bf2b4919e8e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8b981df0-3895-4971-9c48-d2c05ba08a11", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "39c73ce552e9dffa4033153f575b41b9403a7c17b2a72d9359093b69075597d4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c9431384-f193-4772-851c-d1712f8afe7a", "node_type": "1", "metadata": {}, "hash": "24e250d9c85f1c6a41e31647d52bd284b2be0591786dc40ebf9bf7768a010e0d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7188, "end_char_idx": 8885, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
04617287218570709, -0.020941112190485, -0.039556972682476044, 0.023238683119416237, -0.012885774485766888, -0.0464496873319149, 0.012110689654946327, 0.017840776592493057, -0.016387494280934334, -0.02390304207801819, 0.014948051422834396, -0.014242171309888363, -0.06770914047956467, 0.012027645483613014, -0.01006225310266018, -0.04456733912229538, -0.012345983646810055, -0.024498196318745613, -0.021591629832983017, 0.03573691472411156, -0.013300998136401176, -0.017397871240973473, 0.020235233008861542, 0.03252585232257843, 0.010871939361095428, 0.04827667027711868, 0.012449788860976696, 0.02195149101316929, 0.024221379309892654, -0.0514046885073185, 0.010235263034701347, 0.047086361795663834, 0.02377847395837307, -0.011882317252457142, 0.003674729261547327, 0.029287109151482582, -0.02719714865088463, -0.03260889649391174, -0.021148724481463432, -0.006574374623596668, 0.06078874319791794, -0.0018460152205079794, 0.009038034826517105, 0.03593068569898605, 0.00554669601842761, 0.05655346065759659, -0.05544620007276535, 0.007384060882031918, -0.06632506102323532, -0.023709271103143692, -0.03305180370807648, -0.010657407343387604, -0.017577800899744034, -0.06604824215173721, 0.047750718891620636, 0.009432497434318066, 0.02484421618282795, 0.0011963631259277463, -0.03487878665328026, -0.02575770765542984, -0.009847721084952354, 0.04841507971286774, 0.028428979218006134, -0.003920403309166431, -0.010581282898783684, 0.03554314374923706, 0.06654651463031769, 0.03548778221011162, 0.011951521039009094, -0.008574368432164192, -0.01248439121991396, -0.017563961446285248, -0.017909979447722435, -0.0011816573096439242, 0.017550120130181313, 0.002157432958483696, 0.02314179763197899,


key: _node_content
string_value:
{"id_": "c9431384-f193-4772-851c-d1712f8afe7a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "daeab515-6b4f-40c7-a1a5-a97592b75399", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d7cfae8881809a7e19fe70260803d9ad55217e7fdcfba2e97c643bf2b4919e8e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f8aaea3e-5146-476e-aba7-9f135099b206", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "f9f63d18b95c7730ac5bf9dde79e73155d44e463e245114c546bdeb6da4d41af", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3da1df00-abc6-4db2-81b6-6ca2cc0b01e2", "node_type": "1", "metadata": {}, "hash": "a53e459beeebe80b0efcf9015b223c05fc1a09006d7df48ad88a2742fe3b3468", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8003, "end_char_idx": 9690, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
03260889649391174, -0.021148724481463432, -0.006574374623596668, 0.06078874319791794, -0.0018460152205079794, 0.009038034826517105, 0.03593068569898605, 0.00554669601842761, 0.05655346065759659, -0.05544620007276535, 0.007384060882031918, -0.06632506102323532, -0.023709271103143692, -0.03305180370807648, -0.010657407343387604, -0.017577800899744034, -0.06604824215173721, 0.047750718891620636, 0.009432497434318066, 0.02484421618282795, 0.0011963631259277463, -0.03487878665328026, -0.02575770765542984, -0.009847721084952354, 0.04841507971286774, 0.028428979218006134, -0.003920403309166431, -0.010581282898783684, 0.03554314374923706, 0.06654651463031769, 0.03548778221011162, 0.011951521039009094, -0.008574368432164192, -0.01248439121991396, -0.017563961446285248, -0.017909979447722435, -0.0011816573096439242, 0.017550120130181313, 0.002157432958483696, 0.02314179763197899, -0.03496183082461357, 0.02264353074133396, -0.005792370066046715, -0.008048418909311295, 0.003657428314909339, -0.015736976638436317, 0.007937692105770111, -0.06925930827856064, 0.008865024894475937, 0.031169455498456955, 0.012671241536736488, 0.030809596180915833, -0.031473953276872635, -0.041162505745887756, -0.07523852586746216, 0.032664261758327484, 0.05024206265807152, 0.02274041622877121, 0.012006884440779686, 0.016816558316349983, -0.008435960859060287, -0.0019152191234752536, 0.0034999893978238106, -0.006055344827473164, 0.02200685441493988, 0.01652590185403824, -0.004269883036613464, -0.05325935408473015, 0.002250858349725604, -0.023736951872706413, 0.02142554149031639, -0.008256030268967152, -0.0020121047273278236, 0.022366713732481003, 0.005346004385501146, 0.010512079112231731, -0.


key: _node_content
string_value:
{"id_": "3da1df00-abc6-4db2-81b6-6ca2cc0b01e2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "daeab515-6b4f-40c7-a1a5-a97592b75399", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d7cfae8881809a7e19fe70260803d9ad55217e7fdcfba2e97c643bf2b4919e8e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c9431384-f193-4772-851c-d1712f8afe7a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "5cdf9592e7f4f39b2477a858389131550e3d439720d68c7915d02672f0eeed62", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c728981f-2e8c-4ac5-9dd3-c98cfa6d4b17", "node_type": "1", "metadata": {}, "hash": "9e811cd4bcea7915ea4199e3d66452796bf9443f63eaa9748823e851e5d19bd4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8821, "end_char_idx": 10504, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0.017550120130181313, 0.002157432958483696, 0.02314179763197899, -0.03496183082461357, 0.02264353074133396, -0.005792370066046715, -0.008048418909311295, 0.003657428314909339, -0.015736976638436317, 0.007937692105770111, -0.06925930827856064, 0.008865024894475937, 0.031169455498456955, 0.012671241536736488, 0.030809596180915833, -0.031473953276872635, -0.041162505745887756, -0.07523852586746216, 0.032664261758327484, 0.05024206265807152, 0.02274041622877121, 0.012006884440779686, 0.016816558316349983, -0.008435960859060287, -0.0019152191234752536, 0.0034999893978238106, -0.006055344827473164, 0.02200685441493988, 0.01652590185403824, -0.004269883036613464, -0.05325935408473015, 0.002250858349725604, -0.023736951872706413, 0.02142554149031639, -0.008256030268967152, -0.0020121047273278236, 0.022366713732481003, 0.005346004385501146, 0.010512079112231731, -0.03396529331803322, -0.008781980723142624, 0.051044829189777374, 0.04420747980475426, -0.0197508055716753, -0.05840812623500824, 0.03313484787940979, 0.0055328551679849625, -0.023307887837290764, 0.020954953506588936, 0.007079563569277525, -0.025646980851888657, 0.003546701977029443, -0.02420753985643387, 0.000534167920704931, -0.019363263621926308, -0.01060896459966898, 0.044705748558044434, 0.01579234004020691, 0.0025484352372586727, 0.007785443682223558, 0.033909931778907776, -0.03496183082461357, 0.014408260583877563, -0.03833898529410362, -0.02743244357407093, 0.016262926161289215, -0.012159132398664951, 0.022394396364688873, 0.04127323254942894, 0.010539760813117027, 0.030837276950478554, 0.026629677042365074, 0.006875411607325077, 0.02231135033071041, 0.017300985753536224, 0.024608921259641647, 0.


key: _node_content
string_value:
{"id_": "c728981f-2e8c-4ac5-9dd3-c98cfa6d4b17", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "daeab515-6b4f-40c7-a1a5-a97592b75399", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d7cfae8881809a7e19fe70260803d9ad55217e7fdcfba2e97c643bf2b4919e8e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3da1df00-abc6-4db2-81b6-6ca2cc0b01e2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "6310e172dc9e4429bac28d5efdf0c2ba4377bc7f7aad5c1a3ccb5f7c85665a89", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ebbf488e-59d3-4abb-a324-fff27532ad1d", "node_type": "1", "metadata": {}, "hash": "bf220bff5051e92ba3f118210f7e7e3627d36b02115dc04f2b6f7104c17b228f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9643, "end_char_idx": 11325, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0.005346004385501146, 0.010512079112231731, -0.03396529331803322, -0.008781980723142624, 0.051044829189777374, 0.04420747980475426, -0.0197508055716753, -0.05840812623500824, 0.03313484787940979, 0.0055328551679849625, -0.023307887837290764, 0.020954953506588936, 0.007079563569277525, -0.025646980851888657, 0.003546701977029443, -0.02420753985643387, 0.000534167920704931, -0.019363263621926308, -0.01060896459966898, 0.044705748558044434, 0.01579234004020691, 0.0025484352372586727, 0.007785443682223558, 0.033909931778907776, -0.03496183082461357, 0.014408260583877563, -0.03833898529410362, -0.02743244357407093, 0.016262926161289215, -0.012159132398664951, 0.022394396364688873, 0.04127323254942894, 0.010539760813117027, 0.030837276950478554, 0.026629677042365074, 0.006875411607325077, 0.02231135033071041, 0.017300985753536224, 0.024608921259641647, 0.017176419496536255, -0.028428979218006134, 0.033937614411115646, -0.02316948026418686, 0.03283035010099411, -0.033273257315158844, -0.031446270644664764, -0.04728013277053833, -0.032359763979911804, -0.024193698540329933, 0.08221428096294403, 0.008961910381913185, -0.041411638259887695, -0.04027669504284859, 0.021079521626234055, -0.03795144334435463, 0.058242037892341614, -0.0028321712743490934, 0.012795808725059032, 0.03249816969037056, 0.015114140696823597, 0.01883731409907341, 0.016664309427142143, -0.04525937885046005, -0.0077716028317809105, -0.04343239590525627, 0.005761228036135435, -0.026726562529802322, 0.027086423709988594, -0.008484403602778912, 0.004775071982294321, 0.05544620007276535, 0.021051838994026184, 0.02923174574971199, -0.05719013884663582, -0.007397901266813278, 0.015183345414698124, 0.


key: _node_content
string_value:
{"id_": "ebbf488e-59d3-4abb-a324-fff27532ad1d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "daeab515-6b4f-40c7-a1a5-a97592b75399", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d7cfae8881809a7e19fe70260803d9ad55217e7fdcfba2e97c643bf2b4919e8e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c728981f-2e8c-4ac5-9dd3-c98cfa6d4b17", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "caab143076edc1a414c12cdcab5295baebc5e94bc51e48244a6798c6ccb5ddb0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4a3a36b5-88c9-4bd3-a6f9-973938ecc702", "node_type": "1", "metadata": {}, "hash": "08011c8861847e943dfde0301c8653cb39d71fefb22f341033bb5ed46f75dc9f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10458, "end_char_idx": 12147, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0.017300985753536224, 0.024608921259641647, 0.017176419496536255, -0.028428979218006134, 0.033937614411115646, -0.02316948026418686, 0.03283035010099411, -0.033273257315158844, -0.031446270644664764, -0.04728013277053833, -0.032359763979911804, -0.024193698540329933, 0.08221428096294403, 0.008961910381913185, -0.041411638259887695, -0.04027669504284859, 0.021079521626234055, -0.03795144334435463, 0.058242037892341614, -0.0028321712743490934, 0.012795808725059032, 0.03249816969037056, 0.015114140696823597, 0.01883731409907341, 0.016664309427142143, -0.04525937885046005, -0.0077716028317809105, -0.04343239590525627, 0.005761228036135435, -0.026726562529802322, 0.027086423709988594, -0.008484403602778912, 0.004775071982294321, 0.05544620007276535, 0.021051838994026184, 0.02923174574971199, -0.05719013884663582, -0.007397901266813278, 0.015183345414698124, 0.01507261861115694, 0.003287187311798334, -0.014948051422834396, -0.007889249362051487, 0.027695417404174805, 0.029093338176608086, 0.014532827772200108, -0.023307887837290764, 0.03695490583777428, 0.035072557628154755, 0.021010316908359528, 0.04326630383729935, -0.0334116630256176, 0.0013771584490314126, -0.023183321580290794, 0.01615219935774803, -0.018117591738700867, -0.02304491214454174, 0.0030034510418772697, 0.06455343961715698, -0.006110708229243755, -0.0051972162909805775, 0.025439368560910225, 0.0034550067503005266, -0.03103104792535305, 0.05458806827664375, 0.004903099499642849, 0.0028217907529324293, -0.01820063777267933, -0.008512085303664207, -0.01135636679828167, -0.06261572986841202, -0.013356360606849194, -0.03197222203016281, 0.020927272737026215, 0.01716257818043232, -0.011944600380957127, -0.


key: _node_content
string_value:
{"id_": "4a3a36b5-88c9-4bd3-a6f9-973938ecc702", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "daeab515-6b4f-40c7-a1a5-a97592b75399", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d7cfae8881809a7e19fe70260803d9ad55217e7fdcfba2e97c643bf2b4919e8e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ebbf488e-59d3-4abb-a324-fff27532ad1d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "1f865ba01be700db96a802405e955d4d6f346f7b200a07c50eb241e8513d65e8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d4e99148-035a-45a4-9925-1ad2ec6791bf", "node_type": "1", "metadata": {}, "hash": "c1f5e4c0020da528c62053b91aa6d49b0a781ceffe5c0196fbfc806becdc1c97", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11278, "end_char_idx": 12969, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
-0.007397901266813278, 0.015183345414698124, 0.01507261861115694, 0.003287187311798334, -0.014948051422834396, -0.007889249362051487, 0.027695417404174805, 0.029093338176608086, 0.014532827772200108, -0.023307887837290764, 0.03695490583777428, 0.035072557628154755, 0.021010316908359528, 0.04326630383729935, -0.0334116630256176, 0.0013771584490314126, -0.023183321580290794, 0.01615219935774803, -0.018117591738700867, -0.02304491214454174, 0.0030034510418772697, 0.06455343961715698, -0.006110708229243755, -0.0051972162909805775, 0.025439368560910225, 0.0034550067503005266, -0.03103104792535305, 0.05458806827664375, 0.004903099499642849, 0.0028217907529324293, -0.01820063777267933, -0.008512085303664207, -0.01135636679828167, -0.06261572986841202, -0.013356360606849194, -0.03197222203016281, 0.020927272737026215, 0.01716257818043232, -0.011944600380957127, -0.048691894859075546, 0.028456661850214005, -0.035404738038778305, -0.012131450697779655, 0.04669881984591484, 0.013314838521182537, -0.021660834550857544, 0.04066423699259758, -0.006176451686769724, -0.00014338191249407828, 0.007051881868392229, 0.017633164301514626, 0.024124493822455406, 0.03922479599714279, -0.0070311203598976135, 0.029397834092378616, -0.010491318069398403, 0.0386434830725193, 0.0016418634913861752, -0.04393066465854645, 0.06665723770856857, -0.0022162562236189842, 0.006643578410148621, -0.034159064292907715, -0.048055216670036316, 0.004778532311320305, -0.03382688760757446, -0.0206089336425066, 0.011197198182344437, -0.05658114328980446, 0.05212441086769104, -0.021300973370671272, 0.014657394960522652, 0.003185111563652754, -0.00039078600821085274, 0.019930735230445862, 0.011806192807853222,


key: _node_content
string_value:
{"id_": "d4e99148-035a-45a4-9925-1ad2ec6791bf", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "daeab515-6b4f-40c7-a1a5-a97592b75399", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d7cfae8881809a7e19fe70260803d9ad55217e7fdcfba2e97c643bf2b4919e8e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4a3a36b5-88c9-4bd3-a6f9-973938ecc702", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "96b835634902e3b16e7a94601ade9f5e65991881ed3dd48faf9b83bf757a830d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "26ed6eda-58f5-434f-9275-211734cf7262", "node_type": "1", "metadata": {}, "hash": "3a08bc9148f6161b9c5bb8689dd26069768a657987e05108616820bc8b3c0051", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12100, "end_char_idx": 13799, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0.01716257818043232, -0.011944600380957127, -0.048691894859075546, 0.028456661850214005, -0.035404738038778305, -0.012131450697779655, 0.04669881984591484, 0.013314838521182537, -0.021660834550857544, 0.04066423699259758, -0.006176451686769724, -0.00014338191249407828, 0.007051881868392229, 0.017633164301514626, 0.024124493822455406, 0.03922479599714279, -0.0070311203598976135, 0.029397834092378616, -0.010491318069398403, 0.0386434830725193, 0.0016418634913861752, -0.04393066465854645, 0.06665723770856857, -0.0022162562236189842, 0.006643578410148621, -0.034159064292907715, -0.048055216670036316, 0.004778532311320305, -0.03382688760757446, -0.0206089336425066, 0.011197198182344437, -0.05658114328980446, 0.05212441086769104, -0.021300973370671272, 0.014657394960522652, 0.003185111563652754, -0.00039078600821085274, 0.019930735230445862, 0.011806192807853222, 0.061231646686792374, 0.018892675638198853, -0.012491311877965927, -0.008906546980142593, -0.057079412043094635, -0.012989579699933529, -0.025965319946408272, -0.018311362713575363, 0.04271267354488373, 0.031141774728894234, -0.06964685022830963, 0.011557058431208134, -0.0566365085542202, -0.016927285119891167, -0.0031366688199341297, 0.027930710464715958, -0.006508630700409412, -0.018518975004553795, 0.015418638475239277, -0.0327473059296608, -0.007522468455135822, -0.019986098632216454, -0.006335620768368244, -0.005356385372579098, 9.104643686441705e-05, -0.0473078154027462, -0.007730080280452967, -0.03994451463222504, -0.00055146892555058, 0.02548089250922203, 0.005280260927975178, 0.002705874154344201, -0.025356324389576912, 0.01835288479924202, -0.024678125977516174, 0.009792357683181763, -0.03546009957790375, 0.


key: _node_content
string_value:
{"id_": "26ed6eda-58f5-434f-9275-211734cf7262", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "daeab515-6b4f-40c7-a1a5-a97592b75399", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d7cfae8881809a7e19fe70260803d9ad55217e7fdcfba2e97c643bf2b4919e8e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d4e99148-035a-45a4-9925-1ad2ec6791bf", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "64541e57dc6528163948ff7681e32fffab3a6f9b826ee9c84fcc3440077c2b36", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "91b0d828-5530-4f7e-9d07-50d8bf3333f7", "node_type": "1", "metadata": {}, "hash": "0d3e90c392cf1de2d4742edd5c2361e899bcf497b77a0eb5bcaefcea3851f571", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12926, "end_char_idx": 14620, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0.019930735230445862, 0.011806192807853222, 0.061231646686792374, 0.018892675638198853, -0.012491311877965927, -0.008906546980142593, -0.057079412043094635, -0.012989579699933529, -0.025965319946408272, -0.018311362713575363, 0.04271267354488373, 0.031141774728894234, -0.06964685022830963, 0.011557058431208134, -0.0566365085542202, -0.016927285119891167, -0.0031366688199341297, 0.027930710464715958, -0.006508630700409412, -0.018518975004553795, 0.015418638475239277, -0.0327473059296608, -0.007522468455135822, -0.019986098632216454, -0.006335620768368244, -0.005356385372579098, 9.104643686441705e-05, -0.0473078154027462, -0.007730080280452967, -0.03994451463222504, -0.00055146892555058, 0.02548089250922203, 0.005280260927975178, 0.002705874154344201, -0.025356324389576912, 0.01835288479924202, -0.024678125977516174, 0.009792357683181763, -0.03546009957790375, 0.04575764760375023, 0.021688515320420265, 0.039252474904060364, 0.0332178920507431, 0.014311375096440315, -0.017148736864328384, 0.005525934975594282, 0.013833868317306042, -0.013003421016037464, 0.024249061942100525, -0.04213136062026024, -0.044152114540338516, 0.015709295868873596, 0.01685808040201664, 0.024899577721953392, -0.006923854351043701, -0.034214429557323456, -0.01743939332664013, 0.022325191646814346, 0.046477366238832474, -0.014110683463513851, -0.05702405050396919, 0.011411730200052261, -0.02453971840441227, 0.010553601197898388, 0.00746018486097455, 0.025384007021784782, -0.00020728743402287364, -0.045674603432416916, 0.05912784859538078, 0.0036124459002166986, 0.09826959669589996, -0.007799284532666206, 0.01802070625126362, 0.019916893914341927, -0.015307911671698093, -0.0057473876513540745, 0.


key: _node_content
string_value:
{"id_": "91b0d828-5530-4f7e-9d07-50d8bf3333f7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "daeab515-6b4f-40c7-a1a5-a97592b75399", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "d7cfae8881809a7e19fe70260803d9ad55217e7fdcfba2e97c643bf2b4919e8e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "26ed6eda-58f5-434f-9275-211734cf7262", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/Semantic_Search.ipynb", "title": "Semantic Search with Pinecone and OpenAI", "tokens": 34426, "retrieve_doc": false, "source": "OpenAI_Cookbooks"}, "hash": "30f5af940b55f739e2ae03f607522f0fd9d982f45a5481c68e8e2afb4795f7d1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "50cc6067-4799-4b96-b25d-ee560698758c", "node_type": "1", "metadata": {}, "hash": "3fa5b476d68f3d52f0595a8f2705ec91a122c218f00fd4d156a335d8c9af15fb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13753, "end_char_idx": 15443, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0.009792357683181763, -0.03546009957790375, 0.04575764760375023, 0.021688515320420265, 0.039252474904060364, 0.0332178920507431, 0.014311375096440315, -0.017148736864328384, 0.005525934975594282, 0.013833868317306042, -0.013003421016037464, 0.024249061942100525, -0.04213136062026024, -0.044152114540338516, 0.015709295868873596, 0.01685808040201664, 0.024899577721953392, -0.006923854351043701, -0.034214429557323456, -0.01743939332664013, 0.022325191646814346, 0.046477366238832474, -0.014110683463513851, -0.05702405050396919, 0.011411730200052261, -0.02453971840441227, 0.010553601197898388, 0.00746018486097455, 0.025384007021784782, -0.00020728743402287364, -0.045674603432416916, 0.05912784859538078, 0.0036124459002166986, 0.09826959669589996, -0.007799284532666206, 0.01802070625126362, 0.019916893914341927, -0.015307911671698093, -0.0057473876513540745, 0.018186796456575394, 0.029674651101231575, -0.015903066843748093, -0.01807606965303421, -0.04487183690071106, 0.0194186270236969, -0.04498256370425224, 0.020858068019151688, -0.004321786109358072, 0.02139785885810852, -0.005131472367793322, -0.012110689654946327, 0.020622774958610535, 0.04387529939413071, -0.009266408160328865, -0.057633042335510254, 0.009619347751140594, 0.011217959225177765, 0.034214429557323456, -0.02661583572626114, 0.027667736634612083, 0.010172979906201363, 0.004107254091650248, -0.002717984840273857, 0.03277498856186867, 0.02112104371190071, 0.026408225297927856, -0.0239860862493515, -0.007494787219911814, -0.002707604318857193, 0.0024273283779621124, 0.012235256843268871, -0.016235245391726494, -0.003214523196220398, -0.025287121534347534, -0.0018702365923672915, -0.017024170607328415, 0.


