# embedding_metadata

key: _node_content
string_value:
{"id_": "d845f904-c5e5-4e1c-bf9c-404e98134c0a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL", "tokens": 6051, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL", "tokens": 6051, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "723f825cd19bd5d01794b8111cf19db861b8b27b9247f5207316ec1dc1f05a13", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7c5f12e6-6bd5-4f7a-8137-c7b661480300", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL", "tokens": 6051, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "572205a30c09ff140fbca2a7f25b0da8acc2d55af121297633fe50672e167e13", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2df7ff6b-ba7e-4504-b96b-068510c45d63", "node_type": "1", "metadata": {}, "hash": "c26e01f72c6f2863c881361edd6a6c855ab22d8284c6cc1a8dae242198d2c4cf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17414, "end_char_idx": 20295, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
q_topic = generate_quote("politics and virtue")
print("\nA new generated quote:")
print(q_topic)
```

    ** quotes found:
    **    - Happiness is the reward of virtue. (aristotle)
    **    - Our moral virtues benefit mainly other people; intellectual virtues, on the other hand, benefit primarily ourselves; therefore the former make us universally popular, the latter unpopular. (schopenhauer)
    ** end of logging
    
    A new generated quote:
    Virtuous politics purifies society, while corrupt politics breeds chaos and decay.


Use inspiration from just a single philosopher:


```python
q_topic = generate_quote("animals", author="schopenhauer")
print("\nA new generated quote:")
print(q_topic)
```

    ** quotes found:
    **    - Because Christian morality leaves animals out of account, they are at once outlawed in philosophical morals; they are mere 'things,' mere means to any ends whatsoever. They can therefore be used for vivisection, hunting, coursing, bullfights, and horse racing, and can be whipped to death as they struggle along with heavy carts of stone. Shame on such a morality that is worthy of pariahs, and that fails to recognize the eternal essence that exists in every living thing, and shines forth with inscrutable significance from all eyes that see the sun! (schopenhauer)
    **    - The assumption that animals are without rights, and the illusion that our treatment of them has no moral significance, is a positively outrageous example of Western crudity and barbarity. Universal compassion is the only guarantee of morality. (schopenhauer)
    ** end of logging
    
    A new generated quote:
    The true measure of humanity lies not in our dominion over animals, but in our ability to show compassion and respect for all living beings.


## (Optional) **Partitioning**

There's an interesting topic to examine before completing this quickstart. While, generally, tags and quotes can be in any relationship (e.g. a quote having multiple tags), _authors_ are effectively an exact grouping (they define a "disjoint partitioning" on the set of quotes): each quote has exactly one author (for us, at least).

Now, suppose you know in advance your application will usually (or always) run queries on a _single author_. Then you can take full advantage of the underlying database structure: if you group quotes in **partitions** (one per author), vector queries on just an author will use less resources and return much faster.

We'll not dive into the details here, which have to do with the Cassandra storage internals: the important message is that **if your queries are run within a group, consider partitioning accordingly to boost performance**.

You'll now see this choice in action.

First, you need a different table abstraction from CassIO:


```python
from cassio.table import ClusteredMetadataVectorCassandraTable
```


key: _node_content
string_value:
{"id_": "2df7ff6b-ba7e-4504-b96b-068510c45d63", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL", "tokens": 6051, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL", "tokens": 6051, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "723f825cd19bd5d01794b8111cf19db861b8b27b9247f5207316ec1dc1f05a13", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d845f904-c5e5-4e1c-bf9c-404e98134c0a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL", "tokens": 6051, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "39cf7c9d6d0b297719c69bbd675abb83a181a9d317136a1af41b50f85b9b4731", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "efb11482-3ae1-47ee-8975-755de9e7792b", "node_type": "1", "metadata": {}, "hash": "06cf7088182946ccae7ef6824b5c7de4c4cc7556832bfe7598b65f1ca1a76eab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19211, "end_char_idx": 21300, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## (Optional) **Partitioning**

There's an interesting topic to examine before completing this quickstart. While, generally, tags and quotes can be in any relationship (e.g. a quote having multiple tags), _authors_ are effectively an exact grouping (they define a "disjoint partitioning" on the set of quotes): each quote has exactly one author (for us, at least).

Now, suppose you know in advance your application will usually (or always) run queries on a _single author_. Then you can take full advantage of the underlying database structure: if you group quotes in **partitions** (one per author), vector queries on just an author will use less resources and return much faster.

We'll not dive into the details here, which have to do with the Cassandra storage internals: the important message is that **if your queries are run within a group, consider partitioning accordingly to boost performance**.

You'll now see this choice in action.

First, you need a different table abstraction from CassIO:


```python
from cassio.table import ClusteredMetadataVectorCassandraTable
```


```python
v_table_partitioned = ClusteredMetadataVectorCassandraTable(table="philosophers_cassio_partitioned", vector_dimension=1536)
```

Now repeat the compute-embeddings-and-insert step on the new table.

Compared to what you have seen earlier, there is a crucial difference in that now the quote's author is stored as the _partition id_ for the inserted row, instead of being added to the catch-all "metadata" dictionary.

While you are at it, by way of demonstration, you will insert all quotes by a given author _concurrently_: with CassIO, this is done by usng the asynchronous `put_async` method for each quote, collecting the resulting list of `Future` objects, and calling the `result()` method on them all afterwards, to ensure they all have executed. Cassandra / Astra DB well supports a high degree of concurrency in I/O operations.

_(Note: one could have cached the embeddings computed previously to save a few API tokens -- here, however, we wanted to keep the code easier to inspect.)_


key: _node_content
string_value:
{"id_": "efb11482-3ae1-47ee-8975-755de9e7792b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL", "tokens": 6051, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL", "tokens": 6051, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "723f825cd19bd5d01794b8111cf19db861b8b27b9247f5207316ec1dc1f05a13", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2df7ff6b-ba7e-4504-b96b-068510c45d63", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL", "tokens": 6051, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "df9f374bc5ae43309d4f8bfdbe82dd35f2a864770dcf3d676d6b811699f50b2e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1aca468d-bc24-45fc-9de8-1de72112573f", "node_type": "1", "metadata": {}, "hash": "b8d19c4159f80dc3a9c720af1ea488944a32a179071927843ff77b665d6dcd38", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20219, "end_char_idx": 23002, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from cassio.table import ClusteredMetadataVectorCassandraTable
```


```python
v_table_partitioned = ClusteredMetadataVectorCassandraTable(table="philosophers_cassio_partitioned", vector_dimension=1536)
```

Now repeat the compute-embeddings-and-insert step on the new table.

Compared to what you have seen earlier, there is a crucial difference in that now the quote's author is stored as the _partition id_ for the inserted row, instead of being added to the catch-all "metadata" dictionary.

While you are at it, by way of demonstration, you will insert all quotes by a given author _concurrently_: with CassIO, this is done by usng the asynchronous `put_async` method for each quote, collecting the resulting list of `Future` objects, and calling the `result()` method on them all afterwards, to ensure they all have executed. Cassandra / Astra DB well supports a high degree of concurrency in I/O operations.

_(Note: one could have cached the embeddings computed previously to save a few API tokens -- here, however, we wanted to keep the code easier to inspect.)_


```python
BATCH_SIZE = 50

num_batches = ((len(philo_dataset) + BATCH_SIZE - 1) // BATCH_SIZE)

quotes_list = philo_dataset["quote"]
authors_list = philo_dataset["author"]
tags_list = philo_dataset["tags"]

print("Starting to store entries:")
for batch_i in range(num_batches):
    b_start = batch_i * BATCH_SIZE
    b_end = (batch_i + 1) * BATCH_SIZE
    # compute the embedding vectors for this batch
    b_emb_results = client.embeddings.create(
        input=quotes_list[b_start : b_end],
        model=embedding_model_name,
    )
    # prepare the rows for insertion
    futures = []
    print("B ", end="")
    for entry_idx, emb_result in zip(range(b_start, b_end), b_emb_results.data):
        if tags_list[entry_idx]:
            tags = {
                tag
                for tag in tags_list[entry_idx].split(";")
            }
        else:
            tags = set()
        author = authors_list[entry_idx]
        quote = quotes_list[entry_idx]
        futures.append(v_table_partitioned.put_async(
            partition_id=author,
            row_id=f"q_{author}_{entry_idx}",
            body_blob=quote,
            vector=emb_result.embedding,
            metadata={tag: True for tag in tags},
        ))
    #
    for future in futures:
        future.result()
    #
    print(f" done ({len(b_emb_results.data)})")

print("\nFinished storing entries.")
```

    Starting to store entries:
    B  done (50)
    B  done (50)
    B  done (50)
    B  done (50)
    B  done (50)
    B  done (50)
    B  done (50)
    B  done (50)
    B  done (50)
    
    Finished storing entries.


With this new table, the similarity search changes accordingly (note the arguments to `ann_search`):


key: _node_content
string_value:
{"id_": "1aca468d-bc24-45fc-9de8-1de72112573f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL", "tokens": 6051, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL", "tokens": 6051, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "723f825cd19bd5d01794b8111cf19db861b8b27b9247f5207316ec1dc1f05a13", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "efb11482-3ae1-47ee-8975-755de9e7792b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL", "tokens": 6051, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e9594886c16e10c019bddf58f236ff9c58aebc9c1c421e64605d29d14ebdc121", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 22902, "end_char_idx": 26172, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
With this new table, the similarity search changes accordingly (note the arguments to `ann_search`):


```python
def find_quote_and_author_p(query_quote, n, author=None, tags=None):
    query_vector = client.embeddings.create(
        input=[query_quote],
        model=embedding_model_name,
    ).data[0].embedding
    metadata = {}
    partition_id = None
    if author:
        partition_id = author
    if tags:
        for tag in tags:
            metadata[tag] = True
    #
    results = v_table_partitioned.ann_search(
        query_vector,
        n=n,
        partition_id=partition_id,
        metadata=metadata,
    )
    return [
        (result["body_blob"], result["partition_id"])
        for result in results
    ]
```

That's it: the new table still supports the "generic" similarity searches all right ...


```python
find_quote_and_author_p("We struggle all our life for nothing", 3)
```




    [('Life to the great majority is only a constant struggle for mere existence, with the certainty of losing it at last.',
      'schopenhauer'),
     ('We give up leisure in order that we may have leisure, just as we go to war in order that we may have peace.',
      'aristotle'),
     ('Perhaps the gods are kind to us, by making life more disagreeable as we grow older. In the end death seems less intolerable than the manifold burdens we carry',
      'freud')]



... but it's when an author is specified that you would notice a _huge_ performance advantage:


```python
find_quote_and_author_p("We struggle all our life for nothing", 2, author="nietzsche")
```




    [('To live is to suffer, to survive is to find some meaning in the suffering.',
      'nietzsche'),
     ('What makes us heroic?--Confronting simultaneously our supreme suffering and our supreme hope.',
      'nietzsche')]



Well, you _would_ notice a performance gain, if you had a realistic-size dataset. In this demo, with a few tens of entries, there's no noticeable difference -- but you get the idea.

## Conclusion

Congratulations! You have learned how to use OpenAI for vector embeddings and Cassandra / Astra DB through CQL for storage in order to build a sophisticated philosophical search engine and quote generator.

This example used [CassIO](https://cassio.org) to interface with the Vector Store - but this is not the only choice. Check the [README](https://github.com/openai/openai-cookbook/tree/main/examples/vector_databases/cassandra_astradb) for other options and integration with popular frameworks.

To find out more on how Astra DB's Vector Search capabilities can be a key ingredient in your ML/GenAI applications, visit [Astra DB](https://docs.datastax.com/en/astra/home/astra.html)'s web page on the topic.

## Cleanup

If you want to remove all resources used for this demo, run this cell (_warning: this will delete the tables and the data inserted in them!_):


```python
# we peek at CassIO's config to get a direct handle to the DB connection
session = cassio.config.resolve_session()
keyspace = cassio.config.resolve_keyspace()

session.execute(f"DROP TABLE IF EXISTS {keyspace}.philosophers_cassio;")
session.execute(f"DROP TABLE IF EXISTS {keyspace}.philosophers_cassio_partitioned;")
```




    <cassandra.cluster.ResultSet at 0x7fdcc42e8f10>


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/README.ipynb


key: _node_content
string_value:
{"id_": "0e39c9ce-ea3b-4de8-8c29-80717d19d8ba", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/README.ipynb", "title": "RAG with Astra DB and Cassandra", "tokens": 669, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3cf88420-64a3-475f-a53a-07bcf964fd02", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/README.ipynb", "title": "RAG with Astra DB and Cassandra", "tokens": 669, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "75a90d81b31b38a6276c58f70dec4f8a6484ba5b0f1488314322d7da05d6bcd1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2631, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# RAG with Astra DB and Cassandra

The demos in this directory show how to use the Vector
Search capabilities available today in **DataStax Astra DB**, a serverless
Database-as-a-Service built on Apache Cassandra®.

These example notebooks demonstrate implementation of
the same GenAI standard RAG workload with different libraries and APIs.

To use [Astra DB](https://docs.datastax.com/en/astra/home/astra.html)
with its HTTP API interface, head to the "AstraPy" notebook (`astrapy`
is the Python client to interact with the database).

If you prefer CQL access to the database (either with
[Astra DB](https://docs.datastax.com/en/astra-serverless/docs/vector-search/overview.html)
or a Cassandra cluster
[supporting vector search](https://cassandra.apache.org/doc/trunk/cassandra/vector-search/overview.html)),
check the "CQL" or "CassIO" notebooks -- they differ in the level of abstraction you get to work at.

If you want to know more about Astra DB and its Vector Search capabilities,
head over to [datastax.com](https://docs.datastax.com/en/astra/home/astra.html).

### Example notebooks

The following examples show how easily OpenAI and DataStax Astra DB can
work together to power vector-based AI applications. You can run them either
with your local Jupyter engine or as Colab notebooks:

| Use case | Target database | Framework | Notebook | Google Colab |
| -------- | --------------- | --------- | -------- | ------------ |
| Search/generate quotes | Astra DB | AstraPy | [Notebook](./Philosophical_Quotes_AstraPy.ipynb) | [![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb) |
| Search/generate quotes | Cassandra / Astra DB through CQL | CassIO | [Notebook](./Philosophical_Quotes_cassIO.ipynb) | [![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb) |
| Search/generate quotes | Cassandra / Astra DB through CQL | Plain Cassandra language | [Notebook](./Philosophical_Quotes_CQL.ipynb) | [![Colab](https://colab.research.google.com/assets/colab-badge.svg)](https://colab.research.google.com/github/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb) |

### Vector similarity, visual representation

![3_vector_space](https://user-images.githubusercontent.com/14221764/262321363-c8c625c1-8be9-450e-8c68-b1ed518f990d.png)


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Using_MyScale_for_embeddings_search.ipynb


key: _node_content
string_value:
{"id_": "05004896-9f2e-472f-9a14-3f47bc18bbe0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Using_MyScale_for_embeddings_search.ipynb", "title": "Using MyScale for Embeddings Search", "tokens": 2754, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "db892807-c113-4bfd-9704-8d93dd9aff1a", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Using_MyScale_for_embeddings_search.ipynb", "title": "Using MyScale for Embeddings Search", "tokens": 2754, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "fd236f6738bea8e07f4ddca6cb76596d7f2c7a3b7052ead7cb5c93156357c356", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e5626353-aa0f-441a-8d70-f68a4fed54cc", "node_type": "1", "metadata": {}, "hash": "de6fcb349e7fc5d729b2ec4d246c9b47eb380869b5124a394c53b81a7a83c9c9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3615, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Using MyScale for Embeddings Search

This notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.

### What is a Vector Database

A vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.

### Why use a Vector Database

Vector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.


### Demo Flow
The demo flow is:
- **Setup**: Import packages and set any required variables
- **Load data**: Load a dataset and embed it using OpenAI embeddings
- **MyScale**
    - *Setup*: Set up the MyScale Python client. For more details go [here](https://docs.myscale.com/en/python-client/)
    - *Index Data*: We'll create a table and index it for __content__.
    - *Search Data*: Run a few example queries with various goals in mind.

Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.

## Setup

Import the required libraries and set the embedding model that we'd like to use.


```python
# We'll need to install the MyScale client
!pip install clickhouse-connect

#Install wget to pull zip file
!pip install wget
```


```python
import openai

from typing import List, Iterator
import pandas as pd
import numpy as np
import os
import wget
from ast import literal_eval

# MyScale's client library for Python
import clickhouse_connect

# I've set this to our new embeddings model, this can be changed to the embedding model of your choice
EMBEDDING_MODEL = "text-embedding-3-small"

# Ignore unclosed SSL socket warnings - optional in case you get these errors
import warnings

warnings.filterwarnings(action="ignore", message="unclosed", category=ResourceWarning)
warnings.filterwarnings("ignore", category=DeprecationWarning) 
```

## Load data

In this section we'll load embedded data that we've prepared previous to this session.


```python
embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```


```python
import zipfile
with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip","r") as zip_ref:
    zip_ref.extractall("../data")
```


```python
article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')
```


```python
article_df.head()
```


key: _node_content
string_value:
{"id_": "e5626353-aa0f-441a-8d70-f68a4fed54cc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Using_MyScale_for_embeddings_search.ipynb", "title": "Using MyScale for Embeddings Search", "tokens": 2754, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "db892807-c113-4bfd-9704-8d93dd9aff1a", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Using_MyScale_for_embeddings_search.ipynb", "title": "Using MyScale for Embeddings Search", "tokens": 2754, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "fd236f6738bea8e07f4ddca6cb76596d7f2c7a3b7052ead7cb5c93156357c356", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "05004896-9f2e-472f-9a14-3f47bc18bbe0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Using_MyScale_for_embeddings_search.ipynb", "title": "Using MyScale for Embeddings Search", "tokens": 2754, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e471cb199bbb6499ed8c87a093f68be47521ee8a5f800c81b13f25d00f7df59b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dbc212a8-890c-4a9e-a788-20c4be128fa1", "node_type": "1", "metadata": {}, "hash": "55d75d734c6c465942a4bbe99c9e2550ca1dde90d5af183d906f3fc23ae149c3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2269, "end_char_idx": 4631, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# We'll need to install the MyScale client
!pip install clickhouse-connect

#Install wget to pull zip file
!pip install wget
```


```python
import openai

from typing import List, Iterator
import pandas as pd
import numpy as np
import os
import wget
from ast import literal_eval

# MyScale's client library for Python
import clickhouse_connect

# I've set this to our new embeddings model, this can be changed to the embedding model of your choice
EMBEDDING_MODEL = "text-embedding-3-small"

# Ignore unclosed SSL socket warnings - optional in case you get these errors
import warnings

warnings.filterwarnings(action="ignore", message="unclosed", category=ResourceWarning)
warnings.filterwarnings("ignore", category=DeprecationWarning) 
```

## Load data

In this section we'll load embedded data that we've prepared previous to this session.


```python
embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```


```python
import zipfile
with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip","r") as zip_ref:
    zip_ref.extractall("../data")
```


```python
article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')
```


```python
article_df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>url</th>
      <th>title</th>
      <th>text</th>
      <th>title_vector</th>
      <th>content_vector</th>
      <th>vector_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>https://simple.wikipedia.org/wiki/April</td>
      <td>April</td>
      <td>April is the fourth month of the year in the J...</td>
      <td>[0.001009464613161981, -0.020700545981526375, ...</td>
      <td>[-0.011253940872848034, -0.013491976074874401,...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>https://simple.wikipedia.org/wiki/August</td>
      <td>August</td>
      <td>August (Aug.)


key: _node_content
string_value:
{"id_": "dbc212a8-890c-4a9e-a788-20c4be128fa1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Using_MyScale_for_embeddings_search.ipynb", "title": "Using MyScale for Embeddings Search", "tokens": 2754, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "db892807-c113-4bfd-9704-8d93dd9aff1a", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Using_MyScale_for_embeddings_search.ipynb", "title": "Using MyScale for Embeddings Search", "tokens": 2754, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "fd236f6738bea8e07f4ddca6cb76596d7f2c7a3b7052ead7cb5c93156357c356", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e5626353-aa0f-441a-8d70-f68a4fed54cc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Using_MyScale_for_embeddings_search.ipynb", "title": "Using MyScale for Embeddings Search", "tokens": 2754, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a3e97206888675368b528c22d59c7fef64d48483dff473c9ba0afe3b73012aab", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dd5ce74c-81a2-4c94-8423-776fc1237d32", "node_type": "1", "metadata": {}, "hash": "0eb9f1fbcad7c87e0ae1cc80d55f22c888212d66f0901dbcd88725f848899c3c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3484, "end_char_idx": 5662, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: _node_content
string_value:
{"id_": "dd5ce74c-81a2-4c94-8423-776fc1237d32", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Using_MyScale_for_embeddings_search.ipynb", "title": "Using MyScale for Embeddings Search", "tokens": 2754, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "db892807-c113-4bfd-9704-8d93dd9aff1a", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Using_MyScale_for_embeddings_search.ipynb", "title": "Using MyScale for Embeddings Search", "tokens": 2754, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "fd236f6738bea8e07f4ddca6cb76596d7f2c7a3b7052ead7cb5c93156357c356", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dbc212a8-890c-4a9e-a788-20c4be128fa1", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Using_MyScale_for_embeddings_search.ipynb", "title": "Using MyScale for Embeddings Search", "tokens": 2754, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "9c2e89c85f6d1531055a4490587a8ee7a4f4b55ba5b4847f5b570539af44ab8b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "76657275-cb4a-4138-8dc4-4a1363aee852", "node_type": "1", "metadata": {}, "hash": "838f0cc3d0ead1c682db56092fdb0a44134572846028a508c3f613d3bf701d6e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4632, "end_char_idx": 6830, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: _node_content
string_value:
{"id_": "76657275-cb4a-4138-8dc4-4a1363aee852", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Using_MyScale_for_embeddings_search.ipynb", "title": "Using MyScale for Embeddings Search", "tokens": 2754, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "db892807-c113-4bfd-9704-8d93dd9aff1a", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Using_MyScale_for_embeddings_search.ipynb", "title": "Using MyScale for Embeddings Search", "tokens": 2754, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "fd236f6738bea8e07f4ddca6cb76596d7f2c7a3b7052ead7cb5c93156357c356", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dd5ce74c-81a2-4c94-8423-776fc1237d32", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Using_MyScale_for_embeddings_search.ipynb", "title": "Using MyScale for Embeddings Search", "tokens": 2754, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a56655af6402a8182db44ddf3b0cebb90c6b748b9983e5bd48449b2e03e07f04", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "09b77a37-4a4f-48d9-b734-f0625cf4434d", "node_type": "1", "metadata": {}, "hash": "05f076a3140590c1ce448473600f589e8e8bd58f97fcb49d5f379d071502612b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5663, "end_char_idx": 8125, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Air is a...</td>
      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>
      <td>[0.021524671465158463, 0.018522677943110466, -...</td>
      <td>4</td>
    </tr>
  </tbody>
</table>
</div>




```python
# Read vectors from strings back into a list
article_df['title_vector'] = article_df.title_vector.apply(literal_eval)
article_df['content_vector'] = article_df.content_vector.apply(literal_eval)

# Set vector_id to be a string
article_df['vector_id'] = article_df['vector_id'].apply(str)
```


```python
article_df.info(show_counts=True)
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 25000 entries, 0 to 24999
    Data columns (total 7 columns):
     #   Column          Non-Null Count  Dtype 
    ---  ------          --------------  ----- 
     0   id              25000 non-null  int64 
     1   url             25000 non-null  object
     2   title           25000 non-null  object
     3   text            25000 non-null  object
     4   title_vector    25000 non-null  object
     5   content_vector  25000 non-null  object
     6   vector_id       25000 non-null  object
    dtypes: int64(1), object(6)
    memory usage: 1.3+ MB


## MyScale
The next vector database we'll consider is [MyScale](https://myscale.com).

[MyScale](https://myscale.com) is a database built on Clickhouse that combines vector search and SQL analytics to offer a high-performance, streamlined, and fully managed experience. It's designed to facilitate joint queries and analyses on both structured and vector data, with comprehensive SQL support for all data processing.

Deploy and execute vector search with SQL on your cluster within two minutes by using [MyScale Console](https://console.myscale.com).

### Connect to MyScale

Follow the [connections details](https://docs.myscale.com/en/cluster-management/) section to retrieve the cluster host, username, and password information from the MyScale console, and use it to create a connection to your cluster as shown below:


```python
# initialize client
client = clickhouse_connect.get_client(host='YOUR_CLUSTER_HOST', port=8443, username='YOUR_USERNAME', password='YOUR_CLUSTER_PASSWORD')
```

### Index data

We will create an SQL table called `articles` in MyScale to store the embeddings data. The table will include a vector index with a cosine distance metric and a constraint for the length of the embeddings. Use the following code to create and insert data into the articles table:


key: _node_content
string_value:
{"id_": "09b77a37-4a4f-48d9-b734-f0625cf4434d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Using_MyScale_for_embeddings_search.ipynb", "title": "Using MyScale for Embeddings Search", "tokens": 2754, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "db892807-c113-4bfd-9704-8d93dd9aff1a", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Using_MyScale_for_embeddings_search.ipynb", "title": "Using MyScale for Embeddings Search", "tokens": 2754, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "fd236f6738bea8e07f4ddca6cb76596d7f2c7a3b7052ead7cb5c93156357c356", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "76657275-cb4a-4138-8dc4-4a1363aee852", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Using_MyScale_for_embeddings_search.ipynb", "title": "Using MyScale for Embeddings Search", "tokens": 2754, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d47d5c114761eea9fc5e27ae93bb6ec88b57f8abd19c1a53d16afcd3254b4e6a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2305173c-947b-49d3-89e1-ecf6bfcc9c49", "node_type": "1", "metadata": {}, "hash": "61fa91af72ee4f5fd8778ca60338ac801a3bf99a3055b2aa620ede2d6135e6ef", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6833, "end_char_idx": 9998, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## MyScale
The next vector database we'll consider is [MyScale](https://myscale.com).

[MyScale](https://myscale.com) is a database built on Clickhouse that combines vector search and SQL analytics to offer a high-performance, streamlined, and fully managed experience. It's designed to facilitate joint queries and analyses on both structured and vector data, with comprehensive SQL support for all data processing.

Deploy and execute vector search with SQL on your cluster within two minutes by using [MyScale Console](https://console.myscale.com).

### Connect to MyScale

Follow the [connections details](https://docs.myscale.com/en/cluster-management/) section to retrieve the cluster host, username, and password information from the MyScale console, and use it to create a connection to your cluster as shown below:


```python
# initialize client
client = clickhouse_connect.get_client(host='YOUR_CLUSTER_HOST', port=8443, username='YOUR_USERNAME', password='YOUR_CLUSTER_PASSWORD')
```

### Index data

We will create an SQL table called `articles` in MyScale to store the embeddings data. The table will include a vector index with a cosine distance metric and a constraint for the length of the embeddings. Use the following code to create and insert data into the articles table:


```python
# create articles table with vector index
embedding_len=len(article_df['content_vector'][0]) # 1536

client.command(f"""
CREATE TABLE IF NOT EXISTS default.articles
(
    id UInt64,
    url String,
    title String,
    text String,
    content_vector Array(Float32),
    CONSTRAINT cons_vector_len CHECK length(content_vector) = {embedding_len},
    VECTOR INDEX article_content_index content_vector TYPE HNSWFLAT('metric_type=Cosine')
)
ENGINE = MergeTree ORDER BY id
""")

# insert data into the table in batches
from tqdm.auto import tqdm

batch_size = 100
total_records = len(article_df)

# we only need subset of columns
article_df = article_df[['id', 'url', 'title', 'text', 'content_vector']]

# upload data in batches
data = article_df.to_records(index=False).tolist()
column_names = article_df.columns.tolist()

for i in tqdm(range(0, total_records, batch_size)):
    i_end = min(i + batch_size, total_records)
    client.insert("default.articles", data[i:i_end], column_names=column_names)
```


      0%|          | 0/250 [00:00<?, ?it/s]


We need to check the build status of the vector index before proceeding with the search, as it is automatically built in the background.


```python
# check count of inserted data
print(f"articles count: {client.command('SELECT count(*) FROM default.articles')}")

# check the status of the vector index, make sure vector index is ready with 'Built' status
get_index_status="SELECT status FROM system.vector_indices WHERE name='article_content_index'"
print(f"index build status: {client.command(get_index_status)}")
```

    articles count: 25000
    index build status: InProgress


### Search data

Once indexed in MyScale, we can perform vector search to find similar content. First, we will use the OpenAI API to generate embeddings for our query. Then, we will perform the vector search using MyScale.


key: _node_content
string_value:
{"id_": "2305173c-947b-49d3-89e1-ecf6bfcc9c49", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Using_MyScale_for_embeddings_search.ipynb", "title": "Using MyScale for Embeddings Search", "tokens": 2754, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "db892807-c113-4bfd-9704-8d93dd9aff1a", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Using_MyScale_for_embeddings_search.ipynb", "title": "Using MyScale for Embeddings Search", "tokens": 2754, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "fd236f6738bea8e07f4ddca6cb76596d7f2c7a3b7052ead7cb5c93156357c356", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "09b77a37-4a4f-48d9-b734-f0625cf4434d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Using_MyScale_for_embeddings_search.ipynb", "title": "Using MyScale for Embeddings Search", "tokens": 2754, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "cfd9467fd965c3588e2a5d9696ed8d394806b2e017ae63e0a64e1470d05d9642", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9151, "end_char_idx": 10771, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0%|          | 0/250 [00:00<?, ?it/s]


We need to check the build status of the vector index before proceeding with the search, as it is automatically built in the background.


```python
# check count of inserted data
print(f"articles count: {client.command('SELECT count(*) FROM default.articles')}")

# check the status of the vector index, make sure vector index is ready with 'Built' status
get_index_status="SELECT status FROM system.vector_indices WHERE name='article_content_index'"
print(f"index build status: {client.command(get_index_status)}")
```

    articles count: 25000
    index build status: InProgress


### Search data

Once indexed in MyScale, we can perform vector search to find similar content. First, we will use the OpenAI API to generate embeddings for our query. Then, we will perform the vector search using MyScale.


```python
query = "Famous battles in Scottish history"

# creates embedding vector from user query
embed = openai.Embedding.create(
    input=query,
    model="text-embedding-3-small",
)["data"][0]["embedding"]

# query the database to find the top K similar content to the given query
top_k = 10
results = client.query(f"""
SELECT id, url, title, distance(content_vector, {embed}) as dist
FROM default.articles
ORDER BY dist
LIMIT {top_k}
""")

# display results
for i, r in enumerate(results.named_results()):
    print(i+1, r['title'])
```

    1 Battle of Bannockburn
    2 Wars of Scottish Independence
    3 1651
    4 First War of Scottish Independence
    5 Robert I of Scotland
    6 841
    7 1716
    8 1314
    9 1263
    10 William Wallace



```python

```


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Getting_started_with_MyScale_and_OpenAI.ipynb


key: _node_content
string_value:
{"id_": "b003fc65-fd93-4c6f-ac2a-785f0ca687a9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Getting_started_with_MyScale_and_OpenAI.ipynb", "title": "Using MyScale as a vector database for OpenAI embeddings", "tokens": 1436, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f3e34571-f8ca-4a99-b961-b408dfc39dcc", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Getting_started_with_MyScale_and_OpenAI.ipynb", "title": "Using MyScale as a vector database for OpenAI embeddings", "tokens": 1436, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b632e784f373b0fcc7ae9a4af37d40088bd2c40a65431319bd4b3f20aa87cfb8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fddb1dc3-2a22-4e44-b400-0286e5aad4eb", "node_type": "1", "metadata": {}, "hash": "8a76c3a6e64bd14b5ccee4ff2e05899e0b53e7a20c7c6b09d8dbd1d6b8eec0f0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3102, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Using MyScale as a vector database for OpenAI embeddings

This notebook provides a step-by-step guide on using MyScale as a vector database for OpenAI embeddings. The process includes:

1. Utilizing precomputed embeddings generated by OpenAI API.
2. Storing these embeddings in a cloud instance of MyScale.
3. Converting raw text query to an embedding using OpenAI API.
4. Leveraging MyScale to perform nearest neighbor search within the created collection.

### What is MyScale

[MyScale](https://myscale.com) is a database built on Clickhouse that combines vector search and SQL analytics to offer a high-performance, streamlined, and fully managed experience. It's designed to facilitate joint queries and analyses on both structured and vector data, with comprehensive SQL support for all data processing.


### Deployment options

- Deploy and execute vector search with SQL on your cluster within two minutes by using [MyScale Console](https://console.myscale.com).


## Prerequisites

To follow this guide, you will need to have the following:

1. A MyScale cluster deployed by following the [quickstart guide](https://docs.myscale.com/en/quickstart/).
2. The 'clickhouse-connect' library to interact with MyScale.
3. An [OpenAI API key](https://beta.openai.com/account/api-keys) for vectorization of queries.

### Install requirements

This notebook requires the `openai`, `clickhouse-connect`, as well as some other dependencies. Use the following command to install them:


```python
! pip install openai clickhouse-connect wget pandas
```

### Prepare your OpenAI API key

To use the OpenAI API, you'll need to set up an API key. If you don't have one already, you can obtain it from [OpenAI](https://platform.openai.com/account/api-keys).


```python
import openai

# get API key from on OpenAI website
openai.api_key = "OPENAI_API_KEY"

# check we have authenticated
openai.Engine.list()
```

## Connect to MyScale

Follow the [connections details](https://docs.myscale.com/en/cluster-management/) section to retrieve the cluster host, username, and password information from the MyScale console, and use it to create a connection to your cluster as shown below:


```python
import clickhouse_connect

# initialize client
client = clickhouse_connect.get_client(host='YOUR_CLUSTER_HOST', port=8443, username='YOUR_USERNAME', password='YOUR_CLUSTER_PASSWORD')
```

## Load data

We need to load the dataset of precomputed vector embeddings for Wikipedia articles provided by OpenAI. Use the `wget` package to download the dataset.


```python
import wget

embeddings_url = "https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip"

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```

After the download is complete, extract the file using the `zipfile` package:


```python
import zipfile

with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip", "r") as zip_ref:
    zip_ref.extractall("../data")
```

Now, we can load the data from `vector_database_wikipedia_articles_embedded.csv` into a Pandas DataFrame:


key: _node_content
string_value:
{"id_": "fddb1dc3-2a22-4e44-b400-0286e5aad4eb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Getting_started_with_MyScale_and_OpenAI.ipynb", "title": "Using MyScale as a vector database for OpenAI embeddings", "tokens": 1436, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f3e34571-f8ca-4a99-b961-b408dfc39dcc", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Getting_started_with_MyScale_and_OpenAI.ipynb", "title": "Using MyScale as a vector database for OpenAI embeddings", "tokens": 1436, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b632e784f373b0fcc7ae9a4af37d40088bd2c40a65431319bd4b3f20aa87cfb8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b003fc65-fd93-4c6f-ac2a-785f0ca687a9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Getting_started_with_MyScale_and_OpenAI.ipynb", "title": "Using MyScale as a vector database for OpenAI embeddings", "tokens": 1436, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f33a141bc23a2733bfc6ef917a21145de72bc70eb786f9e0bf3d430e8425c106", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0dbab7de-5272-45d5-ad41-e65cdd88881f", "node_type": "1", "metadata": {}, "hash": "6b2f834918e39c5a8b4a72408f93ee070cbc9cf14f0ad2e37a9a14582618fc74", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1486, "end_char_idx": 4833, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
! pip install openai clickhouse-connect wget pandas
```

### Prepare your OpenAI API key

To use the OpenAI API, you'll need to set up an API key. If you don't have one already, you can obtain it from [OpenAI](https://platform.openai.com/account/api-keys).


```python
import openai

# get API key from on OpenAI website
openai.api_key = "OPENAI_API_KEY"

# check we have authenticated
openai.Engine.list()
```

## Connect to MyScale

Follow the [connections details](https://docs.myscale.com/en/cluster-management/) section to retrieve the cluster host, username, and password information from the MyScale console, and use it to create a connection to your cluster as shown below:


```python
import clickhouse_connect

# initialize client
client = clickhouse_connect.get_client(host='YOUR_CLUSTER_HOST', port=8443, username='YOUR_USERNAME', password='YOUR_CLUSTER_PASSWORD')
```

## Load data

We need to load the dataset of precomputed vector embeddings for Wikipedia articles provided by OpenAI. Use the `wget` package to download the dataset.


```python
import wget

embeddings_url = "https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip"

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```

After the download is complete, extract the file using the `zipfile` package:


```python
import zipfile

with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip", "r") as zip_ref:
    zip_ref.extractall("../data")
```

Now, we can load the data from `vector_database_wikipedia_articles_embedded.csv` into a Pandas DataFrame:


```python
import pandas as pd

from ast import literal_eval

# read data from csv
article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')
article_df = article_df[['id', 'url', 'title', 'text', 'content_vector']]

# read vectors from strings back into a list
article_df["content_vector"] = article_df.content_vector.apply(literal_eval)
article_df.head()
```

## Index data

We will create an SQL table called `articles` in MyScale to store the embeddings data. The table will include a vector index with a cosine distance metric and a constraint for the length of the embeddings. Use the following code to create and insert data into the articles table:


```python
# create articles table with vector index
embedding_len=len(article_df['content_vector'][0]) # 1536

client.command(f"""
CREATE TABLE IF NOT EXISTS default.articles
(
    id UInt64,
    url String,
    title String,
    text String,
    content_vector Array(Float32),
    CONSTRAINT cons_vector_len CHECK length(content_vector) = {embedding_len},
    VECTOR INDEX article_content_index content_vector TYPE HNSWFLAT('metric_type=Cosine')
)
ENGINE = MergeTree ORDER BY id
""")

# insert data into the table in batches
from tqdm.auto import tqdm

batch_size = 100
total_records = len(article_df)

# upload data in batches
data = article_df.to_records(index=False).tolist()
column_names = article_df.columns.tolist() 

for i in tqdm(range(0, total_records, batch_size)):
    i_end = min(i + batch_size, total_records)
    client.insert("default.articles", data[i:i_end], column_names=column_names)
```

We need to check the build status of the vector index before proceeding with the search, as it is automatically built in the background.


key: _node_content
string_value:
{"id_": "0dbab7de-5272-45d5-ad41-e65cdd88881f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Getting_started_with_MyScale_and_OpenAI.ipynb", "title": "Using MyScale as a vector database for OpenAI embeddings", "tokens": 1436, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "f3e34571-f8ca-4a99-b961-b408dfc39dcc", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Getting_started_with_MyScale_and_OpenAI.ipynb", "title": "Using MyScale as a vector database for OpenAI embeddings", "tokens": 1436, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b632e784f373b0fcc7ae9a4af37d40088bd2c40a65431319bd4b3f20aa87cfb8", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fddb1dc3-2a22-4e44-b400-0286e5aad4eb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/myscale/Getting_started_with_MyScale_and_OpenAI.ipynb", "title": "Using MyScale as a vector database for OpenAI embeddings", "tokens": 1436, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "21498eff6a29b2ceeca6a0f120bb954b1a15c391662db77734e59ce4e4f1239b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3105, "end_char_idx": 6286, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import pandas as pd

from ast import literal_eval

# read data from csv
article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')
article_df = article_df[['id', 'url', 'title', 'text', 'content_vector']]

# read vectors from strings back into a list
article_df["content_vector"] = article_df.content_vector.apply(literal_eval)
article_df.head()
```

## Index data

We will create an SQL table called `articles` in MyScale to store the embeddings data. The table will include a vector index with a cosine distance metric and a constraint for the length of the embeddings. Use the following code to create and insert data into the articles table:


```python
# create articles table with vector index
embedding_len=len(article_df['content_vector'][0]) # 1536

client.command(f"""
CREATE TABLE IF NOT EXISTS default.articles
(
    id UInt64,
    url String,
    title String,
    text String,
    content_vector Array(Float32),
    CONSTRAINT cons_vector_len CHECK length(content_vector) = {embedding_len},
    VECTOR INDEX article_content_index content_vector TYPE HNSWFLAT('metric_type=Cosine')
)
ENGINE = MergeTree ORDER BY id
""")

# insert data into the table in batches
from tqdm.auto import tqdm

batch_size = 100
total_records = len(article_df)

# upload data in batches
data = article_df.to_records(index=False).tolist()
column_names = article_df.columns.tolist() 

for i in tqdm(range(0, total_records, batch_size)):
    i_end = min(i + batch_size, total_records)
    client.insert("default.articles", data[i:i_end], column_names=column_names)
```

We need to check the build status of the vector index before proceeding with the search, as it is automatically built in the background.


```python
# check count of inserted data
print(f"articles count: {client.command('SELECT count(*) FROM default.articles')}")

# check the status of the vector index, make sure vector index is ready with 'Built' status
get_index_status="SELECT status FROM system.vector_indices WHERE name='article_content_index'"
print(f"index build status: {client.command(get_index_status)}")
```

    articles count: 25000
    index build status: Built


## Search data

Once indexed in MyScale, we can perform vector search to find similar content. First, we will use the OpenAI API to generate embeddings for our query. Then, we will perform the vector search using MyScale.


```python
import openai

query = "Famous battles in Scottish history"

# creates embedding vector from user query
embed = openai.Embedding.create(
    input=query,
    model="text-embedding-3-small",
)["data"][0]["embedding"]

# query the database to find the top K similar content to the given query
top_k = 10
results = client.query(f"""
SELECT id, url, title, distance(content_vector, {embed}) as dist
FROM default.articles
ORDER BY dist
LIMIT {top_k}
""")

# display results
for i, r in enumerate(results.named_results()):
    print(i+1, r['title'])
```

    1 Battle of Bannockburn
    2 Wars of Scottish Independence
    3 1651
    4 First War of Scottish Independence
    5 Robert I of Scotland
    6 841
    7 1716
    8 1314
    9 1263
    10 William Wallace



```python

```


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb


key: _node_content
string_value:
{"id_": "183a08f5-cf64-4c67-96c9-e672864361b8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82976ccf-0844-46c4-bf27-ab24c7c51718", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b6c5f2c6d1fb2f4a10d0884c8ce3c2f5bff33cf9a04dcde31e91b606e959845a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3ea895ac-b9bf-464d-bdb1-6a428eab38a9", "node_type": "1", "metadata": {}, "hash": "94393d94dbf96973f2a08f44bb33ee84468dd8905e7524ff1ae7a5c76f79ab3b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2545, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Getting Started with Zilliz and OpenAI
### Finding your next book

In this notebook we will be going over generating embeddings of book descriptions with OpenAI and using those embeddings within Zilliz to find relevant books. The dataset in this example is sourced from HuggingFace datasets, and contains a little over 1 million title-description pairs.

Lets begin by first downloading the required libraries for this notebook:
- `openai` is used for communicating with the OpenAI embedding service
- `pymilvus` is used for communicating with the Zilliz instance
- `datasets` is used for downloading the dataset
- `tqdm` is used for the progress bars



```python
! pip install openai pymilvus datasets tqdm
```

    Looking in indexes: https://pypi.org/simple, https://pypi.ngc.nvidia.com
    Requirement already satisfied: openai in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (0.27.2)
    Requirement already satisfied: pymilvus in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (2.2.2)
    Requirement already satisfied: datasets in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (2.10.1)
    Requirement already satisfied: tqdm in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (4.64.1)
    Requirement already satisfied: requests>=2.20 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from openai) (2.28.2)
    Requirement already satisfied: aiohttp in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from openai) (3.8.4)
    Requirement already satisfied: ujson<=5.4.0,>=2.0.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pymilvus) (5.1.0)
    Requirement already satisfied: grpcio-tools<=1.48.0,>=1.47.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pymilvus) (1.47.2)
    Requirement already satisfied: grpcio<=1.48.0,>=1.47.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pymilvus) (1.47.2)
    Requirement already satisfied: mmh3<=3.0.0,>=2.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pymilvus) (3.0.0)
    Requirement already satisfied: pandas>=1.2.4 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pymilvus) (1.5.3)
    Requirement already satisfied: numpy>=1.17 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (1.23.


key: _node_content
string_value:
{"id_": "3ea895ac-b9bf-464d-bdb1-6a428eab38a9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82976ccf-0844-46c4-bf27-ab24c7c51718", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b6c5f2c6d1fb2f4a10d0884c8ce3c2f5bff33cf9a04dcde31e91b606e959845a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "183a08f5-cf64-4c67-96c9-e672864361b8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "acf9e7fed1517ea47eb2d8d4e7bbc02e1edc7ec037a8499def13d609157bdffb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ac49cf29-ba02-44c4-a96b-f69e9ec1b18e", "node_type": "1", "metadata": {}, "hash": "793c1173d057a697c7b7b16d1877283157031865aa64b7da6ce53cd3487f6780", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1422, "end_char_idx": 3680, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
9/site-packages (from openai) (2.28.2)
    Requirement already satisfied: aiohttp in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from openai) (3.8.4)
    Requirement already satisfied: ujson<=5.4.0,>=2.0.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pymilvus) (5.1.0)
    Requirement already satisfied: grpcio-tools<=1.48.0,>=1.47.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pymilvus) (1.47.2)
    Requirement already satisfied: grpcio<=1.48.0,>=1.47.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pymilvus) (1.47.2)
    Requirement already satisfied: mmh3<=3.0.0,>=2.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pymilvus) (3.0.0)
    Requirement already satisfied: pandas>=1.2.4 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pymilvus) (1.5.3)
    Requirement already satisfied: numpy>=1.17 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (1.23.5)
    Requirement already satisfied: xxhash in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (3.2.0)
    Requirement already satisfied: responses<0.19 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (0.18.0)
    Requirement already satisfied: dill<0.3.7,>=0.3.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (0.3.6)
    Requirement already satisfied: huggingface-hub<1.0.0,>=0.2.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (0.12.1)
    Requirement already satisfied: pyarrow>=6.0.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (10.0.1)
    Requirement already satisfied: multiprocess in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (0.70.14)
    Requirement already satisfied: pyyaml>=5.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (5.4.1)
    Requirement already satisfied: fsspec[http]>=2021.11.


key: _node_content
string_value:
{"id_": "ac49cf29-ba02-44c4-a96b-f69e9ec1b18e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82976ccf-0844-46c4-bf27-ab24c7c51718", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b6c5f2c6d1fb2f4a10d0884c8ce3c2f5bff33cf9a04dcde31e91b606e959845a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3ea895ac-b9bf-464d-bdb1-6a428eab38a9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ac7b42538ae6baa8b0a5410a6c47ec16faa159d891e112e7d4abda375e9a29cc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f37cdeab-9bd7-4280-9310-554f35733af9", "node_type": "1", "metadata": {}, "hash": "b38132ee4c7934debc6ef3b4d91632fe3f883876a8c25bef4fe996afcfa643c3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2442, "end_char_idx": 4804, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
17 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (1.23.5)
    Requirement already satisfied: xxhash in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (3.2.0)
    Requirement already satisfied: responses<0.19 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (0.18.0)
    Requirement already satisfied: dill<0.3.7,>=0.3.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (0.3.6)
    Requirement already satisfied: huggingface-hub<1.0.0,>=0.2.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (0.12.1)
    Requirement already satisfied: pyarrow>=6.0.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (10.0.1)
    Requirement already satisfied: multiprocess in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (0.70.14)
    Requirement already satisfied: pyyaml>=5.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (5.4.1)
    Requirement already satisfied: fsspec[http]>=2021.11.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (2023.1.0)
    Requirement already satisfied: packaging in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (23.0)
    Requirement already satisfied: frozenlist>=1.1.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (1.3.3)
    Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (4.0.2)
    Requirement already satisfied: aiosignal>=1.1.2 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (1.3.1)
    Requirement already satisfied: attrs>=17.3.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (22.2.0)
    Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (3.0.1)
    Requirement already satisfied: yarl<2.0,


key: _node_content
string_value:
{"id_": "f37cdeab-9bd7-4280-9310-554f35733af9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82976ccf-0844-46c4-bf27-ab24c7c51718", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b6c5f2c6d1fb2f4a10d0884c8ce3c2f5bff33cf9a04dcde31e91b606e959845a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ac49cf29-ba02-44c4-a96b-f69e9ec1b18e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a57ab2c1e9cb75fd97f5f25b6f71b7585a6a65c5182a99479d5909c2d254406e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fca21344-8502-40b3-86dc-c0809cbcc646", "node_type": "1", "metadata": {}, "hash": "86ee24fa828472193a329b5d1293c42eed72a0982f37a71454a210f8f4c3a8cb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3583, "end_char_idx": 5805, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
9/site-packages (from datasets) (5.4.1)
    Requirement already satisfied: fsspec[http]>=2021.11.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (2023.1.0)
    Requirement already satisfied: packaging in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from datasets) (23.0)
    Requirement already satisfied: frozenlist>=1.1.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (1.3.3)
    Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (4.0.2)
    Requirement already satisfied: aiosignal>=1.1.2 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (1.3.1)
    Requirement already satisfied: attrs>=17.3.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (22.2.0)
    Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (3.0.1)
    Requirement already satisfied: yarl<2.0,>=1.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (1.8.2)
    Requirement already satisfied: multidict<7.0,>=4.5 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (6.0.4)
    Requirement already satisfied: six>=1.5.2 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from grpcio<=1.48.0,>=1.47.0->pymilvus) (1.16.0)
    Requirement already satisfied: protobuf<4.0dev,>=3.12.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from grpcio-tools<=1.48.0,>=1.47.0->pymilvus) (3.20.1)
    Requirement already satisfied: setuptools in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from grpcio-tools<=1.48.0,>=1.47.0->pymilvus) (65.6.3)
    Requirement already satisfied: typing-extensions>=3.7.4.3 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from huggingface-hub<1.0.0,>=0.2.


key: _node_content
string_value:
{"id_": "fca21344-8502-40b3-86dc-c0809cbcc646", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82976ccf-0844-46c4-bf27-ab24c7c51718", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b6c5f2c6d1fb2f4a10d0884c8ce3c2f5bff33cf9a04dcde31e91b606e959845a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f37cdeab-9bd7-4280-9310-554f35733af9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "5f71d277c5719a9d2b93766401ca64f86f9aa3bc64bf00f9e06e29288437a415", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9c7fe386-48d0-4a33-8ef8-96d0aebbf422", "node_type": "1", "metadata": {}, "hash": "57939f9c9a9f8dba31243cd8861a5cc10c2daee5cb239cf4c2188d4daf93ab76", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4713, "end_char_idx": 6854, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
9/site-packages (from aiohttp->openai) (3.0.1)
    Requirement already satisfied: yarl<2.0,>=1.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (1.8.2)
    Requirement already satisfied: multidict<7.0,>=4.5 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from aiohttp->openai) (6.0.4)
    Requirement already satisfied: six>=1.5.2 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from grpcio<=1.48.0,>=1.47.0->pymilvus) (1.16.0)
    Requirement already satisfied: protobuf<4.0dev,>=3.12.0 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from grpcio-tools<=1.48.0,>=1.47.0->pymilvus) (3.20.1)
    Requirement already satisfied: setuptools in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from grpcio-tools<=1.48.0,>=1.47.0->pymilvus) (65.6.3)
    Requirement already satisfied: typing-extensions>=3.7.4.3 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from huggingface-hub<1.0.0,>=0.2.0->datasets) (4.5.0)
    Requirement already satisfied: filelock in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from huggingface-hub<1.0.0,>=0.2.0->datasets) (3.9.0)
    Requirement already satisfied: python-dateutil>=2.8.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pandas>=1.2.4->pymilvus) (2.8.2)
    Requirement already satisfied: pytz>=2020.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pandas>=1.2.4->pymilvus) (2022.7.1)
    Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from requests>=2.20->openai) (1.26.14)
    Requirement already satisfied: idna<4,>=2.5 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from requests>=2.20->openai) (3.4)
    Requirement already satisfied: certifi>=2017.4.17 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from requests>=2.20->openai) (2022.12.


key: _node_content
string_value:
{"id_": "9c7fe386-48d0-4a33-8ef8-96d0aebbf422", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82976ccf-0844-46c4-bf27-ab24c7c51718", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b6c5f2c6d1fb2f4a10d0884c8ce3c2f5bff33cf9a04dcde31e91b606e959845a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fca21344-8502-40b3-86dc-c0809cbcc646", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "928535087a30fa5d4401c38142b0894aeb6dfe0ceb6c52e010b473073171a03b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ad5b77fa-247a-4809-b93a-ab45a32aa30c", "node_type": "1", "metadata": {}, "hash": "522f0f87a1b2028b2c7c178f4b0e20f9999c6b94114c7d38e9052b39684d96ad", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5755, "end_char_idx": 8209, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
9/site-packages (from huggingface-hub<1.0.0,>=0.2.0->datasets) (4.5.0)
    Requirement already satisfied: filelock in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from huggingface-hub<1.0.0,>=0.2.0->datasets) (3.9.0)
    Requirement already satisfied: python-dateutil>=2.8.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pandas>=1.2.4->pymilvus) (2.8.2)
    Requirement already satisfied: pytz>=2020.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pandas>=1.2.4->pymilvus) (2022.7.1)
    Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from requests>=2.20->openai) (1.26.14)
    Requirement already satisfied: idna<4,>=2.5 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from requests>=2.20->openai) (3.4)
    Requirement already satisfied: certifi>=2017.4.17 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from requests>=2.20->openai) (2022.12.7)


To get Zilliz up and running take a look [here](https://zilliz.com/doc/quick_start). With your account and database set up, proceed to set the following values:
- URI: The URI your database is running on
- USER: Your database username
- PASSWORD: Your database password
- COLLECTION_NAME: What to name the collection within Zilliz
- DIMENSION: The dimension of the embeddings
- OPENAI_ENGINE: Which embedding model to use
- openai.api_key: Your OpenAI account key
- INDEX_PARAM: The index settings to use for the collection
- QUERY_PARAM: The search parameters to use
- BATCH_SIZE: How many texts to embed and insert at once


```python
import openai

URI = 'your_uri'
TOKEN = 'your_token' # TOKEN == user:password or api_key
COLLECTION_NAME = 'book_search'
DIMENSION = 1536
OPENAI_ENGINE = 'text-embedding-3-small'
openai.api_key = 'sk-your-key'

INDEX_PARAM = {
    'metric_type':'L2',
    'index_type':"AUTOINDEX",
    'params':{}
}

QUERY_PARAM = {
    "metric_type": "L2",
    "params": {},
}

BATCH_SIZE = 1000
```

## Zilliz
This segment deals with Zilliz and setting up the database for this use case. Within Zilliz we need to setup a collection and index it.


```python
from pymilvus import connections, utility, FieldSchema, Collection, CollectionSchema, DataType

# Connect to Zilliz Database
connections.connect(uri=URI, token=TOKEN)
```


key: _node_content
string_value:
{"id_": "ad5b77fa-247a-4809-b93a-ab45a32aa30c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82976ccf-0844-46c4-bf27-ab24c7c51718", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b6c5f2c6d1fb2f4a10d0884c8ce3c2f5bff33cf9a04dcde31e91b606e959845a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9c7fe386-48d0-4a33-8ef8-96d0aebbf422", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1e317fac7235ba2ad3b5ae617dfb0685762c58a84f847ef69b202e1a81dc9e97", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7765b3da-4c39-4f0d-b5e6-c174d5c8d0a9", "node_type": "1", "metadata": {}, "hash": "f22d726b9ef6cee8ebf19bf63ec633a4cf9dac904383b68174236ca0c3e623c4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6732, "end_char_idx": 9384, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
4.17 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from requests>=2.20->openai) (2022.12.7)


To get Zilliz up and running take a look [here](https://zilliz.com/doc/quick_start). With your account and database set up, proceed to set the following values:
- URI: The URI your database is running on
- USER: Your database username
- PASSWORD: Your database password
- COLLECTION_NAME: What to name the collection within Zilliz
- DIMENSION: The dimension of the embeddings
- OPENAI_ENGINE: Which embedding model to use
- openai.api_key: Your OpenAI account key
- INDEX_PARAM: The index settings to use for the collection
- QUERY_PARAM: The search parameters to use
- BATCH_SIZE: How many texts to embed and insert at once


```python
import openai

URI = 'your_uri'
TOKEN = 'your_token' # TOKEN == user:password or api_key
COLLECTION_NAME = 'book_search'
DIMENSION = 1536
OPENAI_ENGINE = 'text-embedding-3-small'
openai.api_key = 'sk-your-key'

INDEX_PARAM = {
    'metric_type':'L2',
    'index_type':"AUTOINDEX",
    'params':{}
}

QUERY_PARAM = {
    "metric_type": "L2",
    "params": {},
}

BATCH_SIZE = 1000
```

## Zilliz
This segment deals with Zilliz and setting up the database for this use case. Within Zilliz we need to setup a collection and index it.


```python
from pymilvus import connections, utility, FieldSchema, Collection, CollectionSchema, DataType

# Connect to Zilliz Database
connections.connect(uri=URI, token=TOKEN)
```


```python
# Remove collection if it already exists
if utility.has_collection(COLLECTION_NAME):
    utility.drop_collection(COLLECTION_NAME)
```


```python
# Create collection which includes the id, title, and embedding.
fields = [
    FieldSchema(name='id', dtype=DataType.INT64, is_primary=True, auto_id=True),
    FieldSchema(name='title', dtype=DataType.VARCHAR, max_length=64000),
    FieldSchema(name='description', dtype=DataType.VARCHAR, max_length=64000),
    FieldSchema(name='embedding', dtype=DataType.FLOAT_VECTOR, dim=DIMENSION)
]
schema = CollectionSchema(fields=fields)
collection = Collection(name=COLLECTION_NAME, schema=schema)
```


```python
# Create the index on the collection and load it.
collection.create_index(field_name="embedding", index_params=INDEX_PARAM)
collection.load()
```

## Dataset
With Zilliz up and running we can begin grabbing our data. `Hugging Face Datasets` is a hub that holds many different user datasets, and for this example we are using Skelebor's book dataset. This dataset contains title-description pairs for over 1 million books. We are going to embed each description and store it within Zilliz along with its title.


key: _node_content
string_value:
{"id_": "7765b3da-4c39-4f0d-b5e6-c174d5c8d0a9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82976ccf-0844-46c4-bf27-ab24c7c51718", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b6c5f2c6d1fb2f4a10d0884c8ce3c2f5bff33cf9a04dcde31e91b606e959845a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ad5b77fa-247a-4809-b93a-ab45a32aa30c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "cd1ef2bb697a5a9a54ad66d7746e1477f9bf6b2cdcaba543d9e100dc6ba1989c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "62a9303b-d27a-4532-8b98-7ca3066e4138", "node_type": "1", "metadata": {}, "hash": "7e32e021cb43564c9d7db39cffb1dc1bdce380941588321cef36d0d6a183688e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8029, "end_char_idx": 10865, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from pymilvus import connections, utility, FieldSchema, Collection, CollectionSchema, DataType

# Connect to Zilliz Database
connections.connect(uri=URI, token=TOKEN)
```


```python
# Remove collection if it already exists
if utility.has_collection(COLLECTION_NAME):
    utility.drop_collection(COLLECTION_NAME)
```


```python
# Create collection which includes the id, title, and embedding.
fields = [
    FieldSchema(name='id', dtype=DataType.INT64, is_primary=True, auto_id=True),
    FieldSchema(name='title', dtype=DataType.VARCHAR, max_length=64000),
    FieldSchema(name='description', dtype=DataType.VARCHAR, max_length=64000),
    FieldSchema(name='embedding', dtype=DataType.FLOAT_VECTOR, dim=DIMENSION)
]
schema = CollectionSchema(fields=fields)
collection = Collection(name=COLLECTION_NAME, schema=schema)
```


```python
# Create the index on the collection and load it.
collection.create_index(field_name="embedding", index_params=INDEX_PARAM)
collection.load()
```

## Dataset
With Zilliz up and running we can begin grabbing our data. `Hugging Face Datasets` is a hub that holds many different user datasets, and for this example we are using Skelebor's book dataset. This dataset contains title-description pairs for over 1 million books. We are going to embed each description and store it within Zilliz along with its title.


```python
import datasets

# Download the dataset and only use the `train` portion (file is around 800Mb)
dataset = datasets.load_dataset('Skelebor/book_titles_and_descriptions_en_clean', split='train')
```

    /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
      from .autonotebook import tqdm as notebook_tqdm
    Found cached dataset parquet (/Users/filiphaltmayer/.cache/huggingface/datasets/Skelebor___parquet/Skelebor--book_titles_and_descriptions_en_clean-3596935b1d8a7747/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)


## Insert the Data
Now that we have our data on our machine we can begin embedding it and inserting it into Zilliz. The embedding function takes in text and returns the embeddings in a list format.


```python
# Simple function that converts the texts to embeddings
def embed(texts):
    embeddings = openai.Embedding.create(
        input=texts,
        engine=OPENAI_ENGINE
    )
    return [x['embedding'] for x in embeddings['data']]

```

This next step does the actual inserting. Due to having so many datapoints, if you want to immediately test it out you can stop the inserting cell block early and move along. Doing this will probably decrease the accuracy of the results due to less datapoints, but it should still be good enough.


key: _node_content
string_value:
{"id_": "62a9303b-d27a-4532-8b98-7ca3066e4138", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82976ccf-0844-46c4-bf27-ab24c7c51718", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b6c5f2c6d1fb2f4a10d0884c8ce3c2f5bff33cf9a04dcde31e91b606e959845a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7765b3da-4c39-4f0d-b5e6-c174d5c8d0a9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "6a216533def6b31b3162179683aad1e56edc1d23239a115d730358e607c2831b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fcc7c584-24b2-4986-9137-b9c798c7002b", "node_type": "1", "metadata": {}, "hash": "742f91e58dcf19a4db25e3f50037b18c0f8b68482423e20deeaff18718ff21a5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9387, "end_char_idx": 11852, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import datasets

# Download the dataset and only use the `train` portion (file is around 800Mb)
dataset = datasets.load_dataset('Skelebor/book_titles_and_descriptions_en_clean', split='train')
```

    /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
      from .autonotebook import tqdm as notebook_tqdm
    Found cached dataset parquet (/Users/filiphaltmayer/.cache/huggingface/datasets/Skelebor___parquet/Skelebor--book_titles_and_descriptions_en_clean-3596935b1d8a7747/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)


## Insert the Data
Now that we have our data on our machine we can begin embedding it and inserting it into Zilliz. The embedding function takes in text and returns the embeddings in a list format.


```python
# Simple function that converts the texts to embeddings
def embed(texts):
    embeddings = openai.Embedding.create(
        input=texts,
        engine=OPENAI_ENGINE
    )
    return [x['embedding'] for x in embeddings['data']]

```

This next step does the actual inserting. Due to having so many datapoints, if you want to immediately test it out you can stop the inserting cell block early and move along. Doing this will probably decrease the accuracy of the results due to less datapoints, but it should still be good enough.


```python
from tqdm import tqdm

data = [
    [], # title
    [], # description
]

# Embed and insert in batches
for i in tqdm(range(0, len(dataset))):
    data[0].append(dataset[i]['title'])
    data[1].append(dataset[i]['description'])
    if len(data[0]) % BATCH_SIZE == 0:
        data.append(embed(data[1]))
        collection.insert(data)
        data = [[],[]]

# Embed and insert the remainder 
if len(data[0]) != 0:
    data.append(embed(data[1]))
    collection.insert(data)
    data = [[],[]]

```

      0%|          | 2999/1032335 [00:19<1:49:30, 156.66it/s]



    ---------------------------------------------------------------------------

    KeyboardInterrupt                         Traceback (most recent call last)

    Cell In[10], line 14
         12     if len(data[0]) % BATCH_SIZE == 0:
         13         data.append(embed(data[1]))
    ---> 14         collection.insert(data)
         15         data = [[],[]]
         17 # Embed and insert the remainder


key: _node_content
string_value:
{"id_": "fcc7c584-24b2-4986-9137-b9c798c7002b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82976ccf-0844-46c4-bf27-ab24c7c51718", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b6c5f2c6d1fb2f4a10d0884c8ce3c2f5bff33cf9a04dcde31e91b606e959845a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "62a9303b-d27a-4532-8b98-7ca3066e4138", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0c524b21667ce6a15cfd6da8b6b2ac0c814db9eb93a49c09de289c92089a3283", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6013faef-6501-4699-bbe6-6b3347c408bc", "node_type": "1", "metadata": {}, "hash": "4d2e5cfe3a0819a852554f9530ee7b0906b71dc9572645ca59703f21d5bc8f51", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10325, "end_char_idx": 13105, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Simple function that converts the texts to embeddings
def embed(texts):
    embeddings = openai.Embedding.create(
        input=texts,
        engine=OPENAI_ENGINE
    )
    return [x['embedding'] for x in embeddings['data']]

```

This next step does the actual inserting. Due to having so many datapoints, if you want to immediately test it out you can stop the inserting cell block early and move along. Doing this will probably decrease the accuracy of the results due to less datapoints, but it should still be good enough.


```python
from tqdm import tqdm

data = [
    [], # title
    [], # description
]

# Embed and insert in batches
for i in tqdm(range(0, len(dataset))):
    data[0].append(dataset[i]['title'])
    data[1].append(dataset[i]['description'])
    if len(data[0]) % BATCH_SIZE == 0:
        data.append(embed(data[1]))
        collection.insert(data)
        data = [[],[]]

# Embed and insert the remainder 
if len(data[0]) != 0:
    data.append(embed(data[1]))
    collection.insert(data)
    data = [[],[]]

```

      0%|          | 2999/1032335 [00:19<1:49:30, 156.66it/s]



    ---------------------------------------------------------------------------

    KeyboardInterrupt                         Traceback (most recent call last)

    Cell In[10], line 14
         12     if len(data[0]) % BATCH_SIZE == 0:
         13         data.append(embed(data[1]))
    ---> 14         collection.insert(data)
         15         data = [[],[]]
         17 # Embed and insert the remainder 


    File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/orm/collection.py:430, in Collection.insert(self, data, partition_name, timeout, **kwargs)
        427 entities = Prepare.prepare_insert_data(data, self._schema)
        429 conn = self._get_connection()
    --> 430 res = conn.batch_insert(self._name, entities, partition_name,
        431                         timeout=timeout, schema=self._schema_dict, **kwargs)
        433 if kwargs.get("_async", False):
        434     return MutationFuture(res)


    File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/decorators.py:105, in error_handler.<locals>.wrapper.<locals>.handler(*args, **kwargs)
        103 try:
        104     record_dict["RPC start"] = str(datetime.datetime.now())
    --> 105     return func(*args, **kwargs)
        106 except MilvusException as e:
        107     record_dict["RPC error"] = str(datetime.datetime.now())


    File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/decorators.py:136, in tracing_request.<locals>.wrapper.<locals>.handler(self, *args, **kwargs)
        134 if req_id:
        135     self.set_onetime_request_id(req_id)
    --> 136 ret = func(self, *args, **kwargs)
        137 return ret


key: _node_content
string_value:
{"id_": "6013faef-6501-4699-bbe6-6b3347c408bc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82976ccf-0844-46c4-bf27-ab24c7c51718", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b6c5f2c6d1fb2f4a10d0884c8ce3c2f5bff33cf9a04dcde31e91b606e959845a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fcc7c584-24b2-4986-9137-b9c798c7002b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c49403231652a2d90380782417a821323b4e2fda7eb780d97f689992e3569e60", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8b6d4e64-0006-42df-b1df-b31818ffedbb", "node_type": "1", "metadata": {}, "hash": "6767872afd7037435e62778bcd115e3866e6674730df2c9595788366ee9913ca", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11860, "end_char_idx": 14841, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/orm/collection.py:430, in Collection.insert(self, data, partition_name, timeout, **kwargs)
        427 entities = Prepare.prepare_insert_data(data, self._schema)
        429 conn = self._get_connection()
    --> 430 res = conn.batch_insert(self._name, entities, partition_name,
        431                         timeout=timeout, schema=self._schema_dict, **kwargs)
        433 if kwargs.get("_async", False):
        434     return MutationFuture(res)


    File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/decorators.py:105, in error_handler.<locals>.wrapper.<locals>.handler(*args, **kwargs)
        103 try:
        104     record_dict["RPC start"] = str(datetime.datetime.now())
    --> 105     return func(*args, **kwargs)
        106 except MilvusException as e:
        107     record_dict["RPC error"] = str(datetime.datetime.now())


    File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/decorators.py:136, in tracing_request.<locals>.wrapper.<locals>.handler(self, *args, **kwargs)
        134 if req_id:
        135     self.set_onetime_request_id(req_id)
    --> 136 ret = func(self, *args, **kwargs)
        137 return ret


    File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/decorators.py:50, in retry_on_rpc_failure.<locals>.wrapper.<locals>.handler(self, *args, **kwargs)
         48 while True:
         49     try:
    ---> 50         return func(self, *args, **kwargs)
         51     except grpc.RpcError as e:
         52         # DEADLINE_EXCEEDED means that the task wat not completed
         53         # UNAVAILABLE means that the service is not reachable currently
         54         # Reference: https://grpc.github.io/grpc/python/grpc.html#grpc-status-code
         55         if e.code() != grpc.StatusCode.DEADLINE_EXCEEDED and e.code() != grpc.StatusCode.UNAVAILABLE:


    File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/client/grpc_handler.py:378, in GrpcHandler.batch_insert(self, collection_name, entities, partition_name, timeout, **kwargs)
        375     f.add_callback(ts_utils.update_ts_on_mutation(collection_name))
        376     return f
    --> 378 response = rf.result()
        379 if response.status.error_code == 0:
        380     m = MutationResult(response)


    File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/grpc/_channel.py:733, in _MultiThreadedRendezvous.result(self, timeout)
        728 """Returns the result of the computation or raises its exception.
        729 
        730 See grpc.Future.result for the full API contract.
        731 """
        732 with self._state.condition:
    --> 733     timed_out = _common.wait(self._state.condition.wait,
        734                              self._is_complete,
        735                              timeout=timeout)
        736     if timed_out:
        737         raise grpc.FutureTimeoutError()


key: _node_content
string_value:
{"id_": "8b6d4e64-0006-42df-b1df-b31818ffedbb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82976ccf-0844-46c4-bf27-ab24c7c51718", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b6c5f2c6d1fb2f4a10d0884c8ce3c2f5bff33cf9a04dcde31e91b606e959845a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6013faef-6501-4699-bbe6-6b3347c408bc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a500f5c67d85a5d1c70232d760ab8ec669bb5ecf9fc4fd551fc1aba1fe0312d8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "95aa88f8-b557-42bf-898f-ef48616428a5", "node_type": "1", "metadata": {}, "hash": "7244597802fabfc2c67ceb47db7815fdd92be822f4dd1f6847aaafb29e408955", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13800, "end_char_idx": 16697, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/client/grpc_handler.py:378, in GrpcHandler.batch_insert(self, collection_name, entities, partition_name, timeout, **kwargs)
        375     f.add_callback(ts_utils.update_ts_on_mutation(collection_name))
        376     return f
    --> 378 response = rf.result()
        379 if response.status.error_code == 0:
        380     m = MutationResult(response)


    File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/grpc/_channel.py:733, in _MultiThreadedRendezvous.result(self, timeout)
        728 """Returns the result of the computation or raises its exception.
        729 
        730 See grpc.Future.result for the full API contract.
        731 """
        732 with self._state.condition:
    --> 733     timed_out = _common.wait(self._state.condition.wait,
        734                              self._is_complete,
        735                              timeout=timeout)
        736     if timed_out:
        737         raise grpc.FutureTimeoutError()


    File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/grpc/_common.py:141, in wait(wait_fn, wait_complete_fn, timeout, spin_cb)
        139 if timeout is None:
        140     while not wait_complete_fn():
    --> 141         _wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)
        142 else:
        143     end = time.time() + timeout


    File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/grpc/_common.py:106, in _wait_once(wait_fn, timeout, spin_cb)
        105 def _wait_once(wait_fn, timeout, spin_cb):
    --> 106     wait_fn(timeout=timeout)
        107     if spin_cb is not None:
        108         spin_cb()


    File ~/miniconda3/envs/haystack/lib/python3.9/threading.py:316, in Condition.wait(self, timeout)
        314 else:
        315     if timeout > 0:
    --> 316         gotit = waiter.acquire(True, timeout)
        317     else:
        318         gotit = waiter.acquire(False)


    KeyboardInterrupt: 


## Query the Database
With our data safely inserted in Zilliz, we can now perform a query. The query takes in a string or a list of strings and searches them. The results print out your provided description and the results that include the result score, the result title, and the result book description.


```python
import textwrap

def query(queries, top_k = 5):
    if type(queries) != list:
        queries = [queries]
    res = collection.search(embed(queries), anns_field='embedding', param=QUERY_PARAM, limit = top_k, output_fields=['title', 'description'])
    for i, hit in enumerate(res):
        print('Description:', queries[i])
        print('Results:')
        for ii, hits in enumerate(hit):
            print('\t' + 'Rank:', ii + 1, 'Score:', hits.score, 'Title:', hits.entity.get('title'))
            print(textwrap.fill(hits.entity.get('description'), 88))
            print()
```


key: _node_content
string_value:
{"id_": "95aa88f8-b557-42bf-898f-ef48616428a5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82976ccf-0844-46c4-bf27-ab24c7c51718", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b6c5f2c6d1fb2f4a10d0884c8ce3c2f5bff33cf9a04dcde31e91b606e959845a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8b6d4e64-0006-42df-b1df-b31818ffedbb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a94d6b4a4ae88434b4592024b806ef13cb652ba4cacb99d1e7280e9ac8bedb0f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "60dd6255-bd32-4e1d-a39a-3191683b8ded", "node_type": "1", "metadata": {}, "hash": "b34ff5d24f9ea8c12c0c6d6cca57060dee9c0292f3dc74eb1c42d8b41eba08a5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15199, "end_char_idx": 18111, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/grpc/_common.py:106, in _wait_once(wait_fn, timeout, spin_cb)
        105 def _wait_once(wait_fn, timeout, spin_cb):
    --> 106     wait_fn(timeout=timeout)
        107     if spin_cb is not None:
        108         spin_cb()


    File ~/miniconda3/envs/haystack/lib/python3.9/threading.py:316, in Condition.wait(self, timeout)
        314 else:
        315     if timeout > 0:
    --> 316         gotit = waiter.acquire(True, timeout)
        317     else:
        318         gotit = waiter.acquire(False)


    KeyboardInterrupt: 


## Query the Database
With our data safely inserted in Zilliz, we can now perform a query. The query takes in a string or a list of strings and searches them. The results print out your provided description and the results that include the result score, the result title, and the result book description.


```python
import textwrap

def query(queries, top_k = 5):
    if type(queries) != list:
        queries = [queries]
    res = collection.search(embed(queries), anns_field='embedding', param=QUERY_PARAM, limit = top_k, output_fields=['title', 'description'])
    for i, hit in enumerate(res):
        print('Description:', queries[i])
        print('Results:')
        for ii, hits in enumerate(hit):
            print('\t' + 'Rank:', ii + 1, 'Score:', hits.score, 'Title:', hits.entity.get('title'))
            print(textwrap.fill(hits.entity.get('description'), 88))
            print()
```


```python
query('Book about a k-9 from europe')
```

    Description: Book about a k-9 from europe
    Results:
    	Rank: 1 Score: 0.3047754764556885 Title: Bark M For Murder
    Who let the dogs out? Evildoers beware! Four of mystery fiction's top storytellers are
    setting the hounds on your trail -- in an incomparable quartet of crime stories with a
    canine edge. Man's (and woman's) best friends take the lead in this phenomenal
    collection of tales tense and surprising, humorous and thrilling: New York
    Timesbestselling author J.A. Jance's spellbinding saga of a scam-busting septuagenarian
    and her two golden retrievers; Anthony Award winner Virginia Lanier's pureblood thriller
    featuring bloodhounds and bloody murder; Chassie West's suspenseful stunner about a
    life-saving German shepherd and a ghastly forgotten crime; rising star Lee Charles
    Kelley's edge-of-your-seat yarn that pits an ex-cop/kennel owner and a yappy toy poodle
    against a craven killer.
    
    	Rank: 2 Score: 0.3283390402793884 Title: Texas K-9 Unit Christmas: Holiday Hero\Rescuing Christmas
    CHRISTMAS COMES WRAPPED IN DANGER Holiday Hero by Shirlee McCoy Emma Fairchild never
    expected to find trouble in sleepy Sagebrush, Texas. But when she's attacked and left
    for dead in her own diner, her childhood friend turned K-9 cop Lucas Harwood offers a
    chance at justice--and love.


key: _node_content
string_value:
{"id_": "60dd6255-bd32-4e1d-a39a-3191683b8ded", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82976ccf-0844-46c4-bf27-ab24c7c51718", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b6c5f2c6d1fb2f4a10d0884c8ce3c2f5bff33cf9a04dcde31e91b606e959845a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "95aa88f8-b557-42bf-898f-ef48616428a5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "22908863c812d315cafd96e524bd210c76cdffdbdeb593aa84eb2bd4e73a1528", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "21d603a9-631e-42ae-85f1-ad5674d9f99c", "node_type": "1", "metadata": {}, "hash": "1032f31ac641bc77c7a1c47e864023e9ab6f983354b3e7c91a6c5029a5e32cab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16700, "end_char_idx": 19777, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
query('Book about a k-9 from europe')
```

    Description: Book about a k-9 from europe
    Results:
    	Rank: 1 Score: 0.3047754764556885 Title: Bark M For Murder
    Who let the dogs out? Evildoers beware! Four of mystery fiction's top storytellers are
    setting the hounds on your trail -- in an incomparable quartet of crime stories with a
    canine edge. Man's (and woman's) best friends take the lead in this phenomenal
    collection of tales tense and surprising, humorous and thrilling: New York
    Timesbestselling author J.A. Jance's spellbinding saga of a scam-busting septuagenarian
    and her two golden retrievers; Anthony Award winner Virginia Lanier's pureblood thriller
    featuring bloodhounds and bloody murder; Chassie West's suspenseful stunner about a
    life-saving German shepherd and a ghastly forgotten crime; rising star Lee Charles
    Kelley's edge-of-your-seat yarn that pits an ex-cop/kennel owner and a yappy toy poodle
    against a craven killer.
    
    	Rank: 2 Score: 0.3283390402793884 Title: Texas K-9 Unit Christmas: Holiday Hero\Rescuing Christmas
    CHRISTMAS COMES WRAPPED IN DANGER Holiday Hero by Shirlee McCoy Emma Fairchild never
    expected to find trouble in sleepy Sagebrush, Texas. But when she's attacked and left
    for dead in her own diner, her childhood friend turned K-9 cop Lucas Harwood offers a
    chance at justice--and love. Rescuing Christmas by Terri Reed She escaped a kidnapper,
    but now a killer has set his sights on K-9 dog trainer Lily Anderson. When fellow
    officer Jarrod Evans appoints himself her bodyguard, Lily knows more than her life is at
    risk--so is her heart. Texas K-9 Unit: These lawmen solve the toughest cases with the
    help of their brave canine partners
    
    	Rank: 3 Score: 0.33899369835853577 Title: Dogs on Duty: Soldiers' Best Friends on the Battlefield and Beyond
    When the news of the raid on Osama Bin Laden's compound broke, the SEAL team member that
    stole the show was a highly trained canine companion. Throughout history, dogs have been
    key contributors to military units. Dorothy Hinshaw Patent follows man's best friend
    onto the battlefield, showing readers why dogs are uniquely qualified for the job at
    hand, how they are trained, how they contribute to missions, and what happens when they
    retire. With full-color photographs throughout and sidebars featuring heroic canines
    throughout history, Dogs on Duty provides a fascinating look at these exceptional
    soldiers and companions.
    
    	Rank: 4 Score: 0.34207457304000854 Title: Toute Allure: Falling in Love in Rural France
    After saying goodbye to life as a successful fashion editor in London, Karen Wheeler is
    now happy in her small village house in rural France. Her idyll is complete when she
    meets the love of her life - he has shaggy hair, four paws and a wet nose!
    
    	Rank: 5 Score: 0.343595951795578 Title: Otherwise Alone (Evan Arden, #1)
    Librarian's note: This is an alternate cover edition for ASIN: B00AP5NNWC.


key: _node_content
string_value:
{"id_": "21d603a9-631e-42ae-85f1-ad5674d9f99c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "82976ccf-0844-46c4-bf27-ab24c7c51718", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b6c5f2c6d1fb2f4a10d0884c8ce3c2f5bff33cf9a04dcde31e91b606e959845a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "60dd6255-bd32-4e1d-a39a-3191683b8ded", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ad833553ddf2fbece188891294282e32c0dcf0ce21bb81e12a010e36dd51b461", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18244, "end_char_idx": 20528, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
When fellow
    officer Jarrod Evans appoints himself her bodyguard, Lily knows more than her life is at
    risk--so is her heart. Texas K-9 Unit: These lawmen solve the toughest cases with the
    help of their brave canine partners
    
    	Rank: 3 Score: 0.33899369835853577 Title: Dogs on Duty: Soldiers' Best Friends on the Battlefield and Beyond
    When the news of the raid on Osama Bin Laden's compound broke, the SEAL team member that
    stole the show was a highly trained canine companion. Throughout history, dogs have been
    key contributors to military units. Dorothy Hinshaw Patent follows man's best friend
    onto the battlefield, showing readers why dogs are uniquely qualified for the job at
    hand, how they are trained, how they contribute to missions, and what happens when they
    retire. With full-color photographs throughout and sidebars featuring heroic canines
    throughout history, Dogs on Duty provides a fascinating look at these exceptional
    soldiers and companions.
    
    	Rank: 4 Score: 0.34207457304000854 Title: Toute Allure: Falling in Love in Rural France
    After saying goodbye to life as a successful fashion editor in London, Karen Wheeler is
    now happy in her small village house in rural France. Her idyll is complete when she
    meets the love of her life - he has shaggy hair, four paws and a wet nose!
    
    	Rank: 5 Score: 0.343595951795578 Title: Otherwise Alone (Evan Arden, #1)
    Librarian's note: This is an alternate cover edition for ASIN: B00AP5NNWC. Lieutenant
    Evan Arden sits in a shack in the middle of nowhere, waiting for orders that will send
    him back home - if he ever gets them. Other than his loyal Great Pyrenees, there's no
    one around to break up the monotony. The tedium is excruciating, but it is suddenly
    interrupted when a young woman stumbles up his path. "It's only 50-something pages, but
    in that short amount of time, the author's awesome writing packs in a whole lotta
    character detail. And sets the stage for the series, perfectly." -Maryse.net, 4.5 Stars
    He has two choices - pick her off from a distance with his trusty sniper-rifle, or dare
    let her approach his cabin and enter his life. Why not? It's been ages, and he is
    otherwise alone...


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb


key: _node_content
string_value:
{"id_": "0b6bd6b3-573d-4cb6-ac9d-15a729b03faa", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb", "title": "Filtered Search with Zilliz and OpenAI", "tokens": 2129, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c9d0f0c4-5522-4412-94ab-c7bcdbda0824", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb", "title": "Filtered Search with Zilliz and OpenAI", "tokens": 2129, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d0f85e91114fa67387a09f45e431bf361b2a131fdcb2296fc32a8ec39dcd23d7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "69cd49ba-c288-47fd-a9aa-04136148f1d3", "node_type": "1", "metadata": {}, "hash": "66d256d33eb3c79f27435d94c053e76543a6145910d02f39de86f2b8140f98b1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2882, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Filtered Search with Zilliz and OpenAI
### Finding your next movie

In this notebook we will be going over generating embeddings of movie descriptions with OpenAI and using those embeddings within Zilliz to find relevant movies. To narrow our search results and try something new, we are going to be using filtering to do metadata searches. The dataset in this example is sourced from HuggingFace datasets, and contains a little over 8 thousand movie entries.

Lets begin by first downloading the required libraries for this notebook:
- `openai` is used for communicating with the OpenAI embedding service
- `pymilvus` is used for communicating with the Zilliz server
- `datasets` is used for downloading the dataset
- `tqdm` is used for the progress bars



```python
! pip install openai pymilvus datasets tqdm
```

To get Zilliz up and running take a look [here](https://zilliz.com/doc/quick_start). With your account and database set up, proceed to set the following values:
- URI: The URI your database is running on
- USER: Your database username
- PASSWORD: Your database password
- COLLECTION_NAME: What to name the collection within Zilliz
- DIMENSION: The dimension of the embeddings
- OPENAI_ENGINE: Which embedding model to use
- openai.api_key: Your OpenAI account key
- INDEX_PARAM: The index settings to use for the collection
- QUERY_PARAM: The search parameters to use
- BATCH_SIZE: How many texts to embed and insert at once


```python
import openai

URI = 'your_uri'
TOKEN = 'your_token' # TOKEN == user:password or api_key
COLLECTION_NAME = 'book_search'
DIMENSION = 1536
OPENAI_ENGINE = 'text-embedding-3-small'
openai.api_key = 'sk-your_key'

INDEX_PARAM = {
    'metric_type':'L2',
    'index_type':"AUTOINDEX",
    'params':{}
}

QUERY_PARAM = {
    "metric_type": "L2",
    "params": {},
}

BATCH_SIZE = 1000
```


```python
from pymilvus import connections, utility, FieldSchema, Collection, CollectionSchema, DataType

# Connect to Zilliz Database
connections.connect(uri=URI, token=TOKEN)
```


```python
# Remove collection if it already exists
if utility.has_collection(COLLECTION_NAME):
    utility.drop_collection(COLLECTION_NAME)
```


```python
# Create collection which includes the id, title, and embedding.
fields = [
    FieldSchema(name='id', dtype=DataType.INT64, is_primary=True, auto_id=True),
    FieldSchema(name='title', dtype=DataType.VARCHAR, max_length=64000),
    FieldSchema(name='type', dtype=DataType.VARCHAR, max_length=64000),
    FieldSchema(name='release_year', dtype=DataType.INT64),
    FieldSchema(name='rating', dtype=DataType.VARCHAR, max_length=64000),
    FieldSchema(name='description', dtype=DataType.VARCHAR, max_length=64000),
    FieldSchema(name='embedding', dtype=DataType.FLOAT_VECTOR, dim=DIMENSION)
]
schema = CollectionSchema(fields=fields)
collection = Collection(name=COLLECTION_NAME, schema=schema)
```


key: _node_content
string_value:
{"id_": "69cd49ba-c288-47fd-a9aa-04136148f1d3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb", "title": "Filtered Search with Zilliz and OpenAI", "tokens": 2129, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c9d0f0c4-5522-4412-94ab-c7bcdbda0824", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb", "title": "Filtered Search with Zilliz and OpenAI", "tokens": 2129, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d0f85e91114fa67387a09f45e431bf361b2a131fdcb2296fc32a8ec39dcd23d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0b6bd6b3-573d-4cb6-ac9d-15a729b03faa", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb", "title": "Filtered Search with Zilliz and OpenAI", "tokens": 2129, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d0106273a2e5b0e83612017133d48552f8674ca9ad94fd98cde5740f13fbb5a9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a1b0eba0-7764-40ff-9afd-463eaefe3ddc", "node_type": "1", "metadata": {}, "hash": "23ca51f9dc3af3a89cca8455eb3a1adabb3f17dca4dd50d1f684a27b8455bbc3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1447, "end_char_idx": 4309, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import openai

URI = 'your_uri'
TOKEN = 'your_token' # TOKEN == user:password or api_key
COLLECTION_NAME = 'book_search'
DIMENSION = 1536
OPENAI_ENGINE = 'text-embedding-3-small'
openai.api_key = 'sk-your_key'

INDEX_PARAM = {
    'metric_type':'L2',
    'index_type':"AUTOINDEX",
    'params':{}
}

QUERY_PARAM = {
    "metric_type": "L2",
    "params": {},
}

BATCH_SIZE = 1000
```


```python
from pymilvus import connections, utility, FieldSchema, Collection, CollectionSchema, DataType

# Connect to Zilliz Database
connections.connect(uri=URI, token=TOKEN)
```


```python
# Remove collection if it already exists
if utility.has_collection(COLLECTION_NAME):
    utility.drop_collection(COLLECTION_NAME)
```


```python
# Create collection which includes the id, title, and embedding.
fields = [
    FieldSchema(name='id', dtype=DataType.INT64, is_primary=True, auto_id=True),
    FieldSchema(name='title', dtype=DataType.VARCHAR, max_length=64000),
    FieldSchema(name='type', dtype=DataType.VARCHAR, max_length=64000),
    FieldSchema(name='release_year', dtype=DataType.INT64),
    FieldSchema(name='rating', dtype=DataType.VARCHAR, max_length=64000),
    FieldSchema(name='description', dtype=DataType.VARCHAR, max_length=64000),
    FieldSchema(name='embedding', dtype=DataType.FLOAT_VECTOR, dim=DIMENSION)
]
schema = CollectionSchema(fields=fields)
collection = Collection(name=COLLECTION_NAME, schema=schema)
```


```python
# Create the index on the collection and load it.
collection.create_index(field_name="embedding", index_params=INDEX_PARAM)
collection.load()
```

## Dataset
With Zilliz up and running we can begin grabbing our data. `Hugging Face Datasets` is a hub that holds many different user datasets, and for this example we are using HuggingLearners's netflix-shows dataset. This dataset contains movies and their metadata pairs for over 8 thousand movies. We are going to embed each description and store it within Zilliz along with its title, type, release_year and rating.


```python
import datasets

# Download the dataset 
dataset = datasets.load_dataset('hugginglearners/netflix-shows', split='train')
```

    /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
      from .autonotebook import tqdm as notebook_tqdm
    Found cached dataset csv (/Users/filiphaltmayer/.cache/huggingface/datasets/hugginglearners___csv/hugginglearners--netflix-shows-03475319fc65a05a/0.0.0/6b34fb8fcf56f7c8ba51dc895bfa2bfbe43546f190a60fcf74bb5e8afdcc2317)


## Insert the Data
Now that we have our data on our machine we can begin embedding it and inserting it into Zilliz. The embedding function takes in text and returns the embeddings in a list format.


key: _node_content
string_value:
{"id_": "a1b0eba0-7764-40ff-9afd-463eaefe3ddc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb", "title": "Filtered Search with Zilliz and OpenAI", "tokens": 2129, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c9d0f0c4-5522-4412-94ab-c7bcdbda0824", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb", "title": "Filtered Search with Zilliz and OpenAI", "tokens": 2129, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d0f85e91114fa67387a09f45e431bf361b2a131fdcb2296fc32a8ec39dcd23d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "69cd49ba-c288-47fd-a9aa-04136148f1d3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb", "title": "Filtered Search with Zilliz and OpenAI", "tokens": 2129, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f431ad6c9a065a7bec0794b1c5845332110396fe17f08562a5cf51e08c2ae961", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4f32c77e-bbb8-4526-bdd6-1a76a4cdd0eb", "node_type": "1", "metadata": {}, "hash": "6977b5f65d14be92a5be5b1422a62ce599bc7442852163224dec30fcbf2071f5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2885, "end_char_idx": 5570, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Create the index on the collection and load it.
collection.create_index(field_name="embedding", index_params=INDEX_PARAM)
collection.load()
```

## Dataset
With Zilliz up and running we can begin grabbing our data. `Hugging Face Datasets` is a hub that holds many different user datasets, and for this example we are using HuggingLearners's netflix-shows dataset. This dataset contains movies and their metadata pairs for over 8 thousand movies. We are going to embed each description and store it within Zilliz along with its title, type, release_year and rating.


```python
import datasets

# Download the dataset 
dataset = datasets.load_dataset('hugginglearners/netflix-shows', split='train')
```

    /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html
      from .autonotebook import tqdm as notebook_tqdm
    Found cached dataset csv (/Users/filiphaltmayer/.cache/huggingface/datasets/hugginglearners___csv/hugginglearners--netflix-shows-03475319fc65a05a/0.0.0/6b34fb8fcf56f7c8ba51dc895bfa2bfbe43546f190a60fcf74bb5e8afdcc2317)


## Insert the Data
Now that we have our data on our machine we can begin embedding it and inserting it into Zilliz. The embedding function takes in text and returns the embeddings in a list format. 


```python
# Simple function that converts the texts to embeddings
def embed(texts):
    embeddings = openai.Embedding.create(
        input=texts,
        engine=OPENAI_ENGINE
    )
    return [x['embedding'] for x in embeddings['data']]

```

This next step does the actual inserting. We iterate through all the entries and create batches that we insert once we hit our set batch size. After the loop is over we insert the last remaning batch if it exists. 


```python
from tqdm import tqdm

data = [
    [], # title
    [], # type
    [], # release_year
    [], # rating
    [], # description
]

# Embed and insert in batches
for i in tqdm(range(0, len(dataset))):
    data[0].append(dataset[i]['title'] or '')
    data[1].append(dataset[i]['type'] or '')
    data[2].append(dataset[i]['release_year'] or -1)
    data[3].append(dataset[i]['rating'] or '')
    data[4].append(dataset[i]['description'] or '')
    if len(data[0]) % BATCH_SIZE == 0:
        data.append(embed(data[4]))
        collection.insert(data)
        data = [[],[],[],[],[]]

# Embed and insert the remainder 
if len(data[0]) != 0:
    data.append(embed(data[4]))
    collection.insert(data)
    data = [[],[],[],[],[]]

```

    100%|██████████| 8807/8807 [00:54<00:00, 162.59it/s]


key: _node_content
string_value:
{"id_": "4f32c77e-bbb8-4526-bdd6-1a76a4cdd0eb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb", "title": "Filtered Search with Zilliz and OpenAI", "tokens": 2129, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c9d0f0c4-5522-4412-94ab-c7bcdbda0824", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb", "title": "Filtered Search with Zilliz and OpenAI", "tokens": 2129, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d0f85e91114fa67387a09f45e431bf361b2a131fdcb2296fc32a8ec39dcd23d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a1b0eba0-7764-40ff-9afd-463eaefe3ddc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb", "title": "Filtered Search with Zilliz and OpenAI", "tokens": 2129, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "5948435e6e9920d8bf1a070d2ae7eef4900083da62a79e3bec6880150500f441", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "10fb0bc3-a4ed-4ac8-b73b-f76585f5c72a", "node_type": "1", "metadata": {}, "hash": "e20f81c66c012596e97af277c1e6bdc1d41643a1307e046504d224a4c55ed57c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4112, "end_char_idx": 6032, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Insert the Data
Now that we have our data on our machine we can begin embedding it and inserting it into Zilliz. The embedding function takes in text and returns the embeddings in a list format. 


```python
# Simple function that converts the texts to embeddings
def embed(texts):
    embeddings = openai.Embedding.create(
        input=texts,
        engine=OPENAI_ENGINE
    )
    return [x['embedding'] for x in embeddings['data']]

```

This next step does the actual inserting. We iterate through all the entries and create batches that we insert once we hit our set batch size. After the loop is over we insert the last remaning batch if it exists. 


```python
from tqdm import tqdm

data = [
    [], # title
    [], # type
    [], # release_year
    [], # rating
    [], # description
]

# Embed and insert in batches
for i in tqdm(range(0, len(dataset))):
    data[0].append(dataset[i]['title'] or '')
    data[1].append(dataset[i]['type'] or '')
    data[2].append(dataset[i]['release_year'] or -1)
    data[3].append(dataset[i]['rating'] or '')
    data[4].append(dataset[i]['description'] or '')
    if len(data[0]) % BATCH_SIZE == 0:
        data.append(embed(data[4]))
        collection.insert(data)
        data = [[],[],[],[],[]]

# Embed and insert the remainder 
if len(data[0]) != 0:
    data.append(embed(data[4]))
    collection.insert(data)
    data = [[],[],[],[],[]]

```

    100%|██████████| 8807/8807 [00:54<00:00, 162.59it/s]


## Query the Database
With our data safely inserted into Zilliz, we can now perform a query. The query takes in a tuple of the movie description you are searching for and the filter to use. More info about the filter can be found [here](https://milvus.io/docs/boolean.md). The search first prints out your description and filter expression. After that for each result we print the score, title, type, release year, rating and description of the result movies.


key: _node_content
string_value:
{"id_": "10fb0bc3-a4ed-4ac8-b73b-f76585f5c72a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb", "title": "Filtered Search with Zilliz and OpenAI", "tokens": 2129, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c9d0f0c4-5522-4412-94ab-c7bcdbda0824", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb", "title": "Filtered Search with Zilliz and OpenAI", "tokens": 2129, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d0f85e91114fa67387a09f45e431bf361b2a131fdcb2296fc32a8ec39dcd23d7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4f32c77e-bbb8-4526-bdd6-1a76a4cdd0eb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb", "title": "Filtered Search with Zilliz and OpenAI", "tokens": 2129, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "cc8670c15140fcc4b75dbf0114574eef60ff2df6a104acc48c7e77184f498605", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4774, "end_char_idx": 8353, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from tqdm import tqdm

data = [
    [], # title
    [], # type
    [], # release_year
    [], # rating
    [], # description
]

# Embed and insert in batches
for i in tqdm(range(0, len(dataset))):
    data[0].append(dataset[i]['title'] or '')
    data[1].append(dataset[i]['type'] or '')
    data[2].append(dataset[i]['release_year'] or -1)
    data[3].append(dataset[i]['rating'] or '')
    data[4].append(dataset[i]['description'] or '')
    if len(data[0]) % BATCH_SIZE == 0:
        data.append(embed(data[4]))
        collection.insert(data)
        data = [[],[],[],[],[]]

# Embed and insert the remainder 
if len(data[0]) != 0:
    data.append(embed(data[4]))
    collection.insert(data)
    data = [[],[],[],[],[]]

```

    100%|██████████| 8807/8807 [00:54<00:00, 162.59it/s]


## Query the Database
With our data safely inserted into Zilliz, we can now perform a query. The query takes in a tuple of the movie description you are searching for and the filter to use. More info about the filter can be found [here](https://milvus.io/docs/boolean.md). The search first prints out your description and filter expression. After that for each result we print the score, title, type, release year, rating and description of the result movies. 


```python
import textwrap

def query(query, top_k = 5):
    text, expr = query
    res = collection.search(embed(text), anns_field='embedding', expr = expr, param=QUERY_PARAM, limit = top_k, output_fields=['title', 'type', 'release_year', 'rating', 'description'])
    for i, hit in enumerate(res):
        print('Description:', text, 'Expression:', expr)
        print('Results:')
        for ii, hits in enumerate(hit):
            print('\t' + 'Rank:', ii + 1, 'Score:', hits.score, 'Title:', hits.entity.get('title'))
            print('\t\t' + 'Type:', hits.entity.get('type'), 'Release Year:', hits.entity.get('release_year'), 'Rating:', hits.entity.get('rating'))
            print(textwrap.fill(hits.entity.get('description'), 88))
            print()

my_query = ('movie about a fluffly animal', 'release_year < 2019 and rating like \"PG%\"')

query(my_query)
```

    Description: movie about a fluffly animal Expression: release_year < 2019 and rating like "PG%"
    Results:
    	Rank: 1 Score: 0.30085673928260803 Title: The Lamb
    		Type: Movie Release Year: 2017 Rating: PG
    A big-dreaming donkey escapes his menial existence and befriends some free-spirited
    animal pals in this imaginative retelling of the Nativity Story.
    
    	Rank: 2 Score: 0.3352621793746948 Title: Puss in Boots
    		Type: Movie Release Year: 2011 Rating: PG
    The fabled feline heads to the Land of Giants with friends Humpty Dumpty and Kitty
    Softpaws on a quest to nab its greatest treasure: the Golden Goose.
    
    	Rank: 3 Score: 0.3415083587169647 Title: Show Dogs
    		Type: Movie Release Year: 2018 Rating: PG
    A rough and tough police dog must go undercover with an FBI agent as a prim and proper
    pet at a dog show to save a baby panda from an illegal sale.
    
    	Rank: 4 Score: 0.3428957462310791 Title: Open Season 2
    		Type: Movie Release Year: 2008 Rating: PG
    Elliot the buck and his forest-dwelling cohorts must rescue their dachshund pal from
    some spoiled pets bent on returning him to domesticity.
    
    	Rank: 5 Score: 0.34376364946365356 Title: Stuart Little 2
    		Type: Movie Release Year: 2002 Rating: PG
    Zany misadventures are in store as lovable city mouse Stuart and his human brother,
    George, raise the roof in this sequel to the 1999 blockbuster.


key: _node_content
string_value:
{"id_": "ed87eb49-4d0a-473b-8aa2-c26dccc31d85", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/README.ipynb", "title": "Weaviate <> OpenAI", "tokens": 656, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b32f3766-b2d0-4926-a170-e81dabc98982", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/README.ipynb", "title": "Weaviate <> OpenAI", "tokens": 656, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "2639cae11e4853760568f45836d92e5458cd5fe2358ca607a2daf8c96401bcac", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2427, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Weaviate <> OpenAI

[​Weaviate](https://weaviate.io) is an open-source vector search engine ([docs](https://weaviate.io/developers/weaviate) - [Github](https://github.com/weaviate/weaviate)) that can store and search through OpenAI embeddings _and_ data objects. The database allows you to do similarity search, hybrid search (the combining of multiple search techniques, such as keyword-based and vector search), and generative search (like Q&A). Weaviate also supports a wide variety of OpenAI-based modules (e.g., [`text2vec-openai`](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai), [`qna-openai`](https://weaviate.io/developers/weaviate/modules/reader-generator-modules/qna-openai)), allowing you to vectorize and query data fast and efficiently.

You can run Weaviate (including the OpenAI modules if desired) in three ways:

1. Open source inside a Docker-container ([example](./docker-compose.yml))
2. Using the Weaviate Cloud Service ([get started](https://weaviate.io/developers/weaviate/quickstart/installation#weaviate-cloud-service))
3. In a Kubernetes cluster ([learn more](https://weaviate.io/developers/weaviate/installation/kubernetes))

### Examples

This folder contains a variety of Weaviate and OpenAI examples. 

| Name | Description | language | Google Colab |
| --- | --- | --- | --- |
| [Getting Started with Weaviate and OpenAI](./getting-started-with-weaviate-and-openai.ipynb) | A simple getting started for *semantic vector search* using the OpenAI vectorization module in Weaviate (`text2vec-openai`) | Python Notebook | [link](https://colab.research.google.com/drive/1RxpDE_ruCnoBB3TfwAZqdjYgHJhtdwhK) |
| [Hybrid Search with Weaviate and OpenAI](./hybrid-search-with-weaviate-and-openai.ipynb) | A simple getting started for *hybrid search* using the OpenAI vectorization module in Weaviate (`text2vec-openai`) | Python Notebook | [link](https://colab.research.google.com/drive/1E75BALWoKrOjvUhaznJKQO0A-B1QUPZ4) |
| [Question Answering with Weaviate and OpenAI](./question-answering-with-weaviate-and-openai.ipynb) | A simple getting started for *question answering (Q&A)* using the OpenAI Q&A module in Weaviate (`qna-openai`) | Python Notebook | [link](https://colab.research.google.com/drive/1pUerUZrJaknEboDxDxsuf3giCK0MJJgm) |
| [Docker-compose example](./docker-compose.yml) | A Docker-compose file with all OpenAI modules enabled | Docker |


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb


key: _node_content
string_value:
{"id_": "a9290059-8d12-4e43-a35f-edf803e27379", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with Generative OpenAI module for Generative Search", "tokens": 1437, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cc97083f-90c7-47bf-8818-1628a9885bd0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with Generative OpenAI module for Generative Search", "tokens": 1437, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c55b140e49ffd8bf154c3993b6cfe9a40882529fed33ac3076de7503e3efa9dc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "407a9d21-7552-4763-acc4-520d7252deba", "node_type": "1", "metadata": {}, "hash": "1ce7d8725919552baff6195ac90683ac0f975634d96a66b9a06ebfd77b453965", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2368, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Using Weaviate with Generative OpenAI module for Generative Search

This notebook is prepared for a scenario where:
* Your data is already in Weaviate
* You want to use Weaviate with the Generative OpenAI module ([generative-openai](https://weaviate.io/developers/weaviate/modules/reader-generator-modules/generative-openai)).



## Prerequisites

This cookbook only coveres Generative Search examples, however, it doesn't cover the configuration and data imports.

In order to make the most of this cookbook, please complete the [Getting Started cookbook](./getting-started-with-weaviate-and-openai.ipynb) first, where you will learn the essentials of working with Weaviate and import the demo data.

Checklist:
* completed [Getting Started cookbook](./getting-started-with-weaviate-and-openai.ipynb),
* crated a `Weaviate` instance,
* imported data into your `Weaviate` instance,
* you have an [OpenAI API key](https://beta.openai.com/account/api-keys)

===========================================================
## Prepare your OpenAI API key

The `OpenAI API key` is used for vectorization of your data at import, and for running queries.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it to your environment variables as `OPENAI_API_KEY`.


```python
# Export OpenAI API Key
!export OPENAI_API_KEY="your key"
```


```python
# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.
import os

# Note. alternatively you can set a temporary env variable like this:
# os.environ["OPENAI_API_KEY"] = 'your-key-goes-here'

if os.getenv("OPENAI_API_KEY") is not None:
    print ("OPENAI_API_KEY is ready")
else:
    print ("OPENAI_API_KEY environment variable not found")
```

## Connect to your Weaviate instance

In this section, we will:

1. test env variable `OPENAI_API_KEY` – **make sure** you completed the step in [#Prepare-your-OpenAI-API-key](#Prepare-your-OpenAI-API-key)
2. connect to your Weaviate with your `OpenAI API Key`
3. and test the client connection

### The client 

After this step, the `client` object will be used to perform all Weaviate-related operations.


key: _node_content
string_value:
{"id_": "407a9d21-7552-4763-acc4-520d7252deba", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with Generative OpenAI module for Generative Search", "tokens": 1437, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cc97083f-90c7-47bf-8818-1628a9885bd0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with Generative OpenAI module for Generative Search", "tokens": 1437, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c55b140e49ffd8bf154c3993b6cfe9a40882529fed33ac3076de7503e3efa9dc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a9290059-8d12-4e43-a35f-edf803e27379", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with Generative OpenAI module for Generative Search", "tokens": 1437, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "572f9a73fab59d392e2bbf4c6d9a0eab855991a8e10b57b65525409004299e81", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "301d8f48-8816-4c5e-85cd-c7aa8d4d25e5", "node_type": "1", "metadata": {}, "hash": "b453ba6928b02ba5269a64dcce6593340e8a21704980a8bf8e42d928f3c77b32", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1382, "end_char_idx": 3993, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Export OpenAI API Key
!export OPENAI_API_KEY="your key"
```


```python
# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.
import os

# Note. alternatively you can set a temporary env variable like this:
# os.environ["OPENAI_API_KEY"] = 'your-key-goes-here'

if os.getenv("OPENAI_API_KEY") is not None:
    print ("OPENAI_API_KEY is ready")
else:
    print ("OPENAI_API_KEY environment variable not found")
```

## Connect to your Weaviate instance

In this section, we will:

1. test env variable `OPENAI_API_KEY` – **make sure** you completed the step in [#Prepare-your-OpenAI-API-key](#Prepare-your-OpenAI-API-key)
2. connect to your Weaviate with your `OpenAI API Key`
3. and test the client connection

### The client 

After this step, the `client` object will be used to perform all Weaviate-related operations.


```python
import weaviate
from datasets import load_dataset
import os

# Connect to your Weaviate instance
client = weaviate.Client(
    url="https://your-wcs-instance-name.weaviate.network/",
    # url="http://localhost:8080/",
    auth_client_secret=weaviate.auth.AuthApiKey(api_key="<YOUR-WEAVIATE-API-KEY>"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)
    additional_headers={
        "X-OpenAI-Api-Key": os.getenv("OPENAI_API_KEY")
    }
)

# Check if your instance is live and ready
# This should return `True`
client.is_ready()
```

## Generative Search
Weaviate offers a [Generative Search OpenAI](https://weaviate.io/developers/weaviate/modules/reader-generator-modules/generative-openai) module, which generates responses based on the data stored in your Weaviate instance.

The way you construct a generative search query is very similar to a standard semantic search query in Weaviate. 

For example:
* search in "Articles", 
* return "title", "content", "url"
* look for objects related to "football clubs"
* limit results to 5 objects

```
    result = (
        client.query
        .get("Articles", ["title", "content", "url"])
        .with_near_text("concepts": "football clubs")
        .with_limit(5)
        # generative query will go here
        .do()
    )
```

Now, you can add `with_generate()` function to apply generative transformation. `with_generate` takes either:
- `single_prompt` - to generate a response for each returned object,
- `grouped_task` – to generate a single response from all returned objects.


key: _node_content
string_value:
{"id_": "301d8f48-8816-4c5e-85cd-c7aa8d4d25e5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with Generative OpenAI module for Generative Search", "tokens": 1437, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cc97083f-90c7-47bf-8818-1628a9885bd0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with Generative OpenAI module for Generative Search", "tokens": 1437, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c55b140e49ffd8bf154c3993b6cfe9a40882529fed33ac3076de7503e3efa9dc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "407a9d21-7552-4763-acc4-520d7252deba", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with Generative OpenAI module for Generative Search", "tokens": 1437, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "6ce400a82648f2b0c97c27824001e9fc526b0ef4e1491504ac58148f945cd628", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "894d3c34-a6f5-47b5-9a31-e4377a10ce06", "node_type": "1", "metadata": {}, "hash": "1a56001b4b2cb993978ef2b2bbe89365ccc57984f685c4eade9cfeec3ed1e028", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2371, "end_char_idx": 4989, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import weaviate
from datasets import load_dataset
import os

# Connect to your Weaviate instance
client = weaviate.Client(
    url="https://your-wcs-instance-name.weaviate.network/",
    # url="http://localhost:8080/",
    auth_client_secret=weaviate.auth.AuthApiKey(api_key="<YOUR-WEAVIATE-API-KEY>"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)
    additional_headers={
        "X-OpenAI-Api-Key": os.getenv("OPENAI_API_KEY")
    }
)

# Check if your instance is live and ready
# This should return `True`
client.is_ready()
```

## Generative Search
Weaviate offers a [Generative Search OpenAI](https://weaviate.io/developers/weaviate/modules/reader-generator-modules/generative-openai) module, which generates responses based on the data stored in your Weaviate instance.

The way you construct a generative search query is very similar to a standard semantic search query in Weaviate. 

For example:
* search in "Articles", 
* return "title", "content", "url"
* look for objects related to "football clubs"
* limit results to 5 objects

```
    result = (
        client.query
        .get("Articles", ["title", "content", "url"])
        .with_near_text("concepts": "football clubs")
        .with_limit(5)
        # generative query will go here
        .do()
    )
```

Now, you can add `with_generate()` function to apply generative transformation. `with_generate` takes either:
- `single_prompt` - to generate a response for each returned object,
- `grouped_task` – to generate a single response from all returned objects.



```python
def generative_search_per_item(query, collection_name):
    prompt = "Summarize in a short tweet the following content: {content}"

    result = (
        client.query
        .get(collection_name, ["title", "content", "url"])
        .with_near_text({ "concepts": [query], "distance": 0.7 })
        .with_limit(5)
        .with_generate(single_prompt=prompt)
        .do()
    )
    
    # Check for errors
    if ("errors" in result):
        print ("\033[91mYou probably have run out of OpenAI API calls for the current minute – the limit is set at 60 per minute.")
        raise Exception(result["errors"][0]['message'])
    
    return result["data"]["Get"][collection_name]
```


```python
query_result = generative_search_per_item("football clubs", "Article")

for i, article in enumerate(query_result):
    print(f"{i+1}. { article['title']}")
    print(article['_additional']['generate']['singleResult']) # print generated response
    print("-----------------------")
```


key: _node_content
string_value:
{"id_": "894d3c34-a6f5-47b5-9a31-e4377a10ce06", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with Generative OpenAI module for Generative Search", "tokens": 1437, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cc97083f-90c7-47bf-8818-1628a9885bd0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with Generative OpenAI module for Generative Search", "tokens": 1437, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c55b140e49ffd8bf154c3993b6cfe9a40882529fed33ac3076de7503e3efa9dc", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "301d8f48-8816-4c5e-85cd-c7aa8d4d25e5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with Generative OpenAI module for Generative Search", "tokens": 1437, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8026485fae1a9630ba4d4d03fe29dce0ed600f2d3612cb1440c39cd68fa2f822", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3997, "end_char_idx": 6062, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def generative_search_per_item(query, collection_name):
    prompt = "Summarize in a short tweet the following content: {content}"

    result = (
        client.query
        .get(collection_name, ["title", "content", "url"])
        .with_near_text({ "concepts": [query], "distance": 0.7 })
        .with_limit(5)
        .with_generate(single_prompt=prompt)
        .do()
    )
    
    # Check for errors
    if ("errors" in result):
        print ("\033[91mYou probably have run out of OpenAI API calls for the current minute – the limit is set at 60 per minute.")
        raise Exception(result["errors"][0]['message'])
    
    return result["data"]["Get"][collection_name]
```


```python
query_result = generative_search_per_item("football clubs", "Article")

for i, article in enumerate(query_result):
    print(f"{i+1}. { article['title']}")
    print(article['_additional']['generate']['singleResult']) # print generated response
    print("-----------------------")
```


```python
def generative_search_group(query, collection_name):
    generateTask = "Explain what these have in common"

    result = (
        client.query
        .get(collection_name, ["title", "content", "url"])
        .with_near_text({ "concepts": [query], "distance": 0.7 })
        .with_generate(grouped_task=generateTask)
        .with_limit(5)
        .do()
    )
    
    # Check for errors
    if ("errors" in result):
        print ("\033[91mYou probably have run out of OpenAI API calls for the current minute – the limit is set at 60 per minute.")
        raise Exception(result["errors"][0]['message'])
    
    return result["data"]["Get"][collection_name]
```


```python
query_result = generative_search_group("football clubs", "Article")

print (query_result[0]['_additional']['generate']['groupedResult'])
```

Thanks for following along, you're now equipped to set up your own vector databases and use embeddings to do all kinds of cool things - enjoy! For more complex use cases please continue to work through other cookbook examples in this repo.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb


key: _node_content
string_value:
{"id_": "d90ad18e-ed91-4beb-8058-54e538dd8442", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d546cc36-3cb0-4b97-b112-2d55137b6e70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "2cea98b33f97d6f94b1d07489015308121dd5728643bf82174fff9a506d8893a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ca13af90-c0d5-4167-aa39-905883a46317", "node_type": "1", "metadata": {}, "hash": "d603f0f3e3cd1bab94ec3b0fc9295d0056729ecb95964e818bedd7a285afe1aa", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2501, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Using Weaviate with OpenAI vectorize module for Hybrid Search

This notebook is prepared for a scenario where:
* Your data is not vectorized
* You want to run Hybrid Search ([learn more](https://weaviate.io/blog/hybrid-search-explained)) on your data
* You want to use Weaviate with the OpenAI module ([text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai)), to generate vector embeddings for you.

This notebook takes you through a simple flow to set up a Weaviate instance, connect to it (with OpenAI API key), configure data schema, import data (which will automatically generate vector embeddings for your data), and run hybrid search (mixing of vector and BM25 search).

This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.

## What is Weaviate

Weaviate is an open-source vector search engine that stores data objects together with their vectors. This allows for combining vector search with structured filtering.

Weaviate uses KNN algorithms to create an vector-optimized index, which allows your queries to run extremely fast. Learn more [here](https://weaviate.io/blog/why-is-vector-search-so-fast).

Weaviate let you use your favorite ML-models, and scale seamlessly into billions of data objects.

### Deployment options

Whatever your scenario or production setup, Weaviate has an option for you. You can deploy Weaviate in the following setups:
* Self-hosted – you can deploy Weaviate with docker locally, or any server you want.
* SaaS – you can use [Weaviate Cloud Service (WCS)](https://console.weaviate.io/) to host your Weaviate instances.
* Hybrid-SaaS – you can deploy Weaviate in your own private Cloud Service 

### Programming languages

Weaviate offers four [client libraries](https://weaviate.io/developers/weaviate/client-libraries), which allow you to communicate from your apps:
* [Python](https://weaviate.io/developers/weaviate/client-libraries/python)
* [JavaScript](https://weaviate.io/developers/weaviate/client-libraries/javascript)
* [Java](https://weaviate.io/developers/weaviate/client-libraries/java)
* [Go](https://weaviate.io/developers/weaviate/client-libraries/go)

Additionally, Weaviate has a [REST layer](https://weaviate.io/developers/weaviate/api/rest/objects). Basically you can call Weaviate from any language that supports REST requests.


key: _node_content
string_value:
{"id_": "ca13af90-c0d5-4167-aa39-905883a46317", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d546cc36-3cb0-4b97-b112-2d55137b6e70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "2cea98b33f97d6f94b1d07489015308121dd5728643bf82174fff9a506d8893a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d90ad18e-ed91-4beb-8058-54e538dd8442", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8c8ccd5d803f1e80586b0e474dff027cec9950c540df3499f7080a0962897705", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2ae2f2c1-3df4-4695-a620-c60d8f0cc4e9", "node_type": "1", "metadata": {}, "hash": "e9e14de57b1163a11ac517d604cc2b7989120d0b88ff7e048f8015b486c07521", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 943, "end_char_idx": 4103, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## What is Weaviate

Weaviate is an open-source vector search engine that stores data objects together with their vectors. This allows for combining vector search with structured filtering.

Weaviate uses KNN algorithms to create an vector-optimized index, which allows your queries to run extremely fast. Learn more [here](https://weaviate.io/blog/why-is-vector-search-so-fast).

Weaviate let you use your favorite ML-models, and scale seamlessly into billions of data objects.

### Deployment options

Whatever your scenario or production setup, Weaviate has an option for you. You can deploy Weaviate in the following setups:
* Self-hosted – you can deploy Weaviate with docker locally, or any server you want.
* SaaS – you can use [Weaviate Cloud Service (WCS)](https://console.weaviate.io/) to host your Weaviate instances.
* Hybrid-SaaS – you can deploy Weaviate in your own private Cloud Service 

### Programming languages

Weaviate offers four [client libraries](https://weaviate.io/developers/weaviate/client-libraries), which allow you to communicate from your apps:
* [Python](https://weaviate.io/developers/weaviate/client-libraries/python)
* [JavaScript](https://weaviate.io/developers/weaviate/client-libraries/javascript)
* [Java](https://weaviate.io/developers/weaviate/client-libraries/java)
* [Go](https://weaviate.io/developers/weaviate/client-libraries/go)

Additionally, Weaviate has a [REST layer](https://weaviate.io/developers/weaviate/api/rest/objects). Basically you can call Weaviate from any language that supports REST requests.

## Demo Flow
The demo flow is:
- **Prerequisites Setup**: Create a Weaviate instance and install required libraries
- **Connect**: Connect to your Weaviate instance 
- **Schema Configuration**: Configure the schema of your data
    - *Note*: Here we can define which OpenAI Embedding Model to use
    - *Note*: Here we can configure which properties to index
- **Import data**: Load a demo dataset and import it into Weaviate
    - *Note*: The import process will automatically index your data - based on the configuration in the schema
    - *Note*: You don't need to explicitly vectorize your data, Weaviate will communicate with OpenAI to do it for you
- **Run Queries**: Query 
    - *Note*: You don't need to explicitly vectorize your queries, Weaviate will communicate with OpenAI to do it for you

Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.

## OpenAI Module in Weaviate
All Weaviate instances come equipped with the [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) module.

This module is responsible for handling vectorization during import (or any CRUD operations) and when you run a query.

### No need to manually vectorize data
This is great news for you. With [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) you don't need to manually vectorize your data, as Weaviate will call OpenAI for you whenever necessary.


key: _node_content
string_value:
{"id_": "2ae2f2c1-3df4-4695-a620-c60d8f0cc4e9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d546cc36-3cb0-4b97-b112-2d55137b6e70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "2cea98b33f97d6f94b1d07489015308121dd5728643bf82174fff9a506d8893a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ca13af90-c0d5-4167-aa39-905883a46317", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d5968fdd60f56879354d352bdb074f5128715a456f0d40c9087b5e150ae93686", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d6cb9acd-c9f0-4845-aedf-d0f31db097d6", "node_type": "1", "metadata": {}, "hash": "914020a4f805cf6e0c1e7d53e8af89e270260391245b35f78f20f83f7eb9f703", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2423, "end_char_idx": 5476, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Basically you can call Weaviate from any language that supports REST requests.

## Demo Flow
The demo flow is:
- **Prerequisites Setup**: Create a Weaviate instance and install required libraries
- **Connect**: Connect to your Weaviate instance 
- **Schema Configuration**: Configure the schema of your data
    - *Note*: Here we can define which OpenAI Embedding Model to use
    - *Note*: Here we can configure which properties to index
- **Import data**: Load a demo dataset and import it into Weaviate
    - *Note*: The import process will automatically index your data - based on the configuration in the schema
    - *Note*: You don't need to explicitly vectorize your data, Weaviate will communicate with OpenAI to do it for you
- **Run Queries**: Query 
    - *Note*: You don't need to explicitly vectorize your queries, Weaviate will communicate with OpenAI to do it for you

Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.

## OpenAI Module in Weaviate
All Weaviate instances come equipped with the [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) module.

This module is responsible for handling vectorization during import (or any CRUD operations) and when you run a query.

### No need to manually vectorize data
This is great news for you. With [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) you don't need to manually vectorize your data, as Weaviate will call OpenAI for you whenever necessary.

All you need to do is:
1. provide your OpenAI API Key – when you connected to the Weaviate Client
2. define which OpenAI vectorizer to use in your Schema

## Prerequisites

Before we start this project, we need setup the following:

* create a `Weaviate` instance
* install libraries
    * `weaviate-client`
    * `datasets`
    * `apache-beam`
* get your [OpenAI API key](https://beta.openai.com/account/api-keys)

===========================================================
### Create a Weaviate instance

To create a Weaviate instance we have 2 options:

1. (Recommended path) [Weaviate Cloud Service](https://console.weaviate.io/) – to host your Weaviate instance in the cloud. The free sandbox should be more than enough for this cookbook.
2. Install and run Weaviate locally with Docker.

#### Option 1 – WCS Installation Steps

Use [Weaviate Cloud Service](https://console.weaviate.io/) (WCS) to create a free Weaviate cluster.
1. create a free account and/or login to [WCS](https://console.weaviate.io/)
2. create a `Weaviate Cluster` with the following settings:
    * Sandbox: `Sandbox Free`
    * Weaviate Version: Use default (latest)
    * OIDC Authentication: `Disabled`
3. your instance should be ready in a minute or two
4. make a note of the `Cluster Id`. The link will take you to the full path of your cluster (you will need it later to connect to it).


key: _node_content
string_value:
{"id_": "d6cb9acd-c9f0-4845-aedf-d0f31db097d6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d546cc36-3cb0-4b97-b112-2d55137b6e70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "2cea98b33f97d6f94b1d07489015308121dd5728643bf82174fff9a506d8893a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2ae2f2c1-3df4-4695-a620-c60d8f0cc4e9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "2c97351177c17abc6cecb8aef3b44cc39dbdfb047c579f11e1be943bae536309", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4d2c3649-aa60-4c5a-9730-6f6681cf3ce2", "node_type": "1", "metadata": {}, "hash": "0f67a17142f8df7346f11f04c1fe1c199dc9ae6e932dbc53902ebb7b407f029f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4105, "end_char_idx": 7308, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
All you need to do is:
1. provide your OpenAI API Key – when you connected to the Weaviate Client
2. define which OpenAI vectorizer to use in your Schema

## Prerequisites

Before we start this project, we need setup the following:

* create a `Weaviate` instance
* install libraries
    * `weaviate-client`
    * `datasets`
    * `apache-beam`
* get your [OpenAI API key](https://beta.openai.com/account/api-keys)

===========================================================
### Create a Weaviate instance

To create a Weaviate instance we have 2 options:

1. (Recommended path) [Weaviate Cloud Service](https://console.weaviate.io/) – to host your Weaviate instance in the cloud. The free sandbox should be more than enough for this cookbook.
2. Install and run Weaviate locally with Docker.

#### Option 1 – WCS Installation Steps

Use [Weaviate Cloud Service](https://console.weaviate.io/) (WCS) to create a free Weaviate cluster.
1. create a free account and/or login to [WCS](https://console.weaviate.io/)
2. create a `Weaviate Cluster` with the following settings:
    * Sandbox: `Sandbox Free`
    * Weaviate Version: Use default (latest)
    * OIDC Authentication: `Disabled`
3. your instance should be ready in a minute or two
4. make a note of the `Cluster Id`. The link will take you to the full path of your cluster (you will need it later to connect to it). It should be something like: `https://your-project-name.weaviate.network` 

#### Option 2 – local Weaviate instance with Docker

Install and run Weaviate locally with Docker.
1. Download the [./docker-compose.yml](./docker-compose.yml) file
2. Then open your terminal, navigate to where your docker-compose.yml file is located, and start docker with: `docker-compose up -d`
3. Once this is ready, your instance should be available at [http://localhost:8080](http://localhost:8080)

Note. To shut down your docker instance you can call: `docker-compose down`

##### Learn more
To learn more, about using Weaviate with Docker see the [installation documentation](https://weaviate.io/developers/weaviate/installation/docker-compose).

===========================================================    
## Install required libraries

Before running this project make sure to have the following libraries:

### Weaviate Python client

The [Weaviate Python client](https://weaviate.io/developers/weaviate/client-libraries/python) allows you to communicate with your Weaviate instance from your Python project.

### datasets & apache-beam

To load sample data, you need the `datasets` library and its' dependency `apache-beam`.


```python
# Install the Weaviate client for Python
!pip install weaviate-client>3.11.0

# Install datasets and apache-beam to load the sample datasets
!pip install datasets apache-beam
```

===========================================================
## Prepare your OpenAI API key

The `OpenAI API key` is used for vectorization of your data at import, and for running queries.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it to your environment variables as `OPENAI_API_KEY`.


key: _node_content
string_value:
{"id_": "4d2c3649-aa60-4c5a-9730-6f6681cf3ce2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d546cc36-3cb0-4b97-b112-2d55137b6e70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "2cea98b33f97d6f94b1d07489015308121dd5728643bf82174fff9a506d8893a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d6cb9acd-c9f0-4845-aedf-d0f31db097d6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "2bec301bd576fe5c6910f08391bf9324c31d98bb2880686ac34ccda2dbcac14b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1fdd736f-7847-4751-a923-c1c0947a35e9", "node_type": "1", "metadata": {}, "hash": "509856b015946197b089046edafeacd7d5815dd72cffb2e1a5e1db1b7ce35bb3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5652, "end_char_idx": 8292, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
1. Download the [./docker-compose.yml](./docker-compose.yml) file
2. Then open your terminal, navigate to where your docker-compose.yml file is located, and start docker with: `docker-compose up -d`
3. Once this is ready, your instance should be available at [http://localhost:8080](http://localhost:8080)

Note. To shut down your docker instance you can call: `docker-compose down`

##### Learn more
To learn more, about using Weaviate with Docker see the [installation documentation](https://weaviate.io/developers/weaviate/installation/docker-compose).

===========================================================    
## Install required libraries

Before running this project make sure to have the following libraries:

### Weaviate Python client

The [Weaviate Python client](https://weaviate.io/developers/weaviate/client-libraries/python) allows you to communicate with your Weaviate instance from your Python project.

### datasets & apache-beam

To load sample data, you need the `datasets` library and its' dependency `apache-beam`.


```python
# Install the Weaviate client for Python
!pip install weaviate-client>3.11.0

# Install datasets and apache-beam to load the sample datasets
!pip install datasets apache-beam
```

===========================================================
## Prepare your OpenAI API key

The `OpenAI API key` is used for vectorization of your data at import, and for running queries.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it to your environment variables as `OPENAI_API_KEY`.


```python
# Export OpenAI API Key
!export OPENAI_API_KEY="your key"
```


```python
# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.
import os

# Note. alternatively you can set a temporary env variable like this:
# os.environ['OPENAI_API_KEY'] = 'your-key-goes-here'

if os.getenv("OPENAI_API_KEY") is not None:
    print ("OPENAI_API_KEY is ready")
else:
    print ("OPENAI_API_KEY environment variable not found")
```

## Connect to your Weaviate instance

In this section, we will:

1. test env variable `OPENAI_API_KEY` – **make sure** you completed the step in [#Prepare-your-OpenAI-API-key](#Prepare-your-OpenAI-API-key)
2. connect to your Weaviate your `OpenAI API Key`
3. and test the client connection

### The client 

After this step, the `client` object will be used to perform all Weaviate-related operations.


key: _node_content
string_value:
{"id_": "1fdd736f-7847-4751-a923-c1c0947a35e9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d546cc36-3cb0-4b97-b112-2d55137b6e70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "2cea98b33f97d6f94b1d07489015308121dd5728643bf82174fff9a506d8893a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4d2c3649-aa60-4c5a-9730-6f6681cf3ce2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4080ca34e04657bf53552de36309b68fde7e3f62c736896aa2503959e24889a9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8535306a-44d1-4348-9805-39a3c1f7b9a9", "node_type": "1", "metadata": {}, "hash": "7dd05b88a771409019501c297fa54b8586656eb118e2be5f24f062c3d1027c76", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6697, "end_char_idx": 9779, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Install the Weaviate client for Python
!pip install weaviate-client>3.11.0

# Install datasets and apache-beam to load the sample datasets
!pip install datasets apache-beam
```

===========================================================
## Prepare your OpenAI API key

The `OpenAI API key` is used for vectorization of your data at import, and for running queries.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it to your environment variables as `OPENAI_API_KEY`.


```python
# Export OpenAI API Key
!export OPENAI_API_KEY="your key"
```


```python
# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.
import os

# Note. alternatively you can set a temporary env variable like this:
# os.environ['OPENAI_API_KEY'] = 'your-key-goes-here'

if os.getenv("OPENAI_API_KEY") is not None:
    print ("OPENAI_API_KEY is ready")
else:
    print ("OPENAI_API_KEY environment variable not found")
```

## Connect to your Weaviate instance

In this section, we will:

1. test env variable `OPENAI_API_KEY` – **make sure** you completed the step in [#Prepare-your-OpenAI-API-key](#Prepare-your-OpenAI-API-key)
2. connect to your Weaviate your `OpenAI API Key`
3. and test the client connection

### The client 

After this step, the `client` object will be used to perform all Weaviate-related operations.


```python
import weaviate
from datasets import load_dataset
import os

# Connect to your Weaviate instance
client = weaviate.Client(
    url="https://your-wcs-instance-name.weaviate.network/",
#   url="http://localhost:8080/",
    auth_client_secret=weaviate.auth.AuthApiKey(api_key="<YOUR-WEAVIATE-API-KEY>"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)
    additional_headers={
        "X-OpenAI-Api-Key": os.getenv("OPENAI_API_KEY")
    }
)

# Check if your instance is live and ready
# This should return `True`
client.is_ready()
```

# Schema

In this section, we will:
1. configure the data schema for your data
2. select OpenAI module

> This is the second and final step, which requires OpenAI specific configuration.
> After this step, the rest of instructions wlll only touch on Weaviate, as the OpenAI tasks will be handled automatically.


## What is a schema

In Weaviate you create __schemas__ to capture each of the entities you will be searching.

A schema is how you tell Weaviate:
* what embedding model should be used to vectorize the data
* what your data is made of (property names and types)
* which properties should be vectorized and indexed

In this cookbook we will use a dataset for `Articles`, which contains:
* `title`
* `content`
* `url`

We want to vectorize `title` and `content`, but not the `url`.

To vectorize and query the data, we will use `text-embedding-3-small`.


key: _node_content
string_value:
{"id_": "8535306a-44d1-4348-9805-39a3c1f7b9a9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d546cc36-3cb0-4b97-b112-2d55137b6e70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "2cea98b33f97d6f94b1d07489015308121dd5728643bf82174fff9a506d8893a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1fdd736f-7847-4751-a923-c1c0947a35e9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "fd148de5c55e6e9476caae1430681e92804e9a1021f7af290c5aaa71bd63c47d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a69960d9-8228-4413-9ec3-4c72cbff718d", "node_type": "1", "metadata": {}, "hash": "a1f48ce52b5853c71c290635b828a01c4627bb0c89fdad704cfadb21bef7981b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8295, "end_char_idx": 11261, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import weaviate
from datasets import load_dataset
import os

# Connect to your Weaviate instance
client = weaviate.Client(
    url="https://your-wcs-instance-name.weaviate.network/",
#   url="http://localhost:8080/",
    auth_client_secret=weaviate.auth.AuthApiKey(api_key="<YOUR-WEAVIATE-API-KEY>"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)
    additional_headers={
        "X-OpenAI-Api-Key": os.getenv("OPENAI_API_KEY")
    }
)

# Check if your instance is live and ready
# This should return `True`
client.is_ready()
```

# Schema

In this section, we will:
1. configure the data schema for your data
2. select OpenAI module

> This is the second and final step, which requires OpenAI specific configuration.
> After this step, the rest of instructions wlll only touch on Weaviate, as the OpenAI tasks will be handled automatically.


## What is a schema

In Weaviate you create __schemas__ to capture each of the entities you will be searching.

A schema is how you tell Weaviate:
* what embedding model should be used to vectorize the data
* what your data is made of (property names and types)
* which properties should be vectorized and indexed

In this cookbook we will use a dataset for `Articles`, which contains:
* `title`
* `content`
* `url`

We want to vectorize `title` and `content`, but not the `url`.

To vectorize and query the data, we will use `text-embedding-3-small`.


```python
# Clear up the schema, so that we can recreate it
client.schema.delete_all()
client.schema.get()

# Define the Schema object to use `text-embedding-3-small` on `title` and `content`, but skip it for `url`
article_schema = {
    "class": "Article",
    "description": "A collection of articles",
    "vectorizer": "text2vec-openai",
    "moduleConfig": {
        "text2vec-openai": {
          "model": "ada",
          "modelVersion": "002",
          "type": "text"
        }
    },
    "properties": [{
        "name": "title",
        "description": "Title of the article",
        "dataType": ["string"]
    },
    {
        "name": "content",
        "description": "Contents of the article",
        "dataType": ["text"]
    },
    {
        "name": "url",
        "description": "URL to the article",
        "dataType": ["string"],
        "moduleConfig": { "text2vec-openai": { "skip": True } }
    }]
}

# add the Article schema
client.schema.create_class(article_schema)

# get the schema to make sure it worked
client.schema.get()
```

## Import data

In this section we will:
1. load the Simple Wikipedia dataset
2. configure Weaviate Batch import (to make the import more efficient)
3. import the data into Weaviate

> Note: <br/>
> Like mentioned before. We don't need to manually vectorize the data.<br/>
> The [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) module will take care of that.


key: _node_content
string_value:
{"id_": "a69960d9-8228-4413-9ec3-4c72cbff718d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d546cc36-3cb0-4b97-b112-2d55137b6e70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "2cea98b33f97d6f94b1d07489015308121dd5728643bf82174fff9a506d8893a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8535306a-44d1-4348-9805-39a3c1f7b9a9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0c2ef8d31f31e9e2230201ff4183465536d55823fa1a4170df8c0237ee762c2f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "abb7e479-1a06-4a51-ac3e-d3bce5c76d42", "node_type": "1", "metadata": {}, "hash": "bd52c142d903f5d4f49c2dcef340cb0228ae08a4910a138e08d9f063452ebe3a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9782, "end_char_idx": 12815, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Clear up the schema, so that we can recreate it
client.schema.delete_all()
client.schema.get()

# Define the Schema object to use `text-embedding-3-small` on `title` and `content`, but skip it for `url`
article_schema = {
    "class": "Article",
    "description": "A collection of articles",
    "vectorizer": "text2vec-openai",
    "moduleConfig": {
        "text2vec-openai": {
          "model": "ada",
          "modelVersion": "002",
          "type": "text"
        }
    },
    "properties": [{
        "name": "title",
        "description": "Title of the article",
        "dataType": ["string"]
    },
    {
        "name": "content",
        "description": "Contents of the article",
        "dataType": ["text"]
    },
    {
        "name": "url",
        "description": "URL to the article",
        "dataType": ["string"],
        "moduleConfig": { "text2vec-openai": { "skip": True } }
    }]
}

# add the Article schema
client.schema.create_class(article_schema)

# get the schema to make sure it worked
client.schema.get()
```

## Import data

In this section we will:
1. load the Simple Wikipedia dataset
2. configure Weaviate Batch import (to make the import more efficient)
3. import the data into Weaviate

> Note: <br/>
> Like mentioned before. We don't need to manually vectorize the data.<br/>
> The [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) module will take care of that.


```python
### STEP 1 - load the dataset

from datasets import load_dataset
from typing import List, Iterator

# We'll use the datasets library to pull the Simple Wikipedia dataset for embedding
dataset = list(load_dataset("wikipedia", "20220301.simple")["train"])

# For testing, limited to 2.5k articles for demo purposes
dataset = dataset[:2_500]

# Limited to 25k articles for larger demo purposes
# dataset = dataset[:25_000]

# for free OpenAI acounts, you can use 50 objects
# dataset = dataset[:50]
```


```python
### Step 2 - configure Weaviate Batch, with
# - starting batch size of 100
# - dynamically increase/decrease based on performance
# - add timeout retries if something goes wrong

client.batch.configure(
    batch_size=10, 
    dynamic=True,
    timeout_retries=3,
#   callback=None,
)
```


```python
### Step 3 - import data

print("Importing Articles")

counter=0

with client.batch as batch:
    for article in dataset:
        if (counter %10 == 0):
            print(f"Import {counter} / {len(dataset)} ")

        properties = {
            "title": article["title"],
            "content": article["text"],
            "url": article["url"]
        }
        
        batch.add_data_object(properties, "Article")
        counter = counter+1

print("Importing Articles complete")       
```


```python
# Test that all data has loaded – get object count
result = (
    client.query.aggregate("Article")
    .with_fields("meta { count }")
    .do()
)
print("Object count: ", result["data"]["Aggregate"]["Article"], "\n")
```


key: _node_content
string_value:
{"id_": "abb7e479-1a06-4a51-ac3e-d3bce5c76d42", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d546cc36-3cb0-4b97-b112-2d55137b6e70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "2cea98b33f97d6f94b1d07489015308121dd5728643bf82174fff9a506d8893a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a69960d9-8228-4413-9ec3-4c72cbff718d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "62d6bd0529c35692ecb62f8efe791012a4300fb053ab7187b6736fdfef6285e8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6b334884-13a2-40ec-9bd7-fdfa18966982", "node_type": "1", "metadata": {}, "hash": "d945252274c65e9f4dd3d0419f1fa9359c7b1ed41db97618bf6174f834c44516", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11264, "end_char_idx": 14327, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
### STEP 1 - load the dataset

from datasets import load_dataset
from typing import List, Iterator

# We'll use the datasets library to pull the Simple Wikipedia dataset for embedding
dataset = list(load_dataset("wikipedia", "20220301.simple")["train"])

# For testing, limited to 2.5k articles for demo purposes
dataset = dataset[:2_500]

# Limited to 25k articles for larger demo purposes
# dataset = dataset[:25_000]

# for free OpenAI acounts, you can use 50 objects
# dataset = dataset[:50]
```


```python
### Step 2 - configure Weaviate Batch, with
# - starting batch size of 100
# - dynamically increase/decrease based on performance
# - add timeout retries if something goes wrong

client.batch.configure(
    batch_size=10, 
    dynamic=True,
    timeout_retries=3,
#   callback=None,
)
```


```python
### Step 3 - import data

print("Importing Articles")

counter=0

with client.batch as batch:
    for article in dataset:
        if (counter %10 == 0):
            print(f"Import {counter} / {len(dataset)} ")

        properties = {
            "title": article["title"],
            "content": article["text"],
            "url": article["url"]
        }
        
        batch.add_data_object(properties, "Article")
        counter = counter+1

print("Importing Articles complete")       
```


```python
# Test that all data has loaded – get object count
result = (
    client.query.aggregate("Article")
    .with_fields("meta { count }")
    .do()
)
print("Object count: ", result["data"]["Aggregate"]["Article"], "\n")
```


```python
# Test one article has worked by checking one object
test_article = (
    client.query
    .get("Article", ["title", "url", "content"])
    .with_limit(1)
    .do()
)["data"]["Get"]["Article"][0]

print(test_article['title'])
print(test_article['url'])
print(test_article['content'])
```

### Search Data

As above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors

Learn more about the `alpha` setting [here](https://weaviate.io/developers/weaviate/api/graphql/vector-search-parameters#hybrid)


```python
def hybrid_query_weaviate(query, collection_name, alpha_val):
    
    nearText = {
        "concepts": [query],
        "distance": 0.7,
    }

    properties = [
        "title", "content", "url",
        "_additional { score }"
    ]

    result = (
        client.query
        .get(collection_name, properties)
        .with_hybrid(nearText, alpha=alpha_val)
        .with_limit(10)
        .do()
    )
    
    # Check for errors
    if ("errors" in result):
        print ("\033[91mYou probably have run out of OpenAI API calls for the current minute – the limit is set at 60 per minute.")
        raise Exception(result["errors"][0]['message'])
    
    return result["data"]["Get"][collection_name]
```


```python
query_result = hybrid_query_weaviate("modern art in Europe", "Article", 0.5)

for i, article in enumerate(query_result):
    print(f"{i+1}. { article['title']} (Score: {article['_additional']['score']})")
```


key: _node_content
string_value:
{"id_": "6b334884-13a2-40ec-9bd7-fdfa18966982", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d546cc36-3cb0-4b97-b112-2d55137b6e70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "2cea98b33f97d6f94b1d07489015308121dd5728643bf82174fff9a506d8893a", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "abb7e479-1a06-4a51-ac3e-d3bce5c76d42", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d78df84e8e414622da549d10ae3dce5d171298b3f39d3e57be0c72c0207ff310", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12818, "end_char_idx": 14803, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Test one article has worked by checking one object
test_article = (
    client.query
    .get("Article", ["title", "url", "content"])
    .with_limit(1)
    .do()
)["data"]["Get"]["Article"][0]

print(test_article['title'])
print(test_article['url'])
print(test_article['content'])
```

### Search Data

As above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors

Learn more about the `alpha` setting [here](https://weaviate.io/developers/weaviate/api/graphql/vector-search-parameters#hybrid)


```python
def hybrid_query_weaviate(query, collection_name, alpha_val):
    
    nearText = {
        "concepts": [query],
        "distance": 0.7,
    }

    properties = [
        "title", "content", "url",
        "_additional { score }"
    ]

    result = (
        client.query
        .get(collection_name, properties)
        .with_hybrid(nearText, alpha=alpha_val)
        .with_limit(10)
        .do()
    )
    
    # Check for errors
    if ("errors" in result):
        print ("\033[91mYou probably have run out of OpenAI API calls for the current minute – the limit is set at 60 per minute.")
        raise Exception(result["errors"][0]['message'])
    
    return result["data"]["Get"][collection_name]
```


```python
query_result = hybrid_query_weaviate("modern art in Europe", "Article", 0.5)

for i, article in enumerate(query_result):
    print(f"{i+1}. { article['title']} (Score: {article['_additional']['score']})")
```


```python
query_result = hybrid_query_weaviate("Famous battles in Scottish history", "Article", 0.5)

for i, article in enumerate(query_result):
    print(f"{i+1}. { article['title']} (Score: {article['_additional']['score']})")
```

Thanks for following along, you're now equipped to set up your own vector databases and use embeddings to do all kinds of cool things - enjoy! For more complex use cases please continue to work through other cookbook examples in this repo.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb


key: _node_content
string_value:
{"id_": "df973c06-c675-47fa-a971-0721ada05b1b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c43259c-d8fe-4f22-8c64-b59f6cbd4b72", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "991bca91f9d90b969f92ab9705826a1252f072df241beb79d175e5a8ac376966", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d2ed3058-c95b-47ac-9cde-7b91950e067c", "node_type": "1", "metadata": {}, "hash": "5f42683117a6882ddfb0c1fe03a12a35edf18df643f375084ed63d3e1f80a4e1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2407, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Using Weaviate with OpenAI vectorize module for Embeddings Search

This notebook is prepared for a scenario where:
* Your data is not vectorized
* You want to run Vector Search on your data
* You want to use Weaviate with the OpenAI module ([text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai)), to generate vector embeddings for you.

This notebook takes you through a simple flow to set up a Weaviate instance, connect to it (with OpenAI API key), configure data schema, import data (which will automatically generate vector embeddings for your data), and run semantic search.

This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.

## What is Weaviate

Weaviate is an open-source vector search engine that stores data objects together with their vectors. This allows for combining vector search with structured filtering.

Weaviate uses KNN algorithms to create an vector-optimized index, which allows your queries to run extremely fast. Learn more [here](https://weaviate.io/blog/why-is-vector-search-so-fast).

Weaviate let you use your favorite ML-models, and scale seamlessly into billions of data objects.

### Deployment options

Whatever your scenario or production setup, Weaviate has an option for you. You can deploy Weaviate in the following setups:
* Self-hosted – you can deploy Weaviate with docker locally, or any server you want.
* SaaS – you can use [Weaviate Cloud Service (WCS)](https://console.weaviate.io/) to host your Weaviate instances.
* Hybrid-SaaS – you can deploy Weaviate in your own private Cloud Service.

### Programming languages

Weaviate offers four [client libraries](https://weaviate.io/developers/weaviate/client-libraries), which allow you to communicate from your apps:
* [Python](https://weaviate.io/developers/weaviate/client-libraries/python)
* [JavaScript](https://weaviate.io/developers/weaviate/client-libraries/javascript)
* [Java](https://weaviate.io/developers/weaviate/client-libraries/java)
* [Go](https://weaviate.io/developers/weaviate/client-libraries/go)

Additionally, Weaviate has a [REST layer](https://weaviate.io/developers/weaviate/api/rest/objects). Basically you can call Weaviate from any language that supports REST requests.


key: _node_content
string_value:
{"id_": "d2ed3058-c95b-47ac-9cde-7b91950e067c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c43259c-d8fe-4f22-8c64-b59f6cbd4b72", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "991bca91f9d90b969f92ab9705826a1252f072df241beb79d175e5a8ac376966", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "df973c06-c675-47fa-a971-0721ada05b1b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "07d332f1dc607db67c7b906eb0b44ab27abfaa4a48b8bb23d519ae8582444c07", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2a38bb43-64f6-427c-b625-e4c1fbc708f5", "node_type": "1", "metadata": {}, "hash": "91e24283b44225b5528d7257722b3aab3d33708477e2e4aac032f4ad58933a12", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 849, "end_char_idx": 4013, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## What is Weaviate

Weaviate is an open-source vector search engine that stores data objects together with their vectors. This allows for combining vector search with structured filtering.

Weaviate uses KNN algorithms to create an vector-optimized index, which allows your queries to run extremely fast. Learn more [here](https://weaviate.io/blog/why-is-vector-search-so-fast).

Weaviate let you use your favorite ML-models, and scale seamlessly into billions of data objects.

### Deployment options

Whatever your scenario or production setup, Weaviate has an option for you. You can deploy Weaviate in the following setups:
* Self-hosted – you can deploy Weaviate with docker locally, or any server you want.
* SaaS – you can use [Weaviate Cloud Service (WCS)](https://console.weaviate.io/) to host your Weaviate instances.
* Hybrid-SaaS – you can deploy Weaviate in your own private Cloud Service.

### Programming languages

Weaviate offers four [client libraries](https://weaviate.io/developers/weaviate/client-libraries), which allow you to communicate from your apps:
* [Python](https://weaviate.io/developers/weaviate/client-libraries/python)
* [JavaScript](https://weaviate.io/developers/weaviate/client-libraries/javascript)
* [Java](https://weaviate.io/developers/weaviate/client-libraries/java)
* [Go](https://weaviate.io/developers/weaviate/client-libraries/go)

Additionally, Weaviate has a [REST layer](https://weaviate.io/developers/weaviate/api/rest/objects). Basically you can call Weaviate from any language that supports REST requests.

## Demo Flow
The demo flow is:
- **Prerequisites Setup**: Create a Weaviate instance and install the required libraries
- **Connect**: Connect to your Weaviate instance 
- **Schema Configuration**: Configure the schema of your data
    - *Note*: Here we can define which OpenAI Embedding Model to use
    - *Note*: Here we can configure which properties to index
- **Import data**: Load a demo dataset and import it into Weaviate
    - *Note*: The import process will automatically index your data - based on the configuration in the schema
    - *Note*: You don't need to explicitly vectorize your data, Weaviate will communicate with OpenAI to do it for you
- **Run Queries**: Query 
    - *Note*: You don't need to explicitly vectorize your queries, Weaviate will communicate with OpenAI to do it for you

Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.

## OpenAI Module in Weaviate
All Weaviate instances come equipped with the [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) module.

This module is responsible for handling vectorization during import (or any CRUD operations) and when you run a query.

### No need to manually vectorize data
This is great news for you. With [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) you don't need to manually vectorize your data, as Weaviate will call OpenAI for you whenever necessary.


key: _node_content
string_value:
{"id_": "2a38bb43-64f6-427c-b625-e4c1fbc708f5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c43259c-d8fe-4f22-8c64-b59f6cbd4b72", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "991bca91f9d90b969f92ab9705826a1252f072df241beb79d175e5a8ac376966", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d2ed3058-c95b-47ac-9cde-7b91950e067c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3bdae287b4cd5045df99877f477d6c6f1bd019aa36c8b37aa9c41ee7014bb40c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "eb6a8088-3972-4866-a889-c53f73d43077", "node_type": "1", "metadata": {}, "hash": "4c24e0a97e6fa87af6f52ae729261b8a382095973148c261c75712da4c640237", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2329, "end_char_idx": 5386, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Basically you can call Weaviate from any language that supports REST requests.

## Demo Flow
The demo flow is:
- **Prerequisites Setup**: Create a Weaviate instance and install the required libraries
- **Connect**: Connect to your Weaviate instance 
- **Schema Configuration**: Configure the schema of your data
    - *Note*: Here we can define which OpenAI Embedding Model to use
    - *Note*: Here we can configure which properties to index
- **Import data**: Load a demo dataset and import it into Weaviate
    - *Note*: The import process will automatically index your data - based on the configuration in the schema
    - *Note*: You don't need to explicitly vectorize your data, Weaviate will communicate with OpenAI to do it for you
- **Run Queries**: Query 
    - *Note*: You don't need to explicitly vectorize your queries, Weaviate will communicate with OpenAI to do it for you

Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.

## OpenAI Module in Weaviate
All Weaviate instances come equipped with the [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) module.

This module is responsible for handling vectorization during import (or any CRUD operations) and when you run a query.

### No need to manually vectorize data
This is great news for you. With [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) you don't need to manually vectorize your data, as Weaviate will call OpenAI for you whenever necessary.

All you need to do is:
1. provide your OpenAI API Key – when you connected to the Weaviate Client
2. define which OpenAI vectorizer to use in your Schema

## Prerequisites

Before we start this project, we need setup the following:

* create a `Weaviate` instance
* install libraries
    * `weaviate-client`
    * `datasets`
    * `apache-beam`
* get your [OpenAI API key](https://beta.openai.com/account/api-keys)

===========================================================
### Create a Weaviate instance

To create a Weaviate instance we have 2 options:

1. (Recommended path) [Weaviate Cloud Service](https://console.weaviate.io/) – to host your Weaviate instance in the cloud. The free sandbox should be more than enough for this cookbook.
2. Install and run Weaviate locally with Docker.

#### Option 1 – WCS Installation Steps

Use [Weaviate Cloud Service](https://console.weaviate.io/) (WCS) to create a free Weaviate cluster.
1. create a free account and/or login to [WCS](https://console.weaviate.io/)
2. create a `Weaviate Cluster` with the following settings:
    * Sandbox: `Sandbox Free`
    * Weaviate Version: Use default (latest)
    * OIDC Authentication: `Disabled`
3. your instance should be ready in a minute or two
4. make a note of the `Cluster Id`. The link will take you to the full path of your cluster (you will need it later to connect to it).


key: _node_content
string_value:
{"id_": "eb6a8088-3972-4866-a889-c53f73d43077", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c43259c-d8fe-4f22-8c64-b59f6cbd4b72", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "991bca91f9d90b969f92ab9705826a1252f072df241beb79d175e5a8ac376966", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2a38bb43-64f6-427c-b625-e4c1fbc708f5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "68318732e0f68e624f7838d88d280e8075e47c55c4984aedc4577e1352e83614", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "31a25d77-c405-4111-ac0f-cc0c0f172fbf", "node_type": "1", "metadata": {}, "hash": "9fc0c29ae9da2787f6e42fcc65f01d10a280af533d1481233c659dd7ec018426", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4015, "end_char_idx": 7218, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
All you need to do is:
1. provide your OpenAI API Key – when you connected to the Weaviate Client
2. define which OpenAI vectorizer to use in your Schema

## Prerequisites

Before we start this project, we need setup the following:

* create a `Weaviate` instance
* install libraries
    * `weaviate-client`
    * `datasets`
    * `apache-beam`
* get your [OpenAI API key](https://beta.openai.com/account/api-keys)

===========================================================
### Create a Weaviate instance

To create a Weaviate instance we have 2 options:

1. (Recommended path) [Weaviate Cloud Service](https://console.weaviate.io/) – to host your Weaviate instance in the cloud. The free sandbox should be more than enough for this cookbook.
2. Install and run Weaviate locally with Docker.

#### Option 1 – WCS Installation Steps

Use [Weaviate Cloud Service](https://console.weaviate.io/) (WCS) to create a free Weaviate cluster.
1. create a free account and/or login to [WCS](https://console.weaviate.io/)
2. create a `Weaviate Cluster` with the following settings:
    * Sandbox: `Sandbox Free`
    * Weaviate Version: Use default (latest)
    * OIDC Authentication: `Disabled`
3. your instance should be ready in a minute or two
4. make a note of the `Cluster Id`. The link will take you to the full path of your cluster (you will need it later to connect to it). It should be something like: `https://your-project-name.weaviate.network` 

#### Option 2 – local Weaviate instance with Docker

Install and run Weaviate locally with Docker.
1. Download the [./docker-compose.yml](./docker-compose.yml) file
2. Then open your terminal, navigate to where your docker-compose.yml file is located, and start docker with: `docker-compose up -d`
3. Once this is ready, your instance should be available at [http://localhost:8080](http://localhost:8080)

Note. To shut down your docker instance you can call: `docker-compose down`

##### Learn more
To learn more, about using Weaviate with Docker see the [installation documentation](https://weaviate.io/developers/weaviate/installation/docker-compose).

===========================================================    
## Install required libraries

Before running this project make sure to have the following libraries:

### Weaviate Python client

The [Weaviate Python client](https://weaviate.io/developers/weaviate/client-libraries/python) allows you to communicate with your Weaviate instance from your Python project.

### datasets & apache-beam

To load sample data, you need the `datasets` library and its dependency `apache-beam`.


```python
# Install the Weaviate client for Python
!pip install weaviate-client>=3.11.0

# Install datasets and apache-beam to load the sample datasets
!pip install datasets apache-beam
```

===========================================================
## Prepare your OpenAI API key

The `OpenAI API key` is used for vectorization of your data at import, and for running queries.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it to your environment variables as `OPENAI_API_KEY`.


key: _node_content
string_value:
{"id_": "31a25d77-c405-4111-ac0f-cc0c0f172fbf", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c43259c-d8fe-4f22-8c64-b59f6cbd4b72", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "991bca91f9d90b969f92ab9705826a1252f072df241beb79d175e5a8ac376966", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "eb6a8088-3972-4866-a889-c53f73d43077", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "5df42f8960e2462514758ead1f88b7786e9877f90df7210bafd55522a7ccf32a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8ad38bcb-2064-4dcd-b52b-e15b6433a275", "node_type": "1", "metadata": {}, "hash": "e6127a721c83a5f23e7175dcaa40d0529657321f96e63bb58772853cb2a5afaa", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5562, "end_char_idx": 8207, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
1. Download the [./docker-compose.yml](./docker-compose.yml) file
2. Then open your terminal, navigate to where your docker-compose.yml file is located, and start docker with: `docker-compose up -d`
3. Once this is ready, your instance should be available at [http://localhost:8080](http://localhost:8080)

Note. To shut down your docker instance you can call: `docker-compose down`

##### Learn more
To learn more, about using Weaviate with Docker see the [installation documentation](https://weaviate.io/developers/weaviate/installation/docker-compose).

===========================================================    
## Install required libraries

Before running this project make sure to have the following libraries:

### Weaviate Python client

The [Weaviate Python client](https://weaviate.io/developers/weaviate/client-libraries/python) allows you to communicate with your Weaviate instance from your Python project.

### datasets & apache-beam

To load sample data, you need the `datasets` library and its dependency `apache-beam`.


```python
# Install the Weaviate client for Python
!pip install weaviate-client>=3.11.0

# Install datasets and apache-beam to load the sample datasets
!pip install datasets apache-beam
```

===========================================================
## Prepare your OpenAI API key

The `OpenAI API key` is used for vectorization of your data at import, and for running queries.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it to your environment variables as `OPENAI_API_KEY`.


```python
# Export OpenAI API Key
!export OPENAI_API_KEY="your key"
```


```python
# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.
import os

# Note. alternatively you can set a temporary env variable like this:
# os.environ["OPENAI_API_KEY"] = 'your-key-goes-here'

if os.getenv("OPENAI_API_KEY") is not None:
    print ("OPENAI_API_KEY is ready")
else:
    print ("OPENAI_API_KEY environment variable not found")
```

## Connect to your Weaviate instance

In this section, we will:

1. test env variable `OPENAI_API_KEY` – **make sure** you completed the step in [#Prepare-your-OpenAI-API-key](#Prepare-your-OpenAI-API-key)
2. connect to your Weaviate with your `OpenAI API Key`
3. and test the client connection

### The client 

After this step, the `client` object will be used to perform all Weaviate-related operations.


key: _node_content
string_value:
{"id_": "8ad38bcb-2064-4dcd-b52b-e15b6433a275", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c43259c-d8fe-4f22-8c64-b59f6cbd4b72", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "991bca91f9d90b969f92ab9705826a1252f072df241beb79d175e5a8ac376966", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "31a25d77-c405-4111-ac0f-cc0c0f172fbf", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b3c99bec20674073ab3b4cd78b957f191d5f2108e4d7e7de0a61940de6c6255d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cba0d9c2-0f3f-49b5-939e-69cbfe0bb4fc", "node_type": "1", "metadata": {}, "hash": "e1850fb6310398320760051daae4cd5d30ac778e8caba3f319dd260a2e97e4df", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6606, "end_char_idx": 9696, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Install the Weaviate client for Python
!pip install weaviate-client>=3.11.0

# Install datasets and apache-beam to load the sample datasets
!pip install datasets apache-beam
```

===========================================================
## Prepare your OpenAI API key

The `OpenAI API key` is used for vectorization of your data at import, and for running queries.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it to your environment variables as `OPENAI_API_KEY`.


```python
# Export OpenAI API Key
!export OPENAI_API_KEY="your key"
```


```python
# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.
import os

# Note. alternatively you can set a temporary env variable like this:
# os.environ["OPENAI_API_KEY"] = 'your-key-goes-here'

if os.getenv("OPENAI_API_KEY") is not None:
    print ("OPENAI_API_KEY is ready")
else:
    print ("OPENAI_API_KEY environment variable not found")
```

## Connect to your Weaviate instance

In this section, we will:

1. test env variable `OPENAI_API_KEY` – **make sure** you completed the step in [#Prepare-your-OpenAI-API-key](#Prepare-your-OpenAI-API-key)
2. connect to your Weaviate with your `OpenAI API Key`
3. and test the client connection

### The client 

After this step, the `client` object will be used to perform all Weaviate-related operations.


```python
import weaviate
from datasets import load_dataset
import os

# Connect to your Weaviate instance
client = weaviate.Client(
    url="https://your-wcs-instance-name.weaviate.network/",
    # url="http://localhost:8080/",
    auth_client_secret=weaviate.auth.AuthApiKey(api_key="<YOUR-WEAVIATE-API-KEY>"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)
    additional_headers={
        "X-OpenAI-Api-Key": os.getenv("OPENAI_API_KEY")
    }
)

# Check if your instance is live and ready
# This should return `True`
client.is_ready()
```

# Schema

In this section, we will:
1. configure the data schema for your data
2. select OpenAI module

> This is the second and final step, which requires OpenAI specific configuration.
> After this step, the rest of instructions wlll only touch on Weaviate, as the OpenAI tasks will be handled automatically.


## What is a schema

In Weaviate you create __schemas__ to capture each of the entities you will be searching.

A schema is how you tell Weaviate:
* what embedding model should be used to vectorize the data
* what your data is made of (property names and types)
* which properties should be vectorized and indexed

In this cookbook we will use a dataset for `Articles`, which contains:
* `title`
* `content`
* `url`

We want to vectorize `title` and `content`, but not the `url`.

To vectorize and query the data, we will use `text-embedding-3-small`.


key: _node_content
string_value:
{"id_": "cba0d9c2-0f3f-49b5-939e-69cbfe0bb4fc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c43259c-d8fe-4f22-8c64-b59f6cbd4b72", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "991bca91f9d90b969f92ab9705826a1252f072df241beb79d175e5a8ac376966", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8ad38bcb-2064-4dcd-b52b-e15b6433a275", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7a707e17d1aa49da414720c29a253410cb1dcc52b7b83066107a44c8b674bc62", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7854ac72-6431-4d77-8535-16948b689042", "node_type": "1", "metadata": {}, "hash": "a1f48ce52b5853c71c290635b828a01c4627bb0c89fdad704cfadb21bef7981b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8210, "end_char_idx": 11178, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import weaviate
from datasets import load_dataset
import os

# Connect to your Weaviate instance
client = weaviate.Client(
    url="https://your-wcs-instance-name.weaviate.network/",
    # url="http://localhost:8080/",
    auth_client_secret=weaviate.auth.AuthApiKey(api_key="<YOUR-WEAVIATE-API-KEY>"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)
    additional_headers={
        "X-OpenAI-Api-Key": os.getenv("OPENAI_API_KEY")
    }
)

# Check if your instance is live and ready
# This should return `True`
client.is_ready()
```

# Schema

In this section, we will:
1. configure the data schema for your data
2. select OpenAI module

> This is the second and final step, which requires OpenAI specific configuration.
> After this step, the rest of instructions wlll only touch on Weaviate, as the OpenAI tasks will be handled automatically.


## What is a schema

In Weaviate you create __schemas__ to capture each of the entities you will be searching.

A schema is how you tell Weaviate:
* what embedding model should be used to vectorize the data
* what your data is made of (property names and types)
* which properties should be vectorized and indexed

In this cookbook we will use a dataset for `Articles`, which contains:
* `title`
* `content`
* `url`

We want to vectorize `title` and `content`, but not the `url`.

To vectorize and query the data, we will use `text-embedding-3-small`.


```python
# Clear up the schema, so that we can recreate it
client.schema.delete_all()
client.schema.get()

# Define the Schema object to use `text-embedding-3-small` on `title` and `content`, but skip it for `url`
article_schema = {
    "class": "Article",
    "description": "A collection of articles",
    "vectorizer": "text2vec-openai",
    "moduleConfig": {
        "text2vec-openai": {
          "model": "ada",
          "modelVersion": "002",
          "type": "text"
        }
    },
    "properties": [{
        "name": "title",
        "description": "Title of the article",
        "dataType": ["string"]
    },
    {
        "name": "content",
        "description": "Contents of the article",
        "dataType": ["text"]
    },
    {
        "name": "url",
        "description": "URL to the article",
        "dataType": ["string"],
        "moduleConfig": { "text2vec-openai": { "skip": True } }
    }]
}

# add the Article schema
client.schema.create_class(article_schema)

# get the schema to make sure it worked
client.schema.get()
```

## Import data

In this section we will:
1. load the Simple Wikipedia dataset
2. configure Weaviate Batch import (to make the import more efficient)
3. import the data into Weaviate

> Note: <br/>
> Like mentioned before. We don't need to manually vectorize the data.<br/>
> The [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) module will take care of that.


key: _node_content
string_value:
{"id_": "7854ac72-6431-4d77-8535-16948b689042", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c43259c-d8fe-4f22-8c64-b59f6cbd4b72", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "991bca91f9d90b969f92ab9705826a1252f072df241beb79d175e5a8ac376966", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cba0d9c2-0f3f-49b5-939e-69cbfe0bb4fc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "2fd6d2ac4702bca3f4485e6ebf54ca040487d7f9aad837d338d0ba15f077cebd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "073843ec-cfef-49f8-a21e-cb0131b59690", "node_type": "1", "metadata": {}, "hash": "9f2b67e6cb5a1ec3cbeebe42850a2d75b0af30299e32337e52da40c3c1269f48", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9699, "end_char_idx": 12732, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: _node_content
string_value:
{"id_": "073843ec-cfef-49f8-a21e-cb0131b59690", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c43259c-d8fe-4f22-8c64-b59f6cbd4b72", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "991bca91f9d90b969f92ab9705826a1252f072df241beb79d175e5a8ac376966", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7854ac72-6431-4d77-8535-16948b689042", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "76a231381198a8fb1f1e1e65a88c63faef76fa3d291e46a837aaa0a9a7e8f5a6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b0d21866-68c3-4ced-8949-6e1f6acfdc2c", "node_type": "1", "metadata": {}, "hash": "1f9e449e149d81370862abe9ae406b159c3e90a9b8ed34797995edff92c82a07", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11181, "end_char_idx": 14095, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
### STEP 1 - load the dataset

from datasets import load_dataset
from typing import List, Iterator

# We'll use the datasets library to pull the Simple Wikipedia dataset for embedding
dataset = list(load_dataset("wikipedia", "20220301.simple")["train"])

# For testing, limited to 2.5k articles for demo purposes
dataset = dataset[:2_500]

# Limited to 25k articles for larger demo purposes
# dataset = dataset[:25_000]

# for free OpenAI acounts, you can use 50 objects
# dataset = dataset[:50]
```


```python
### Step 2 - configure Weaviate Batch, with
# - starting batch size of 100
# - dynamically increase/decrease based on performance
# - add timeout retries if something goes wrong

client.batch.configure(
    batch_size=10, 
    dynamic=True,
    timeout_retries=3,
#   callback=None,
)
```


```python
### Step 3 - import data

print("Importing Articles")

counter=0

with client.batch as batch:
    for article in dataset:
        if (counter %10 == 0):
            print(f"Import {counter} / {len(dataset)} ")

        properties = {
            "title": article["title"],
            "content": article["text"],
            "url": article["url"]
        }
        
        batch.add_data_object(properties, "Article")
        counter = counter+1

print("Importing Articles complete")       
```


```python
# Test that all data has loaded – get object count
result = (
    client.query.aggregate("Article")
    .with_fields("meta { count }")
    .do()
)
print("Object count: ", result["data"]["Aggregate"]["Article"], "\n")
```


```python
# Test one article has worked by checking one object
test_article = (
    client.query
    .get("Article", ["title", "url", "content"])
    .with_limit(1)
    .do()
)["data"]["Get"]["Article"][0]

print(test_article['title'])
print(test_article['url'])
print(test_article['content'])
```

### Search Data

As above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors


```python
def query_weaviate(query, collection_name):
    
    nearText = {
        "concepts": [query],
        "distance": 0.7,
    }

    properties = [
        "title", "content", "url",
        "_additional {certainty distance}"
    ]

    result = (
        client.query
        .get(collection_name, properties)
        .with_near_text(nearText)
        .with_limit(10)
        .do()
    )
    
    # Check for errors
    if ("errors" in result):
        print ("\033[91mYou probably have run out of OpenAI API calls for the current minute – the limit is set at 60 per minute.")
        raise Exception(result["errors"][0]['message'])
    
    return result["data"]["Get"][collection_name]
```


```python
query_result = query_weaviate("modern art in Europe", "Article")

for i, article in enumerate(query_result):
    print(f"{i+1}. { article['title']} (Score: {round(article['_additional']['certainty'],3) })")
```


key: _node_content
string_value:
{"id_": "b0d21866-68c3-4ced-8949-6e1f6acfdc2c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5c43259c-d8fe-4f22-8c64-b59f6cbd4b72", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "991bca91f9d90b969f92ab9705826a1252f072df241beb79d175e5a8ac376966", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "073843ec-cfef-49f8-a21e-cb0131b59690", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "2936d2f49008e5a4a0f8b6ccafb6260e35dcc549958fc5a2355a673a66a05334", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12502, "end_char_idx": 14573, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Test that all data has loaded – get object count
result = (
    client.query.aggregate("Article")
    .with_fields("meta { count }")
    .do()
)
print("Object count: ", result["data"]["Aggregate"]["Article"], "\n")
```


```python
# Test one article has worked by checking one object
test_article = (
    client.query
    .get("Article", ["title", "url", "content"])
    .with_limit(1)
    .do()
)["data"]["Get"]["Article"][0]

print(test_article['title'])
print(test_article['url'])
print(test_article['content'])
```

### Search Data

As above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors


```python
def query_weaviate(query, collection_name):
    
    nearText = {
        "concepts": [query],
        "distance": 0.7,
    }

    properties = [
        "title", "content", "url",
        "_additional {certainty distance}"
    ]

    result = (
        client.query
        .get(collection_name, properties)
        .with_near_text(nearText)
        .with_limit(10)
        .do()
    )
    
    # Check for errors
    if ("errors" in result):
        print ("\033[91mYou probably have run out of OpenAI API calls for the current minute – the limit is set at 60 per minute.")
        raise Exception(result["errors"][0]['message'])
    
    return result["data"]["Get"][collection_name]
```


```python
query_result = query_weaviate("modern art in Europe", "Article")

for i, article in enumerate(query_result):
    print(f"{i+1}. { article['title']} (Score: {round(article['_additional']['certainty'],3) })")
```


```python
query_result = query_weaviate("Famous battles in Scottish history", "Article")

for i, article in enumerate(query_result):
    print(f"{i+1}. { article['title']} (Score: {round(article['_additional']['certainty'],3) })")
```

Thanks for following along, you're now equipped to set up your own vector databases and use embeddings to do all kinds of cool things - enjoy! For more complex use cases please continue to work through other cookbook examples in this repo.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb


key: _node_content
string_value:
{"id_": "46f7b4ec-d57d-4ffd-87c0-09246f2a0cdc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "105ad3e7-dd4e-4fc3-ad89-828cb68fd275", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f1e50e0a23cbd05a6303dce4eadd3e2a409b3265e3110a49323df38bcf426110", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "88541259-ca10-4629-945a-9a4f7289b4be", "node_type": "1", "metadata": {}, "hash": "ea33eec6ce7fe9964cdec82a5a01d5024ec7994955bd2f2cb55509a0cc3dd62d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3520, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Using Weaviate for Embeddings Search

This notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.

### What is a Vector Database

A vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.

### Why use a Vector Database

Vector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.


### Demo Flow
The demo flow is:
- **Setup**: Import packages and set any required variables
- **Load data**: Load a dataset and embed it using OpenAI embeddings
- **Weaviate**
    - *Setup*: Here we'll set up the Python client for Weaviate. For more details go [here](https://weaviate.io/developers/weaviate/current/client-libraries/python.html)
    - *Index Data*: We'll create an index with __title__ search vectors in it
    - *Search Data*: We'll run a few searches to confirm it works

Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.

## Setup

Import the required libraries and set the embedding model that we'd like to use.


```python
# We'll need to install the Weaviate client
!pip install weaviate-client

#Install wget to pull zip file
!pip install wget
```


```python
import openai

from typing import List, Iterator
import pandas as pd
import numpy as np
import os
import wget
from ast import literal_eval

# Weaviate's client library for Python
import weaviate

# I've set this to our new embeddings model, this can be changed to the embedding model of your choice
EMBEDDING_MODEL = "text-embedding-3-small"

# Ignore unclosed SSL socket warnings - optional in case you get these errors
import warnings

warnings.filterwarnings(action="ignore", message="unclosed", category=ResourceWarning)
warnings.filterwarnings("ignore", category=DeprecationWarning) 
```

## Load data

In this section we'll load embedded data that we've prepared previous to this session.


```python
embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```


```python
import zipfile
with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip","r") as zip_ref:
    zip_ref.extractall("../data")
```


key: _node_content
string_value:
{"id_": "88541259-ca10-4629-945a-9a4f7289b4be", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "105ad3e7-dd4e-4fc3-ad89-828cb68fd275", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f1e50e0a23cbd05a6303dce4eadd3e2a409b3265e3110a49323df38bcf426110", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "46f7b4ec-d57d-4ffd-87c0-09246f2a0cdc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9ead60d1e11a0d721d70d926e8130ee20d34929a126b60cd1229df5ac2decdda", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8ea2a501-e502-4073-9df9-284491264d2c", "node_type": "1", "metadata": {}, "hash": "55d75d734c6c465942a4bbe99c9e2550ca1dde90d5af183d906f3fc23ae149c3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2319, "end_char_idx": 4670, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# We'll need to install the Weaviate client
!pip install weaviate-client

#Install wget to pull zip file
!pip install wget
```


```python
import openai

from typing import List, Iterator
import pandas as pd
import numpy as np
import os
import wget
from ast import literal_eval

# Weaviate's client library for Python
import weaviate

# I've set this to our new embeddings model, this can be changed to the embedding model of your choice
EMBEDDING_MODEL = "text-embedding-3-small"

# Ignore unclosed SSL socket warnings - optional in case you get these errors
import warnings

warnings.filterwarnings(action="ignore", message="unclosed", category=ResourceWarning)
warnings.filterwarnings("ignore", category=DeprecationWarning) 
```

## Load data

In this section we'll load embedded data that we've prepared previous to this session.


```python
embeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'

# The file is ~700 MB so this will take some time
wget.download(embeddings_url)
```


```python
import zipfile
with zipfile.ZipFile("vector_database_wikipedia_articles_embedded.zip","r") as zip_ref:
    zip_ref.extractall("../data")
```


```python
article_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')
```


```python
article_df.head()
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>id</th>
      <th>url</th>
      <th>title</th>
      <th>text</th>
      <th>title_vector</th>
      <th>content_vector</th>
      <th>vector_id</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>1</td>
      <td>https://simple.wikipedia.org/wiki/April</td>
      <td>April</td>
      <td>April is the fourth month of the year in the J...</td>
      <td>[0.001009464613161981, -0.020700545981526375, ...</td>
      <td>[-0.011253940872848034, -0.013491976074874401,...</td>
      <td>0</td>
    </tr>
    <tr>
      <th>1</th>
      <td>2</td>
      <td>https://simple.wikipedia.org/wiki/August</td>
      <td>August</td>
      <td>August (Aug.)


key: _node_content
string_value:
{"id_": "8ea2a501-e502-4073-9df9-284491264d2c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "105ad3e7-dd4e-4fc3-ad89-828cb68fd275", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f1e50e0a23cbd05a6303dce4eadd3e2a409b3265e3110a49323df38bcf426110", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "88541259-ca10-4629-945a-9a4f7289b4be", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "acb6e82dcf4dcfb26dc8899aeddf5df26228d31ae68655ca2020ff2b9f72ea9a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "631c409c-feab-46bf-9c1c-2aba15d00cd2", "node_type": "1", "metadata": {}, "hash": "0eb9f1fbcad7c87e0ae1cc80d55f22c888212d66f0901dbcd88725f848899c3c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3523, "end_char_idx": 5701, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: _node_content
string_value:
{"id_": "631c409c-feab-46bf-9c1c-2aba15d00cd2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "105ad3e7-dd4e-4fc3-ad89-828cb68fd275", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f1e50e0a23cbd05a6303dce4eadd3e2a409b3265e3110a49323df38bcf426110", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8ea2a501-e502-4073-9df9-284491264d2c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9c7eb5d14437d9a303d1fc6534fa1eb459d4cbf8873d93da2ce41ef3b74d4772", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "884ddc2d-69ae-4d33-9b7d-69e49fa4cc28", "node_type": "1", "metadata": {}, "hash": "56e2aa1bde5ed15fe8073faa23455e10c4418bcfff784ccff7bb57b05c25c908", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4671, "end_char_idx": 6869, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: _node_content
string_value:
{"id_": "884ddc2d-69ae-4d33-9b7d-69e49fa4cc28", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "105ad3e7-dd4e-4fc3-ad89-828cb68fd275", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f1e50e0a23cbd05a6303dce4eadd3e2a409b3265e3110a49323df38bcf426110", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "631c409c-feab-46bf-9c1c-2aba15d00cd2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4320c7c17eb3368c7bfc18cc765fcc9a783c8fba72ee141f67ffd98304297e89", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6168404b-425e-4936-8735-6ca773f8a578", "node_type": "1", "metadata": {}, "hash": "686d4ea9bbc87de3b274fe0ef6b95b9180e4ab82f9db022f1d6861c6c8def44d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5702, "end_char_idx": 7367, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Air is a...</td>
      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>
      <td>[0.021524671465158463, 0.018522677943110466, -...</td>
      <td>4</td>
    </tr>
  </tbody>
</table>
</div>




```python
# Read vectors from strings back into a list
article_df['title_vector'] = article_df.title_vector.apply(literal_eval)
article_df['content_vector'] = article_df.content_vector.apply(literal_eval)

# Set vector_id to be a string
article_df['vector_id'] = article_df['vector_id'].apply(str)
```


```python
article_df.info(show_counts=True)
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 25000 entries, 0 to 24999
    Data columns (total 7 columns):
     #   Column          Non-Null Count  Dtype 
    ---  ------          --------------  ----- 
     0   id              25000 non-null  int64 
     1   url             25000 non-null  object
     2   title           25000 non-null  object
     3   text            25000 non-null  object
     4   title_vector    25000 non-null  object
     5   content_vector  25000 non-null  object
     6   vector_id       25000 non-null  object
    dtypes: int64(1), object(6)
    memory usage: 1.3+ MB


## Weaviate

Another vector database option we'll explore is **Weaviate**, which offers both a managed, [SaaS](https://console.weaviate.io/) option, as well as a self-hosted [open source](https://github.com/weaviate/weaviate) option. As we've already looked at a cloud vector database, we'll try the self-hosted option here.

For this we will:
- Set up a local deployment of Weaviate
- Create indices in Weaviate
- Store our data there
- Fire some similarity search queries
- Try a real use case


key: _node_content
string_value:
{"id_": "6168404b-425e-4936-8735-6ca773f8a578", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "105ad3e7-dd4e-4fc3-ad89-828cb68fd275", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f1e50e0a23cbd05a6303dce4eadd3e2a409b3265e3110a49323df38bcf426110", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "884ddc2d-69ae-4d33-9b7d-69e49fa4cc28", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "2227bc53c26ec91854caaf023a6210198f8ca58076b71aadcde3ce81a581636c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d5a16a3d-9ee7-4c84-9148-7b1c6c0f6d80", "node_type": "1", "metadata": {}, "hash": "90fecac5f3411c5c078a789fd288daf8432efa3aa262124fce90eb352c3ffb0b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5907, "end_char_idx": 9346, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Read vectors from strings back into a list
article_df['title_vector'] = article_df.title_vector.apply(literal_eval)
article_df['content_vector'] = article_df.content_vector.apply(literal_eval)

# Set vector_id to be a string
article_df['vector_id'] = article_df['vector_id'].apply(str)
```


```python
article_df.info(show_counts=True)
```

    <class 'pandas.core.frame.DataFrame'>
    RangeIndex: 25000 entries, 0 to 24999
    Data columns (total 7 columns):
     #   Column          Non-Null Count  Dtype 
    ---  ------          --------------  ----- 
     0   id              25000 non-null  int64 
     1   url             25000 non-null  object
     2   title           25000 non-null  object
     3   text            25000 non-null  object
     4   title_vector    25000 non-null  object
     5   content_vector  25000 non-null  object
     6   vector_id       25000 non-null  object
    dtypes: int64(1), object(6)
    memory usage: 1.3+ MB


## Weaviate

Another vector database option we'll explore is **Weaviate**, which offers both a managed, [SaaS](https://console.weaviate.io/) option, as well as a self-hosted [open source](https://github.com/weaviate/weaviate) option. As we've already looked at a cloud vector database, we'll try the self-hosted option here.

For this we will:
- Set up a local deployment of Weaviate
- Create indices in Weaviate
- Store our data there
- Fire some similarity search queries
- Try a real use case


### Bring your own vectors approach
In this cookbook, we provide the data with already generated vectors. This is a good approach for scenarios, where your data is already vectorized.

### Automated vectorization with OpenAI module
For scenarios, where your data is not vectorized yet, you can delegate the vectorization task with OpenAI to Weaviate.
Weaviate offers a built-in module [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai), which takes care of the vectorization for you at:
* import
* for any CRUD operations
* for semantic search

Check out the [Getting Started with Weaviate and OpenAI module cookbook](./weaviate/getting-started-with-weaviate-and-openai.ipynb) to learn step by step how to import and vectorize data in one step.

### Setup

To run Weaviate locally, you'll need [Docker](https://www.docker.com/). Following the instructions contained in the Weaviate documentation [here](https://weaviate.io/developers/weaviate/installation/docker-compose), we created an example docker-compose.yml file in this repo saved at [./weaviate/docker-compose.yml](./weaviate/docker-compose.yml).

After starting Docker, you can start Weaviate locally by navigating to the `examples/vector_databases/weaviate/` directory and running `docker-compose up -d`.

#### SaaS
Alternatively you can use [Weaviate Cloud Service](https://console.weaviate.io/) (WCS) to create a free Weaviate cluster.
1. create a free account and/or login to [WCS](https://console.weaviate.io/)
2. create a `Weaviate Cluster` with the following settings:
    * Sandbox: `Sandbox Free`
    * Weaviate Version: Use default (latest)
    * OIDC Authentication: `Disabled`
3. your instance should be ready in a minute or two
4. make a note of the `Cluster Id`. The link will take you to the full path of your cluster (you will need it later to connect to it). It should be something like: `https://your-project-name-suffix.weaviate.network`


key: _node_content
string_value:
{"id_": "d5a16a3d-9ee7-4c84-9148-7b1c6c0f6d80", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "105ad3e7-dd4e-4fc3-ad89-828cb68fd275", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f1e50e0a23cbd05a6303dce4eadd3e2a409b3265e3110a49323df38bcf426110", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6168404b-425e-4936-8735-6ca773f8a578", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "948bfa1d880a92ecb8626ee624642409d1c395f2965032fdb9cd83cad0458c23", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ce56af66-b8b1-494e-9868-9f2c714e359e", "node_type": "1", "metadata": {}, "hash": "982b6e72b18ea116823a5652081d817fc7c69f77f70b1f9af7be82eb4780ea38", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9350, "end_char_idx": 11140, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Option #1 - Self-hosted - Weaviate Open Source 
client = weaviate.Client(
    url="http://localhost:8080",
    additional_headers={
        "X-OpenAI-Api-Key": os.getenv("OPENAI_API_KEY")
    }
)
```


```python
# Option #2 - SaaS - (Weaviate Cloud Service)
client = weaviate.Client(
    url="https://your-wcs-instance-name.weaviate.network",
    additional_headers={
        "X-OpenAI-Api-Key": os.getenv("OPENAI_API_KEY")
    }
)
```


```python
client.is_ready()
```

### Index data

In Weaviate you create __schemas__ to capture each of the entities you will be searching. 

In this case we'll create a schema called **Article** with the **title** vector from above included for us to search by.

The next few steps closely follow the documentation Weaviate provides [here](https://weaviate.io/developers/weaviate/quickstart).



```python
# Clear up the schema, so that we can recreate it
client.schema.delete_all()
client.schema.get()

# Define the Schema object to use `text-embedding-3-small` on `title` and `content`, but skip it for `url`
article_schema = {
    "class": "Article",
    "description": "A collection of articles",
    "vectorizer": "text2vec-openai",
    "moduleConfig": {
        "text2vec-openai": {
          "model": "ada",
          "modelVersion": "002",
          "type": "text"
        }
    },
    "properties": [{
        "name": "title",
        "description": "Title of the article",
        "dataType": ["string"]
    },
    {
        "name": "content",
        "description": "Contents of the article",
        "dataType": ["text"],
        "moduleConfig": { "text2vec-openai": { "skip": True } }
    }]
}

# add the Article schema
client.schema.create_class(article_schema)

# get the schema to make sure it worked
client.schema.get()
```


key: _node_content
string_value:
{"id_": "ce56af66-b8b1-494e-9868-9f2c714e359e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "105ad3e7-dd4e-4fc3-ad89-828cb68fd275", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f1e50e0a23cbd05a6303dce4eadd3e2a409b3265e3110a49323df38bcf426110", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d5a16a3d-9ee7-4c84-9148-7b1c6c0f6d80", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "baddf20f95d1ca3e8ea1ec2dc79d3f99a78645360f9398bf2c6205b6284a9c39", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6db1b5d9-9282-4e03-a40c-f1df8ee7bc0d", "node_type": "1", "metadata": {}, "hash": "e6ee35f80c0a64480f769fb7c0d160afc6cd88a0f192af479dec9b9d33e0cd37", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9564, "end_char_idx": 12819, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Option #2 - SaaS - (Weaviate Cloud Service)
client = weaviate.Client(
    url="https://your-wcs-instance-name.weaviate.network",
    additional_headers={
        "X-OpenAI-Api-Key": os.getenv("OPENAI_API_KEY")
    }
)
```


```python
client.is_ready()
```

### Index data

In Weaviate you create __schemas__ to capture each of the entities you will be searching. 

In this case we'll create a schema called **Article** with the **title** vector from above included for us to search by.

The next few steps closely follow the documentation Weaviate provides [here](https://weaviate.io/developers/weaviate/quickstart).



```python
# Clear up the schema, so that we can recreate it
client.schema.delete_all()
client.schema.get()

# Define the Schema object to use `text-embedding-3-small` on `title` and `content`, but skip it for `url`
article_schema = {
    "class": "Article",
    "description": "A collection of articles",
    "vectorizer": "text2vec-openai",
    "moduleConfig": {
        "text2vec-openai": {
          "model": "ada",
          "modelVersion": "002",
          "type": "text"
        }
    },
    "properties": [{
        "name": "title",
        "description": "Title of the article",
        "dataType": ["string"]
    },
    {
        "name": "content",
        "description": "Contents of the article",
        "dataType": ["text"],
        "moduleConfig": { "text2vec-openai": { "skip": True } }
    }]
}

# add the Article schema
client.schema.create_class(article_schema)

# get the schema to make sure it worked
client.schema.get()
```




    {'classes': [{'class': 'Article',
       'description': 'A collection of articles',
       'invertedIndexConfig': {'bm25': {'b': 0.75, 'k1': 1.2},
        'cleanupIntervalSeconds': 60,
        'stopwords': {'additions': None, 'preset': 'en', 'removals': None}},
       'moduleConfig': {'text2vec-openai': {'model': 'ada',
         'modelVersion': '002',
         'type': 'text',
         'vectorizeClassName': True}},
       'properties': [{'dataType': ['string'],
         'description': 'Title of the article',
         'moduleConfig': {'text2vec-openai': {'skip': False,
           'vectorizePropertyName': False}},
         'name': 'title',
         'tokenization': 'word'},
        {'dataType': ['text'],
         'description': 'Contents of the article',
         'moduleConfig': {'text2vec-openai': {'skip': True,
           'vectorizePropertyName': False}},
         'name': 'content',
         'tokenization': 'word'}],
       'replicationConfig': {'factor': 1},
       'shardingConfig': {'virtualPerPhysical': 128,
        'desiredCount': 1,
        'actualCount': 1,
        'desiredVirtualCount': 128,
        'actualVirtualCount': 128,
        'key': '_id',
        'strategy': 'hash',
        'function': 'murmur3'},
       'vectorIndexConfig': {'skip': False,
        'cleanupIntervalSeconds': 300,
        'maxConnections': 64,
        'efConstruction': 128,
        'ef': -1,
        'dynamicEfMin': 100,
        'dynamicEfMax': 500,
        'dynamicEfFactor': 8,
        'vectorCacheMaxObjects': 1000000000000,
        'flatSearchCutoff': 40000,
        'distance': 'cosine'},
       'vectorIndexType': 'hnsw',
       'vectorizer': 'text2vec-openai'}]}


key: _node_content
string_value:
{"id_": "6db1b5d9-9282-4e03-a40c-f1df8ee7bc0d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "105ad3e7-dd4e-4fc3-ad89-828cb68fd275", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f1e50e0a23cbd05a6303dce4eadd3e2a409b3265e3110a49323df38bcf426110", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ce56af66-b8b1-494e-9868-9f2c714e359e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "dacb84ae0acd83f866a831c1438cd4392b5301b2b9ad7b880fd9d5374c6190ca", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3655ed68-78e3-4f66-9351-8056d183fb1a", "node_type": "1", "metadata": {}, "hash": "b826d1f19ce073fb40a7dd4d7b173da3b4357cb1e4c3230cae13b71d465433f5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
### Step 1 - configure Weaviate Batch, which optimizes CRUD operations in bulk
# - starting batch size of 100
# - dynamically increase/decrease based on performance
# - add timeout retries if something goes wrong

client.batch.configure(
    batch_size=100,
    dynamic=True,
    timeout_retries=3,
)
```




    <weaviate.batch.crud_batch.Batch at 0x3f0ca0fa0>




```python
### Step 2 - import data

print("Uploading data with vectors to Article schema.")

counter=0

with client.batch as batch:
    for k,v in article_df.iterrows():
        
        # print update message every 100 objects        
        if (counter %100 == 0):
            print(f"Import {counter} / {len(article_df)} ")
        
        properties = {
            "title": v["title"],
            "content": v["text"]
        }
        
        vector = v["title_vector"]
        
        batch.add_data_object(properties, "Article", None, vector)
        counter = counter+1

print(f"Importing ({len(article_df)}) Articles complete")  
```

    Uploading data with vectors to Article schema.
    Import 0 / 25000 
    Import 100 / 25000 
    Import 200 / 25000 
    Import 300 / 25000 
    Import 400 / 25000 
    Import 500 / 25000 
    Import 600 / 25000 
    Import 700 / 25000 
    Import 800 / 25000 
    Import 900 / 25000 
    Import 1000 / 25000 
    Import 1100 / 25000 
    Import 1200 / 25000 
    Import 1300 / 25000 
    Import 1400 / 25000 
    Import 1500 / 25000 
    Import 1600 / 25000 
    Import 1700 / 25000 
    Import 1800 / 25000 
    Import 1900 / 25000 
    Import 2000 / 25000 
    Import 2100 / 25000 
    Import 2200 / 25000 
    Import 2300 / 25000 
    Import 2400 / 25000 
    Import 2500 / 25000 
    Import 2600 / 25000 
    Import 2700 / 25000 
    Import 2800 / 25000 
    Import 2900 / 25000 
    Import 3000 / 25000 
    Import 3100 / 25000 
    Import 3200 / 25000 
    Import 3300 / 25000 
    Import 3400 / 25000 
    Import 3500 / 25000 
    Import 3600 / 25000 
    Import 3700 / 25000 
    Import 3800 / 25000 
    Import 3900 / 25000 
    Import 4000 / 25000 
    Import 4100 / 25000 
    Import 4200 /


key: _node_content
string_value:
{"id_": "3655ed68-78e3-4f66-9351-8056d183fb1a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "105ad3e7-dd4e-4fc3-ad89-828cb68fd275", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f1e50e0a23cbd05a6303dce4eadd3e2a409b3265e3110a49323df38bcf426110", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6db1b5d9-9282-4e03-a40c-f1df8ee7bc0d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "2b38e0b97d1a7825ec619572f8637d5d70f8a874a95919c37cf847eda610866d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a8f7bfd3-68d0-4037-8877-0e53e64feb7c", "node_type": "1", "metadata": {}, "hash": "817b9687fbb19a9f5a55424392c1379fb2b6b748f1b47e451a691cded3994200", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14121, "end_char_idx": 15714, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Import 900 / 25000 
    Import 1000 / 25000 
    Import 1100 / 25000 
    Import 1200 / 25000 
    Import 1300 / 25000 
    Import 1400 / 25000 
    Import 1500 / 25000 
    Import 1600 / 25000 
    Import 1700 / 25000 
    Import 1800 / 25000 
    Import 1900 / 25000 
    Import 2000 / 25000 
    Import 2100 / 25000 
    Import 2200 / 25000 
    Import 2300 / 25000 
    Import 2400 / 25000 
    Import 2500 / 25000 
    Import 2600 / 25000 
    Import 2700 / 25000 
    Import 2800 / 25000 
    Import 2900 / 25000 
    Import 3000 / 25000 
    Import 3100 / 25000 
    Import 3200 / 25000 
    Import 3300 / 25000 
    Import 3400 / 25000 
    Import 3500 / 25000 
    Import 3600 / 25000 
    Import 3700 / 25000 
    Import 3800 / 25000 
    Import 3900 / 25000 
    Import 4000 / 25000 
    Import 4100 / 25000 
    Import 4200 / 25000 
    Import 4300 / 25000 
    Import 4400 / 25000 
    Import 4500 / 25000 
    Import 4600 / 25000 
    Import 4700 / 25000 
    Import 4800 / 25000 
    Import 4900 / 25000 
    Import 5000 / 25000 
    Import 5100 / 25000 
    Import 5200 / 25000 
    Import 5300 / 25000 
    Import 5400 / 25000 
    Import 5500 / 25000 
    Import 5600 / 25000 
    Import 5700 / 25000 
    Import 5800 / 25000 
    Import 5900 / 25000 
    Import 6000 / 25000 
    Import 6100 / 25000 
    Import 6200 / 25000 
    Import 6300 / 25000 
    Import 6400 / 25000 
    Import 6500 / 25000 
    Import 6600 / 25000 
    Import 6700 / 25000 
    Import 6800 / 25000 
    Import 6900 / 25000 
    Import 7000 / 25000 
    Import 7100 / 25000 
    Import 7200 / 25000


key: _node_content
string_value:
{"id_": "a8f7bfd3-68d0-4037-8877-0e53e64feb7c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "105ad3e7-dd4e-4fc3-ad89-828cb68fd275", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f1e50e0a23cbd05a6303dce4eadd3e2a409b3265e3110a49323df38bcf426110", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3655ed68-78e3-4f66-9351-8056d183fb1a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "447e2a62615131256367c7ff71ec12aba63fad8e7f6f4df0ead7903ea7b212d5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7430c852-9a75-410d-ac75-015ee60e5560", "node_type": "1", "metadata": {}, "hash": "4d2127694d0da5e7bdd5b1f8131be3765976240e348bcd183cd3fc898e763d42", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14895, "end_char_idx": 16487, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Import 4000 / 25000 
    Import 4100 / 25000 
    Import 4200 / 25000 
    Import 4300 / 25000 
    Import 4400 / 25000 
    Import 4500 / 25000 
    Import 4600 / 25000 
    Import 4700 / 25000 
    Import 4800 / 25000 
    Import 4900 / 25000 
    Import 5000 / 25000 
    Import 5100 / 25000 
    Import 5200 / 25000 
    Import 5300 / 25000 
    Import 5400 / 25000 
    Import 5500 / 25000 
    Import 5600 / 25000 
    Import 5700 / 25000 
    Import 5800 / 25000 
    Import 5900 / 25000 
    Import 6000 / 25000 
    Import 6100 / 25000 
    Import 6200 / 25000 
    Import 6300 / 25000 
    Import 6400 / 25000 
    Import 6500 / 25000 
    Import 6600 / 25000 
    Import 6700 / 25000 
    Import 6800 / 25000 
    Import 6900 / 25000 
    Import 7000 / 25000 
    Import 7100 / 25000 
    Import 7200 / 25000 
    Import 7300 / 25000 
    Import 7400 / 25000 
    Import 7500 / 25000 
    Import 7600 / 25000 
    Import 7700 / 25000 
    Import 7800 / 25000 
    Import 7900 / 25000 
    Import 8000 / 25000 
    Import 8100 / 25000 
    Import 8200 / 25000 
    Import 8300 / 25000 
    Import 8400 / 25000 
    Import 8500 / 25000 
    Import 8600 / 25000 
    Import 8700 / 25000 
    Import 8800 / 25000 
    Import 8900 / 25000 
    Import 9000 / 25000 
    Import 9100 / 25000 
    Import 9200 / 25000 
    Import 9300 / 25000 
    Import 9400 / 25000 
    Import 9500 / 25000 
    Import 9600 / 25000 
    Import 9700 / 25000 
    Import 9800 / 25000 
    Import 9900 / 25000 
    Import 10000 / 25000 
    Import 10100 / 25000 
    Import 10200 / 25000 
    Import 10300 /


key: _node_content
string_value:
{"id_": "7430c852-9a75-410d-ac75-015ee60e5560", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "105ad3e7-dd4e-4fc3-ad89-828cb68fd275", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f1e50e0a23cbd05a6303dce4eadd3e2a409b3265e3110a49323df38bcf426110", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a8f7bfd3-68d0-4037-8877-0e53e64feb7c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "019e0c076cd83801cc45408462e39b354558d5a4515eb7675d8190afee0e7f1a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "547d8b71-6f6d-455f-b91e-c1b392fc21e2", "node_type": "1", "metadata": {}, "hash": "033d7a3b510026426048cd08515a88375fe8f2f51bb087c47a2fbccb5bf770f0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15652, "end_char_idx": 17273, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
7000 / 25000 
    Import 7100 / 25000 
    Import 7200 / 25000 
    Import 7300 / 25000 
    Import 7400 / 25000 
    Import 7500 / 25000 
    Import 7600 / 25000 
    Import 7700 / 25000 
    Import 7800 / 25000 
    Import 7900 / 25000 
    Import 8000 / 25000 
    Import 8100 / 25000 
    Import 8200 / 25000 
    Import 8300 / 25000 
    Import 8400 / 25000 
    Import 8500 / 25000 
    Import 8600 / 25000 
    Import 8700 / 25000 
    Import 8800 / 25000 
    Import 8900 / 25000 
    Import 9000 / 25000 
    Import 9100 / 25000 
    Import 9200 / 25000 
    Import 9300 / 25000 
    Import 9400 / 25000 
    Import 9500 / 25000 
    Import 9600 / 25000 
    Import 9700 / 25000 
    Import 9800 / 25000 
    Import 9900 / 25000 
    Import 10000 / 25000 
    Import 10100 / 25000 
    Import 10200 / 25000 
    Import 10300 / 25000 
    Import 10400 / 25000 
    Import 10500 / 25000 
    Import 10600 / 25000 
    Import 10700 / 25000 
    Import 10800 / 25000 
    Import 10900 / 25000 
    Import 11000 / 25000 
    Import 11100 / 25000 
    Import 11200 / 25000 
    Import 11300 / 25000 
    Import 11400 / 25000 
    Import 11500 / 25000 
    Import 11600 / 25000 
    Import 11700 / 25000 
    Import 11800 / 25000 
    Import 11900 / 25000 
    Import 12000 / 25000 
    Import 12100 / 25000 
    Import 12200 / 25000 
    Import 12300 / 25000 
    Import 12400 / 25000 
    Import 12500 / 25000 
    Import 12600 / 25000 
    Import 12700 / 25000 
    Import 12800 / 25000 
    Import 12900 / 25000 
    Import 13000 / 25000 
    Import 13100 / 25000 
    Import 13200 / 25000 
    Import 13300 / 25000


key: _node_content
string_value:
{"id_": "547d8b71-6f6d-455f-b91e-c1b392fc21e2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "105ad3e7-dd4e-4fc3-ad89-828cb68fd275", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f1e50e0a23cbd05a6303dce4eadd3e2a409b3265e3110a49323df38bcf426110", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7430c852-9a75-410d-ac75-015ee60e5560", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "44aaa4da573f97cddb62a3bab4003752c22131802ca01ec7a41a1cfc55d8ee5d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6251a170-b688-4f40-acbf-bae066d39a0e", "node_type": "1", "metadata": {}, "hash": "3d8b43043e604db0040ed442e3c16eea436617c0f84150730e58c41785e4191f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16421, "end_char_idx": 18073, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Import 10100 / 25000 
    Import 10200 / 25000 
    Import 10300 / 25000 
    Import 10400 / 25000 
    Import 10500 / 25000 
    Import 10600 / 25000 
    Import 10700 / 25000 
    Import 10800 / 25000 
    Import 10900 / 25000 
    Import 11000 / 25000 
    Import 11100 / 25000 
    Import 11200 / 25000 
    Import 11300 / 25000 
    Import 11400 / 25000 
    Import 11500 / 25000 
    Import 11600 / 25000 
    Import 11700 / 25000 
    Import 11800 / 25000 
    Import 11900 / 25000 
    Import 12000 / 25000 
    Import 12100 / 25000 
    Import 12200 / 25000 
    Import 12300 / 25000 
    Import 12400 / 25000 
    Import 12500 / 25000 
    Import 12600 / 25000 
    Import 12700 / 25000 
    Import 12800 / 25000 
    Import 12900 / 25000 
    Import 13000 / 25000 
    Import 13100 / 25000 
    Import 13200 / 25000 
    Import 13300 / 25000 
    Import 13400 / 25000 
    Import 13500 / 25000 
    Import 13600 / 25000 
    Import 13700 / 25000 
    Import 13800 / 25000 
    Import 13900 / 25000 
    Import 14000 / 25000 
    Import 14100 / 25000 
    Import 14200 / 25000 
    Import 14300 / 25000 
    Import 14400 / 25000 
    Import 14500 / 25000 
    Import 14600 / 25000 
    Import 14700 / 25000 
    Import 14800 / 25000 
    Import 14900 / 25000 
    Import 15000 / 25000 
    Import 15100 / 25000 
    Import 15200 / 25000 
    Import 15300 / 25000 
    Import 15400 / 25000 
    Import 15500 / 25000 
    Import 15600 / 25000 
    Import 15700 / 25000 
    Import 15800 / 25000 
    Import 15900 / 25000 
    Import 16000 / 25000 
    Import 16100 / 25000 
    Import 16200 / 25000 
    Import 16300 / 25000 
    Import 16400 /


key: _node_content
string_value:
{"id_": "6251a170-b688-4f40-acbf-bae066d39a0e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "105ad3e7-dd4e-4fc3-ad89-828cb68fd275", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f1e50e0a23cbd05a6303dce4eadd3e2a409b3265e3110a49323df38bcf426110", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "547d8b71-6f6d-455f-b91e-c1b392fc21e2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "e29096b52cd1093d246058098a2bb0c168125df210253fe1758155e273e973f9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9bafec6c-6232-4e25-bc67-13118695ab64", "node_type": "1", "metadata": {}, "hash": "0fb7cc44d0b07d1653fa1a91c95ed4363d079296d63dbc1f2268b9bd5f39e198", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17208, "end_char_idx": 18859, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
13100 / 25000 
    Import 13200 / 25000 
    Import 13300 / 25000 
    Import 13400 / 25000 
    Import 13500 / 25000 
    Import 13600 / 25000 
    Import 13700 / 25000 
    Import 13800 / 25000 
    Import 13900 / 25000 
    Import 14000 / 25000 
    Import 14100 / 25000 
    Import 14200 / 25000 
    Import 14300 / 25000 
    Import 14400 / 25000 
    Import 14500 / 25000 
    Import 14600 / 25000 
    Import 14700 / 25000 
    Import 14800 / 25000 
    Import 14900 / 25000 
    Import 15000 / 25000 
    Import 15100 / 25000 
    Import 15200 / 25000 
    Import 15300 / 25000 
    Import 15400 / 25000 
    Import 15500 / 25000 
    Import 15600 / 25000 
    Import 15700 / 25000 
    Import 15800 / 25000 
    Import 15900 / 25000 
    Import 16000 / 25000 
    Import 16100 / 25000 
    Import 16200 / 25000 
    Import 16300 / 25000 
    Import 16400 / 25000 
    Import 16500 / 25000 
    Import 16600 / 25000 
    Import 16700 / 25000 
    Import 16800 / 25000 
    Import 16900 / 25000 
    Import 17000 / 25000 
    Import 17100 / 25000 
    Import 17200 / 25000 
    Import 17300 / 25000 
    Import 17400 / 25000 
    Import 17500 / 25000 
    Import 17600 / 25000 
    Import 17700 / 25000 
    Import 17800 / 25000 
    Import 17900 / 25000 
    Import 18000 / 25000 
    Import 18100 / 25000 
    Import 18200 / 25000 
    Import 18300 / 25000 
    Import 18400 / 25000 
    Import 18500 / 25000 
    Import 18600 / 25000 
    Import 18700 / 25000 
    Import 18800 / 25000 
    Import 18900 / 25000 
    Import 19000 / 25000 
    Import 19100 / 25000 
    Import 19200 / 25000 
    Import 19300 / 25000 
    Import 19400 / 25000


key: _node_content
string_value:
{"id_": "9bafec6c-6232-4e25-bc67-13118695ab64", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "105ad3e7-dd4e-4fc3-ad89-828cb68fd275", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f1e50e0a23cbd05a6303dce4eadd3e2a409b3265e3110a49323df38bcf426110", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6251a170-b688-4f40-acbf-bae066d39a0e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "6b75675ad6bc95a46b5d295f1ca8153bfffa50c7a0fadf839f3a76a43464a19e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "09fe66f0-49e4-4c82-9ef6-8b856af6b0b5", "node_type": "1", "metadata": {}, "hash": "a6e7db12c0c87c5baf1bbc57d8cf8993016903035c85c18fbf6369d1a533aa51", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18007, "end_char_idx": 19659, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Import 16200 / 25000 
    Import 16300 / 25000 
    Import 16400 / 25000 
    Import 16500 / 25000 
    Import 16600 / 25000 
    Import 16700 / 25000 
    Import 16800 / 25000 
    Import 16900 / 25000 
    Import 17000 / 25000 
    Import 17100 / 25000 
    Import 17200 / 25000 
    Import 17300 / 25000 
    Import 17400 / 25000 
    Import 17500 / 25000 
    Import 17600 / 25000 
    Import 17700 / 25000 
    Import 17800 / 25000 
    Import 17900 / 25000 
    Import 18000 / 25000 
    Import 18100 / 25000 
    Import 18200 / 25000 
    Import 18300 / 25000 
    Import 18400 / 25000 
    Import 18500 / 25000 
    Import 18600 / 25000 
    Import 18700 / 25000 
    Import 18800 / 25000 
    Import 18900 / 25000 
    Import 19000 / 25000 
    Import 19100 / 25000 
    Import 19200 / 25000 
    Import 19300 / 25000 
    Import 19400 / 25000 
    Import 19500 / 25000 
    Import 19600 / 25000 
    Import 19700 / 25000 
    Import 19800 / 25000 
    Import 19900 / 25000 
    Import 20000 / 25000 
    Import 20100 / 25000 
    Import 20200 / 25000 
    Import 20300 / 25000 
    Import 20400 / 25000 
    Import 20500 / 25000 
    Import 20600 / 25000 
    Import 20700 / 25000 
    Import 20800 / 25000 
    Import 20900 / 25000 
    Import 21000 / 25000 
    Import 21100 / 25000 
    Import 21200 / 25000 
    Import 21300 / 25000 
    Import 21400 / 25000 
    Import 21500 / 25000 
    Import 21600 / 25000 
    Import 21700 / 25000 
    Import 21800 / 25000 
    Import 21900 / 25000 
    Import 22000 / 25000 
    Import 22100 / 25000 
    Import 22200 / 25000 
    Import 22300 / 25000 
    Import 22400 / 25000 
    Import 22500 /


key: _node_content
string_value:
{"id_": "09fe66f0-49e4-4c82-9ef6-8b856af6b0b5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "105ad3e7-dd4e-4fc3-ad89-828cb68fd275", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f1e50e0a23cbd05a6303dce4eadd3e2a409b3265e3110a49323df38bcf426110", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9bafec6c-6232-4e25-bc67-13118695ab64", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "505100d167946453d92c31e47a83d5fa5a98b8c889ba5ece3648f19fad4f82a2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "df551ae3-6a7c-45dc-8a55-f0f75a3135c5", "node_type": "1", "metadata": {}, "hash": "11ab5009488c061ec53d52b34bdfac3bb1ed7d14eccb9852acc725e337c4c2ee", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18794, "end_char_idx": 20330, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
19200 / 25000 
    Import 19300 / 25000 
    Import 19400 / 25000 
    Import 19500 / 25000 
    Import 19600 / 25000 
    Import 19700 / 25000 
    Import 19800 / 25000 
    Import 19900 / 25000 
    Import 20000 / 25000 
    Import 20100 / 25000 
    Import 20200 / 25000 
    Import 20300 / 25000 
    Import 20400 / 25000 
    Import 20500 / 25000 
    Import 20600 / 25000 
    Import 20700 / 25000 
    Import 20800 / 25000 
    Import 20900 / 25000 
    Import 21000 / 25000 
    Import 21100 / 25000 
    Import 21200 / 25000 
    Import 21300 / 25000 
    Import 21400 / 25000 
    Import 21500 / 25000 
    Import 21600 / 25000 
    Import 21700 / 25000 
    Import 21800 / 25000 
    Import 21900 / 25000 
    Import 22000 / 25000 
    Import 22100 / 25000 
    Import 22200 / 25000 
    Import 22300 / 25000 
    Import 22400 / 25000 
    Import 22500 / 25000 
    Import 22600 / 25000 
    Import 22700 / 25000 
    Import 22800 / 25000 
    Import 22900 / 25000 
    Import 23000 / 25000 
    Import 23100 / 25000 
    Import 23200 / 25000 
    Import 23300 / 25000 
    Import 23400 / 25000 
    Import 23500 / 25000 
    Import 23600 / 25000 
    Import 23700 / 25000 
    Import 23800 / 25000 
    Import 23900 / 25000 
    Import 24000 / 25000 
    Import 24100 / 25000 
    Import 24200 / 25000 
    Import 24300 / 25000 
    Import 24400 / 25000 
    Import 24500 / 25000 
    Import 24600 / 25000 
    Import 24700 / 25000 
    Import 24800 / 25000 
    Import 24900 / 25000 
    Importing (25000) Articles complete


key: _node_content
string_value:
{"id_": "df551ae3-6a7c-45dc-8a55-f0f75a3135c5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "105ad3e7-dd4e-4fc3-ad89-828cb68fd275", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f1e50e0a23cbd05a6303dce4eadd3e2a409b3265e3110a49323df38bcf426110", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "09fe66f0-49e4-4c82-9ef6-8b856af6b0b5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "1bc086b207ab3a14cdb6e86500020465ccf6cbbe605c94554676030addb6ae41", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b50da944-fc45-4fda-8955-3f570c89ca68", "node_type": "1", "metadata": {}, "hash": "bb61926ac4774cd1afc875939707c911be5d9e348f15974cd3999ad48fa7ef96", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19450, "end_char_idx": 21244, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
/ 25000 
    Import 21800 / 25000 
    Import 21900 / 25000 
    Import 22000 / 25000 
    Import 22100 / 25000 
    Import 22200 / 25000 
    Import 22300 / 25000 
    Import 22400 / 25000 
    Import 22500 / 25000 
    Import 22600 / 25000 
    Import 22700 / 25000 
    Import 22800 / 25000 
    Import 22900 / 25000 
    Import 23000 / 25000 
    Import 23100 / 25000 
    Import 23200 / 25000 
    Import 23300 / 25000 
    Import 23400 / 25000 
    Import 23500 / 25000 
    Import 23600 / 25000 
    Import 23700 / 25000 
    Import 23800 / 25000 
    Import 23900 / 25000 
    Import 24000 / 25000 
    Import 24100 / 25000 
    Import 24200 / 25000 
    Import 24300 / 25000 
    Import 24400 / 25000 
    Import 24500 / 25000 
    Import 24600 / 25000 
    Import 24700 / 25000 
    Import 24800 / 25000 
    Import 24900 / 25000 
    Importing (25000) Articles complete



```python
# Test that all data has loaded – get object count
result = (
    client.query.aggregate("Article")
    .with_fields("meta { count }")
    .do()
)
print("Object count: ", result["data"]["Aggregate"]["Article"])
```

    Object count:  [{'meta': {'count': 25000}}]



```python
# Test one article has worked by checking one object
test_article = (
    client.query
    .get("Article", ["title", "content", "_additional {id}"])
    .with_limit(1)
    .do()
)["data"]["Get"]["Article"][0]

print(test_article["_additional"]["id"])
print(test_article["title"])
print(test_article["content"])
```

    000393f2-1182-4e3d-abcf-4217eda64be0
    Lago d'Origlio
    Lago d'Origlio is a lake in the municipality of Origlio, in Ticino, Switzerland.
    
    Lakes of Ticino


### Search data

As above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors


key: _node_content
string_value:
{"id_": "b50da944-fc45-4fda-8955-3f570c89ca68", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "105ad3e7-dd4e-4fc3-ad89-828cb68fd275", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f1e50e0a23cbd05a6303dce4eadd3e2a409b3265e3110a49323df38bcf426110", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "df551ae3-6a7c-45dc-8a55-f0f75a3135c5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "02d42c19788423a82603826920e188c2d239cbc8ce75c30ed98448747670a008", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "214b9d01-bf85-461b-b44b-046459a2503a", "node_type": "1", "metadata": {}, "hash": "38ff07da86b966c01397abffc90605b7d2f0fd7a0fd9339a7963fa0a561b66e4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19983, "end_char_idx": 21840, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Import 23800 / 25000 
    Import 23900 / 25000 
    Import 24000 / 25000 
    Import 24100 / 25000 
    Import 24200 / 25000 
    Import 24300 / 25000 
    Import 24400 / 25000 
    Import 24500 / 25000 
    Import 24600 / 25000 
    Import 24700 / 25000 
    Import 24800 / 25000 
    Import 24900 / 25000 
    Importing (25000) Articles complete



```python
# Test that all data has loaded – get object count
result = (
    client.query.aggregate("Article")
    .with_fields("meta { count }")
    .do()
)
print("Object count: ", result["data"]["Aggregate"]["Article"])
```

    Object count:  [{'meta': {'count': 25000}}]



```python
# Test one article has worked by checking one object
test_article = (
    client.query
    .get("Article", ["title", "content", "_additional {id}"])
    .with_limit(1)
    .do()
)["data"]["Get"]["Article"][0]

print(test_article["_additional"]["id"])
print(test_article["title"])
print(test_article["content"])
```

    000393f2-1182-4e3d-abcf-4217eda64be0
    Lago d'Origlio
    Lago d'Origlio is a lake in the municipality of Origlio, in Ticino, Switzerland.
    
    Lakes of Ticino


### Search data

As above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors


```python
def query_weaviate(query, collection_name, top_k=20):

    # Creates embedding vector from user query
    embedded_query = openai.Embedding.create(
        input=query,
        model=EMBEDDING_MODEL,
    )["data"][0]['embedding']
    
    near_vector = {"vector": embedded_query}

    # Queries input schema with vectorised user query
    query_result = (
        client.query
        .get(collection_name, ["title", "content", "_additional {certainty distance}"])
        .with_near_vector(near_vector)
        .with_limit(top_k)
        .do()
    )
    
    return query_result
```


key: _node_content
string_value:
{"id_": "214b9d01-bf85-461b-b44b-046459a2503a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "105ad3e7-dd4e-4fc3-ad89-828cb68fd275", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f1e50e0a23cbd05a6303dce4eadd3e2a409b3265e3110a49323df38bcf426110", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b50da944-fc45-4fda-8955-3f570c89ca68", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "e80c1bf4b8f6f1e523c0626337767bdf47edef99f405cda2db22e3bf7d82e901", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "79402333-dc0b-42f7-9f80-a42f7c0b2b6e", "node_type": "1", "metadata": {}, "hash": "250117ee8578b85026bd11c00c797f387ab9033d560b419116d20bdf2b90426f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20295, "end_char_idx": 23310, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Importing (25000) Articles complete



```python
# Test that all data has loaded – get object count
result = (
    client.query.aggregate("Article")
    .with_fields("meta { count }")
    .do()
)
print("Object count: ", result["data"]["Aggregate"]["Article"])
```

    Object count:  [{'meta': {'count': 25000}}]



```python
# Test one article has worked by checking one object
test_article = (
    client.query
    .get("Article", ["title", "content", "_additional {id}"])
    .with_limit(1)
    .do()
)["data"]["Get"]["Article"][0]

print(test_article["_additional"]["id"])
print(test_article["title"])
print(test_article["content"])
```

    000393f2-1182-4e3d-abcf-4217eda64be0
    Lago d'Origlio
    Lago d'Origlio is a lake in the municipality of Origlio, in Ticino, Switzerland.
    
    Lakes of Ticino


### Search data

As above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors


```python
def query_weaviate(query, collection_name, top_k=20):

    # Creates embedding vector from user query
    embedded_query = openai.Embedding.create(
        input=query,
        model=EMBEDDING_MODEL,
    )["data"][0]['embedding']
    
    near_vector = {"vector": embedded_query}

    # Queries input schema with vectorised user query
    query_result = (
        client.query
        .get(collection_name, ["title", "content", "_additional {certainty distance}"])
        .with_near_vector(near_vector)
        .with_limit(top_k)
        .do()
    )
    
    return query_result
```


```python
query_result = query_weaviate("modern art in Europe", "Article")
counter = 0
for article in query_result["data"]["Get"]["Article"]:
    counter += 1
    print(f"{counter}. { article['title']} (Certainty: {round(article['_additional']['certainty'],3) }) (Distance: {round(article['_additional']['distance'],3) })")
```

    1. Museum of Modern Art (Certainty: 0.938) (Distance: 0.125)
    2. Western Europe (Certainty: 0.934) (Distance: 0.133)
    3. Renaissance art (Certainty: 0.932) (Distance: 0.136)
    4. Pop art (Certainty: 0.93) (Distance: 0.14)
    5. Northern Europe (Certainty: 0.927) (Distance: 0.145)
    6. Hellenistic art (Certainty: 0.926) (Distance: 0.147)
    7. Modernist literature (Certainty: 0.924) (Distance: 0.153)
    8. Art film (Certainty: 0.922) (Distance: 0.157)
    9. Central Europe (Certainty: 0.921) (Distance: 0.157)
    10. European (Certainty: 0.921) (Distance: 0.159)
    11. Art (Certainty: 0.921) (Distance: 0.159)
    12. Byzantine art (Certainty: 0.92) (Distance: 0.159)
    13. Postmodernism (Certainty: 0.92) (Distance: 0.16)
    14. Eastern Europe (Certainty: 0.92) (Distance: 0.161)
    15. Europe (Certainty: 0.919) (Distance: 0.161)
    16. Cubism (Certainty: 0.919) (Distance: 0.161)
    17. Impressionism (Certainty: 0.919) (Distance: 0.162)
    18. Bauhaus (Certainty: 0.919) (Distance: 0.162)
    19. Expressionism (Certainty: 0.918) (Distance: 0.163)
    20. Surrealism (Certainty: 0.918) (Distance: 0.163)


key: _node_content
string_value:
{"id_": "79402333-dc0b-42f7-9f80-a42f7c0b2b6e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "105ad3e7-dd4e-4fc3-ad89-828cb68fd275", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f1e50e0a23cbd05a6303dce4eadd3e2a409b3265e3110a49323df38bcf426110", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "214b9d01-bf85-461b-b44b-046459a2503a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9409905c40be96f45e1f3194b0f98bdd05e80a6758e91a6d26effe3d8e417bd3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7dae2690-9fd2-4548-af22-4f2e59270e5b", "node_type": "1", "metadata": {}, "hash": "a1c9811b5079510a370a01716f8f65534cba6264be9dffb3c6b640ede9d73f23", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23314, "end_char_idx": 25369, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
query_result = query_weaviate("Famous battles in Scottish history", "Article")
counter = 0
for article in query_result["data"]["Get"]["Article"]:
    counter += 1
    print(f"{counter}. {article['title']} (Score: {round(article['_additional']['certainty'],3) })")
```

    1. Historic Scotland (Score: 0.946)
    2. First War of Scottish Independence (Score: 0.946)
    3. Battle of Bannockburn (Score: 0.946)
    4. Wars of Scottish Independence (Score: 0.944)
    5. Second War of Scottish Independence (Score: 0.94)
    6. List of Scottish monarchs (Score: 0.937)
    7. Scottish Borders (Score: 0.932)
    8. Braveheart (Score: 0.929)
    9. John of Scotland (Score: 0.929)
    10. Guardians of Scotland (Score: 0.926)
    11. Holyrood Abbey (Score: 0.925)
    12. Scottish (Score: 0.925)
    13. Scots (Score: 0.925)
    14. Robert I of Scotland (Score: 0.924)
    15. Scottish people (Score: 0.924)
    16. Edinburgh Castle (Score: 0.924)
    17. Alexander I of Scotland (Score: 0.924)
    18. Robert Burns (Score: 0.924)
    19. Battle of Bosworth Field (Score: 0.922)
    20. David II of Scotland (Score: 0.922)


### Let Weaviate handle vector embeddings

Weaviate has a [built-in module for OpenAI](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai), which takes care of the steps required to generate a vector embedding for your queries and any CRUD operations.

This allows you to run a vector query with the `with_near_text` filter, which uses your `OPEN_API_KEY`.


```python
def near_text_weaviate(query, collection_name):
    
    nearText = {
        "concepts": [query],
        "distance": 0.7,
    }

    properties = [
        "title", "content",
        "_additional {certainty distance}"
    ]

    query_result = (
        client.query
        .get(collection_name, properties)
        .with_near_text(nearText)
        .with_limit(20)
        .do()
    )["data"]["Get"][collection_name]
    
    print (f"Objects returned: {len(query_result)}")
    
    return query_result
```


key: _node_content
string_value:
{"id_": "7dae2690-9fd2-4548-af22-4f2e59270e5b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "105ad3e7-dd4e-4fc3-ad89-828cb68fd275", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f1e50e0a23cbd05a6303dce4eadd3e2a409b3265e3110a49323df38bcf426110", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "79402333-dc0b-42f7-9f80-a42f7c0b2b6e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0c6f2f8dc094613ed5255307e81210356b95564e294a918b68ea81ab43bb79a8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "db549dca-d8c1-4663-8f28-53d08f3454f2", "node_type": "1", "metadata": {}, "hash": "4b1d951090cca7e1b4232cfac53d912cbda980def8226a40ef6711dc01592866", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24446, "end_char_idx": 26841, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Let Weaviate handle vector embeddings

Weaviate has a [built-in module for OpenAI](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai), which takes care of the steps required to generate a vector embedding for your queries and any CRUD operations.

This allows you to run a vector query with the `with_near_text` filter, which uses your `OPEN_API_KEY`.


```python
def near_text_weaviate(query, collection_name):
    
    nearText = {
        "concepts": [query],
        "distance": 0.7,
    }

    properties = [
        "title", "content",
        "_additional {certainty distance}"
    ]

    query_result = (
        client.query
        .get(collection_name, properties)
        .with_near_text(nearText)
        .with_limit(20)
        .do()
    )["data"]["Get"][collection_name]
    
    print (f"Objects returned: {len(query_result)}")
    
    return query_result
```


```python
query_result = near_text_weaviate("modern art in Europe","Article")
counter = 0
for article in query_result:
    counter += 1
    print(f"{counter}. { article['title']} (Certainty: {round(article['_additional']['certainty'],3) }) (Distance: {round(article['_additional']['distance'],3) })")
```

    Objects returned: 20
    1. Museum of Modern Art (Certainty: 0.938) (Distance: 0.125)
    2. Western Europe (Certainty: 0.934) (Distance: 0.133)
    3. Renaissance art (Certainty: 0.932) (Distance: 0.136)
    4. Pop art (Certainty: 0.93) (Distance: 0.14)
    5. Northern Europe (Certainty: 0.927) (Distance: 0.145)
    6. Hellenistic art (Certainty: 0.926) (Distance: 0.147)
    7. Modernist literature (Certainty: 0.923) (Distance: 0.153)
    8. Art film (Certainty: 0.922) (Distance: 0.157)
    9. Central Europe (Certainty: 0.921) (Distance: 0.157)
    10. European (Certainty: 0.921) (Distance: 0.159)
    11. Art (Certainty: 0.921) (Distance: 0.159)
    12. Byzantine art (Certainty: 0.92) (Distance: 0.159)
    13. Postmodernism (Certainty: 0.92) (Distance: 0.16)
    14. Eastern Europe (Certainty: 0.92) (Distance: 0.161)
    15. Europe (Certainty: 0.919) (Distance: 0.161)
    16. Cubism (Certainty: 0.919) (Distance: 0.161)
    17. Impressionism (Certainty: 0.919) (Distance: 0.162)
    18. Bauhaus (Certainty: 0.919) (Distance: 0.162)
    19. Surrealism (Certainty: 0.918) (Distance: 0.163)
    20. Expressionism (Certainty: 0.918) (Distance: 0.163)


key: _node_content
string_value:
{"id_": "db549dca-d8c1-4663-8f28-53d08f3454f2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "105ad3e7-dd4e-4fc3-ad89-828cb68fd275", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f1e50e0a23cbd05a6303dce4eadd3e2a409b3265e3110a49323df38bcf426110", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7dae2690-9fd2-4548-af22-4f2e59270e5b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "11da1a73d2492a1a89ded16ab4431e28d1da4a5e6284f0d2bfb32aecfe631105", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 26845, "end_char_idx": 28495, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
query_result = near_text_weaviate("Famous battles in Scottish history","Article")
counter = 0
for article in query_result:
    counter += 1
    print(f"{counter}. { article['title']} (Certainty: {round(article['_additional']['certainty'],3) }) (Distance: {round(article['_additional']['distance'],3) })")
```

    Objects returned: 20
    1. Historic Scotland (Certainty: 0.946) (Distance: 0.107)
    2. First War of Scottish Independence (Certainty: 0.946) (Distance: 0.108)
    3. Battle of Bannockburn (Certainty: 0.946) (Distance: 0.109)
    4. Wars of Scottish Independence (Certainty: 0.944) (Distance: 0.111)
    5. Second War of Scottish Independence (Certainty: 0.94) (Distance: 0.121)
    6. List of Scottish monarchs (Certainty: 0.937) (Distance: 0.127)
    7. Scottish Borders (Certainty: 0.932) (Distance: 0.137)
    8. Braveheart (Certainty: 0.929) (Distance: 0.141)
    9. John of Scotland (Certainty: 0.929) (Distance: 0.142)
    10. Guardians of Scotland (Certainty: 0.926) (Distance: 0.148)
    11. Holyrood Abbey (Certainty: 0.925) (Distance: 0.15)
    12. Scottish (Certainty: 0.925) (Distance: 0.15)
    13. Scots (Certainty: 0.925) (Distance: 0.15)
    14. Robert I of Scotland (Certainty: 0.924) (Distance: 0.151)
    15. Scottish people (Certainty: 0.924) (Distance: 0.152)
    16. Edinburgh Castle (Certainty: 0.924) (Distance: 0.153)
    17. Alexander I of Scotland (Certainty: 0.924) (Distance: 0.153)
    18. Robert Burns (Certainty: 0.924) (Distance: 0.153)
    19. Battle of Bosworth Field (Certainty: 0.922) (Distance: 0.155)
    20. David II of Scotland (Certainty: 0.922) (Distance: 0.157)



```python

```


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb


key: _node_content
string_value:
{"id_": "78d7206b-9d6a-49aa-b6ad-5ee3726c45cc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cfb7dcd7-df38-4283-a059-c75f4ee317f0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f75bb8b2745cc6f988eb21bf4d579a3ce42fa6438772074dabd728616f89367e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d6e98e5f-ec9b-412b-92f7-8ca580f59151", "node_type": "1", "metadata": {}, "hash": "1cedf7ad193670c958567e54a7c96fd27833e6fdc3c016f35ac09d9afa067293", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2380, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Question Answering in Weaviate with OpenAI Q&A module

This notebook is prepared for a scenario where:
* Your data is not vectorized
* You want to run Q&A ([learn more](https://weaviate.io/developers/weaviate/modules/reader-generator-modules/qna-openai)) on your data based on the [OpenAI completions](https://beta.openai.com/docs/api-reference/completions) endpoint.
* You want to use Weaviate with the OpenAI module ([text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai)), to generate vector embeddings for you.

This notebook takes you through a simple flow to set up a Weaviate instance, connect to it (with OpenAI API key), configure data schema, import data (which will automatically generate vector embeddings for your data), and run question answering.

## What is Weaviate

Weaviate is an open-source vector search engine that stores data objects together with their vectors. This allows for combining vector search with structured filtering.

Weaviate uses KNN algorithms to create an vector-optimized index, which allows your queries to run extremely fast. Learn more [here](https://weaviate.io/blog/why-is-vector-search-so-fast).

Weaviate let you use your favorite ML-models, and scale seamlessly into billions of data objects.

### Deployment options

Whatever your scenario or production setup, Weaviate has an option for you. You can deploy Weaviate in the following setups:
* Self-hosted – you can deploy Weaviate with docker locally, or any server you want.
* SaaS – you can use [Weaviate Cloud Service (WCS)](https://console.weaviate.io/) to host your Weaviate instances.
* Hybrid-SaaS – you can deploy Weaviate in your own private Cloud Service 

### Programming languages

Weaviate offers four [client libraries](https://weaviate.io/developers/weaviate/client-libraries), which allow you to communicate from your apps:
* [Python](https://weaviate.io/developers/weaviate/client-libraries/python)
* [JavaScript](https://weaviate.io/developers/weaviate/client-libraries/javascript)
* [Java](https://weaviate.io/developers/weaviate/client-libraries/java)
* [Go](https://weaviate.io/developers/weaviate/client-libraries/go)

Additionally, Weaviate has a [REST layer](https://weaviate.io/developers/weaviate/api/rest/objects). Basically you can call Weaviate from any language that supports REST requests.


key: _node_content
string_value:
{"id_": "d6e98e5f-ec9b-412b-92f7-8ca580f59151", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cfb7dcd7-df38-4283-a059-c75f4ee317f0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f75bb8b2745cc6f988eb21bf4d579a3ce42fa6438772074dabd728616f89367e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "78d7206b-9d6a-49aa-b6ad-5ee3726c45cc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3251c45b5f195a0447e1247fae60e2347af92cf8ef05dba1f8a29336839239ff", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0e66670d-3197-4109-bbc1-776d323b6d8a", "node_type": "1", "metadata": {}, "hash": "edf9f6dd3cbbe94a584e4dd8c5d77fb413406bea18b3d7afe040bd62b0577c42", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 822, "end_char_idx": 3997, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## What is Weaviate

Weaviate is an open-source vector search engine that stores data objects together with their vectors. This allows for combining vector search with structured filtering.

Weaviate uses KNN algorithms to create an vector-optimized index, which allows your queries to run extremely fast. Learn more [here](https://weaviate.io/blog/why-is-vector-search-so-fast).

Weaviate let you use your favorite ML-models, and scale seamlessly into billions of data objects.

### Deployment options

Whatever your scenario or production setup, Weaviate has an option for you. You can deploy Weaviate in the following setups:
* Self-hosted – you can deploy Weaviate with docker locally, or any server you want.
* SaaS – you can use [Weaviate Cloud Service (WCS)](https://console.weaviate.io/) to host your Weaviate instances.
* Hybrid-SaaS – you can deploy Weaviate in your own private Cloud Service 

### Programming languages

Weaviate offers four [client libraries](https://weaviate.io/developers/weaviate/client-libraries), which allow you to communicate from your apps:
* [Python](https://weaviate.io/developers/weaviate/client-libraries/python)
* [JavaScript](https://weaviate.io/developers/weaviate/client-libraries/javascript)
* [Java](https://weaviate.io/developers/weaviate/client-libraries/java)
* [Go](https://weaviate.io/developers/weaviate/client-libraries/go)

Additionally, Weaviate has a [REST layer](https://weaviate.io/developers/weaviate/api/rest/objects). Basically you can call Weaviate from any language that supports REST requests.

## Demo Flow
The demo flow is:
- **Prerequisites Setup**: Create a Weaviate instance and install required libraries
- **Connect**: Connect to your Weaviate instance 
- **Schema Configuration**: Configure the schema of your data
    - *Note*: Here we can define which OpenAI Embedding Model to use
    - *Note*: Here we can configure which properties to index
- **Import data**: Load a demo dataset and import it into Weaviate
    - *Note*: The import process will automatically index your data - based on the configuration in the schema
    - *Note*: You don't need to explicitly vectorize your data, Weaviate will communicate with OpenAI to do it for you
- **Run Queries**: Query 
    - *Note*: You don't need to explicitly vectorize your queries, Weaviate will communicate with OpenAI to do it for you
    - *Note*: The `qna-openai` module automatically communicates with the OpenAI completions endpoint

Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases for question answering.

## OpenAI Module in Weaviate
All Weaviate instances come equipped with the [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) and the [qna-openai](https://weaviate.io/developers/weaviate/modules/reader-generator-modules/qna-openai) modules.

The first module is responsible for handling vectorization at import (or any CRUD operations) and when you run a search query. The second module communicates with the OpenAI completions endpoint.

### No need to manually vectorize data
This is great news for you.


key: _node_content
string_value:
{"id_": "0e66670d-3197-4109-bbc1-776d323b6d8a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cfb7dcd7-df38-4283-a059-c75f4ee317f0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f75bb8b2745cc6f988eb21bf4d579a3ce42fa6438772074dabd728616f89367e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d6e98e5f-ec9b-412b-92f7-8ca580f59151", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "090672b836361d41b8d71d8f229c41f414446378082b8b497c1771ce173d2d59", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3c51a78f-2100-4a2b-a5ed-d67712f7cab6", "node_type": "1", "metadata": {}, "hash": "542895e94c9739af8d54a92c8b463901a65e9c5f6da37bdaeb93d209975e184e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2302, "end_char_idx": 5155, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Basically you can call Weaviate from any language that supports REST requests.

## Demo Flow
The demo flow is:
- **Prerequisites Setup**: Create a Weaviate instance and install required libraries
- **Connect**: Connect to your Weaviate instance 
- **Schema Configuration**: Configure the schema of your data
    - *Note*: Here we can define which OpenAI Embedding Model to use
    - *Note*: Here we can configure which properties to index
- **Import data**: Load a demo dataset and import it into Weaviate
    - *Note*: The import process will automatically index your data - based on the configuration in the schema
    - *Note*: You don't need to explicitly vectorize your data, Weaviate will communicate with OpenAI to do it for you
- **Run Queries**: Query 
    - *Note*: You don't need to explicitly vectorize your queries, Weaviate will communicate with OpenAI to do it for you
    - *Note*: The `qna-openai` module automatically communicates with the OpenAI completions endpoint

Once you've run through this notebook you should have a basic understanding of how to setup and use vector databases for question answering.

## OpenAI Module in Weaviate
All Weaviate instances come equipped with the [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) and the [qna-openai](https://weaviate.io/developers/weaviate/modules/reader-generator-modules/qna-openai) modules.

The first module is responsible for handling vectorization at import (or any CRUD operations) and when you run a search query. The second module communicates with the OpenAI completions endpoint.

### No need to manually vectorize data
This is great news for you. With [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) you don't need to manually vectorize your data, as Weaviate will call OpenAI for you whenever necessary.

All you need to do is:
1. provide your OpenAI API Key – when you connected to the Weaviate Client
2. define which OpenAI vectorizer to use in your Schema

## Prerequisites

Before we start this project, we need setup the following:

* create a `Weaviate` instance
* install libraries
    * `weaviate-client`
    * `datasets`
    * `apache-beam`
* get your [OpenAI API key](https://beta.openai.com/account/api-keys)

===========================================================
### Create a Weaviate instance

To create a Weaviate instance we have 2 options:

1. (Recommended path) [Weaviate Cloud Service](https://console.weaviate.io/) – to host your Weaviate instance in the cloud. The free sandbox should be more than enough for this cookbook.
2. Install and run Weaviate locally with Docker.

#### Option 1 – WCS Installation Steps

Use [Weaviate Cloud Service](https://console.weaviate.io/) (WCS) to create a free Weaviate cluster.


key: _node_content
string_value:
{"id_": "3c51a78f-2100-4a2b-a5ed-d67712f7cab6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cfb7dcd7-df38-4283-a059-c75f4ee317f0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f75bb8b2745cc6f988eb21bf4d579a3ce42fa6438772074dabd728616f89367e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0e66670d-3197-4109-bbc1-776d323b6d8a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7b8d0aa86530930a74d21dbb156291800980ff91dedb6d5643c5b99d24f081a2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7ec75470-b9e5-4cb8-ba56-73e40b792f3a", "node_type": "1", "metadata": {}, "hash": "edb5b569c70b90f9bd00e5eff3a7319fe0e77a79206bc15a24c687e3432c8db8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3734, "end_char_idx": 6810, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The first module is responsible for handling vectorization at import (or any CRUD operations) and when you run a search query. The second module communicates with the OpenAI completions endpoint.

### No need to manually vectorize data
This is great news for you. With [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) you don't need to manually vectorize your data, as Weaviate will call OpenAI for you whenever necessary.

All you need to do is:
1. provide your OpenAI API Key – when you connected to the Weaviate Client
2. define which OpenAI vectorizer to use in your Schema

## Prerequisites

Before we start this project, we need setup the following:

* create a `Weaviate` instance
* install libraries
    * `weaviate-client`
    * `datasets`
    * `apache-beam`
* get your [OpenAI API key](https://beta.openai.com/account/api-keys)

===========================================================
### Create a Weaviate instance

To create a Weaviate instance we have 2 options:

1. (Recommended path) [Weaviate Cloud Service](https://console.weaviate.io/) – to host your Weaviate instance in the cloud. The free sandbox should be more than enough for this cookbook.
2. Install and run Weaviate locally with Docker.

#### Option 1 – WCS Installation Steps

Use [Weaviate Cloud Service](https://console.weaviate.io/) (WCS) to create a free Weaviate cluster.
1. create a free account and/or login to [WCS](https://console.weaviate.io/)
2. create a `Weaviate Cluster` with the following settings:
    * Sandbox: `Sandbox Free`
    * Weaviate Version: Use default (latest)
    * OIDC Authentication: `Disabled`
3. your instance should be ready in a minute or two
4. make a note of the `Cluster Id`. The link will take you to the full path of your cluster (you will need it later to connect to it). It should be something like: `https://your-project-name.weaviate.network` 

#### Option 2 – local Weaviate instance with Docker

Install and run Weaviate locally with Docker.
1. Download the [./docker-compose.yml](./docker-compose.yml) file
2. Then open your terminal, navigate to where your docker-compose.yml file is located, and start docker with: `docker-compose up -d`
3. Once this is ready, your instance should be available at [http://localhost:8080](http://localhost:8080)

Note. To shut down your docker instance you can call: `docker-compose down`

##### Learn more
To learn more, about using Weaviate with Docker see the [installation documentation](https://weaviate.io/developers/weaviate/installation/docker-compose).

===========================================================    
## Install required libraries

Before running this project make sure to have the following libraries:

### Weaviate Python client

The [Weaviate Python client](https://weaviate.io/developers/weaviate/client-libraries/python) allows you to communicate with your Weaviate instance from your Python project.

### datasets & apache-beam

To load sample data, you need the `datasets` library and its' dependency `apache-beam`.


key: _node_content
string_value:
{"id_": "7ec75470-b9e5-4cb8-ba56-73e40b792f3a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cfb7dcd7-df38-4283-a059-c75f4ee317f0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f75bb8b2745cc6f988eb21bf4d579a3ce42fa6438772074dabd728616f89367e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3c51a78f-2100-4a2b-a5ed-d67712f7cab6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8d8a89cd988da38d6d673052c040ed7127d0a56c0bc628944a5578e05fb7cc4d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cb7df488-fb8e-40a2-969f-d38488f2db96", "node_type": "1", "metadata": {}, "hash": "c6071153842b32a8ff6bf387dde86cca1ac06f861683f8a8048c8964b5abf50f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5156, "end_char_idx": 7490, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
1. create a free account and/or login to [WCS](https://console.weaviate.io/)
2. create a `Weaviate Cluster` with the following settings:
    * Sandbox: `Sandbox Free`
    * Weaviate Version: Use default (latest)
    * OIDC Authentication: `Disabled`
3. your instance should be ready in a minute or two
4. make a note of the `Cluster Id`. The link will take you to the full path of your cluster (you will need it later to connect to it). It should be something like: `https://your-project-name.weaviate.network` 

#### Option 2 – local Weaviate instance with Docker

Install and run Weaviate locally with Docker.
1. Download the [./docker-compose.yml](./docker-compose.yml) file
2. Then open your terminal, navigate to where your docker-compose.yml file is located, and start docker with: `docker-compose up -d`
3. Once this is ready, your instance should be available at [http://localhost:8080](http://localhost:8080)

Note. To shut down your docker instance you can call: `docker-compose down`

##### Learn more
To learn more, about using Weaviate with Docker see the [installation documentation](https://weaviate.io/developers/weaviate/installation/docker-compose).

===========================================================    
## Install required libraries

Before running this project make sure to have the following libraries:

### Weaviate Python client

The [Weaviate Python client](https://weaviate.io/developers/weaviate/client-libraries/python) allows you to communicate with your Weaviate instance from your Python project.

### datasets & apache-beam

To load sample data, you need the `datasets` library and its' dependency `apache-beam`.


```python
# Install the Weaviate client for Python
!pip install weaviate-client>3.11.0

# Install datasets and apache-beam to load the sample datasets
!pip install datasets apache-beam
```

===========================================================
## Prepare your OpenAI API key

The `OpenAI API key` is used for vectorization of your data at import, and for queries.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it to your environment variables as `OPENAI_API_KEY`.


```python
# Export OpenAI API Key
!export OPENAI_API_KEY="your key"
```


key: _node_content
string_value:
{"id_": "cb7df488-fb8e-40a2-969f-d38488f2db96", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cfb7dcd7-df38-4283-a059-c75f4ee317f0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f75bb8b2745cc6f988eb21bf4d579a3ce42fa6438772074dabd728616f89367e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7ec75470-b9e5-4cb8-ba56-73e40b792f3a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "de99fa83ada4c36caf4fd07be4a82d9970cac51ee7d9f76061934c4542ed28a7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cacad6a0-7e7f-4e49-be73-b80a60247fd9", "node_type": "1", "metadata": {}, "hash": "88f96e7784f267ea98a6b6819a100ebb82099924dc4490279e1f8d07c048f3f6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5768, "end_char_idx": 8400, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
1. Download the [./docker-compose.yml](./docker-compose.yml) file
2. Then open your terminal, navigate to where your docker-compose.yml file is located, and start docker with: `docker-compose up -d`
3. Once this is ready, your instance should be available at [http://localhost:8080](http://localhost:8080)

Note. To shut down your docker instance you can call: `docker-compose down`

##### Learn more
To learn more, about using Weaviate with Docker see the [installation documentation](https://weaviate.io/developers/weaviate/installation/docker-compose).

===========================================================    
## Install required libraries

Before running this project make sure to have the following libraries:

### Weaviate Python client

The [Weaviate Python client](https://weaviate.io/developers/weaviate/client-libraries/python) allows you to communicate with your Weaviate instance from your Python project.

### datasets & apache-beam

To load sample data, you need the `datasets` library and its' dependency `apache-beam`.


```python
# Install the Weaviate client for Python
!pip install weaviate-client>3.11.0

# Install datasets and apache-beam to load the sample datasets
!pip install datasets apache-beam
```

===========================================================
## Prepare your OpenAI API key

The `OpenAI API key` is used for vectorization of your data at import, and for queries.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it to your environment variables as `OPENAI_API_KEY`.


```python
# Export OpenAI API Key
!export OPENAI_API_KEY="your key"
```


```python
# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.
import os

# Note. alternatively you can set a temporary env variable like this:
# os.environ['OPENAI_API_KEY'] = 'your-key-goes-here'

if os.getenv("OPENAI_API_KEY") is not None:
    print ("OPENAI_API_KEY is ready")
else:
    print ("OPENAI_API_KEY environment variable not found")
```

## Connect to your Weaviate instance

In this section, we will:

1. test env variable `OPENAI_API_KEY` – **make sure** you completed the step in [#Prepare-your-OpenAI-API-key](#Prepare-your-OpenAI-API-key)
2. connect to your Weaviate your `OpenAI API Key`
3. and test the client connection

### The client 

After this step, the `client` object will be used to perform all Weaviate-related operations.


key: _node_content
string_value:
{"id_": "cacad6a0-7e7f-4e49-be73-b80a60247fd9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cfb7dcd7-df38-4283-a059-c75f4ee317f0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f75bb8b2745cc6f988eb21bf4d579a3ce42fa6438772074dabd728616f89367e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cb7df488-fb8e-40a2-969f-d38488f2db96", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ff019d7bd0c721010e89a2767ef3b9aebbc9b54f1d0d6358d7d782b5d26feb55", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "84ef8ea9-e73b-4608-a533-f94208586960", "node_type": "1", "metadata": {}, "hash": "4664c256fb8dd63a9972c509f70d3cd812cdfe63596fe35694a1b91fc0f9707c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6813, "end_char_idx": 9333, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Install the Weaviate client for Python
!pip install weaviate-client>3.11.0

# Install datasets and apache-beam to load the sample datasets
!pip install datasets apache-beam
```

===========================================================
## Prepare your OpenAI API key

The `OpenAI API key` is used for vectorization of your data at import, and for queries.

If you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).

Once you get your key, please add it to your environment variables as `OPENAI_API_KEY`.


```python
# Export OpenAI API Key
!export OPENAI_API_KEY="your key"
```


```python
# Test that your OpenAI API key is correctly set as an environment variable
# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.
import os

# Note. alternatively you can set a temporary env variable like this:
# os.environ['OPENAI_API_KEY'] = 'your-key-goes-here'

if os.getenv("OPENAI_API_KEY") is not None:
    print ("OPENAI_API_KEY is ready")
else:
    print ("OPENAI_API_KEY environment variable not found")
```

## Connect to your Weaviate instance

In this section, we will:

1. test env variable `OPENAI_API_KEY` – **make sure** you completed the step in [#Prepare-your-OpenAI-API-key](#Prepare-your-OpenAI-API-key)
2. connect to your Weaviate your `OpenAI API Key`
3. and test the client connection

### The client 

After this step, the `client` object will be used to perform all Weaviate-related operations.


```python
import weaviate
from datasets import load_dataset
import os

# Connect to your Weaviate instance
client = weaviate.Client(
    url="https://your-wcs-instance-name.weaviate.network/",
#   url="http://localhost:8080/",
    auth_client_secret=weaviate.auth.AuthApiKey(api_key="<YOUR-WEAVIATE-API-KEY>"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)
    additional_headers={
        "X-OpenAI-Api-Key": os.getenv("OPENAI_API_KEY")
    }
)

# Check if your instance is live and ready
# This should return `True`
client.is_ready()
```

# Schema

In this section, we will:
1. configure the data schema for your data
2. select OpenAI module

> This is the second and final step, which requires OpenAI specific configuration.
> After this step, the rest of instructions wlll only touch on Weaviate, as the OpenAI tasks will be handled automatically.


key: _node_content
string_value:
{"id_": "84ef8ea9-e73b-4608-a533-f94208586960", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cfb7dcd7-df38-4283-a059-c75f4ee317f0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f75bb8b2745cc6f988eb21bf4d579a3ce42fa6438772074dabd728616f89367e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cacad6a0-7e7f-4e49-be73-b80a60247fd9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "541946926ef127cafd10a467bab8287c631b7b11633c7b2b0c58b284e57a84eb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "510ef9cb-680d-487e-b6e4-363fbd37134a", "node_type": "1", "metadata": {}, "hash": "83c5fd2f64583cc6965b52341845d5b99c7756e036529a54b66f8efb651a89b7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8403, "end_char_idx": 9933, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import weaviate
from datasets import load_dataset
import os

# Connect to your Weaviate instance
client = weaviate.Client(
    url="https://your-wcs-instance-name.weaviate.network/",
#   url="http://localhost:8080/",
    auth_client_secret=weaviate.auth.AuthApiKey(api_key="<YOUR-WEAVIATE-API-KEY>"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)
    additional_headers={
        "X-OpenAI-Api-Key": os.getenv("OPENAI_API_KEY")
    }
)

# Check if your instance is live and ready
# This should return `True`
client.is_ready()
```

# Schema

In this section, we will:
1. configure the data schema for your data
2. select OpenAI module

> This is the second and final step, which requires OpenAI specific configuration.
> After this step, the rest of instructions wlll only touch on Weaviate, as the OpenAI tasks will be handled automatically.


## What is a schema

In Weaviate you create __schemas__ to capture each of the entities you will be searching.

A schema is how you tell Weaviate:
* what embedding model should be used to vectorize the data
* what your data is made of (property names and types)
* which properties should be vectorized and indexed

In this cookbook we will use a dataset for `Articles`, which contains:
* `title`
* `content`
* `url`

We want to vectorize `title` and `content`, but not the `url`.

To vectorize and query the data, we will use `text-embedding-3-small`. For Q&A we will use `gpt-3.5-turbo-instruct`.


key: _node_content
string_value:
{"id_": "510ef9cb-680d-487e-b6e4-363fbd37134a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cfb7dcd7-df38-4283-a059-c75f4ee317f0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f75bb8b2745cc6f988eb21bf4d579a3ce42fa6438772074dabd728616f89367e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "84ef8ea9-e73b-4608-a533-f94208586960", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3dbb2a0cd7c8a9fc40d183601255b5dedd87c22f96d8eee88e891d2b6b2474d1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4d1fa503-9f83-4beb-b681-33270ae32412", "node_type": "1", "metadata": {}, "hash": "65c244219ebf03383e791121f75bcbdf3b72a8343eea35f0ee347031a44f596e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8403, "end_char_idx": 11642, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import weaviate
from datasets import load_dataset
import os

# Connect to your Weaviate instance
client = weaviate.Client(
    url="https://your-wcs-instance-name.weaviate.network/",
#   url="http://localhost:8080/",
    auth_client_secret=weaviate.auth.AuthApiKey(api_key="<YOUR-WEAVIATE-API-KEY>"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)
    additional_headers={
        "X-OpenAI-Api-Key": os.getenv("OPENAI_API_KEY")
    }
)

# Check if your instance is live and ready
# This should return `True`
client.is_ready()
```

# Schema

In this section, we will:
1. configure the data schema for your data
2. select OpenAI module

> This is the second and final step, which requires OpenAI specific configuration.
> After this step, the rest of instructions wlll only touch on Weaviate, as the OpenAI tasks will be handled automatically.


## What is a schema

In Weaviate you create __schemas__ to capture each of the entities you will be searching.

A schema is how you tell Weaviate:
* what embedding model should be used to vectorize the data
* what your data is made of (property names and types)
* which properties should be vectorized and indexed

In this cookbook we will use a dataset for `Articles`, which contains:
* `title`
* `content`
* `url`

We want to vectorize `title` and `content`, but not the `url`.

To vectorize and query the data, we will use `text-embedding-3-small`. For Q&A we will use `gpt-3.5-turbo-instruct`.


```python
# Clear up the schema, so that we can recreate it
client.schema.delete_all()
client.schema.get()

# Define the Schema object to use `text-embedding-3-small` on `title` and `content`, but skip it for `url`
article_schema = {
    "class": "Article",
    "description": "A collection of articles",
    "vectorizer": "text2vec-openai",
    "moduleConfig": {
        "text2vec-openai": {
          "model": "ada",
          "modelVersion": "002",
          "type": "text"
        }, 
        "qna-openai": {
          "model": "gpt-3.5-turbo-instruct",
          "maxTokens": 16,
          "temperature": 0.0,
          "topP": 1,
          "frequencyPenalty": 0.0,
          "presencePenalty": 0.0
        }
    },
    "properties": [{
        "name": "title",
        "description": "Title of the article",
        "dataType": ["string"]
    },
    {
        "name": "content",
        "description": "Contents of the article",
        "dataType": ["text"]
    },
    {
        "name": "url",
        "description": "URL to the article",
        "dataType": ["string"],
        "moduleConfig": { "text2vec-openai": { "skip": True } }
    }]
}

# add the Article schema
client.schema.create_class(article_schema)

# get the schema to make sure it worked
client.schema.get()
```

## Import data

In this section we will:
1. load the Simple Wikipedia dataset
2. configure Weaviate Batch import (to make the import more efficient)
3. import the data into Weaviate

> Note: <br/>
> Like mentioned before. We don't need to manually vectorize the data.<br/>
> The [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) module will take care of that.


key: _node_content
string_value:
{"id_": "4d1fa503-9f83-4beb-b681-33270ae32412", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cfb7dcd7-df38-4283-a059-c75f4ee317f0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f75bb8b2745cc6f988eb21bf4d579a3ce42fa6438772074dabd728616f89367e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "510ef9cb-680d-487e-b6e4-363fbd37134a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e501eaad0e57d95c09bcf36b973e50108a3887c009f399a103527d39cf944155", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "42aa1c4c-1501-4c1a-82e4-dbbc76f4b3bd", "node_type": "1", "metadata": {}, "hash": "49887c2bf6e2a01509dc4d4ef70138967631275af3f7740204460169d0d90914", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11645, "end_char_idx": 14637, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
### STEP 1 - load the dataset

from datasets import load_dataset
from typing import List, Iterator

# We'll use the datasets library to pull the Simple Wikipedia dataset for embedding
dataset = list(load_dataset("wikipedia", "20220301.simple")["train"])

# For testing, limited to 2.5k articles for demo purposes
dataset = dataset[:2_500]

# Limited to 25k articles for larger demo purposes
# dataset = dataset[:25_000]

# for free OpenAI acounts, you can use 50 objects
# dataset = dataset[:50]
```


```python
### Step 2 - configure Weaviate Batch, with
# - starting batch size of 100
# - dynamically increase/decrease based on performance
# - add timeout retries if something goes wrong

client.batch.configure(
    batch_size=10, 
    dynamic=True,
    timeout_retries=3,
#   callback=None,
)
```


```python
### Step 3 - import data

print("Importing Articles")

counter=0

with client.batch as batch:
    for article in dataset:
        if (counter %10 == 0):
            print(f"Import {counter} / {len(dataset)} ")

        properties = {
            "title": article["title"],
            "content": article["text"],
            "url": article["url"]
        }
        
        batch.add_data_object(properties, "Article")
        counter = counter+1

print("Importing Articles complete")
```


```python
# Test that all data has loaded – get object count
result = (
    client.query.aggregate("Article")
    .with_fields("meta { count }")
    .do()
)
print("Object count: ", result["data"]["Aggregate"]["Article"], "\n")
```


```python
# Test one article has worked by checking one object
test_article = (
    client.query
    .get("Article", ["title", "url", "content"])
    .with_limit(1)
    .do()
)["data"]["Get"]["Article"][0]

print(test_article['title'])
print(test_article['url'])
print(test_article['content'])
```

### Question Answering on the Data

As above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors


```python
def qna(query, collection_name):
    
    properties = [
        "title", "content", "url",
        "_additional { answer { hasAnswer property result startPosition endPosition } distance }"
    ]

    ask = {
        "question": query,
        "properties": ["content"]
    }

    result = (
        client.query
        .get(collection_name, properties)
        .with_ask(ask)
        .with_limit(1)
        .do()
    )
    
    # Check for errors
    if ("errors" in result):
        print ("\033[91mYou probably have run out of OpenAI API calls for the current minute – the limit is set at 60 per minute.")
        raise Exception(result["errors"][0]['message'])
    
    return result["data"]["Get"][collection_name]
```


```python
query_result = qna("Did Alanis Morissette win a Grammy?", "Article")

for i, article in enumerate(query_result):
    print(f"{i+1}. { article['_additional']['answer']['result']} (Distance: {round(article['_additional']['distance'],3) })")
```


key: _node_content
string_value:
{"id_": "42aa1c4c-1501-4c1a-82e4-dbbc76f4b3bd", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "cfb7dcd7-df38-4283-a059-c75f4ee317f0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f75bb8b2745cc6f988eb21bf4d579a3ce42fa6438772074dabd728616f89367e", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4d1fa503-9f83-4beb-b681-33270ae32412", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "747e6c8d7475f6adea3aef22f02cdf40542949d2321471121f8ea844f4bc8cc7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13192, "end_char_idx": 15233, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


