# embeddings_queue

created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: a9c16037-652d-4ba4-a013-d38a24160900
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"a9c16037-652d-4ba4-a013-d38a24160900\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f361c56b-00f8-41f8-a1a8-0ab271e7a2a5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7dd27188881361e561cb26f8e74ef0000c681696c90d143322e9840b83deaaed\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0fbfa488-ee1d-4766-9764-8487ccbb8e2f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"903b6146c166bb9f77d80c0aa3925b21d9ea27d7e7f3d1d8884636fe3e606352\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"22e3f572-9b60-48db-842a-31ed2eca41c4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"522f0f87a1b2028b2c7c178f4b0e20f9999c6b94114c7d38e9052b39684d96ad\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5755, \"end_char_idx\": 8209, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "doc_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "ref_doc_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "chroma:document": "9/site-packages (from huggingface-hub<1.0.0,>=0.2.0->datasets) (4.5.0)\n    Requirement already satisfied: filelock in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from huggingface-hub<1.0.0,>=0.2.0->datasets) (3.9.0)\n    Requirement already satisfied: python-dateutil>=2.8.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pandas>=1.2.4->pymilvus) (2.8.2)\n    Requirement already satisfied: pytz>=2020.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from pandas>=1.2.4->pymilvus) (2022.7.1)\n    Requirement already satisfied: urllib3<1.27,>=1.21.1 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from requests>=2.20->openai) (1.26.14)\n    Requirement already satisfied: idna<4,>=2.5 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from requests>=2.20->openai) (3.4)\n    Requirement already satisfied: certifi>=2017.4.17 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from requests>=2.20->openai) (2022.12.7)\n\n\nTo get Zilliz up and running take a look [here](https://zilliz.com/doc/quick_start). With your account and database set up, proceed to set the following values:\n- URI: The URI your database is running on\n- USER: Your database username\n- PASSWORD: Your database password\n- COLLECTION_NAME: What to name the collection within Zilliz\n- DIMENSION: The dimension of the embeddings\n- OPENAI_ENGINE: Which embedding model to use\n- openai.api_key: Your OpenAI account key\n- INDEX_PARAM: The index settings to use for the collection\n- QUERY_PARAM: The search parameters to use\n- BATCH_SIZE: How many texts to embed and insert at once\n\n\n```python\nimport openai\n\nURI = 'your_uri'\nTOKEN = 'your_token' # TOKEN == user:password or api_key\nCOLLECTION_NAME = 'book_search'\nDIMENSION = 1536\nOPENAI_ENGINE = 'text-embedding-3-small'\nopenai.api_key = 'sk-your-key'\n\nINDEX_PARAM = {\n    'metric_type':'L2',\n    'index_type':\"AUTOINDEX\",\n    'params':{}\n}\n\nQUERY_PARAM = {\n    \"metric_type\": \"L2\",\n    \"params\": {},\n}\n\nBATCH_SIZE = 1000\n```\n\n## Zilliz\nThis segment deals with Zilliz and setting up the database for this use case. Within Zilliz we need to setup a collection and index it.\n\n\n```python\nfrom pymilvus import connections, utility, FieldSchema, Collection, CollectionSchema, DataType\n\n# Connect to Zilliz Database\nconnections.connect(uri=URI, token=TOKEN)\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 22e3f572-9b60-48db-842a-31ed2eca41c4
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"22e3f572-9b60-48db-842a-31ed2eca41c4\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f361c56b-00f8-41f8-a1a8-0ab271e7a2a5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7dd27188881361e561cb26f8e74ef0000c681696c90d143322e9840b83deaaed\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a9c16037-652d-4ba4-a013-d38a24160900\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"096755da2f4eb31f774f308d63e0ce00911c0d47d71e7b49f58b6543898c84ed\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"747c347d-0a9a-4ea3-8691-cbf50b6fecbb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f22d726b9ef6cee8ebf19bf63ec633a4cf9dac904383b68174236ca0c3e623c4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6732, \"end_char_idx\": 9384, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "doc_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "ref_doc_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "chroma:document": "4.17 in /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages (from requests>=2.20->openai) (2022.12.7)\n\n\nTo get Zilliz up and running take a look [here](https://zilliz.com/doc/quick_start). With your account and database set up, proceed to set the following values:\n- URI: The URI your database is running on\n- USER: Your database username\n- PASSWORD: Your database password\n- COLLECTION_NAME: What to name the collection within Zilliz\n- DIMENSION: The dimension of the embeddings\n- OPENAI_ENGINE: Which embedding model to use\n- openai.api_key: Your OpenAI account key\n- INDEX_PARAM: The index settings to use for the collection\n- QUERY_PARAM: The search parameters to use\n- BATCH_SIZE: How many texts to embed and insert at once\n\n\n```python\nimport openai\n\nURI = 'your_uri'\nTOKEN = 'your_token' # TOKEN == user:password or api_key\nCOLLECTION_NAME = 'book_search'\nDIMENSION = 1536\nOPENAI_ENGINE = 'text-embedding-3-small'\nopenai.api_key = 'sk-your-key'\n\nINDEX_PARAM = {\n    'metric_type':'L2',\n    'index_type':\"AUTOINDEX\",\n    'params':{}\n}\n\nQUERY_PARAM = {\n    \"metric_type\": \"L2\",\n    \"params\": {},\n}\n\nBATCH_SIZE = 1000\n```\n\n## Zilliz\nThis segment deals with Zilliz and setting up the database for this use case. Within Zilliz we need to setup a collection and index it.\n\n\n```python\nfrom pymilvus import connections, utility, FieldSchema, Collection, CollectionSchema, DataType\n\n# Connect to Zilliz Database\nconnections.connect(uri=URI, token=TOKEN)\n```\n\n\n```python\n# Remove collection if it already exists\nif utility.has_collection(COLLECTION_NAME):\n    utility.drop_collection(COLLECTION_NAME)\n```\n\n\n```python\n# Create collection which includes the id, title, and embedding.\nfields = [\n    FieldSchema(name='id', dtype=DataType.INT64, is_primary=True, auto_id=True),\n    FieldSchema(name='title', dtype=DataType.VARCHAR, max_length=64000),\n    FieldSchema(name='description', dtype=DataType.VARCHAR, max_length=64000),\n    FieldSchema(name='embedding', dtype=DataType.FLOAT_VECTOR, dim=DIMENSION)\n]\nschema = CollectionSchema(fields=fields)\ncollection = Collection(name=COLLECTION_NAME, schema=schema)\n```\n\n\n```python\n# Create the index on the collection and load it.\ncollection.create_index(field_name=\"embedding\", index_params=INDEX_PARAM)\ncollection.load()\n```\n\n## Dataset\nWith Zilliz up and running we can begin grabbing our data. `Hugging Face Datasets` is a hub that holds many different user datasets, and for this example we are using Skelebor's book dataset. This dataset contains title-description pairs for over 1 million books. We are going to embed each description and store it within Zilliz along with its title."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 747c347d-0a9a-4ea3-8691-cbf50b6fecbb
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"747c347d-0a9a-4ea3-8691-cbf50b6fecbb\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f361c56b-00f8-41f8-a1a8-0ab271e7a2a5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7dd27188881361e561cb26f8e74ef0000c681696c90d143322e9840b83deaaed\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"22e3f572-9b60-48db-842a-31ed2eca41c4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"006750b59bd64b8c624c757aaf09c96211168326d700bed979d4e1a627363b16\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e5b48317-f668-415d-acc8-0c408b14703c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7e32e021cb43564c9d7db39cffb1dc1bdce380941588321cef36d0d6a183688e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8029, \"end_char_idx\": 10865, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "doc_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "ref_doc_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "chroma:document": "```python\nfrom pymilvus import connections, utility, FieldSchema, Collection, CollectionSchema, DataType\n\n# Connect to Zilliz Database\nconnections.connect(uri=URI, token=TOKEN)\n```\n\n\n```python\n# Remove collection if it already exists\nif utility.has_collection(COLLECTION_NAME):\n    utility.drop_collection(COLLECTION_NAME)\n```\n\n\n```python\n# Create collection which includes the id, title, and embedding.\nfields = [\n    FieldSchema(name='id', dtype=DataType.INT64, is_primary=True, auto_id=True),\n    FieldSchema(name='title', dtype=DataType.VARCHAR, max_length=64000),\n    FieldSchema(name='description', dtype=DataType.VARCHAR, max_length=64000),\n    FieldSchema(name='embedding', dtype=DataType.FLOAT_VECTOR, dim=DIMENSION)\n]\nschema = CollectionSchema(fields=fields)\ncollection = Collection(name=COLLECTION_NAME, schema=schema)\n```\n\n\n```python\n# Create the index on the collection and load it.\ncollection.create_index(field_name=\"embedding\", index_params=INDEX_PARAM)\ncollection.load()\n```\n\n## Dataset\nWith Zilliz up and running we can begin grabbing our data. `Hugging Face Datasets` is a hub that holds many different user datasets, and for this example we are using Skelebor's book dataset. This dataset contains title-description pairs for over 1 million books. We are going to embed each description and store it within Zilliz along with its title.\n\n\n```python\nimport datasets\n\n# Download the dataset and only use the `train` portion (file is around 800Mb)\ndataset = datasets.load_dataset('Skelebor/book_titles_and_descriptions_en_clean', split='train')\n```\n\n    /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n      from .autonotebook import tqdm as notebook_tqdm\n    Found cached dataset parquet (/Users/filiphaltmayer/.cache/huggingface/datasets/Skelebor___parquet/Skelebor--book_titles_and_descriptions_en_clean-3596935b1d8a7747/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)\n\n\n## Insert the Data\nNow that we have our data on our machine we can begin embedding it and inserting it into Zilliz. The embedding function takes in text and returns the embeddings in a list format.\n\n\n```python\n# Simple function that converts the texts to embeddings\ndef embed(texts):\n    embeddings = openai.Embedding.create(\n        input=texts,\n        engine=OPENAI_ENGINE\n    )\n    return [x['embedding'] for x in embeddings['data']]\n\n```\n\nThis next step does the actual inserting. Due to having so many datapoints, if you want to immediately test it out you can stop the inserting cell block early and move along. Doing this will probably decrease the accuracy of the results due to less datapoints, but it should still be good enough."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: e5b48317-f668-415d-acc8-0c408b14703c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"e5b48317-f668-415d-acc8-0c408b14703c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f361c56b-00f8-41f8-a1a8-0ab271e7a2a5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7dd27188881361e561cb26f8e74ef0000c681696c90d143322e9840b83deaaed\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"747c347d-0a9a-4ea3-8691-cbf50b6fecbb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"d000af0b6c61780f315a55c67b58a11c4d2552648edfd7f2d004d6166eddf6d0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"da530151-a848-4b47-81d9-b7a3bef7b233\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"742f91e58dcf19a4db25e3f50037b18c0f8b68482423e20deeaff18718ff21a5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9387, \"end_char_idx\": 11852, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "doc_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "ref_doc_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "chroma:document": "```python\nimport datasets\n\n# Download the dataset and only use the `train` portion (file is around 800Mb)\ndataset = datasets.load_dataset('Skelebor/book_titles_and_descriptions_en_clean', split='train')\n```\n\n    /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n      from .autonotebook import tqdm as notebook_tqdm\n    Found cached dataset parquet (/Users/filiphaltmayer/.cache/huggingface/datasets/Skelebor___parquet/Skelebor--book_titles_and_descriptions_en_clean-3596935b1d8a7747/0.0.0/2a3b91fbd88a2c90d1dbbb32b460cf621d31bd5b05b934492fdef7d8d6f236ec)\n\n\n## Insert the Data\nNow that we have our data on our machine we can begin embedding it and inserting it into Zilliz. The embedding function takes in text and returns the embeddings in a list format.\n\n\n```python\n# Simple function that converts the texts to embeddings\ndef embed(texts):\n    embeddings = openai.Embedding.create(\n        input=texts,\n        engine=OPENAI_ENGINE\n    )\n    return [x['embedding'] for x in embeddings['data']]\n\n```\n\nThis next step does the actual inserting. Due to having so many datapoints, if you want to immediately test it out you can stop the inserting cell block early and move along. Doing this will probably decrease the accuracy of the results due to less datapoints, but it should still be good enough.\n\n\n```python\nfrom tqdm import tqdm\n\ndata = [\n    [], # title\n    [], # description\n]\n\n# Embed and insert in batches\nfor i in tqdm(range(0, len(dataset))):\n    data[0].append(dataset[i]['title'])\n    data[1].append(dataset[i]['description'])\n    if len(data[0]) % BATCH_SIZE == 0:\n        data.append(embed(data[1]))\n        collection.insert(data)\n        data = [[],[]]\n\n# Embed and insert the remainder \nif len(data[0]) != 0:\n    data.append(embed(data[1]))\n    collection.insert(data)\n    data = [[],[]]\n\n```\n\n      0%|          | 2999/1032335 [00:19<1:49:30, 156.66it/s]\n\n\n\n    ---------------------------------------------------------------------------\n\n    KeyboardInterrupt                         Traceback (most recent call last)\n\n    Cell In[10], line 14\n         12     if len(data[0]) % BATCH_SIZE == 0:\n         13         data.append(embed(data[1]))\n    ---> 14         collection.insert(data)\n         15         data = [[],[]]\n         17 # Embed and insert the remainder"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: da530151-a848-4b47-81d9-b7a3bef7b233
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"da530151-a848-4b47-81d9-b7a3bef7b233\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f361c56b-00f8-41f8-a1a8-0ab271e7a2a5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7dd27188881361e561cb26f8e74ef0000c681696c90d143322e9840b83deaaed\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e5b48317-f668-415d-acc8-0c408b14703c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"2e91b29d085eaef20ba4beab490878dc80d72571ad66a603b816df44312fbd0f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"df047a2b-1489-416b-b1d1-358d55ed42cf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4d2e5cfe3a0819a852554f9530ee7b0906b71dc9572645ca59703f21d5bc8f51\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10325, \"end_char_idx\": 13105, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "doc_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "ref_doc_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "chroma:document": "```python\n# Simple function that converts the texts to embeddings\ndef embed(texts):\n    embeddings = openai.Embedding.create(\n        input=texts,\n        engine=OPENAI_ENGINE\n    )\n    return [x['embedding'] for x in embeddings['data']]\n\n```\n\nThis next step does the actual inserting. Due to having so many datapoints, if you want to immediately test it out you can stop the inserting cell block early and move along. Doing this will probably decrease the accuracy of the results due to less datapoints, but it should still be good enough.\n\n\n```python\nfrom tqdm import tqdm\n\ndata = [\n    [], # title\n    [], # description\n]\n\n# Embed and insert in batches\nfor i in tqdm(range(0, len(dataset))):\n    data[0].append(dataset[i]['title'])\n    data[1].append(dataset[i]['description'])\n    if len(data[0]) % BATCH_SIZE == 0:\n        data.append(embed(data[1]))\n        collection.insert(data)\n        data = [[],[]]\n\n# Embed and insert the remainder \nif len(data[0]) != 0:\n    data.append(embed(data[1]))\n    collection.insert(data)\n    data = [[],[]]\n\n```\n\n      0%|          | 2999/1032335 [00:19<1:49:30, 156.66it/s]\n\n\n\n    ---------------------------------------------------------------------------\n\n    KeyboardInterrupt                         Traceback (most recent call last)\n\n    Cell In[10], line 14\n         12     if len(data[0]) % BATCH_SIZE == 0:\n         13         data.append(embed(data[1]))\n    ---> 14         collection.insert(data)\n         15         data = [[],[]]\n         17 # Embed and insert the remainder \n\n\n    File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/orm/collection.py:430, in Collection.insert(self, data, partition_name, timeout, **kwargs)\n        427 entities = Prepare.prepare_insert_data(data, self._schema)\n        429 conn = self._get_connection()\n    --> 430 res = conn.batch_insert(self._name, entities, partition_name,\n        431                         timeout=timeout, schema=self._schema_dict, **kwargs)\n        433 if kwargs.get(\"_async\", False):\n        434     return MutationFuture(res)\n\n\n    File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/decorators.py:105, in error_handler.<locals>.wrapper.<locals>.handler(*args, **kwargs)\n        103 try:\n        104     record_dict[\"RPC start\"] = str(datetime.datetime.now())\n    --> 105     return func(*args, **kwargs)\n        106 except MilvusException as e:\n        107     record_dict[\"RPC error\"] = str(datetime.datetime.now())\n\n\n    File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/decorators.py:136, in tracing_request.<locals>.wrapper.<locals>.handler(self, *args, **kwargs)\n        134 if req_id:\n        135     self.set_onetime_request_id(req_id)\n    --> 136 ret = func(self, *args, **kwargs)\n        137 return ret"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: df047a2b-1489-416b-b1d1-358d55ed42cf
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"df047a2b-1489-416b-b1d1-358d55ed42cf\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f361c56b-00f8-41f8-a1a8-0ab271e7a2a5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7dd27188881361e561cb26f8e74ef0000c681696c90d143322e9840b83deaaed\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"da530151-a848-4b47-81d9-b7a3bef7b233\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"533da2bcdd9c0823c7dee290674f01d942576583f03d5a24cf8bdefcbf901cbb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"43011f0c-4c2e-4c2e-84f6-0a7d1bbbd053\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6767872afd7037435e62778bcd115e3866e6674730df2c9595788366ee9913ca\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11860, \"end_char_idx\": 14841, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "doc_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "ref_doc_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "chroma:document": "File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/orm/collection.py:430, in Collection.insert(self, data, partition_name, timeout, **kwargs)\n        427 entities = Prepare.prepare_insert_data(data, self._schema)\n        429 conn = self._get_connection()\n    --> 430 res = conn.batch_insert(self._name, entities, partition_name,\n        431                         timeout=timeout, schema=self._schema_dict, **kwargs)\n        433 if kwargs.get(\"_async\", False):\n        434     return MutationFuture(res)\n\n\n    File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/decorators.py:105, in error_handler.<locals>.wrapper.<locals>.handler(*args, **kwargs)\n        103 try:\n        104     record_dict[\"RPC start\"] = str(datetime.datetime.now())\n    --> 105     return func(*args, **kwargs)\n        106 except MilvusException as e:\n        107     record_dict[\"RPC error\"] = str(datetime.datetime.now())\n\n\n    File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/decorators.py:136, in tracing_request.<locals>.wrapper.<locals>.handler(self, *args, **kwargs)\n        134 if req_id:\n        135     self.set_onetime_request_id(req_id)\n    --> 136 ret = func(self, *args, **kwargs)\n        137 return ret\n\n\n    File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/decorators.py:50, in retry_on_rpc_failure.<locals>.wrapper.<locals>.handler(self, *args, **kwargs)\n         48 while True:\n         49     try:\n    ---> 50         return func(self, *args, **kwargs)\n         51     except grpc.RpcError as e:\n         52         # DEADLINE_EXCEEDED means that the task wat not completed\n         53         # UNAVAILABLE means that the service is not reachable currently\n         54         # Reference: https://grpc.github.io/grpc/python/grpc.html#grpc-status-code\n         55         if e.code() != grpc.StatusCode.DEADLINE_EXCEEDED and e.code() != grpc.StatusCode.UNAVAILABLE:\n\n\n    File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/client/grpc_handler.py:378, in GrpcHandler.batch_insert(self, collection_name, entities, partition_name, timeout, **kwargs)\n        375     f.add_callback(ts_utils.update_ts_on_mutation(collection_name))\n        376     return f\n    --> 378 response = rf.result()\n        379 if response.status.error_code == 0:\n        380     m = MutationResult(response)\n\n\n    File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/grpc/_channel.py:733, in _MultiThreadedRendezvous.result(self, timeout)\n        728 \"\"\"Returns the result of the computation or raises its exception.\n        729 \n        730 See grpc.Future.result for the full API contract.\n        731 \"\"\"\n        732 with self._state.condition:\n    --> 733     timed_out = _common.wait(self._state.condition.wait,\n        734                              self._is_complete,\n        735                              timeout=timeout)\n        736     if timed_out:\n        737         raise grpc.FutureTimeoutError()"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 43011f0c-4c2e-4c2e-84f6-0a7d1bbbd053
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"43011f0c-4c2e-4c2e-84f6-0a7d1bbbd053\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f361c56b-00f8-41f8-a1a8-0ab271e7a2a5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7dd27188881361e561cb26f8e74ef0000c681696c90d143322e9840b83deaaed\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"df047a2b-1489-416b-b1d1-358d55ed42cf\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"abfa1a517ab485f3a29896f5c153d4923a5bdda8c645dac02b2624e4ef2d5325\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3c4aebc1-1bc0-400f-97b5-696bb334ccef\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7244597802fabfc2c67ceb47db7815fdd92be822f4dd1f6847aaafb29e408955\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13800, \"end_char_idx\": 16697, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "doc_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "ref_doc_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "chroma:document": "File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/pymilvus/client/grpc_handler.py:378, in GrpcHandler.batch_insert(self, collection_name, entities, partition_name, timeout, **kwargs)\n        375     f.add_callback(ts_utils.update_ts_on_mutation(collection_name))\n        376     return f\n    --> 378 response = rf.result()\n        379 if response.status.error_code == 0:\n        380     m = MutationResult(response)\n\n\n    File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/grpc/_channel.py:733, in _MultiThreadedRendezvous.result(self, timeout)\n        728 \"\"\"Returns the result of the computation or raises its exception.\n        729 \n        730 See grpc.Future.result for the full API contract.\n        731 \"\"\"\n        732 with self._state.condition:\n    --> 733     timed_out = _common.wait(self._state.condition.wait,\n        734                              self._is_complete,\n        735                              timeout=timeout)\n        736     if timed_out:\n        737         raise grpc.FutureTimeoutError()\n\n\n    File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/grpc/_common.py:141, in wait(wait_fn, wait_complete_fn, timeout, spin_cb)\n        139 if timeout is None:\n        140     while not wait_complete_fn():\n    --> 141         _wait_once(wait_fn, MAXIMUM_WAIT_TIMEOUT, spin_cb)\n        142 else:\n        143     end = time.time() + timeout\n\n\n    File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/grpc/_common.py:106, in _wait_once(wait_fn, timeout, spin_cb)\n        105 def _wait_once(wait_fn, timeout, spin_cb):\n    --> 106     wait_fn(timeout=timeout)\n        107     if spin_cb is not None:\n        108         spin_cb()\n\n\n    File ~/miniconda3/envs/haystack/lib/python3.9/threading.py:316, in Condition.wait(self, timeout)\n        314 else:\n        315     if timeout > 0:\n    --> 316         gotit = waiter.acquire(True, timeout)\n        317     else:\n        318         gotit = waiter.acquire(False)\n\n\n    KeyboardInterrupt: \n\n\n## Query the Database\nWith our data safely inserted in Zilliz, we can now perform a query. The query takes in a string or a list of strings and searches them. The results print out your provided description and the results that include the result score, the result title, and the result book description.\n\n\n```python\nimport textwrap\n\ndef query(queries, top_k = 5):\n    if type(queries) != list:\n        queries = [queries]\n    res = collection.search(embed(queries), anns_field='embedding', param=QUERY_PARAM, limit = top_k, output_fields=['title', 'description'])\n    for i, hit in enumerate(res):\n        print('Description:', queries[i])\n        print('Results:')\n        for ii, hits in enumerate(hit):\n            print('\\t' + 'Rank:', ii + 1, 'Score:', hits.score, 'Title:', hits.entity.get('title'))\n            print(textwrap.fill(hits.entity.get('description'), 88))\n            print()\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 3c4aebc1-1bc0-400f-97b5-696bb334ccef
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"3c4aebc1-1bc0-400f-97b5-696bb334ccef\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f361c56b-00f8-41f8-a1a8-0ab271e7a2a5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7dd27188881361e561cb26f8e74ef0000c681696c90d143322e9840b83deaaed\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"43011f0c-4c2e-4c2e-84f6-0a7d1bbbd053\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e171b01de74d77233555c1e042902010813d4415227c2c933f0175c4271487e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1f36a36a-fde8-4cbe-8c27-ef4733004302\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b34ff5d24f9ea8c12c0c6d6cca57060dee9c0292f3dc74eb1c42d8b41eba08a5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15199, \"end_char_idx\": 18111, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "doc_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "ref_doc_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "chroma:document": "File ~/miniconda3/envs/haystack/lib/python3.9/site-packages/grpc/_common.py:106, in _wait_once(wait_fn, timeout, spin_cb)\n        105 def _wait_once(wait_fn, timeout, spin_cb):\n    --> 106     wait_fn(timeout=timeout)\n        107     if spin_cb is not None:\n        108         spin_cb()\n\n\n    File ~/miniconda3/envs/haystack/lib/python3.9/threading.py:316, in Condition.wait(self, timeout)\n        314 else:\n        315     if timeout > 0:\n    --> 316         gotit = waiter.acquire(True, timeout)\n        317     else:\n        318         gotit = waiter.acquire(False)\n\n\n    KeyboardInterrupt: \n\n\n## Query the Database\nWith our data safely inserted in Zilliz, we can now perform a query. The query takes in a string or a list of strings and searches them. The results print out your provided description and the results that include the result score, the result title, and the result book description.\n\n\n```python\nimport textwrap\n\ndef query(queries, top_k = 5):\n    if type(queries) != list:\n        queries = [queries]\n    res = collection.search(embed(queries), anns_field='embedding', param=QUERY_PARAM, limit = top_k, output_fields=['title', 'description'])\n    for i, hit in enumerate(res):\n        print('Description:', queries[i])\n        print('Results:')\n        for ii, hits in enumerate(hit):\n            print('\\t' + 'Rank:', ii + 1, 'Score:', hits.score, 'Title:', hits.entity.get('title'))\n            print(textwrap.fill(hits.entity.get('description'), 88))\n            print()\n```\n\n\n```python\nquery('Book about a k-9 from europe')\n```\n\n    Description: Book about a k-9 from europe\n    Results:\n    \tRank: 1 Score: 0.3047754764556885 Title: Bark M For Murder\n    Who let the dogs out? Evildoers beware! Four of mystery fiction's top storytellers are\n    setting the hounds on your trail -- in an incomparable quartet of crime stories with a\n    canine edge. Man's (and woman's) best friends take the lead in this phenomenal\n    collection of tales tense and surprising, humorous and thrilling: New York\n    Timesbestselling author J.A. Jance's spellbinding saga of a scam-busting septuagenarian\n    and her two golden retrievers; Anthony Award winner Virginia Lanier's pureblood thriller\n    featuring bloodhounds and bloody murder; Chassie West's suspenseful stunner about a\n    life-saving German shepherd and a ghastly forgotten crime; rising star Lee Charles\n    Kelley's edge-of-your-seat yarn that pits an ex-cop/kennel owner and a yappy toy poodle\n    against a craven killer.\n    \n    \tRank: 2 Score: 0.3283390402793884 Title: Texas K-9 Unit Christmas: Holiday Hero\\Rescuing Christmas\n    CHRISTMAS COMES WRAPPED IN DANGER Holiday Hero by Shirlee McCoy Emma Fairchild never\n    expected to find trouble in sleepy Sagebrush, Texas. But when she's attacked and left\n    for dead in her own diner, her childhood friend turned K-9 cop Lucas Harwood offers a\n    chance at justice--and love."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 1f36a36a-fde8-4cbe-8c27-ef4733004302
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"1f36a36a-fde8-4cbe-8c27-ef4733004302\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f361c56b-00f8-41f8-a1a8-0ab271e7a2a5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7dd27188881361e561cb26f8e74ef0000c681696c90d143322e9840b83deaaed\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3c4aebc1-1bc0-400f-97b5-696bb334ccef\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"927eb72f82d892dec177809d3db1bc73df15496f82881649c6d2463b0cd0ab95\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5009d30c-ef65-4da7-9cd3-e6e0615fb77a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1032f31ac641bc77c7a1c47e864023e9ab6f983354b3e7c91a6c5029a5e32cab\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16700, \"end_char_idx\": 19777, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "doc_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "ref_doc_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "chroma:document": "```python\nquery('Book about a k-9 from europe')\n```\n\n    Description: Book about a k-9 from europe\n    Results:\n    \tRank: 1 Score: 0.3047754764556885 Title: Bark M For Murder\n    Who let the dogs out? Evildoers beware! Four of mystery fiction's top storytellers are\n    setting the hounds on your trail -- in an incomparable quartet of crime stories with a\n    canine edge. Man's (and woman's) best friends take the lead in this phenomenal\n    collection of tales tense and surprising, humorous and thrilling: New York\n    Timesbestselling author J.A. Jance's spellbinding saga of a scam-busting septuagenarian\n    and her two golden retrievers; Anthony Award winner Virginia Lanier's pureblood thriller\n    featuring bloodhounds and bloody murder; Chassie West's suspenseful stunner about a\n    life-saving German shepherd and a ghastly forgotten crime; rising star Lee Charles\n    Kelley's edge-of-your-seat yarn that pits an ex-cop/kennel owner and a yappy toy poodle\n    against a craven killer.\n    \n    \tRank: 2 Score: 0.3283390402793884 Title: Texas K-9 Unit Christmas: Holiday Hero\\Rescuing Christmas\n    CHRISTMAS COMES WRAPPED IN DANGER Holiday Hero by Shirlee McCoy Emma Fairchild never\n    expected to find trouble in sleepy Sagebrush, Texas. But when she's attacked and left\n    for dead in her own diner, her childhood friend turned K-9 cop Lucas Harwood offers a\n    chance at justice--and love. Rescuing Christmas by Terri Reed She escaped a kidnapper,\n    but now a killer has set his sights on K-9 dog trainer Lily Anderson. When fellow\n    officer Jarrod Evans appoints himself her bodyguard, Lily knows more than her life is at\n    risk--so is her heart. Texas K-9 Unit: These lawmen solve the toughest cases with the\n    help of their brave canine partners\n    \n    \tRank: 3 Score: 0.33899369835853577 Title: Dogs on Duty: Soldiers' Best Friends on the Battlefield and Beyond\n    When the news of the raid on Osama Bin Laden's compound broke, the SEAL team member that\n    stole the show was a highly trained canine companion. Throughout history, dogs have been\n    key contributors to military units. Dorothy Hinshaw Patent follows man's best friend\n    onto the battlefield, showing readers why dogs are uniquely qualified for the job at\n    hand, how they are trained, how they contribute to missions, and what happens when they\n    retire. With full-color photographs throughout and sidebars featuring heroic canines\n    throughout history, Dogs on Duty provides a fascinating look at these exceptional\n    soldiers and companions.\n    \n    \tRank: 4 Score: 0.34207457304000854 Title: Toute Allure: Falling in Love in Rural France\n    After saying goodbye to life as a successful fashion editor in London, Karen Wheeler is\n    now happy in her small village house in rural France. Her idyll is complete when she\n    meets the love of her life - he has shaggy hair, four paws and a wet nose!\n    \n    \tRank: 5 Score: 0.343595951795578 Title: Otherwise Alone (Evan Arden, #1)\n    Librarian's note: This is an alternate cover edition for ASIN: B00AP5NNWC."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 5009d30c-ef65-4da7-9cd3-e6e0615fb77a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb", "title": "Getting Started with Zilliz and OpenAI", "tokens": 5622, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"5009d30c-ef65-4da7-9cd3-e6e0615fb77a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f361c56b-00f8-41f8-a1a8-0ab271e7a2a5\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7dd27188881361e561cb26f8e74ef0000c681696c90d143322e9840b83deaaed\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1f36a36a-fde8-4cbe-8c27-ef4733004302\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Getting_started_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Getting Started with Zilliz and OpenAI\", \"tokens\": 5622, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"08eaf91ddace0d1165f730525a72730a11945518a5e465cb25333992cc6c0713\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18244, \"end_char_idx\": 20528, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "doc_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "ref_doc_id": "f361c56b-00f8-41f8-a1a8-0ab271e7a2a5", "chroma:document": "When fellow\n    officer Jarrod Evans appoints himself her bodyguard, Lily knows more than her life is at\n    risk--so is her heart. Texas K-9 Unit: These lawmen solve the toughest cases with the\n    help of their brave canine partners\n    \n    \tRank: 3 Score: 0.33899369835853577 Title: Dogs on Duty: Soldiers' Best Friends on the Battlefield and Beyond\n    When the news of the raid on Osama Bin Laden's compound broke, the SEAL team member that\n    stole the show was a highly trained canine companion. Throughout history, dogs have been\n    key contributors to military units. Dorothy Hinshaw Patent follows man's best friend\n    onto the battlefield, showing readers why dogs are uniquely qualified for the job at\n    hand, how they are trained, how they contribute to missions, and what happens when they\n    retire. With full-color photographs throughout and sidebars featuring heroic canines\n    throughout history, Dogs on Duty provides a fascinating look at these exceptional\n    soldiers and companions.\n    \n    \tRank: 4 Score: 0.34207457304000854 Title: Toute Allure: Falling in Love in Rural France\n    After saying goodbye to life as a successful fashion editor in London, Karen Wheeler is\n    now happy in her small village house in rural France. Her idyll is complete when she\n    meets the love of her life - he has shaggy hair, four paws and a wet nose!\n    \n    \tRank: 5 Score: 0.343595951795578 Title: Otherwise Alone (Evan Arden, #1)\n    Librarian's note: This is an alternate cover edition for ASIN: B00AP5NNWC. Lieutenant\n    Evan Arden sits in a shack in the middle of nowhere, waiting for orders that will send\n    him back home - if he ever gets them. Other than his loyal Great Pyrenees, there's no\n    one around to break up the monotony. The tedium is excruciating, but it is suddenly\n    interrupted when a young woman stumbles up his path. \"It's only 50-something pages, but\n    in that short amount of time, the author's awesome writing packs in a whole lotta\n    character detail. And sets the stage for the series, perfectly.\" -Maryse.net, 4.5 Stars\n    He has two choices - pick her off from a distance with his trusty sniper-rifle, or dare\n    let her approach his cabin and enter his life. Why not? It's been ages, and he is\n    otherwise alone..."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 2faf2b77-2165-4124-9464-931e132a4394
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb", "title": "Filtered Search with Zilliz and OpenAI", "tokens": 2129, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"2faf2b77-2165-4124-9464-931e132a4394\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Filtered Search with Zilliz and OpenAI\", \"tokens\": 2129, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"082b235a-e3c5-4993-9cc1-a0527bd3dfb3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Filtered Search with Zilliz and OpenAI\", \"tokens\": 2129, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"8243d90a985bfa201d016f4e4e8754a0224edf957ac88ac834132750b77d1de5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d6b731a5-02a4-4b76-8157-2f9c7cbedd5c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"66d256d33eb3c79f27435d94c053e76543a6145910d02f39de86f2b8140f98b1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2882, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "082b235a-e3c5-4993-9cc1-a0527bd3dfb3", "doc_id": "082b235a-e3c5-4993-9cc1-a0527bd3dfb3", "ref_doc_id": "082b235a-e3c5-4993-9cc1-a0527bd3dfb3", "chroma:document": "# Filtered Search with Zilliz and OpenAI\n### Finding your next movie\n\nIn this notebook we will be going over generating embeddings of movie descriptions with OpenAI and using those embeddings within Zilliz to find relevant movies. To narrow our search results and try something new, we are going to be using filtering to do metadata searches. The dataset in this example is sourced from HuggingFace datasets, and contains a little over 8 thousand movie entries.\n\nLets begin by first downloading the required libraries for this notebook:\n- `openai` is used for communicating with the OpenAI embedding service\n- `pymilvus` is used for communicating with the Zilliz server\n- `datasets` is used for downloading the dataset\n- `tqdm` is used for the progress bars\n\n\n\n```python\n! pip install openai pymilvus datasets tqdm\n```\n\nTo get Zilliz up and running take a look [here](https://zilliz.com/doc/quick_start). With your account and database set up, proceed to set the following values:\n- URI: The URI your database is running on\n- USER: Your database username\n- PASSWORD: Your database password\n- COLLECTION_NAME: What to name the collection within Zilliz\n- DIMENSION: The dimension of the embeddings\n- OPENAI_ENGINE: Which embedding model to use\n- openai.api_key: Your OpenAI account key\n- INDEX_PARAM: The index settings to use for the collection\n- QUERY_PARAM: The search parameters to use\n- BATCH_SIZE: How many texts to embed and insert at once\n\n\n```python\nimport openai\n\nURI = 'your_uri'\nTOKEN = 'your_token' # TOKEN == user:password or api_key\nCOLLECTION_NAME = 'book_search'\nDIMENSION = 1536\nOPENAI_ENGINE = 'text-embedding-3-small'\nopenai.api_key = 'sk-your_key'\n\nINDEX_PARAM = {\n    'metric_type':'L2',\n    'index_type':\"AUTOINDEX\",\n    'params':{}\n}\n\nQUERY_PARAM = {\n    \"metric_type\": \"L2\",\n    \"params\": {},\n}\n\nBATCH_SIZE = 1000\n```\n\n\n```python\nfrom pymilvus import connections, utility, FieldSchema, Collection, CollectionSchema, DataType\n\n# Connect to Zilliz Database\nconnections.connect(uri=URI, token=TOKEN)\n```\n\n\n```python\n# Remove collection if it already exists\nif utility.has_collection(COLLECTION_NAME):\n    utility.drop_collection(COLLECTION_NAME)\n```\n\n\n```python\n# Create collection which includes the id, title, and embedding.\nfields = [\n    FieldSchema(name='id', dtype=DataType.INT64, is_primary=True, auto_id=True),\n    FieldSchema(name='title', dtype=DataType.VARCHAR, max_length=64000),\n    FieldSchema(name='type', dtype=DataType.VARCHAR, max_length=64000),\n    FieldSchema(name='release_year', dtype=DataType.INT64),\n    FieldSchema(name='rating', dtype=DataType.VARCHAR, max_length=64000),\n    FieldSchema(name='description', dtype=DataType.VARCHAR, max_length=64000),\n    FieldSchema(name='embedding', dtype=DataType.FLOAT_VECTOR, dim=DIMENSION)\n]\nschema = CollectionSchema(fields=fields)\ncollection = Collection(name=COLLECTION_NAME, schema=schema)\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: d6b731a5-02a4-4b76-8157-2f9c7cbedd5c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb", "title": "Filtered Search with Zilliz and OpenAI", "tokens": 2129, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"d6b731a5-02a4-4b76-8157-2f9c7cbedd5c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Filtered Search with Zilliz and OpenAI\", \"tokens\": 2129, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"082b235a-e3c5-4993-9cc1-a0527bd3dfb3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Filtered Search with Zilliz and OpenAI\", \"tokens\": 2129, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"8243d90a985bfa201d016f4e4e8754a0224edf957ac88ac834132750b77d1de5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2faf2b77-2165-4124-9464-931e132a4394\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Filtered Search with Zilliz and OpenAI\", \"tokens\": 2129, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"49a3f04c61aed582b5f1f1b885ac7a48cb13f0854fd5eb267d77e45000d66287\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d709c801-e1f1-42f6-a835-127b8dad44f0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"23ca51f9dc3af3a89cca8455eb3a1adabb3f17dca4dd50d1f684a27b8455bbc3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1447, \"end_char_idx\": 4309, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "082b235a-e3c5-4993-9cc1-a0527bd3dfb3", "doc_id": "082b235a-e3c5-4993-9cc1-a0527bd3dfb3", "ref_doc_id": "082b235a-e3c5-4993-9cc1-a0527bd3dfb3", "chroma:document": "```python\nimport openai\n\nURI = 'your_uri'\nTOKEN = 'your_token' # TOKEN == user:password or api_key\nCOLLECTION_NAME = 'book_search'\nDIMENSION = 1536\nOPENAI_ENGINE = 'text-embedding-3-small'\nopenai.api_key = 'sk-your_key'\n\nINDEX_PARAM = {\n    'metric_type':'L2',\n    'index_type':\"AUTOINDEX\",\n    'params':{}\n}\n\nQUERY_PARAM = {\n    \"metric_type\": \"L2\",\n    \"params\": {},\n}\n\nBATCH_SIZE = 1000\n```\n\n\n```python\nfrom pymilvus import connections, utility, FieldSchema, Collection, CollectionSchema, DataType\n\n# Connect to Zilliz Database\nconnections.connect(uri=URI, token=TOKEN)\n```\n\n\n```python\n# Remove collection if it already exists\nif utility.has_collection(COLLECTION_NAME):\n    utility.drop_collection(COLLECTION_NAME)\n```\n\n\n```python\n# Create collection which includes the id, title, and embedding.\nfields = [\n    FieldSchema(name='id', dtype=DataType.INT64, is_primary=True, auto_id=True),\n    FieldSchema(name='title', dtype=DataType.VARCHAR, max_length=64000),\n    FieldSchema(name='type', dtype=DataType.VARCHAR, max_length=64000),\n    FieldSchema(name='release_year', dtype=DataType.INT64),\n    FieldSchema(name='rating', dtype=DataType.VARCHAR, max_length=64000),\n    FieldSchema(name='description', dtype=DataType.VARCHAR, max_length=64000),\n    FieldSchema(name='embedding', dtype=DataType.FLOAT_VECTOR, dim=DIMENSION)\n]\nschema = CollectionSchema(fields=fields)\ncollection = Collection(name=COLLECTION_NAME, schema=schema)\n```\n\n\n```python\n# Create the index on the collection and load it.\ncollection.create_index(field_name=\"embedding\", index_params=INDEX_PARAM)\ncollection.load()\n```\n\n## Dataset\nWith Zilliz up and running we can begin grabbing our data. `Hugging Face Datasets` is a hub that holds many different user datasets, and for this example we are using HuggingLearners's netflix-shows dataset. This dataset contains movies and their metadata pairs for over 8 thousand movies. We are going to embed each description and store it within Zilliz along with its title, type, release_year and rating.\n\n\n```python\nimport datasets\n\n# Download the dataset \ndataset = datasets.load_dataset('hugginglearners/netflix-shows', split='train')\n```\n\n    /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n      from .autonotebook import tqdm as notebook_tqdm\n    Found cached dataset csv (/Users/filiphaltmayer/.cache/huggingface/datasets/hugginglearners___csv/hugginglearners--netflix-shows-03475319fc65a05a/0.0.0/6b34fb8fcf56f7c8ba51dc895bfa2bfbe43546f190a60fcf74bb5e8afdcc2317)\n\n\n## Insert the Data\nNow that we have our data on our machine we can begin embedding it and inserting it into Zilliz. The embedding function takes in text and returns the embeddings in a list format."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: d709c801-e1f1-42f6-a835-127b8dad44f0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb", "title": "Filtered Search with Zilliz and OpenAI", "tokens": 2129, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"d709c801-e1f1-42f6-a835-127b8dad44f0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Filtered Search with Zilliz and OpenAI\", \"tokens\": 2129, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"082b235a-e3c5-4993-9cc1-a0527bd3dfb3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Filtered Search with Zilliz and OpenAI\", \"tokens\": 2129, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"8243d90a985bfa201d016f4e4e8754a0224edf957ac88ac834132750b77d1de5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d6b731a5-02a4-4b76-8157-2f9c7cbedd5c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Filtered Search with Zilliz and OpenAI\", \"tokens\": 2129, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"24e63954feefecc4abbe27f9d7db041742e479a00ccea335b622648bb5d94502\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"35333155-155e-44b5-8b03-be8a36404179\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6977b5f65d14be92a5be5b1422a62ce599bc7442852163224dec30fcbf2071f5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2885, \"end_char_idx\": 5570, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "082b235a-e3c5-4993-9cc1-a0527bd3dfb3", "doc_id": "082b235a-e3c5-4993-9cc1-a0527bd3dfb3", "ref_doc_id": "082b235a-e3c5-4993-9cc1-a0527bd3dfb3", "chroma:document": "```python\n# Create the index on the collection and load it.\ncollection.create_index(field_name=\"embedding\", index_params=INDEX_PARAM)\ncollection.load()\n```\n\n## Dataset\nWith Zilliz up and running we can begin grabbing our data. `Hugging Face Datasets` is a hub that holds many different user datasets, and for this example we are using HuggingLearners's netflix-shows dataset. This dataset contains movies and their metadata pairs for over 8 thousand movies. We are going to embed each description and store it within Zilliz along with its title, type, release_year and rating.\n\n\n```python\nimport datasets\n\n# Download the dataset \ndataset = datasets.load_dataset('hugginglearners/netflix-shows', split='train')\n```\n\n    /Users/filiphaltmayer/miniconda3/envs/haystack/lib/python3.9/site-packages/tqdm/auto.py:22: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n      from .autonotebook import tqdm as notebook_tqdm\n    Found cached dataset csv (/Users/filiphaltmayer/.cache/huggingface/datasets/hugginglearners___csv/hugginglearners--netflix-shows-03475319fc65a05a/0.0.0/6b34fb8fcf56f7c8ba51dc895bfa2bfbe43546f190a60fcf74bb5e8afdcc2317)\n\n\n## Insert the Data\nNow that we have our data on our machine we can begin embedding it and inserting it into Zilliz. The embedding function takes in text and returns the embeddings in a list format. \n\n\n```python\n# Simple function that converts the texts to embeddings\ndef embed(texts):\n    embeddings = openai.Embedding.create(\n        input=texts,\n        engine=OPENAI_ENGINE\n    )\n    return [x['embedding'] for x in embeddings['data']]\n\n```\n\nThis next step does the actual inserting. We iterate through all the entries and create batches that we insert once we hit our set batch size. After the loop is over we insert the last remaning batch if it exists. \n\n\n```python\nfrom tqdm import tqdm\n\ndata = [\n    [], # title\n    [], # type\n    [], # release_year\n    [], # rating\n    [], # description\n]\n\n# Embed and insert in batches\nfor i in tqdm(range(0, len(dataset))):\n    data[0].append(dataset[i]['title'] or '')\n    data[1].append(dataset[i]['type'] or '')\n    data[2].append(dataset[i]['release_year'] or -1)\n    data[3].append(dataset[i]['rating'] or '')\n    data[4].append(dataset[i]['description'] or '')\n    if len(data[0]) % BATCH_SIZE == 0:\n        data.append(embed(data[4]))\n        collection.insert(data)\n        data = [[],[],[],[],[]]\n\n# Embed and insert the remainder \nif len(data[0]) != 0:\n    data.append(embed(data[4]))\n    collection.insert(data)\n    data = [[],[],[],[],[]]\n\n```\n\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 8807/8807 [00:54<00:00, 162.59it/s]"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 35333155-155e-44b5-8b03-be8a36404179
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb", "title": "Filtered Search with Zilliz and OpenAI", "tokens": 2129, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"35333155-155e-44b5-8b03-be8a36404179\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Filtered Search with Zilliz and OpenAI\", \"tokens\": 2129, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"082b235a-e3c5-4993-9cc1-a0527bd3dfb3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Filtered Search with Zilliz and OpenAI\", \"tokens\": 2129, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"8243d90a985bfa201d016f4e4e8754a0224edf957ac88ac834132750b77d1de5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d709c801-e1f1-42f6-a835-127b8dad44f0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Filtered Search with Zilliz and OpenAI\", \"tokens\": 2129, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"2465e3c68f6c59ec550885415674b3494081977c1fd30313eb142d49e07a4d76\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2890546d-cd03-4e52-9ac9-330c3b83c113\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e20f81c66c012596e97af277c1e6bdc1d41643a1307e046504d224a4c55ed57c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4112, \"end_char_idx\": 6032, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "082b235a-e3c5-4993-9cc1-a0527bd3dfb3", "doc_id": "082b235a-e3c5-4993-9cc1-a0527bd3dfb3", "ref_doc_id": "082b235a-e3c5-4993-9cc1-a0527bd3dfb3", "chroma:document": "## Insert the Data\nNow that we have our data on our machine we can begin embedding it and inserting it into Zilliz. The embedding function takes in text and returns the embeddings in a list format. \n\n\n```python\n# Simple function that converts the texts to embeddings\ndef embed(texts):\n    embeddings = openai.Embedding.create(\n        input=texts,\n        engine=OPENAI_ENGINE\n    )\n    return [x['embedding'] for x in embeddings['data']]\n\n```\n\nThis next step does the actual inserting. We iterate through all the entries and create batches that we insert once we hit our set batch size. After the loop is over we insert the last remaning batch if it exists. \n\n\n```python\nfrom tqdm import tqdm\n\ndata = [\n    [], # title\n    [], # type\n    [], # release_year\n    [], # rating\n    [], # description\n]\n\n# Embed and insert in batches\nfor i in tqdm(range(0, len(dataset))):\n    data[0].append(dataset[i]['title'] or '')\n    data[1].append(dataset[i]['type'] or '')\n    data[2].append(dataset[i]['release_year'] or -1)\n    data[3].append(dataset[i]['rating'] or '')\n    data[4].append(dataset[i]['description'] or '')\n    if len(data[0]) % BATCH_SIZE == 0:\n        data.append(embed(data[4]))\n        collection.insert(data)\n        data = [[],[],[],[],[]]\n\n# Embed and insert the remainder \nif len(data[0]) != 0:\n    data.append(embed(data[4]))\n    collection.insert(data)\n    data = [[],[],[],[],[]]\n\n```\n\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 8807/8807 [00:54<00:00, 162.59it/s]\n\n\n## Query the Database\nWith our data safely inserted into Zilliz, we can now perform a query. The query takes in a tuple of the movie description you are searching for and the filter to use. More info about the filter can be found [here](https://milvus.io/docs/boolean.md). The search first prints out your description and filter expression. After that for each result we print the score, title, type, release year, rating and description of the result movies."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 2890546d-cd03-4e52-9ac9-330c3b83c113
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb", "title": "Filtered Search with Zilliz and OpenAI", "tokens": 2129, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"2890546d-cd03-4e52-9ac9-330c3b83c113\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Filtered Search with Zilliz and OpenAI\", \"tokens\": 2129, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"082b235a-e3c5-4993-9cc1-a0527bd3dfb3\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Filtered Search with Zilliz and OpenAI\", \"tokens\": 2129, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"8243d90a985bfa201d016f4e4e8754a0224edf957ac88ac834132750b77d1de5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"35333155-155e-44b5-8b03-be8a36404179\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/zilliz/Filtered_search_with_Zilliz_and_OpenAI.ipynb\", \"title\": \"Filtered Search with Zilliz and OpenAI\", \"tokens\": 2129, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"4dd13e279de583368384f6709c91b43577d41eb2fdc8fd25f090175f689831b4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4774, \"end_char_idx\": 8353, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "082b235a-e3c5-4993-9cc1-a0527bd3dfb3", "doc_id": "082b235a-e3c5-4993-9cc1-a0527bd3dfb3", "ref_doc_id": "082b235a-e3c5-4993-9cc1-a0527bd3dfb3", "chroma:document": "```python\nfrom tqdm import tqdm\n\ndata = [\n    [], # title\n    [], # type\n    [], # release_year\n    [], # rating\n    [], # description\n]\n\n# Embed and insert in batches\nfor i in tqdm(range(0, len(dataset))):\n    data[0].append(dataset[i]['title'] or '')\n    data[1].append(dataset[i]['type'] or '')\n    data[2].append(dataset[i]['release_year'] or -1)\n    data[3].append(dataset[i]['rating'] or '')\n    data[4].append(dataset[i]['description'] or '')\n    if len(data[0]) % BATCH_SIZE == 0:\n        data.append(embed(data[4]))\n        collection.insert(data)\n        data = [[],[],[],[],[]]\n\n# Embed and insert the remainder \nif len(data[0]) != 0:\n    data.append(embed(data[4]))\n    collection.insert(data)\n    data = [[],[],[],[],[]]\n\n```\n\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 8807/8807 [00:54<00:00, 162.59it/s]\n\n\n## Query the Database\nWith our data safely inserted into Zilliz, we can now perform a query. The query takes in a tuple of the movie description you are searching for and the filter to use. More info about the filter can be found [here](https://milvus.io/docs/boolean.md). The search first prints out your description and filter expression. After that for each result we print the score, title, type, release year, rating and description of the result movies. \n\n\n```python\nimport textwrap\n\ndef query(query, top_k = 5):\n    text, expr = query\n    res = collection.search(embed(text), anns_field='embedding', expr = expr, param=QUERY_PARAM, limit = top_k, output_fields=['title', 'type', 'release_year', 'rating', 'description'])\n    for i, hit in enumerate(res):\n        print('Description:', text, 'Expression:', expr)\n        print('Results:')\n        for ii, hits in enumerate(hit):\n            print('\\t' + 'Rank:', ii + 1, 'Score:', hits.score, 'Title:', hits.entity.get('title'))\n            print('\\t\\t' + 'Type:', hits.entity.get('type'), 'Release Year:', hits.entity.get('release_year'), 'Rating:', hits.entity.get('rating'))\n            print(textwrap.fill(hits.entity.get('description'), 88))\n            print()\n\nmy_query = ('movie about a fluffly animal', 'release_year < 2019 and rating like \\\"PG%\\\"')\n\nquery(my_query)\n```\n\n    Description: movie about a fluffly animal Expression: release_year < 2019 and rating like \"PG%\"\n    Results:\n    \tRank: 1 Score: 0.30085673928260803 Title: The Lamb\n    \t\tType: Movie Release Year: 2017 Rating: PG\n    A big-dreaming donkey escapes his menial existence and befriends some free-spirited\n    animal pals in this imaginative retelling of the Nativity Story.\n    \n    \tRank: 2 Score: 0.3352621793746948 Title: Puss in Boots\n    \t\tType: Movie Release Year: 2011 Rating: PG\n    The fabled feline heads to the Land of Giants with friends Humpty Dumpty and Kitty\n    Softpaws on a quest to nab its greatest treasure: the Golden Goose.\n    \n    \tRank: 3 Score: 0.3415083587169647 Title: Show Dogs\n    \t\tType: Movie Release Year: 2018 Rating: PG\n    A rough and tough police dog must go undercover with an FBI agent as a prim and proper\n    pet at a dog show to save a baby panda from an illegal sale.\n    \n    \tRank: 4 Score: 0.3428957462310791 Title: Open Season 2\n    \t\tType: Movie Release Year: 2008 Rating: PG\n    Elliot the buck and his forest-dwelling cohorts must rescue their dachshund pal from\n    some spoiled pets bent on returning him to domesticity.\n    \n    \tRank: 5 Score: 0.34376364946365356 Title: Stuart Little 2\n    \t\tType: Movie Release Year: 2002 Rating: PG\n    Zany misadventures are in store as lovable city mouse Stuart and his human brother,\n    George, raise the roof in this sequel to the 1999 blockbuster."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 275d3f04-7402-4596-85ff-5fa4a3c65349
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/README.ipynb", "title": "Weaviate <> OpenAI", "tokens": 656, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"275d3f04-7402-4596-85ff-5fa4a3c65349\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/README.ipynb\", \"title\": \"Weaviate <> OpenAI\", \"tokens\": 656, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"75527e50-ba2d-45f1-af0d-4cc2774c6931\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/README.ipynb\", \"title\": \"Weaviate <> OpenAI\", \"tokens\": 656, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"c30665db3e544975dd32e4c9c10b37e85f75e60622a605bb96487f1c789c3a30\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2427, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "75527e50-ba2d-45f1-af0d-4cc2774c6931", "doc_id": "75527e50-ba2d-45f1-af0d-4cc2774c6931", "ref_doc_id": "75527e50-ba2d-45f1-af0d-4cc2774c6931", "chroma:document": "# Weaviate <> OpenAI\n\n[\u200bWeaviate](https://weaviate.io) is an open-source vector search engine ([docs](https://weaviate.io/developers/weaviate) - [Github](https://github.com/weaviate/weaviate)) that can store and search through OpenAI embeddings _and_ data objects. The database allows you to do similarity search, hybrid search (the combining of multiple search techniques, such as keyword-based and vector search), and generative search (like Q&A). Weaviate also supports a wide variety of OpenAI-based modules (e.g., [`text2vec-openai`](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai), [`qna-openai`](https://weaviate.io/developers/weaviate/modules/reader-generator-modules/qna-openai)), allowing you to vectorize and query data fast and efficiently.\n\nYou can run Weaviate (including the OpenAI modules if desired) in three ways:\n\n1. Open source inside a Docker-container ([example](./docker-compose.yml))\n2. Using the Weaviate Cloud Service ([get started](https://weaviate.io/developers/weaviate/quickstart/installation#weaviate-cloud-service))\n3. In a Kubernetes cluster ([learn more](https://weaviate.io/developers/weaviate/installation/kubernetes))\n\n### Examples\n\nThis folder contains a variety of Weaviate and OpenAI examples. \n\n| Name | Description | language | Google Colab |\n| --- | --- | --- | --- |\n| [Getting Started with Weaviate and OpenAI](./getting-started-with-weaviate-and-openai.ipynb) | A simple getting started for *semantic vector search* using the OpenAI vectorization module in Weaviate (`text2vec-openai`) | Python Notebook | [link](https://colab.research.google.com/drive/1RxpDE_ruCnoBB3TfwAZqdjYgHJhtdwhK) |\n| [Hybrid Search with Weaviate and OpenAI](./hybrid-search-with-weaviate-and-openai.ipynb) | A simple getting started for *hybrid search* using the OpenAI vectorization module in Weaviate (`text2vec-openai`) | Python Notebook | [link](https://colab.research.google.com/drive/1E75BALWoKrOjvUhaznJKQO0A-B1QUPZ4) |\n| [Question Answering with Weaviate and OpenAI](./question-answering-with-weaviate-and-openai.ipynb) | A simple getting started for *question answering (Q&A)* using the OpenAI Q&A module in Weaviate (`qna-openai`) | Python Notebook | [link](https://colab.research.google.com/drive/1pUerUZrJaknEboDxDxsuf3giCK0MJJgm) |\n| [Docker-compose example](./docker-compose.yml) | A Docker-compose file with all OpenAI modules enabled | Docker |"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 6ff6ed36-d138-4866-9724-07b4f0e93ecc
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with Generative OpenAI module for Generative Search", "tokens": 1437, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"6ff6ed36-d138-4866-9724-07b4f0e93ecc\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with Generative OpenAI module for Generative Search\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"493743fa-5abf-45e8-95ec-7cee1ba9d53b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with Generative OpenAI module for Generative Search\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"65872f24f42605ae6a0ea781103cec7643f3204eeae8f1ce7cb0e6a9b4095b78\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"54006685-b807-4f32-8b37-bb8e0e94bba5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1ce7d8725919552baff6195ac90683ac0f975634d96a66b9a06ebfd77b453965\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2368, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "493743fa-5abf-45e8-95ec-7cee1ba9d53b", "doc_id": "493743fa-5abf-45e8-95ec-7cee1ba9d53b", "ref_doc_id": "493743fa-5abf-45e8-95ec-7cee1ba9d53b", "chroma:document": "# Using Weaviate with Generative OpenAI module for Generative Search\n\nThis notebook is prepared for a scenario where:\n* Your data is already in Weaviate\n* You want to use Weaviate with the Generative OpenAI module ([generative-openai](https://weaviate.io/developers/weaviate/modules/reader-generator-modules/generative-openai)).\n\n\n\n## Prerequisites\n\nThis cookbook only coveres Generative Search examples, however, it doesn't cover the configuration and data imports.\n\nIn order to make the most of this cookbook, please complete the [Getting Started cookbook](./getting-started-with-weaviate-and-openai.ipynb) first, where you will learn the essentials of working with Weaviate and import the demo data.\n\nChecklist:\n* completed [Getting Started cookbook](./getting-started-with-weaviate-and-openai.ipynb),\n* crated a `Weaviate` instance,\n* imported data into your `Weaviate` instance,\n* you have an [OpenAI API key](https://beta.openai.com/account/api-keys)\n\n===========================================================\n## Prepare your OpenAI API key\n\nThe `OpenAI API key` is used for vectorization of your data at import, and for running queries.\n\nIf you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n\nOnce you get your key, please add it to your environment variables as `OPENAI_API_KEY`.\n\n\n```python\n# Export OpenAI API Key\n!export OPENAI_API_KEY=\"your key\"\n```\n\n\n```python\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\nimport os\n\n# Note. alternatively you can set a temporary env variable like this:\n# os.environ[\"OPENAI_API_KEY\"] = 'your-key-goes-here'\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    print (\"OPENAI_API_KEY is ready\")\nelse:\n    print (\"OPENAI_API_KEY environment variable not found\")\n```\n\n## Connect to your Weaviate instance\n\nIn this section, we will:\n\n1. test env variable `OPENAI_API_KEY` \u2013 **make sure** you completed the step in [#Prepare-your-OpenAI-API-key](#Prepare-your-OpenAI-API-key)\n2. connect to your Weaviate with your `OpenAI API Key`\n3. and test the client connection\n\n### The client \n\nAfter this step, the `client` object will be used to perform all Weaviate-related operations."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 54006685-b807-4f32-8b37-bb8e0e94bba5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with Generative OpenAI module for Generative Search", "tokens": 1437, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"54006685-b807-4f32-8b37-bb8e0e94bba5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with Generative OpenAI module for Generative Search\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"493743fa-5abf-45e8-95ec-7cee1ba9d53b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with Generative OpenAI module for Generative Search\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"65872f24f42605ae6a0ea781103cec7643f3204eeae8f1ce7cb0e6a9b4095b78\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6ff6ed36-d138-4866-9724-07b4f0e93ecc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with Generative OpenAI module for Generative Search\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3035cc7b888d4f05173f7a309e8276cbf0d278fbf7fa16d2f2f37996d91f2293\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"19c21758-b882-4b50-adf0-e3d514b3f205\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b453ba6928b02ba5269a64dcce6593340e8a21704980a8bf8e42d928f3c77b32\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1382, \"end_char_idx\": 3993, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "493743fa-5abf-45e8-95ec-7cee1ba9d53b", "doc_id": "493743fa-5abf-45e8-95ec-7cee1ba9d53b", "ref_doc_id": "493743fa-5abf-45e8-95ec-7cee1ba9d53b", "chroma:document": "```python\n# Export OpenAI API Key\n!export OPENAI_API_KEY=\"your key\"\n```\n\n\n```python\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\nimport os\n\n# Note. alternatively you can set a temporary env variable like this:\n# os.environ[\"OPENAI_API_KEY\"] = 'your-key-goes-here'\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    print (\"OPENAI_API_KEY is ready\")\nelse:\n    print (\"OPENAI_API_KEY environment variable not found\")\n```\n\n## Connect to your Weaviate instance\n\nIn this section, we will:\n\n1. test env variable `OPENAI_API_KEY` \u2013 **make sure** you completed the step in [#Prepare-your-OpenAI-API-key](#Prepare-your-OpenAI-API-key)\n2. connect to your Weaviate with your `OpenAI API Key`\n3. and test the client connection\n\n### The client \n\nAfter this step, the `client` object will be used to perform all Weaviate-related operations.\n\n\n```python\nimport weaviate\nfrom datasets import load_dataset\nimport os\n\n# Connect to your Weaviate instance\nclient = weaviate.Client(\n    url=\"https://your-wcs-instance-name.weaviate.network/\",\n    # url=\"http://localhost:8080/\",\n    auth_client_secret=weaviate.auth.AuthApiKey(api_key=\"<YOUR-WEAVIATE-API-KEY>\"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)\n    additional_headers={\n        \"X-OpenAI-Api-Key\": os.getenv(\"OPENAI_API_KEY\")\n    }\n)\n\n# Check if your instance is live and ready\n# This should return `True`\nclient.is_ready()\n```\n\n## Generative Search\nWeaviate offers a [Generative Search OpenAI](https://weaviate.io/developers/weaviate/modules/reader-generator-modules/generative-openai) module, which generates responses based on the data stored in your Weaviate instance.\n\nThe way you construct a generative search query is very similar to a standard semantic search query in Weaviate. \n\nFor example:\n* search in \"Articles\", \n* return \"title\", \"content\", \"url\"\n* look for objects related to \"football clubs\"\n* limit results to 5 objects\n\n```\n    result = (\n        client.query\n        .get(\"Articles\", [\"title\", \"content\", \"url\"])\n        .with_near_text(\"concepts\": \"football clubs\")\n        .with_limit(5)\n        # generative query will go here\n        .do()\n    )\n```\n\nNow, you can add `with_generate()` function to apply generative transformation. `with_generate` takes either:\n- `single_prompt` - to generate a response for each returned object,\n- `grouped_task` \u2013 to generate a single response from all returned objects."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 19c21758-b882-4b50-adf0-e3d514b3f205
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with Generative OpenAI module for Generative Search", "tokens": 1437, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"19c21758-b882-4b50-adf0-e3d514b3f205\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with Generative OpenAI module for Generative Search\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"493743fa-5abf-45e8-95ec-7cee1ba9d53b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with Generative OpenAI module for Generative Search\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"65872f24f42605ae6a0ea781103cec7643f3204eeae8f1ce7cb0e6a9b4095b78\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"54006685-b807-4f32-8b37-bb8e0e94bba5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with Generative OpenAI module for Generative Search\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a5d21b6fde5166cce304a6089ce677802777e564fb3f2728a09161ec8f70cff3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1210848c-f9bd-4845-95b3-7a95760c8e48\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1a56001b4b2cb993978ef2b2bbe89365ccc57984f685c4eade9cfeec3ed1e028\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2371, \"end_char_idx\": 4989, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "493743fa-5abf-45e8-95ec-7cee1ba9d53b", "doc_id": "493743fa-5abf-45e8-95ec-7cee1ba9d53b", "ref_doc_id": "493743fa-5abf-45e8-95ec-7cee1ba9d53b", "chroma:document": "```python\nimport weaviate\nfrom datasets import load_dataset\nimport os\n\n# Connect to your Weaviate instance\nclient = weaviate.Client(\n    url=\"https://your-wcs-instance-name.weaviate.network/\",\n    # url=\"http://localhost:8080/\",\n    auth_client_secret=weaviate.auth.AuthApiKey(api_key=\"<YOUR-WEAVIATE-API-KEY>\"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)\n    additional_headers={\n        \"X-OpenAI-Api-Key\": os.getenv(\"OPENAI_API_KEY\")\n    }\n)\n\n# Check if your instance is live and ready\n# This should return `True`\nclient.is_ready()\n```\n\n## Generative Search\nWeaviate offers a [Generative Search OpenAI](https://weaviate.io/developers/weaviate/modules/reader-generator-modules/generative-openai) module, which generates responses based on the data stored in your Weaviate instance.\n\nThe way you construct a generative search query is very similar to a standard semantic search query in Weaviate. \n\nFor example:\n* search in \"Articles\", \n* return \"title\", \"content\", \"url\"\n* look for objects related to \"football clubs\"\n* limit results to 5 objects\n\n```\n    result = (\n        client.query\n        .get(\"Articles\", [\"title\", \"content\", \"url\"])\n        .with_near_text(\"concepts\": \"football clubs\")\n        .with_limit(5)\n        # generative query will go here\n        .do()\n    )\n```\n\nNow, you can add `with_generate()` function to apply generative transformation. `with_generate` takes either:\n- `single_prompt` - to generate a response for each returned object,\n- `grouped_task` \u2013 to generate a single response from all returned objects.\n\n\n\n```python\ndef generative_search_per_item(query, collection_name):\n    prompt = \"Summarize in a short tweet the following content: {content}\"\n\n    result = (\n        client.query\n        .get(collection_name, [\"title\", \"content\", \"url\"])\n        .with_near_text({ \"concepts\": [query], \"distance\": 0.7 })\n        .with_limit(5)\n        .with_generate(single_prompt=prompt)\n        .do()\n    )\n    \n    # Check for errors\n    if (\"errors\" in result):\n        print (\"\\033[91mYou probably have run out of OpenAI API calls for the current minute \u2013 the limit is set at 60 per minute.\")\n        raise Exception(result[\"errors\"][0]['message'])\n    \n    return result[\"data\"][\"Get\"][collection_name]\n```\n\n\n```python\nquery_result = generative_search_per_item(\"football clubs\", \"Article\")\n\nfor i, article in enumerate(query_result):\n    print(f\"{i+1}. { article['title']}\")\n    print(article['_additional']['generate']['singleResult']) # print generated response\n    print(\"-----------------------\")\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 1210848c-f9bd-4845-95b3-7a95760c8e48
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with Generative OpenAI module for Generative Search", "tokens": 1437, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"1210848c-f9bd-4845-95b3-7a95760c8e48\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with Generative OpenAI module for Generative Search\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"493743fa-5abf-45e8-95ec-7cee1ba9d53b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with Generative OpenAI module for Generative Search\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"65872f24f42605ae6a0ea781103cec7643f3204eeae8f1ce7cb0e6a9b4095b78\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"19c21758-b882-4b50-adf0-e3d514b3f205\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/generative-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with Generative OpenAI module for Generative Search\", \"tokens\": 1437, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"ee4cd214138da6507e11fb361310f94148f7e9bd3bf8c0669192409a0a8e8fdb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3997, \"end_char_idx\": 6062, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "493743fa-5abf-45e8-95ec-7cee1ba9d53b", "doc_id": "493743fa-5abf-45e8-95ec-7cee1ba9d53b", "ref_doc_id": "493743fa-5abf-45e8-95ec-7cee1ba9d53b", "chroma:document": "```python\ndef generative_search_per_item(query, collection_name):\n    prompt = \"Summarize in a short tweet the following content: {content}\"\n\n    result = (\n        client.query\n        .get(collection_name, [\"title\", \"content\", \"url\"])\n        .with_near_text({ \"concepts\": [query], \"distance\": 0.7 })\n        .with_limit(5)\n        .with_generate(single_prompt=prompt)\n        .do()\n    )\n    \n    # Check for errors\n    if (\"errors\" in result):\n        print (\"\\033[91mYou probably have run out of OpenAI API calls for the current minute \u2013 the limit is set at 60 per minute.\")\n        raise Exception(result[\"errors\"][0]['message'])\n    \n    return result[\"data\"][\"Get\"][collection_name]\n```\n\n\n```python\nquery_result = generative_search_per_item(\"football clubs\", \"Article\")\n\nfor i, article in enumerate(query_result):\n    print(f\"{i+1}. { article['title']}\")\n    print(article['_additional']['generate']['singleResult']) # print generated response\n    print(\"-----------------------\")\n```\n\n\n```python\ndef generative_search_group(query, collection_name):\n    generateTask = \"Explain what these have in common\"\n\n    result = (\n        client.query\n        .get(collection_name, [\"title\", \"content\", \"url\"])\n        .with_near_text({ \"concepts\": [query], \"distance\": 0.7 })\n        .with_generate(grouped_task=generateTask)\n        .with_limit(5)\n        .do()\n    )\n    \n    # Check for errors\n    if (\"errors\" in result):\n        print (\"\\033[91mYou probably have run out of OpenAI API calls for the current minute \u2013 the limit is set at 60 per minute.\")\n        raise Exception(result[\"errors\"][0]['message'])\n    \n    return result[\"data\"][\"Get\"][collection_name]\n```\n\n\n```python\nquery_result = generative_search_group(\"football clubs\", \"Article\")\n\nprint (query_result[0]['_additional']['generate']['groupedResult'])\n```\n\nThanks for following along, you're now equipped to set up your own vector databases and use embeddings to do all kinds of cool things - enjoy! For more complex use cases please continue to work through other cookbook examples in this repo."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 3a6c629d-542c-46eb-b189-966b980e046d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"3a6c629d-542c-46eb-b189-966b980e046d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f73f419e-2b6f-4043-b065-53b2c3d27ea0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"4530bedcb85f5dca9e36ad0f00e6c7cc12b4212403ee25931b8f28c67f4b8508\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ded727d3-9fbd-46e0-9325-9860299d8469\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d603f0f3e3cd1bab94ec3b0fc9295d0056729ecb95964e818bedd7a285afe1aa\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2501, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "doc_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "ref_doc_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "chroma:document": "# Using Weaviate with OpenAI vectorize module for Hybrid Search\n\nThis notebook is prepared for a scenario where:\n* Your data is not vectorized\n* You want to run Hybrid Search ([learn more](https://weaviate.io/blog/hybrid-search-explained)) on your data\n* You want to use Weaviate with the OpenAI module ([text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai)), to generate vector embeddings for you.\n\nThis notebook takes you through a simple flow to set up a Weaviate instance, connect to it (with OpenAI API key), configure data schema, import data (which will automatically generate vector embeddings for your data), and run hybrid search (mixing of vector and BM25 search).\n\nThis is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.\n\n## What is Weaviate\n\nWeaviate is an open-source vector search engine that stores data objects together with their vectors. This allows for combining vector search with structured filtering.\n\nWeaviate uses KNN algorithms to create an vector-optimized index, which allows your queries to run extremely fast. Learn more [here](https://weaviate.io/blog/why-is-vector-search-so-fast).\n\nWeaviate let you use your favorite ML-models, and scale seamlessly into billions of data objects.\n\n### Deployment options\n\nWhatever your scenario or production setup, Weaviate has an option for you. You can deploy Weaviate in the following setups:\n* Self-hosted \u2013 you can deploy Weaviate with docker locally, or any server you want.\n* SaaS \u2013 you can use [Weaviate Cloud Service (WCS)](https://console.weaviate.io/) to host your Weaviate instances.\n* Hybrid-SaaS \u2013 you can deploy Weaviate in your own private Cloud Service \n\n### Programming languages\n\nWeaviate offers four [client libraries](https://weaviate.io/developers/weaviate/client-libraries), which allow you to communicate from your apps:\n* [Python](https://weaviate.io/developers/weaviate/client-libraries/python)\n* [JavaScript](https://weaviate.io/developers/weaviate/client-libraries/javascript)\n* [Java](https://weaviate.io/developers/weaviate/client-libraries/java)\n* [Go](https://weaviate.io/developers/weaviate/client-libraries/go)\n\nAdditionally, Weaviate has a [REST layer](https://weaviate.io/developers/weaviate/api/rest/objects). Basically you can call Weaviate from any language that supports REST requests."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: ded727d3-9fbd-46e0-9325-9860299d8469
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"ded727d3-9fbd-46e0-9325-9860299d8469\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f73f419e-2b6f-4043-b065-53b2c3d27ea0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"4530bedcb85f5dca9e36ad0f00e6c7cc12b4212403ee25931b8f28c67f4b8508\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3a6c629d-542c-46eb-b189-966b980e046d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7d115e3726f5c190cf6295da38038287e00181124ca1629848ebc9e139c70204\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"14b419d9-fe29-4c1c-b4ad-2658743bc946\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e9e14de57b1163a11ac517d604cc2b7989120d0b88ff7e048f8015b486c07521\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 943, \"end_char_idx\": 4103, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "doc_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "ref_doc_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "chroma:document": "## What is Weaviate\n\nWeaviate is an open-source vector search engine that stores data objects together with their vectors. This allows for combining vector search with structured filtering.\n\nWeaviate uses KNN algorithms to create an vector-optimized index, which allows your queries to run extremely fast. Learn more [here](https://weaviate.io/blog/why-is-vector-search-so-fast).\n\nWeaviate let you use your favorite ML-models, and scale seamlessly into billions of data objects.\n\n### Deployment options\n\nWhatever your scenario or production setup, Weaviate has an option for you. You can deploy Weaviate in the following setups:\n* Self-hosted \u2013 you can deploy Weaviate with docker locally, or any server you want.\n* SaaS \u2013 you can use [Weaviate Cloud Service (WCS)](https://console.weaviate.io/) to host your Weaviate instances.\n* Hybrid-SaaS \u2013 you can deploy Weaviate in your own private Cloud Service \n\n### Programming languages\n\nWeaviate offers four [client libraries](https://weaviate.io/developers/weaviate/client-libraries), which allow you to communicate from your apps:\n* [Python](https://weaviate.io/developers/weaviate/client-libraries/python)\n* [JavaScript](https://weaviate.io/developers/weaviate/client-libraries/javascript)\n* [Java](https://weaviate.io/developers/weaviate/client-libraries/java)\n* [Go](https://weaviate.io/developers/weaviate/client-libraries/go)\n\nAdditionally, Weaviate has a [REST layer](https://weaviate.io/developers/weaviate/api/rest/objects). Basically you can call Weaviate from any language that supports REST requests.\n\n## Demo Flow\nThe demo flow is:\n- **Prerequisites Setup**: Create a Weaviate instance and install required libraries\n- **Connect**: Connect to your Weaviate instance \n- **Schema Configuration**: Configure the schema of your data\n    - *Note*: Here we can define which OpenAI Embedding Model to use\n    - *Note*: Here we can configure which properties to index\n- **Import data**: Load a demo dataset and import it into Weaviate\n    - *Note*: The import process will automatically index your data - based on the configuration in the schema\n    - *Note*: You don't need to explicitly vectorize your data, Weaviate will communicate with OpenAI to do it for you\n- **Run Queries**: Query \n    - *Note*: You don't need to explicitly vectorize your queries, Weaviate will communicate with OpenAI to do it for you\n\nOnce you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.\n\n## OpenAI Module in Weaviate\nAll Weaviate instances come equipped with the [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) module.\n\nThis module is responsible for handling vectorization during import (or any CRUD operations) and when you run a query.\n\n### No need to manually vectorize data\nThis is great news for you. With [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) you don't need to manually vectorize your data, as Weaviate will call OpenAI for you whenever necessary."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 14b419d9-fe29-4c1c-b4ad-2658743bc946
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"14b419d9-fe29-4c1c-b4ad-2658743bc946\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f73f419e-2b6f-4043-b065-53b2c3d27ea0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"4530bedcb85f5dca9e36ad0f00e6c7cc12b4212403ee25931b8f28c67f4b8508\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ded727d3-9fbd-46e0-9325-9860299d8469\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"34237e7a79fad467615a67ddddb7ca49c484695b63c97817971776c2f1b1ef18\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6d957711-ce2a-4095-8c9d-8973e65f9922\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"914020a4f805cf6e0c1e7d53e8af89e270260391245b35f78f20f83f7eb9f703\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2423, \"end_char_idx\": 5476, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "doc_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "ref_doc_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "chroma:document": "Basically you can call Weaviate from any language that supports REST requests.\n\n## Demo Flow\nThe demo flow is:\n- **Prerequisites Setup**: Create a Weaviate instance and install required libraries\n- **Connect**: Connect to your Weaviate instance \n- **Schema Configuration**: Configure the schema of your data\n    - *Note*: Here we can define which OpenAI Embedding Model to use\n    - *Note*: Here we can configure which properties to index\n- **Import data**: Load a demo dataset and import it into Weaviate\n    - *Note*: The import process will automatically index your data - based on the configuration in the schema\n    - *Note*: You don't need to explicitly vectorize your data, Weaviate will communicate with OpenAI to do it for you\n- **Run Queries**: Query \n    - *Note*: You don't need to explicitly vectorize your queries, Weaviate will communicate with OpenAI to do it for you\n\nOnce you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.\n\n## OpenAI Module in Weaviate\nAll Weaviate instances come equipped with the [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) module.\n\nThis module is responsible for handling vectorization during import (or any CRUD operations) and when you run a query.\n\n### No need to manually vectorize data\nThis is great news for you. With [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) you don't need to manually vectorize your data, as Weaviate will call OpenAI for you whenever necessary.\n\nAll you need to do is:\n1. provide your OpenAI API Key \u2013 when you connected to the Weaviate Client\n2. define which OpenAI vectorizer to use in your Schema\n\n## Prerequisites\n\nBefore we start this project, we need setup the following:\n\n* create a `Weaviate` instance\n* install libraries\n    * `weaviate-client`\n    * `datasets`\n    * `apache-beam`\n* get your [OpenAI API key](https://beta.openai.com/account/api-keys)\n\n===========================================================\n### Create a Weaviate instance\n\nTo create a Weaviate instance we have 2 options:\n\n1. (Recommended path) [Weaviate Cloud Service](https://console.weaviate.io/) \u2013 to host your Weaviate instance in the cloud. The free sandbox should be more than enough for this cookbook.\n2. Install and run Weaviate locally with Docker.\n\n#### Option 1 \u2013 WCS Installation Steps\n\nUse [Weaviate Cloud Service](https://console.weaviate.io/) (WCS) to create a free Weaviate cluster.\n1. create a free account and/or login to [WCS](https://console.weaviate.io/)\n2. create a `Weaviate Cluster` with the following settings:\n    * Sandbox: `Sandbox Free`\n    * Weaviate Version: Use default (latest)\n    * OIDC Authentication: `Disabled`\n3. your instance should be ready in a minute or two\n4. make a note of the `Cluster Id`. The link will take you to the full path of your cluster (you will need it later to connect to it)."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 6d957711-ce2a-4095-8c9d-8973e65f9922
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"6d957711-ce2a-4095-8c9d-8973e65f9922\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f73f419e-2b6f-4043-b065-53b2c3d27ea0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"4530bedcb85f5dca9e36ad0f00e6c7cc12b4212403ee25931b8f28c67f4b8508\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"14b419d9-fe29-4c1c-b4ad-2658743bc946\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7ad7fb8539e9286e0cf00e88c3c24ee05499ef790f3094e86a530be0e7549d53\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cb6b98b8-d126-47fe-8c6a-1778863efe6e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0f67a17142f8df7346f11f04c1fe1c199dc9ae6e932dbc53902ebb7b407f029f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4105, \"end_char_idx\": 7308, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "doc_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "ref_doc_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "chroma:document": "All you need to do is:\n1. provide your OpenAI API Key \u2013 when you connected to the Weaviate Client\n2. define which OpenAI vectorizer to use in your Schema\n\n## Prerequisites\n\nBefore we start this project, we need setup the following:\n\n* create a `Weaviate` instance\n* install libraries\n    * `weaviate-client`\n    * `datasets`\n    * `apache-beam`\n* get your [OpenAI API key](https://beta.openai.com/account/api-keys)\n\n===========================================================\n### Create a Weaviate instance\n\nTo create a Weaviate instance we have 2 options:\n\n1. (Recommended path) [Weaviate Cloud Service](https://console.weaviate.io/) \u2013 to host your Weaviate instance in the cloud. The free sandbox should be more than enough for this cookbook.\n2. Install and run Weaviate locally with Docker.\n\n#### Option 1 \u2013 WCS Installation Steps\n\nUse [Weaviate Cloud Service](https://console.weaviate.io/) (WCS) to create a free Weaviate cluster.\n1. create a free account and/or login to [WCS](https://console.weaviate.io/)\n2. create a `Weaviate Cluster` with the following settings:\n    * Sandbox: `Sandbox Free`\n    * Weaviate Version: Use default (latest)\n    * OIDC Authentication: `Disabled`\n3. your instance should be ready in a minute or two\n4. make a note of the `Cluster Id`. The link will take you to the full path of your cluster (you will need it later to connect to it). It should be something like: `https://your-project-name.weaviate.network` \n\n#### Option 2 \u2013 local Weaviate instance with Docker\n\nInstall and run Weaviate locally with Docker.\n1. Download the [./docker-compose.yml](./docker-compose.yml) file\n2. Then open your terminal, navigate to where your docker-compose.yml file is located, and start docker with: `docker-compose up -d`\n3. Once this is ready, your instance should be available at [http://localhost:8080](http://localhost:8080)\n\nNote. To shut down your docker instance you can call: `docker-compose down`\n\n##### Learn more\nTo learn more, about using Weaviate with Docker see the [installation documentation](https://weaviate.io/developers/weaviate/installation/docker-compose).\n\n===========================================================    \n## Install required libraries\n\nBefore running this project make sure to have the following libraries:\n\n### Weaviate Python client\n\nThe [Weaviate Python client](https://weaviate.io/developers/weaviate/client-libraries/python) allows you to communicate with your Weaviate instance from your Python project.\n\n### datasets & apache-beam\n\nTo load sample data, you need the `datasets` library and its' dependency `apache-beam`.\n\n\n```python\n# Install the Weaviate client for Python\n!pip install weaviate-client>3.11.0\n\n# Install datasets and apache-beam to load the sample datasets\n!pip install datasets apache-beam\n```\n\n===========================================================\n## Prepare your OpenAI API key\n\nThe `OpenAI API key` is used for vectorization of your data at import, and for running queries.\n\nIf you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n\nOnce you get your key, please add it to your environment variables as `OPENAI_API_KEY`."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: cb6b98b8-d126-47fe-8c6a-1778863efe6e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"cb6b98b8-d126-47fe-8c6a-1778863efe6e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f73f419e-2b6f-4043-b065-53b2c3d27ea0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"4530bedcb85f5dca9e36ad0f00e6c7cc12b4212403ee25931b8f28c67f4b8508\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6d957711-ce2a-4095-8c9d-8973e65f9922\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"0486adcb802ab8c8260a4d09fd2f4c322e3bcacf8fccf27de1186678b9cc4367\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0377cd52-9198-4dd0-8032-e6b327238bc3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"509856b015946197b089046edafeacd7d5815dd72cffb2e1a5e1db1b7ce35bb3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5652, \"end_char_idx\": 8292, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "doc_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "ref_doc_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "chroma:document": "1. Download the [./docker-compose.yml](./docker-compose.yml) file\n2. Then open your terminal, navigate to where your docker-compose.yml file is located, and start docker with: `docker-compose up -d`\n3. Once this is ready, your instance should be available at [http://localhost:8080](http://localhost:8080)\n\nNote. To shut down your docker instance you can call: `docker-compose down`\n\n##### Learn more\nTo learn more, about using Weaviate with Docker see the [installation documentation](https://weaviate.io/developers/weaviate/installation/docker-compose).\n\n===========================================================    \n## Install required libraries\n\nBefore running this project make sure to have the following libraries:\n\n### Weaviate Python client\n\nThe [Weaviate Python client](https://weaviate.io/developers/weaviate/client-libraries/python) allows you to communicate with your Weaviate instance from your Python project.\n\n### datasets & apache-beam\n\nTo load sample data, you need the `datasets` library and its' dependency `apache-beam`.\n\n\n```python\n# Install the Weaviate client for Python\n!pip install weaviate-client>3.11.0\n\n# Install datasets and apache-beam to load the sample datasets\n!pip install datasets apache-beam\n```\n\n===========================================================\n## Prepare your OpenAI API key\n\nThe `OpenAI API key` is used for vectorization of your data at import, and for running queries.\n\nIf you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n\nOnce you get your key, please add it to your environment variables as `OPENAI_API_KEY`.\n\n\n```python\n# Export OpenAI API Key\n!export OPENAI_API_KEY=\"your key\"\n```\n\n\n```python\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\nimport os\n\n# Note. alternatively you can set a temporary env variable like this:\n# os.environ['OPENAI_API_KEY'] = 'your-key-goes-here'\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    print (\"OPENAI_API_KEY is ready\")\nelse:\n    print (\"OPENAI_API_KEY environment variable not found\")\n```\n\n## Connect to your Weaviate instance\n\nIn this section, we will:\n\n1. test env variable `OPENAI_API_KEY` \u2013 **make sure** you completed the step in [#Prepare-your-OpenAI-API-key](#Prepare-your-OpenAI-API-key)\n2. connect to your Weaviate your `OpenAI API Key`\n3. and test the client connection\n\n### The client \n\nAfter this step, the `client` object will be used to perform all Weaviate-related operations."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 0377cd52-9198-4dd0-8032-e6b327238bc3
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"0377cd52-9198-4dd0-8032-e6b327238bc3\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f73f419e-2b6f-4043-b065-53b2c3d27ea0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"4530bedcb85f5dca9e36ad0f00e6c7cc12b4212403ee25931b8f28c67f4b8508\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cb6b98b8-d126-47fe-8c6a-1778863efe6e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"afd09fe3689ca2d970ab31522ce46def1b32d97884c1063f86086f6ac27ad67b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b77c0252-7813-47ad-b273-e2a0342ad1bf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7dd05b88a771409019501c297fa54b8586656eb118e2be5f24f062c3d1027c76\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6697, \"end_char_idx\": 9779, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "doc_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "ref_doc_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "chroma:document": "```python\n# Install the Weaviate client for Python\n!pip install weaviate-client>3.11.0\n\n# Install datasets and apache-beam to load the sample datasets\n!pip install datasets apache-beam\n```\n\n===========================================================\n## Prepare your OpenAI API key\n\nThe `OpenAI API key` is used for vectorization of your data at import, and for running queries.\n\nIf you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n\nOnce you get your key, please add it to your environment variables as `OPENAI_API_KEY`.\n\n\n```python\n# Export OpenAI API Key\n!export OPENAI_API_KEY=\"your key\"\n```\n\n\n```python\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\nimport os\n\n# Note. alternatively you can set a temporary env variable like this:\n# os.environ['OPENAI_API_KEY'] = 'your-key-goes-here'\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    print (\"OPENAI_API_KEY is ready\")\nelse:\n    print (\"OPENAI_API_KEY environment variable not found\")\n```\n\n## Connect to your Weaviate instance\n\nIn this section, we will:\n\n1. test env variable `OPENAI_API_KEY` \u2013 **make sure** you completed the step in [#Prepare-your-OpenAI-API-key](#Prepare-your-OpenAI-API-key)\n2. connect to your Weaviate your `OpenAI API Key`\n3. and test the client connection\n\n### The client \n\nAfter this step, the `client` object will be used to perform all Weaviate-related operations.\n\n\n```python\nimport weaviate\nfrom datasets import load_dataset\nimport os\n\n# Connect to your Weaviate instance\nclient = weaviate.Client(\n    url=\"https://your-wcs-instance-name.weaviate.network/\",\n#   url=\"http://localhost:8080/\",\n    auth_client_secret=weaviate.auth.AuthApiKey(api_key=\"<YOUR-WEAVIATE-API-KEY>\"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)\n    additional_headers={\n        \"X-OpenAI-Api-Key\": os.getenv(\"OPENAI_API_KEY\")\n    }\n)\n\n# Check if your instance is live and ready\n# This should return `True`\nclient.is_ready()\n```\n\n# Schema\n\nIn this section, we will:\n1. configure the data schema for your data\n2. select OpenAI module\n\n> This is the second and final step, which requires OpenAI specific configuration.\n> After this step, the rest of instructions wlll only touch on Weaviate, as the OpenAI tasks will be handled automatically.\n\n\n## What is a schema\n\nIn Weaviate you create __schemas__ to capture each of the entities you will be searching.\n\nA schema is how you tell Weaviate:\n* what embedding model should be used to vectorize the data\n* what your data is made of (property names and types)\n* which properties should be vectorized and indexed\n\nIn this cookbook we will use a dataset for `Articles`, which contains:\n* `title`\n* `content`\n* `url`\n\nWe want to vectorize `title` and `content`, but not the `url`.\n\nTo vectorize and query the data, we will use `text-embedding-3-small`."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: b77c0252-7813-47ad-b273-e2a0342ad1bf
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"b77c0252-7813-47ad-b273-e2a0342ad1bf\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f73f419e-2b6f-4043-b065-53b2c3d27ea0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"4530bedcb85f5dca9e36ad0f00e6c7cc12b4212403ee25931b8f28c67f4b8508\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0377cd52-9198-4dd0-8032-e6b327238bc3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"4257731d4b4ac271238c254186d9d8f1393673bead4f8d0dc8366be02502fa48\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"da62f154-3218-42ec-a1b1-ab2555fedcd9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a1f48ce52b5853c71c290635b828a01c4627bb0c89fdad704cfadb21bef7981b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8295, \"end_char_idx\": 11261, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "doc_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "ref_doc_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "chroma:document": "```python\nimport weaviate\nfrom datasets import load_dataset\nimport os\n\n# Connect to your Weaviate instance\nclient = weaviate.Client(\n    url=\"https://your-wcs-instance-name.weaviate.network/\",\n#   url=\"http://localhost:8080/\",\n    auth_client_secret=weaviate.auth.AuthApiKey(api_key=\"<YOUR-WEAVIATE-API-KEY>\"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)\n    additional_headers={\n        \"X-OpenAI-Api-Key\": os.getenv(\"OPENAI_API_KEY\")\n    }\n)\n\n# Check if your instance is live and ready\n# This should return `True`\nclient.is_ready()\n```\n\n# Schema\n\nIn this section, we will:\n1. configure the data schema for your data\n2. select OpenAI module\n\n> This is the second and final step, which requires OpenAI specific configuration.\n> After this step, the rest of instructions wlll only touch on Weaviate, as the OpenAI tasks will be handled automatically.\n\n\n## What is a schema\n\nIn Weaviate you create __schemas__ to capture each of the entities you will be searching.\n\nA schema is how you tell Weaviate:\n* what embedding model should be used to vectorize the data\n* what your data is made of (property names and types)\n* which properties should be vectorized and indexed\n\nIn this cookbook we will use a dataset for `Articles`, which contains:\n* `title`\n* `content`\n* `url`\n\nWe want to vectorize `title` and `content`, but not the `url`.\n\nTo vectorize and query the data, we will use `text-embedding-3-small`.\n\n\n```python\n# Clear up the schema, so that we can recreate it\nclient.schema.delete_all()\nclient.schema.get()\n\n# Define the Schema object to use `text-embedding-3-small` on `title` and `content`, but skip it for `url`\narticle_schema = {\n    \"class\": \"Article\",\n    \"description\": \"A collection of articles\",\n    \"vectorizer\": \"text2vec-openai\",\n    \"moduleConfig\": {\n        \"text2vec-openai\": {\n          \"model\": \"ada\",\n          \"modelVersion\": \"002\",\n          \"type\": \"text\"\n        }\n    },\n    \"properties\": [{\n        \"name\": \"title\",\n        \"description\": \"Title of the article\",\n        \"dataType\": [\"string\"]\n    },\n    {\n        \"name\": \"content\",\n        \"description\": \"Contents of the article\",\n        \"dataType\": [\"text\"]\n    },\n    {\n        \"name\": \"url\",\n        \"description\": \"URL to the article\",\n        \"dataType\": [\"string\"],\n        \"moduleConfig\": { \"text2vec-openai\": { \"skip\": True } }\n    }]\n}\n\n# add the Article schema\nclient.schema.create_class(article_schema)\n\n# get the schema to make sure it worked\nclient.schema.get()\n```\n\n## Import data\n\nIn this section we will:\n1. load the Simple Wikipedia dataset\n2. configure Weaviate Batch import (to make the import more efficient)\n3. import the data into Weaviate\n\n> Note: <br/>\n> Like mentioned before. We don't need to manually vectorize the data.<br/>\n> The [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) module will take care of that."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: da62f154-3218-42ec-a1b1-ab2555fedcd9
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"da62f154-3218-42ec-a1b1-ab2555fedcd9\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f73f419e-2b6f-4043-b065-53b2c3d27ea0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"4530bedcb85f5dca9e36ad0f00e6c7cc12b4212403ee25931b8f28c67f4b8508\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b77c0252-7813-47ad-b273-e2a0342ad1bf\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"61977f71be99473a071e5db90dee17d1ee6e582457b793c445a7ff1140edd7b3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d7030f5c-1f5d-4556-87a0-437098dfdbf9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bd52c142d903f5d4f49c2dcef340cb0228ae08a4910a138e08d9f063452ebe3a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9782, \"end_char_idx\": 12815, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "doc_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "ref_doc_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "chroma:document": "```python\n# Clear up the schema, so that we can recreate it\nclient.schema.delete_all()\nclient.schema.get()\n\n# Define the Schema object to use `text-embedding-3-small` on `title` and `content`, but skip it for `url`\narticle_schema = {\n    \"class\": \"Article\",\n    \"description\": \"A collection of articles\",\n    \"vectorizer\": \"text2vec-openai\",\n    \"moduleConfig\": {\n        \"text2vec-openai\": {\n          \"model\": \"ada\",\n          \"modelVersion\": \"002\",\n          \"type\": \"text\"\n        }\n    },\n    \"properties\": [{\n        \"name\": \"title\",\n        \"description\": \"Title of the article\",\n        \"dataType\": [\"string\"]\n    },\n    {\n        \"name\": \"content\",\n        \"description\": \"Contents of the article\",\n        \"dataType\": [\"text\"]\n    },\n    {\n        \"name\": \"url\",\n        \"description\": \"URL to the article\",\n        \"dataType\": [\"string\"],\n        \"moduleConfig\": { \"text2vec-openai\": { \"skip\": True } }\n    }]\n}\n\n# add the Article schema\nclient.schema.create_class(article_schema)\n\n# get the schema to make sure it worked\nclient.schema.get()\n```\n\n## Import data\n\nIn this section we will:\n1. load the Simple Wikipedia dataset\n2. configure Weaviate Batch import (to make the import more efficient)\n3. import the data into Weaviate\n\n> Note: <br/>\n> Like mentioned before. We don't need to manually vectorize the data.<br/>\n> The [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) module will take care of that.\n\n\n```python\n### STEP 1 - load the dataset\n\nfrom datasets import load_dataset\nfrom typing import List, Iterator\n\n# We'll use the datasets library to pull the Simple Wikipedia dataset for embedding\ndataset = list(load_dataset(\"wikipedia\", \"20220301.simple\")[\"train\"])\n\n# For testing, limited to 2.5k articles for demo purposes\ndataset = dataset[:2_500]\n\n# Limited to 25k articles for larger demo purposes\n# dataset = dataset[:25_000]\n\n# for free OpenAI acounts, you can use 50 objects\n# dataset = dataset[:50]\n```\n\n\n```python\n### Step 2 - configure Weaviate Batch, with\n# - starting batch size of 100\n# - dynamically increase/decrease based on performance\n# - add timeout retries if something goes wrong\n\nclient.batch.configure(\n    batch_size=10, \n    dynamic=True,\n    timeout_retries=3,\n#   callback=None,\n)\n```\n\n\n```python\n### Step 3 - import data\n\nprint(\"Importing Articles\")\n\ncounter=0\n\nwith client.batch as batch:\n    for article in dataset:\n        if (counter %10 == 0):\n            print(f\"Import {counter} / {len(dataset)} \")\n\n        properties = {\n            \"title\": article[\"title\"],\n            \"content\": article[\"text\"],\n            \"url\": article[\"url\"]\n        }\n        \n        batch.add_data_object(properties, \"Article\")\n        counter = counter+1\n\nprint(\"Importing Articles complete\")       \n```\n\n\n```python\n# Test that all data has loaded \u2013 get object count\nresult = (\n    client.query.aggregate(\"Article\")\n    .with_fields(\"meta { count }\")\n    .do()\n)\nprint(\"Object count: \", result[\"data\"][\"Aggregate\"][\"Article\"], \"\\n\")\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: d7030f5c-1f5d-4556-87a0-437098dfdbf9
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"d7030f5c-1f5d-4556-87a0-437098dfdbf9\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f73f419e-2b6f-4043-b065-53b2c3d27ea0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"4530bedcb85f5dca9e36ad0f00e6c7cc12b4212403ee25931b8f28c67f4b8508\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"da62f154-3218-42ec-a1b1-ab2555fedcd9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"d2258b54ced6240d09ca2d4591376185b4a4b749e432f1772342e3c86f8e622b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"65973c5f-7233-46ec-bb28-8d71f103177b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d945252274c65e9f4dd3d0419f1fa9359c7b1ed41db97618bf6174f834c44516\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11264, \"end_char_idx\": 14327, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "doc_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "ref_doc_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "chroma:document": "```python\n### STEP 1 - load the dataset\n\nfrom datasets import load_dataset\nfrom typing import List, Iterator\n\n# We'll use the datasets library to pull the Simple Wikipedia dataset for embedding\ndataset = list(load_dataset(\"wikipedia\", \"20220301.simple\")[\"train\"])\n\n# For testing, limited to 2.5k articles for demo purposes\ndataset = dataset[:2_500]\n\n# Limited to 25k articles for larger demo purposes\n# dataset = dataset[:25_000]\n\n# for free OpenAI acounts, you can use 50 objects\n# dataset = dataset[:50]\n```\n\n\n```python\n### Step 2 - configure Weaviate Batch, with\n# - starting batch size of 100\n# - dynamically increase/decrease based on performance\n# - add timeout retries if something goes wrong\n\nclient.batch.configure(\n    batch_size=10, \n    dynamic=True,\n    timeout_retries=3,\n#   callback=None,\n)\n```\n\n\n```python\n### Step 3 - import data\n\nprint(\"Importing Articles\")\n\ncounter=0\n\nwith client.batch as batch:\n    for article in dataset:\n        if (counter %10 == 0):\n            print(f\"Import {counter} / {len(dataset)} \")\n\n        properties = {\n            \"title\": article[\"title\"],\n            \"content\": article[\"text\"],\n            \"url\": article[\"url\"]\n        }\n        \n        batch.add_data_object(properties, \"Article\")\n        counter = counter+1\n\nprint(\"Importing Articles complete\")       \n```\n\n\n```python\n# Test that all data has loaded \u2013 get object count\nresult = (\n    client.query.aggregate(\"Article\")\n    .with_fields(\"meta { count }\")\n    .do()\n)\nprint(\"Object count: \", result[\"data\"][\"Aggregate\"][\"Article\"], \"\\n\")\n```\n\n\n```python\n# Test one article has worked by checking one object\ntest_article = (\n    client.query\n    .get(\"Article\", [\"title\", \"url\", \"content\"])\n    .with_limit(1)\n    .do()\n)[\"data\"][\"Get\"][\"Article\"][0]\n\nprint(test_article['title'])\nprint(test_article['url'])\nprint(test_article['content'])\n```\n\n### Search Data\n\nAs above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors\n\nLearn more about the `alpha` setting [here](https://weaviate.io/developers/weaviate/api/graphql/vector-search-parameters#hybrid)\n\n\n```python\ndef hybrid_query_weaviate(query, collection_name, alpha_val):\n    \n    nearText = {\n        \"concepts\": [query],\n        \"distance\": 0.7,\n    }\n\n    properties = [\n        \"title\", \"content\", \"url\",\n        \"_additional { score }\"\n    ]\n\n    result = (\n        client.query\n        .get(collection_name, properties)\n        .with_hybrid(nearText, alpha=alpha_val)\n        .with_limit(10)\n        .do()\n    )\n    \n    # Check for errors\n    if (\"errors\" in result):\n        print (\"\\033[91mYou probably have run out of OpenAI API calls for the current minute \u2013 the limit is set at 60 per minute.\")\n        raise Exception(result[\"errors\"][0]['message'])\n    \n    return result[\"data\"][\"Get\"][collection_name]\n```\n\n\n```python\nquery_result = hybrid_query_weaviate(\"modern art in Europe\", \"Article\", 0.5)\n\nfor i, article in enumerate(query_result):\n    print(f\"{i+1}. { article['title']} (Score: {article['_additional']['score']})\")\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 65973c5f-7233-46ec-bb28-8d71f103177b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Hybrid Search", "tokens": 3544, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"65973c5f-7233-46ec-bb28-8d71f103177b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"f73f419e-2b6f-4043-b065-53b2c3d27ea0\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"4530bedcb85f5dca9e36ad0f00e6c7cc12b4212403ee25931b8f28c67f4b8508\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d7030f5c-1f5d-4556-87a0-437098dfdbf9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/hybrid-search-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Hybrid Search\", \"tokens\": 3544, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"66a322683383dda0430ee41955089b74a1745bd0114c2991a5aaa8d1bdf4983f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12818, \"end_char_idx\": 14803, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "doc_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "ref_doc_id": "f73f419e-2b6f-4043-b065-53b2c3d27ea0", "chroma:document": "```python\n# Test one article has worked by checking one object\ntest_article = (\n    client.query\n    .get(\"Article\", [\"title\", \"url\", \"content\"])\n    .with_limit(1)\n    .do()\n)[\"data\"][\"Get\"][\"Article\"][0]\n\nprint(test_article['title'])\nprint(test_article['url'])\nprint(test_article['content'])\n```\n\n### Search Data\n\nAs above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors\n\nLearn more about the `alpha` setting [here](https://weaviate.io/developers/weaviate/api/graphql/vector-search-parameters#hybrid)\n\n\n```python\ndef hybrid_query_weaviate(query, collection_name, alpha_val):\n    \n    nearText = {\n        \"concepts\": [query],\n        \"distance\": 0.7,\n    }\n\n    properties = [\n        \"title\", \"content\", \"url\",\n        \"_additional { score }\"\n    ]\n\n    result = (\n        client.query\n        .get(collection_name, properties)\n        .with_hybrid(nearText, alpha=alpha_val)\n        .with_limit(10)\n        .do()\n    )\n    \n    # Check for errors\n    if (\"errors\" in result):\n        print (\"\\033[91mYou probably have run out of OpenAI API calls for the current minute \u2013 the limit is set at 60 per minute.\")\n        raise Exception(result[\"errors\"][0]['message'])\n    \n    return result[\"data\"][\"Get\"][collection_name]\n```\n\n\n```python\nquery_result = hybrid_query_weaviate(\"modern art in Europe\", \"Article\", 0.5)\n\nfor i, article in enumerate(query_result):\n    print(f\"{i+1}. { article['title']} (Score: {article['_additional']['score']})\")\n```\n\n\n```python\nquery_result = hybrid_query_weaviate(\"Famous battles in Scottish history\", \"Article\", 0.5)\n\nfor i, article in enumerate(query_result):\n    print(f\"{i+1}. { article['title']} (Score: {article['_additional']['score']})\")\n```\n\nThanks for following along, you're now equipped to set up your own vector databases and use embeddings to do all kinds of cool things - enjoy! For more complex use cases please continue to work through other cookbook examples in this repo."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 46135e54-802b-4f79-a42c-217711b502fc
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"46135e54-802b-4f79-a42c-217711b502fc\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d37c0f79-0dbd-46d1-960f-f8658ba0441d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9186d36e3d16e88ea64cbc8d3e9795fb1ee4170dc196a43047f899e68bed2a34\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"675c9432-5ff6-475b-a702-542f66b8cfd9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5f42683117a6882ddfb0c1fe03a12a35edf18df643f375084ed63d3e1f80a4e1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2407, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "doc_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "ref_doc_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "chroma:document": "# Using Weaviate with OpenAI vectorize module for Embeddings Search\n\nThis notebook is prepared for a scenario where:\n* Your data is not vectorized\n* You want to run Vector Search on your data\n* You want to use Weaviate with the OpenAI module ([text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai)), to generate vector embeddings for you.\n\nThis notebook takes you through a simple flow to set up a Weaviate instance, connect to it (with OpenAI API key), configure data schema, import data (which will automatically generate vector embeddings for your data), and run semantic search.\n\nThis is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.\n\n## What is Weaviate\n\nWeaviate is an open-source vector search engine that stores data objects together with their vectors. This allows for combining vector search with structured filtering.\n\nWeaviate uses KNN algorithms to create an vector-optimized index, which allows your queries to run extremely fast. Learn more [here](https://weaviate.io/blog/why-is-vector-search-so-fast).\n\nWeaviate let you use your favorite ML-models, and scale seamlessly into billions of data objects.\n\n### Deployment options\n\nWhatever your scenario or production setup, Weaviate has an option for you. You can deploy Weaviate in the following setups:\n* Self-hosted \u2013 you can deploy Weaviate with docker locally, or any server you want.\n* SaaS \u2013 you can use [Weaviate Cloud Service (WCS)](https://console.weaviate.io/) to host your Weaviate instances.\n* Hybrid-SaaS \u2013 you can deploy Weaviate in your own private Cloud Service.\n\n### Programming languages\n\nWeaviate offers four [client libraries](https://weaviate.io/developers/weaviate/client-libraries), which allow you to communicate from your apps:\n* [Python](https://weaviate.io/developers/weaviate/client-libraries/python)\n* [JavaScript](https://weaviate.io/developers/weaviate/client-libraries/javascript)\n* [Java](https://weaviate.io/developers/weaviate/client-libraries/java)\n* [Go](https://weaviate.io/developers/weaviate/client-libraries/go)\n\nAdditionally, Weaviate has a [REST layer](https://weaviate.io/developers/weaviate/api/rest/objects). Basically you can call Weaviate from any language that supports REST requests."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 675c9432-5ff6-475b-a702-542f66b8cfd9
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"675c9432-5ff6-475b-a702-542f66b8cfd9\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d37c0f79-0dbd-46d1-960f-f8658ba0441d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9186d36e3d16e88ea64cbc8d3e9795fb1ee4170dc196a43047f899e68bed2a34\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"46135e54-802b-4f79-a42c-217711b502fc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"eaa1545e3cc3d51704dac5dc4681b62d7d35cfbac5ddd8e35367e93b9ca8a243\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dd437a19-2760-49ff-b5bc-cfa2a9d436c8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"91e24283b44225b5528d7257722b3aab3d33708477e2e4aac032f4ad58933a12\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 849, \"end_char_idx\": 4013, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "doc_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "ref_doc_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "chroma:document": "## What is Weaviate\n\nWeaviate is an open-source vector search engine that stores data objects together with their vectors. This allows for combining vector search with structured filtering.\n\nWeaviate uses KNN algorithms to create an vector-optimized index, which allows your queries to run extremely fast. Learn more [here](https://weaviate.io/blog/why-is-vector-search-so-fast).\n\nWeaviate let you use your favorite ML-models, and scale seamlessly into billions of data objects.\n\n### Deployment options\n\nWhatever your scenario or production setup, Weaviate has an option for you. You can deploy Weaviate in the following setups:\n* Self-hosted \u2013 you can deploy Weaviate with docker locally, or any server you want.\n* SaaS \u2013 you can use [Weaviate Cloud Service (WCS)](https://console.weaviate.io/) to host your Weaviate instances.\n* Hybrid-SaaS \u2013 you can deploy Weaviate in your own private Cloud Service.\n\n### Programming languages\n\nWeaviate offers four [client libraries](https://weaviate.io/developers/weaviate/client-libraries), which allow you to communicate from your apps:\n* [Python](https://weaviate.io/developers/weaviate/client-libraries/python)\n* [JavaScript](https://weaviate.io/developers/weaviate/client-libraries/javascript)\n* [Java](https://weaviate.io/developers/weaviate/client-libraries/java)\n* [Go](https://weaviate.io/developers/weaviate/client-libraries/go)\n\nAdditionally, Weaviate has a [REST layer](https://weaviate.io/developers/weaviate/api/rest/objects). Basically you can call Weaviate from any language that supports REST requests.\n\n## Demo Flow\nThe demo flow is:\n- **Prerequisites Setup**: Create a Weaviate instance and install the required libraries\n- **Connect**: Connect to your Weaviate instance \n- **Schema Configuration**: Configure the schema of your data\n    - *Note*: Here we can define which OpenAI Embedding Model to use\n    - *Note*: Here we can configure which properties to index\n- **Import data**: Load a demo dataset and import it into Weaviate\n    - *Note*: The import process will automatically index your data - based on the configuration in the schema\n    - *Note*: You don't need to explicitly vectorize your data, Weaviate will communicate with OpenAI to do it for you\n- **Run Queries**: Query \n    - *Note*: You don't need to explicitly vectorize your queries, Weaviate will communicate with OpenAI to do it for you\n\nOnce you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.\n\n## OpenAI Module in Weaviate\nAll Weaviate instances come equipped with the [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) module.\n\nThis module is responsible for handling vectorization during import (or any CRUD operations) and when you run a query.\n\n### No need to manually vectorize data\nThis is great news for you. With [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) you don't need to manually vectorize your data, as Weaviate will call OpenAI for you whenever necessary."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: dd437a19-2760-49ff-b5bc-cfa2a9d436c8
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"dd437a19-2760-49ff-b5bc-cfa2a9d436c8\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d37c0f79-0dbd-46d1-960f-f8658ba0441d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9186d36e3d16e88ea64cbc8d3e9795fb1ee4170dc196a43047f899e68bed2a34\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"675c9432-5ff6-475b-a702-542f66b8cfd9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f350c1268d965cd82d8796d8f98b9049b7207015674f48bc7915f365d569be89\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"238cb912-0ac2-4ce9-b9a7-52e777fdf0a8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4c24e0a97e6fa87af6f52ae729261b8a382095973148c261c75712da4c640237\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2329, \"end_char_idx\": 5386, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "doc_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "ref_doc_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "chroma:document": "Basically you can call Weaviate from any language that supports REST requests.\n\n## Demo Flow\nThe demo flow is:\n- **Prerequisites Setup**: Create a Weaviate instance and install the required libraries\n- **Connect**: Connect to your Weaviate instance \n- **Schema Configuration**: Configure the schema of your data\n    - *Note*: Here we can define which OpenAI Embedding Model to use\n    - *Note*: Here we can configure which properties to index\n- **Import data**: Load a demo dataset and import it into Weaviate\n    - *Note*: The import process will automatically index your data - based on the configuration in the schema\n    - *Note*: You don't need to explicitly vectorize your data, Weaviate will communicate with OpenAI to do it for you\n- **Run Queries**: Query \n    - *Note*: You don't need to explicitly vectorize your queries, Weaviate will communicate with OpenAI to do it for you\n\nOnce you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.\n\n## OpenAI Module in Weaviate\nAll Weaviate instances come equipped with the [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) module.\n\nThis module is responsible for handling vectorization during import (or any CRUD operations) and when you run a query.\n\n### No need to manually vectorize data\nThis is great news for you. With [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) you don't need to manually vectorize your data, as Weaviate will call OpenAI for you whenever necessary.\n\nAll you need to do is:\n1. provide your OpenAI API Key \u2013 when you connected to the Weaviate Client\n2. define which OpenAI vectorizer to use in your Schema\n\n## Prerequisites\n\nBefore we start this project, we need setup the following:\n\n* create a `Weaviate` instance\n* install libraries\n    * `weaviate-client`\n    * `datasets`\n    * `apache-beam`\n* get your [OpenAI API key](https://beta.openai.com/account/api-keys)\n\n===========================================================\n### Create a Weaviate instance\n\nTo create a Weaviate instance we have 2 options:\n\n1. (Recommended path) [Weaviate Cloud Service](https://console.weaviate.io/) \u2013 to host your Weaviate instance in the cloud. The free sandbox should be more than enough for this cookbook.\n2. Install and run Weaviate locally with Docker.\n\n#### Option 1 \u2013 WCS Installation Steps\n\nUse [Weaviate Cloud Service](https://console.weaviate.io/) (WCS) to create a free Weaviate cluster.\n1. create a free account and/or login to [WCS](https://console.weaviate.io/)\n2. create a `Weaviate Cluster` with the following settings:\n    * Sandbox: `Sandbox Free`\n    * Weaviate Version: Use default (latest)\n    * OIDC Authentication: `Disabled`\n3. your instance should be ready in a minute or two\n4. make a note of the `Cluster Id`. The link will take you to the full path of your cluster (you will need it later to connect to it)."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 238cb912-0ac2-4ce9-b9a7-52e777fdf0a8
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"238cb912-0ac2-4ce9-b9a7-52e777fdf0a8\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d37c0f79-0dbd-46d1-960f-f8658ba0441d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9186d36e3d16e88ea64cbc8d3e9795fb1ee4170dc196a43047f899e68bed2a34\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dd437a19-2760-49ff-b5bc-cfa2a9d436c8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"914f8a1bbcf8bbf2950aa50d32456b563a852ef7a70b82559ce488f0d31d9bd7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c4958325-4761-4f82-9d8a-a44094fc9b99\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9fc0c29ae9da2787f6e42fcc65f01d10a280af533d1481233c659dd7ec018426\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4015, \"end_char_idx\": 7218, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "doc_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "ref_doc_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "chroma:document": "All you need to do is:\n1. provide your OpenAI API Key \u2013 when you connected to the Weaviate Client\n2. define which OpenAI vectorizer to use in your Schema\n\n## Prerequisites\n\nBefore we start this project, we need setup the following:\n\n* create a `Weaviate` instance\n* install libraries\n    * `weaviate-client`\n    * `datasets`\n    * `apache-beam`\n* get your [OpenAI API key](https://beta.openai.com/account/api-keys)\n\n===========================================================\n### Create a Weaviate instance\n\nTo create a Weaviate instance we have 2 options:\n\n1. (Recommended path) [Weaviate Cloud Service](https://console.weaviate.io/) \u2013 to host your Weaviate instance in the cloud. The free sandbox should be more than enough for this cookbook.\n2. Install and run Weaviate locally with Docker.\n\n#### Option 1 \u2013 WCS Installation Steps\n\nUse [Weaviate Cloud Service](https://console.weaviate.io/) (WCS) to create a free Weaviate cluster.\n1. create a free account and/or login to [WCS](https://console.weaviate.io/)\n2. create a `Weaviate Cluster` with the following settings:\n    * Sandbox: `Sandbox Free`\n    * Weaviate Version: Use default (latest)\n    * OIDC Authentication: `Disabled`\n3. your instance should be ready in a minute or two\n4. make a note of the `Cluster Id`. The link will take you to the full path of your cluster (you will need it later to connect to it). It should be something like: `https://your-project-name.weaviate.network` \n\n#### Option 2 \u2013 local Weaviate instance with Docker\n\nInstall and run Weaviate locally with Docker.\n1. Download the [./docker-compose.yml](./docker-compose.yml) file\n2. Then open your terminal, navigate to where your docker-compose.yml file is located, and start docker with: `docker-compose up -d`\n3. Once this is ready, your instance should be available at [http://localhost:8080](http://localhost:8080)\n\nNote. To shut down your docker instance you can call: `docker-compose down`\n\n##### Learn more\nTo learn more, about using Weaviate with Docker see the [installation documentation](https://weaviate.io/developers/weaviate/installation/docker-compose).\n\n===========================================================    \n## Install required libraries\n\nBefore running this project make sure to have the following libraries:\n\n### Weaviate Python client\n\nThe [Weaviate Python client](https://weaviate.io/developers/weaviate/client-libraries/python) allows you to communicate with your Weaviate instance from your Python project.\n\n### datasets & apache-beam\n\nTo load sample data, you need the `datasets` library and its dependency `apache-beam`.\n\n\n```python\n# Install the Weaviate client for Python\n!pip install weaviate-client>=3.11.0\n\n# Install datasets and apache-beam to load the sample datasets\n!pip install datasets apache-beam\n```\n\n===========================================================\n## Prepare your OpenAI API key\n\nThe `OpenAI API key` is used for vectorization of your data at import, and for running queries.\n\nIf you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n\nOnce you get your key, please add it to your environment variables as `OPENAI_API_KEY`."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: c4958325-4761-4f82-9d8a-a44094fc9b99
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"c4958325-4761-4f82-9d8a-a44094fc9b99\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d37c0f79-0dbd-46d1-960f-f8658ba0441d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9186d36e3d16e88ea64cbc8d3e9795fb1ee4170dc196a43047f899e68bed2a34\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"238cb912-0ac2-4ce9-b9a7-52e777fdf0a8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"dbbbdc169c9384206130875a86f591859018c69d08101777f693632fb6f8577e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"21bba6fc-08dd-484f-9c18-e0920ada98aa\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e6127a721c83a5f23e7175dcaa40d0529657321f96e63bb58772853cb2a5afaa\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5562, \"end_char_idx\": 8207, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "doc_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "ref_doc_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "chroma:document": "1. Download the [./docker-compose.yml](./docker-compose.yml) file\n2. Then open your terminal, navigate to where your docker-compose.yml file is located, and start docker with: `docker-compose up -d`\n3. Once this is ready, your instance should be available at [http://localhost:8080](http://localhost:8080)\n\nNote. To shut down your docker instance you can call: `docker-compose down`\n\n##### Learn more\nTo learn more, about using Weaviate with Docker see the [installation documentation](https://weaviate.io/developers/weaviate/installation/docker-compose).\n\n===========================================================    \n## Install required libraries\n\nBefore running this project make sure to have the following libraries:\n\n### Weaviate Python client\n\nThe [Weaviate Python client](https://weaviate.io/developers/weaviate/client-libraries/python) allows you to communicate with your Weaviate instance from your Python project.\n\n### datasets & apache-beam\n\nTo load sample data, you need the `datasets` library and its dependency `apache-beam`.\n\n\n```python\n# Install the Weaviate client for Python\n!pip install weaviate-client>=3.11.0\n\n# Install datasets and apache-beam to load the sample datasets\n!pip install datasets apache-beam\n```\n\n===========================================================\n## Prepare your OpenAI API key\n\nThe `OpenAI API key` is used for vectorization of your data at import, and for running queries.\n\nIf you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n\nOnce you get your key, please add it to your environment variables as `OPENAI_API_KEY`.\n\n\n```python\n# Export OpenAI API Key\n!export OPENAI_API_KEY=\"your key\"\n```\n\n\n```python\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\nimport os\n\n# Note. alternatively you can set a temporary env variable like this:\n# os.environ[\"OPENAI_API_KEY\"] = 'your-key-goes-here'\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    print (\"OPENAI_API_KEY is ready\")\nelse:\n    print (\"OPENAI_API_KEY environment variable not found\")\n```\n\n## Connect to your Weaviate instance\n\nIn this section, we will:\n\n1. test env variable `OPENAI_API_KEY` \u2013 **make sure** you completed the step in [#Prepare-your-OpenAI-API-key](#Prepare-your-OpenAI-API-key)\n2. connect to your Weaviate with your `OpenAI API Key`\n3. and test the client connection\n\n### The client \n\nAfter this step, the `client` object will be used to perform all Weaviate-related operations."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 21bba6fc-08dd-484f-9c18-e0920ada98aa
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"21bba6fc-08dd-484f-9c18-e0920ada98aa\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d37c0f79-0dbd-46d1-960f-f8658ba0441d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9186d36e3d16e88ea64cbc8d3e9795fb1ee4170dc196a43047f899e68bed2a34\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c4958325-4761-4f82-9d8a-a44094fc9b99\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9045829a956c277a2f154d7c8615a1354ee3334412b7ce8b197763c96d06e004\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f7df4b3d-be5d-4b26-bff6-876e875aa910\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e1850fb6310398320760051daae4cd5d30ac778e8caba3f319dd260a2e97e4df\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6606, \"end_char_idx\": 9696, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "doc_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "ref_doc_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "chroma:document": "```python\n# Install the Weaviate client for Python\n!pip install weaviate-client>=3.11.0\n\n# Install datasets and apache-beam to load the sample datasets\n!pip install datasets apache-beam\n```\n\n===========================================================\n## Prepare your OpenAI API key\n\nThe `OpenAI API key` is used for vectorization of your data at import, and for running queries.\n\nIf you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n\nOnce you get your key, please add it to your environment variables as `OPENAI_API_KEY`.\n\n\n```python\n# Export OpenAI API Key\n!export OPENAI_API_KEY=\"your key\"\n```\n\n\n```python\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\nimport os\n\n# Note. alternatively you can set a temporary env variable like this:\n# os.environ[\"OPENAI_API_KEY\"] = 'your-key-goes-here'\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    print (\"OPENAI_API_KEY is ready\")\nelse:\n    print (\"OPENAI_API_KEY environment variable not found\")\n```\n\n## Connect to your Weaviate instance\n\nIn this section, we will:\n\n1. test env variable `OPENAI_API_KEY` \u2013 **make sure** you completed the step in [#Prepare-your-OpenAI-API-key](#Prepare-your-OpenAI-API-key)\n2. connect to your Weaviate with your `OpenAI API Key`\n3. and test the client connection\n\n### The client \n\nAfter this step, the `client` object will be used to perform all Weaviate-related operations.\n\n\n```python\nimport weaviate\nfrom datasets import load_dataset\nimport os\n\n# Connect to your Weaviate instance\nclient = weaviate.Client(\n    url=\"https://your-wcs-instance-name.weaviate.network/\",\n    # url=\"http://localhost:8080/\",\n    auth_client_secret=weaviate.auth.AuthApiKey(api_key=\"<YOUR-WEAVIATE-API-KEY>\"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)\n    additional_headers={\n        \"X-OpenAI-Api-Key\": os.getenv(\"OPENAI_API_KEY\")\n    }\n)\n\n# Check if your instance is live and ready\n# This should return `True`\nclient.is_ready()\n```\n\n# Schema\n\nIn this section, we will:\n1. configure the data schema for your data\n2. select OpenAI module\n\n> This is the second and final step, which requires OpenAI specific configuration.\n> After this step, the rest of instructions wlll only touch on Weaviate, as the OpenAI tasks will be handled automatically.\n\n\n## What is a schema\n\nIn Weaviate you create __schemas__ to capture each of the entities you will be searching.\n\nA schema is how you tell Weaviate:\n* what embedding model should be used to vectorize the data\n* what your data is made of (property names and types)\n* which properties should be vectorized and indexed\n\nIn this cookbook we will use a dataset for `Articles`, which contains:\n* `title`\n* `content`\n* `url`\n\nWe want to vectorize `title` and `content`, but not the `url`.\n\nTo vectorize and query the data, we will use `text-embedding-3-small`."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: f7df4b3d-be5d-4b26-bff6-876e875aa910
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"f7df4b3d-be5d-4b26-bff6-876e875aa910\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d37c0f79-0dbd-46d1-960f-f8658ba0441d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9186d36e3d16e88ea64cbc8d3e9795fb1ee4170dc196a43047f899e68bed2a34\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"21bba6fc-08dd-484f-9c18-e0920ada98aa\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3ba69cad2205be10e64ca62538ac1ee9fbd46a83d48417bc55b380aab09b3d09\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5213a45c-a31b-4317-a300-c04c75a0fe69\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a1f48ce52b5853c71c290635b828a01c4627bb0c89fdad704cfadb21bef7981b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8210, \"end_char_idx\": 11178, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "doc_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "ref_doc_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "chroma:document": "```python\nimport weaviate\nfrom datasets import load_dataset\nimport os\n\n# Connect to your Weaviate instance\nclient = weaviate.Client(\n    url=\"https://your-wcs-instance-name.weaviate.network/\",\n    # url=\"http://localhost:8080/\",\n    auth_client_secret=weaviate.auth.AuthApiKey(api_key=\"<YOUR-WEAVIATE-API-KEY>\"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)\n    additional_headers={\n        \"X-OpenAI-Api-Key\": os.getenv(\"OPENAI_API_KEY\")\n    }\n)\n\n# Check if your instance is live and ready\n# This should return `True`\nclient.is_ready()\n```\n\n# Schema\n\nIn this section, we will:\n1. configure the data schema for your data\n2. select OpenAI module\n\n> This is the second and final step, which requires OpenAI specific configuration.\n> After this step, the rest of instructions wlll only touch on Weaviate, as the OpenAI tasks will be handled automatically.\n\n\n## What is a schema\n\nIn Weaviate you create __schemas__ to capture each of the entities you will be searching.\n\nA schema is how you tell Weaviate:\n* what embedding model should be used to vectorize the data\n* what your data is made of (property names and types)\n* which properties should be vectorized and indexed\n\nIn this cookbook we will use a dataset for `Articles`, which contains:\n* `title`\n* `content`\n* `url`\n\nWe want to vectorize `title` and `content`, but not the `url`.\n\nTo vectorize and query the data, we will use `text-embedding-3-small`.\n\n\n```python\n# Clear up the schema, so that we can recreate it\nclient.schema.delete_all()\nclient.schema.get()\n\n# Define the Schema object to use `text-embedding-3-small` on `title` and `content`, but skip it for `url`\narticle_schema = {\n    \"class\": \"Article\",\n    \"description\": \"A collection of articles\",\n    \"vectorizer\": \"text2vec-openai\",\n    \"moduleConfig\": {\n        \"text2vec-openai\": {\n          \"model\": \"ada\",\n          \"modelVersion\": \"002\",\n          \"type\": \"text\"\n        }\n    },\n    \"properties\": [{\n        \"name\": \"title\",\n        \"description\": \"Title of the article\",\n        \"dataType\": [\"string\"]\n    },\n    {\n        \"name\": \"content\",\n        \"description\": \"Contents of the article\",\n        \"dataType\": [\"text\"]\n    },\n    {\n        \"name\": \"url\",\n        \"description\": \"URL to the article\",\n        \"dataType\": [\"string\"],\n        \"moduleConfig\": { \"text2vec-openai\": { \"skip\": True } }\n    }]\n}\n\n# add the Article schema\nclient.schema.create_class(article_schema)\n\n# get the schema to make sure it worked\nclient.schema.get()\n```\n\n## Import data\n\nIn this section we will:\n1. load the Simple Wikipedia dataset\n2. configure Weaviate Batch import (to make the import more efficient)\n3. import the data into Weaviate\n\n> Note: <br/>\n> Like mentioned before. We don't need to manually vectorize the data.<br/>\n> The [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) module will take care of that."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 5213a45c-a31b-4317-a300-c04c75a0fe69
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"5213a45c-a31b-4317-a300-c04c75a0fe69\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d37c0f79-0dbd-46d1-960f-f8658ba0441d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9186d36e3d16e88ea64cbc8d3e9795fb1ee4170dc196a43047f899e68bed2a34\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f7df4b3d-be5d-4b26-bff6-876e875aa910\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"d08014f6c991cd5cef8dcf4619ce60c70080f9307b9cea134adb536878c9d365\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"47e0584b-3571-47a4-b3db-b97c65137d6a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9f2b67e6cb5a1ec3cbeebe42850a2d75b0af30299e32337e52da40c3c1269f48\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9699, \"end_char_idx\": 12732, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "doc_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "ref_doc_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "chroma:document": "```python\n# Clear up the schema, so that we can recreate it\nclient.schema.delete_all()\nclient.schema.get()\n\n# Define the Schema object to use `text-embedding-3-small` on `title` and `content`, but skip it for `url`\narticle_schema = {\n    \"class\": \"Article\",\n    \"description\": \"A collection of articles\",\n    \"vectorizer\": \"text2vec-openai\",\n    \"moduleConfig\": {\n        \"text2vec-openai\": {\n          \"model\": \"ada\",\n          \"modelVersion\": \"002\",\n          \"type\": \"text\"\n        }\n    },\n    \"properties\": [{\n        \"name\": \"title\",\n        \"description\": \"Title of the article\",\n        \"dataType\": [\"string\"]\n    },\n    {\n        \"name\": \"content\",\n        \"description\": \"Contents of the article\",\n        \"dataType\": [\"text\"]\n    },\n    {\n        \"name\": \"url\",\n        \"description\": \"URL to the article\",\n        \"dataType\": [\"string\"],\n        \"moduleConfig\": { \"text2vec-openai\": { \"skip\": True } }\n    }]\n}\n\n# add the Article schema\nclient.schema.create_class(article_schema)\n\n# get the schema to make sure it worked\nclient.schema.get()\n```\n\n## Import data\n\nIn this section we will:\n1. load the Simple Wikipedia dataset\n2. configure Weaviate Batch import (to make the import more efficient)\n3. import the data into Weaviate\n\n> Note: <br/>\n> Like mentioned before. We don't need to manually vectorize the data.<br/>\n> The [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) module will take care of that.\n\n\n```python\n### STEP 1 - load the dataset\n\nfrom datasets import load_dataset\nfrom typing import List, Iterator\n\n# We'll use the datasets library to pull the Simple Wikipedia dataset for embedding\ndataset = list(load_dataset(\"wikipedia\", \"20220301.simple\")[\"train\"])\n\n# For testing, limited to 2.5k articles for demo purposes\ndataset = dataset[:2_500]\n\n# Limited to 25k articles for larger demo purposes\n# dataset = dataset[:25_000]\n\n# for free OpenAI acounts, you can use 50 objects\n# dataset = dataset[:50]\n```\n\n\n```python\n### Step 2 - configure Weaviate Batch, with\n# - starting batch size of 100\n# - dynamically increase/decrease based on performance\n# - add timeout retries if something goes wrong\n\nclient.batch.configure(\n    batch_size=10, \n    dynamic=True,\n    timeout_retries=3,\n#   callback=None,\n)\n```\n\n\n```python\n### Step 3 - import data\n\nprint(\"Importing Articles\")\n\ncounter=0\n\nwith client.batch as batch:\n    for article in dataset:\n        if (counter %10 == 0):\n            print(f\"Import {counter} / {len(dataset)} \")\n\n        properties = {\n            \"title\": article[\"title\"],\n            \"content\": article[\"text\"],\n            \"url\": article[\"url\"]\n        }\n        \n        batch.add_data_object(properties, \"Article\")\n        counter = counter+1\n\nprint(\"Importing Articles complete\")       \n```\n\n\n```python\n# Test that all data has loaded \u2013 get object count\nresult = (\n    client.query.aggregate(\"Article\")\n    .with_fields(\"meta { count }\")\n    .do()\n)\nprint(\"Object count: \", result[\"data\"][\"Aggregate\"][\"Article\"], \"\\n\")\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 47e0584b-3571-47a4-b3db-b97c65137d6a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"47e0584b-3571-47a4-b3db-b97c65137d6a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d37c0f79-0dbd-46d1-960f-f8658ba0441d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9186d36e3d16e88ea64cbc8d3e9795fb1ee4170dc196a43047f899e68bed2a34\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5213a45c-a31b-4317-a300-c04c75a0fe69\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"2576a3e21f502e83682287e411e906c9669e358fa49a029f2931f011cbab9a9b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"982e41c0-2ae1-43c0-9e3e-902d537b1578\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1f9e449e149d81370862abe9ae406b159c3e90a9b8ed34797995edff92c82a07\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11181, \"end_char_idx\": 14095, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "doc_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "ref_doc_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "chroma:document": "```python\n### STEP 1 - load the dataset\n\nfrom datasets import load_dataset\nfrom typing import List, Iterator\n\n# We'll use the datasets library to pull the Simple Wikipedia dataset for embedding\ndataset = list(load_dataset(\"wikipedia\", \"20220301.simple\")[\"train\"])\n\n# For testing, limited to 2.5k articles for demo purposes\ndataset = dataset[:2_500]\n\n# Limited to 25k articles for larger demo purposes\n# dataset = dataset[:25_000]\n\n# for free OpenAI acounts, you can use 50 objects\n# dataset = dataset[:50]\n```\n\n\n```python\n### Step 2 - configure Weaviate Batch, with\n# - starting batch size of 100\n# - dynamically increase/decrease based on performance\n# - add timeout retries if something goes wrong\n\nclient.batch.configure(\n    batch_size=10, \n    dynamic=True,\n    timeout_retries=3,\n#   callback=None,\n)\n```\n\n\n```python\n### Step 3 - import data\n\nprint(\"Importing Articles\")\n\ncounter=0\n\nwith client.batch as batch:\n    for article in dataset:\n        if (counter %10 == 0):\n            print(f\"Import {counter} / {len(dataset)} \")\n\n        properties = {\n            \"title\": article[\"title\"],\n            \"content\": article[\"text\"],\n            \"url\": article[\"url\"]\n        }\n        \n        batch.add_data_object(properties, \"Article\")\n        counter = counter+1\n\nprint(\"Importing Articles complete\")       \n```\n\n\n```python\n# Test that all data has loaded \u2013 get object count\nresult = (\n    client.query.aggregate(\"Article\")\n    .with_fields(\"meta { count }\")\n    .do()\n)\nprint(\"Object count: \", result[\"data\"][\"Aggregate\"][\"Article\"], \"\\n\")\n```\n\n\n```python\n# Test one article has worked by checking one object\ntest_article = (\n    client.query\n    .get(\"Article\", [\"title\", \"url\", \"content\"])\n    .with_limit(1)\n    .do()\n)[\"data\"][\"Get\"][\"Article\"][0]\n\nprint(test_article['title'])\nprint(test_article['url'])\nprint(test_article['content'])\n```\n\n### Search Data\n\nAs above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors\n\n\n```python\ndef query_weaviate(query, collection_name):\n    \n    nearText = {\n        \"concepts\": [query],\n        \"distance\": 0.7,\n    }\n\n    properties = [\n        \"title\", \"content\", \"url\",\n        \"_additional {certainty distance}\"\n    ]\n\n    result = (\n        client.query\n        .get(collection_name, properties)\n        .with_near_text(nearText)\n        .with_limit(10)\n        .do()\n    )\n    \n    # Check for errors\n    if (\"errors\" in result):\n        print (\"\\033[91mYou probably have run out of OpenAI API calls for the current minute \u2013 the limit is set at 60 per minute.\")\n        raise Exception(result[\"errors\"][0]['message'])\n    \n    return result[\"data\"][\"Get\"][collection_name]\n```\n\n\n```python\nquery_result = query_weaviate(\"modern art in Europe\", \"Article\")\n\nfor i, article in enumerate(query_result):\n    print(f\"{i+1}. { article['title']} (Score: {round(article['_additional']['certainty'],3) })\")\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 982e41c0-2ae1-43c0-9e3e-902d537b1578
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb", "title": "Using Weaviate with OpenAI vectorize module for Embeddings Search", "tokens": 3475, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"982e41c0-2ae1-43c0-9e3e-902d537b1578\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d37c0f79-0dbd-46d1-960f-f8658ba0441d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9186d36e3d16e88ea64cbc8d3e9795fb1ee4170dc196a43047f899e68bed2a34\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"47e0584b-3571-47a4-b3db-b97c65137d6a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/getting-started-with-weaviate-and-openai.ipynb\", \"title\": \"Using Weaviate with OpenAI vectorize module for Embeddings Search\", \"tokens\": 3475, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"e9050a30f1840e9f67c07a06f00e64177dfd2e34b6b93e880c9a2996bb0fd0b1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12502, \"end_char_idx\": 14573, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "doc_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "ref_doc_id": "d37c0f79-0dbd-46d1-960f-f8658ba0441d", "chroma:document": "```python\n# Test that all data has loaded \u2013 get object count\nresult = (\n    client.query.aggregate(\"Article\")\n    .with_fields(\"meta { count }\")\n    .do()\n)\nprint(\"Object count: \", result[\"data\"][\"Aggregate\"][\"Article\"], \"\\n\")\n```\n\n\n```python\n# Test one article has worked by checking one object\ntest_article = (\n    client.query\n    .get(\"Article\", [\"title\", \"url\", \"content\"])\n    .with_limit(1)\n    .do()\n)[\"data\"][\"Get\"][\"Article\"][0]\n\nprint(test_article['title'])\nprint(test_article['url'])\nprint(test_article['content'])\n```\n\n### Search Data\n\nAs above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors\n\n\n```python\ndef query_weaviate(query, collection_name):\n    \n    nearText = {\n        \"concepts\": [query],\n        \"distance\": 0.7,\n    }\n\n    properties = [\n        \"title\", \"content\", \"url\",\n        \"_additional {certainty distance}\"\n    ]\n\n    result = (\n        client.query\n        .get(collection_name, properties)\n        .with_near_text(nearText)\n        .with_limit(10)\n        .do()\n    )\n    \n    # Check for errors\n    if (\"errors\" in result):\n        print (\"\\033[91mYou probably have run out of OpenAI API calls for the current minute \u2013 the limit is set at 60 per minute.\")\n        raise Exception(result[\"errors\"][0]['message'])\n    \n    return result[\"data\"][\"Get\"][collection_name]\n```\n\n\n```python\nquery_result = query_weaviate(\"modern art in Europe\", \"Article\")\n\nfor i, article in enumerate(query_result):\n    print(f\"{i+1}. { article['title']} (Score: {round(article['_additional']['certainty'],3) })\")\n```\n\n\n```python\nquery_result = query_weaviate(\"Famous battles in Scottish history\", \"Article\")\n\nfor i, article in enumerate(query_result):\n    print(f\"{i+1}. { article['title']} (Score: {round(article['_additional']['certainty'],3) })\")\n```\n\nThanks for following along, you're now equipped to set up your own vector databases and use embeddings to do all kinds of cool things - enjoy! For more complex use cases please continue to work through other cookbook examples in this repo."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: c492d33d-7fe9-4ee6-9924-6d770bf81bf0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"c492d33d-7fe9-4ee6-9924-6d770bf81bf0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ecef48c5-024b-4970-bd02-56a7cb90f695\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"60bcb85d-5e6e-4ab0-a9fc-5187130928dd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ea33eec6ce7fe9964cdec82a5a01d5024ec7994955bd2f2cb55509a0cc3dd62d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3520, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "ref_doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "chroma:document": "# Using Weaviate for Embeddings Search\n\nThis notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.\n\n### What is a Vector Database\n\nA vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.\n\n### Why use a Vector Database\n\nVector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.\n\n\n### Demo Flow\nThe demo flow is:\n- **Setup**: Import packages and set any required variables\n- **Load data**: Load a dataset and embed it using OpenAI embeddings\n- **Weaviate**\n    - *Setup*: Here we'll set up the Python client for Weaviate. For more details go [here](https://weaviate.io/developers/weaviate/current/client-libraries/python.html)\n    - *Index Data*: We'll create an index with __title__ search vectors in it\n    - *Search Data*: We'll run a few searches to confirm it works\n\nOnce you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.\n\n## Setup\n\nImport the required libraries and set the embedding model that we'd like to use.\n\n\n```python\n# We'll need to install the Weaviate client\n!pip install weaviate-client\n\n#Install wget to pull zip file\n!pip install wget\n```\n\n\n```python\nimport openai\n\nfrom typing import List, Iterator\nimport pandas as pd\nimport numpy as np\nimport os\nimport wget\nfrom ast import literal_eval\n\n# Weaviate's client library for Python\nimport weaviate\n\n# I've set this to our new embeddings model, this can be changed to the embedding model of your choice\nEMBEDDING_MODEL = \"text-embedding-3-small\"\n\n# Ignore unclosed SSL socket warnings - optional in case you get these errors\nimport warnings\n\nwarnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning) \n```\n\n## Load data\n\nIn this section we'll load embedded data that we've prepared previous to this session.\n\n\n```python\nembeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'\n\n# The file is ~700 MB so this will take some time\nwget.download(embeddings_url)\n```\n\n\n```python\nimport zipfile\nwith zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n    zip_ref.extractall(\"../data\")\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 60bcb85d-5e6e-4ab0-a9fc-5187130928dd
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"60bcb85d-5e6e-4ab0-a9fc-5187130928dd\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ecef48c5-024b-4970-bd02-56a7cb90f695\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c492d33d-7fe9-4ee6-9924-6d770bf81bf0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"c55639b4894da83201da04224277c0e2b3d41084bb7df908bfe8894b45c129de\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d1e3c7cd-30c3-402d-a9c5-b8bbb99209ec\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"55d75d734c6c465942a4bbe99c9e2550ca1dde90d5af183d906f3fc23ae149c3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2319, \"end_char_idx\": 4670, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "ref_doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "chroma:document": "```python\n# We'll need to install the Weaviate client\n!pip install weaviate-client\n\n#Install wget to pull zip file\n!pip install wget\n```\n\n\n```python\nimport openai\n\nfrom typing import List, Iterator\nimport pandas as pd\nimport numpy as np\nimport os\nimport wget\nfrom ast import literal_eval\n\n# Weaviate's client library for Python\nimport weaviate\n\n# I've set this to our new embeddings model, this can be changed to the embedding model of your choice\nEMBEDDING_MODEL = \"text-embedding-3-small\"\n\n# Ignore unclosed SSL socket warnings - optional in case you get these errors\nimport warnings\n\nwarnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning) \n```\n\n## Load data\n\nIn this section we'll load embedded data that we've prepared previous to this session.\n\n\n```python\nembeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'\n\n# The file is ~700 MB so this will take some time\nwget.download(embeddings_url)\n```\n\n\n```python\nimport zipfile\nwith zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n    zip_ref.extractall(\"../data\")\n```\n\n\n```python\narticle_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')\n```\n\n\n```python\narticle_df.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>url</th>\n      <th>title</th>\n      <th>text</th>\n      <th>title_vector</th>\n      <th>content_vector</th>\n      <th>vector_id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>https://simple.wikipedia.org/wiki/April</td>\n      <td>April</td>\n      <td>April is the fourth month of the year in the J...</td>\n      <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n      <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>https://simple.wikipedia.org/wiki/August</td>\n      <td>August</td>\n      <td>August (Aug.)"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: d1e3c7cd-30c3-402d-a9c5-b8bbb99209ec
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"d1e3c7cd-30c3-402d-a9c5-b8bbb99209ec\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ecef48c5-024b-4970-bd02-56a7cb90f695\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"60bcb85d-5e6e-4ab0-a9fc-5187130928dd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"e52ab9730fc78a7c173e9f5cc18540aaa7e7c8a7178a7f4071ad2c19a019dbd5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1db9ed07-ec32-4a10-a0d9-b645f4631e1b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0eb9f1fbcad7c87e0ae1cc80d55f22c888212d66f0901dbcd88725f848899c3c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3523, \"end_char_idx\": 5701, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "ref_doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "chroma:document": "```python\narticle_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')\n```\n\n\n```python\narticle_df.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>url</th>\n      <th>title</th>\n      <th>text</th>\n      <th>title_vector</th>\n      <th>content_vector</th>\n      <th>vector_id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>https://simple.wikipedia.org/wiki/April</td>\n      <td>April</td>\n      <td>April is the fourth month of the year in the J...</td>\n      <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n      <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>https://simple.wikipedia.org/wiki/August</td>\n      <td>August</td>\n      <td>August (Aug.) is the eighth month of the year ...</td>\n      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>6</td>\n      <td>https://simple.wikipedia.org/wiki/Art</td>\n      <td>Art</td>\n      <td>Art is a creative activity that expresses imag...</td>\n      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>8</td>\n      <td>https://simple.wikipedia.org/wiki/A</td>\n      <td>A</td>\n      <td>A or a is the first letter of the English alph...</td>\n      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n      <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>9</td>\n      <td>https://simple.wikipedia.org/wiki/Air</td>\n      <td>Air</td>\n      <td>Air refers to the Earth's atmosphere."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 1db9ed07-ec32-4a10-a0d9-b645f4631e1b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"1db9ed07-ec32-4a10-a0d9-b645f4631e1b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ecef48c5-024b-4970-bd02-56a7cb90f695\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d1e3c7cd-30c3-402d-a9c5-b8bbb99209ec\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"6e7f6ae44a256c97413473634539eeb442908ff2e22cf217d14a8da3e65baee4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"83fcc0ee-9cc4-4d03-9f26-bda3c77705c1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"56e2aa1bde5ed15fe8073faa23455e10c4418bcfff784ccff7bb57b05c25c908\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4671, \"end_char_idx\": 6869, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "ref_doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "chroma:document": "is the eighth month of the year ...</td>\n      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>6</td>\n      <td>https://simple.wikipedia.org/wiki/Art</td>\n      <td>Art</td>\n      <td>Art is a creative activity that expresses imag...</td>\n      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>8</td>\n      <td>https://simple.wikipedia.org/wiki/A</td>\n      <td>A</td>\n      <td>A or a is the first letter of the English alph...</td>\n      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n      <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>9</td>\n      <td>https://simple.wikipedia.org/wiki/Air</td>\n      <td>Air</td>\n      <td>Air refers to the Earth's atmosphere. Air is a...</td>\n      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n      <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n\n```python\n# Read vectors from strings back into a list\narticle_df['title_vector'] = article_df.title_vector.apply(literal_eval)\narticle_df['content_vector'] = article_df.content_vector.apply(literal_eval)\n\n# Set vector_id to be a string\narticle_df['vector_id'] = article_df['vector_id'].apply(str)\n```\n\n\n```python\narticle_df.info(show_counts=True)\n```\n\n    <class 'pandas.core.frame.DataFrame'>\n    RangeIndex: 25000 entries, 0 to 24999\n    Data columns (total 7 columns):\n     #   Column          Non-Null Count  Dtype \n    ---  ------          --------------  ----- \n     0   id              25000 non-null  int64 \n     1   url             25000 non-null  object\n     2   title           25000 non-null  object\n     3   text            25000 non-null  object\n     4   title_vector    25000 non-null  object\n     5   content_vector  25000 non-null  object\n     6   vector_id       25000 non-null  object\n    dtypes: int64(1), object(6)\n    memory usage: 1.3+ MB"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 83fcc0ee-9cc4-4d03-9f26-bda3c77705c1
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"83fcc0ee-9cc4-4d03-9f26-bda3c77705c1\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ecef48c5-024b-4970-bd02-56a7cb90f695\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1db9ed07-ec32-4a10-a0d9-b645f4631e1b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f120b37e3b2b0aa4b98e8e3c26ef91952e7bdbdc599479b8d831bafefaa73cdb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2f25fbd0-5d9b-4b0d-915c-8f4cde303639\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"686d4ea9bbc87de3b274fe0ef6b95b9180e4ab82f9db022f1d6861c6c8def44d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5702, \"end_char_idx\": 7367, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "ref_doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "chroma:document": "Air is a...</td>\n      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n      <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n\n```python\n# Read vectors from strings back into a list\narticle_df['title_vector'] = article_df.title_vector.apply(literal_eval)\narticle_df['content_vector'] = article_df.content_vector.apply(literal_eval)\n\n# Set vector_id to be a string\narticle_df['vector_id'] = article_df['vector_id'].apply(str)\n```\n\n\n```python\narticle_df.info(show_counts=True)\n```\n\n    <class 'pandas.core.frame.DataFrame'>\n    RangeIndex: 25000 entries, 0 to 24999\n    Data columns (total 7 columns):\n     #   Column          Non-Null Count  Dtype \n    ---  ------          --------------  ----- \n     0   id              25000 non-null  int64 \n     1   url             25000 non-null  object\n     2   title           25000 non-null  object\n     3   text            25000 non-null  object\n     4   title_vector    25000 non-null  object\n     5   content_vector  25000 non-null  object\n     6   vector_id       25000 non-null  object\n    dtypes: int64(1), object(6)\n    memory usage: 1.3+ MB\n\n\n## Weaviate\n\nAnother vector database option we'll explore is **Weaviate**, which offers both a managed, [SaaS](https://console.weaviate.io/) option, as well as a self-hosted [open source](https://github.com/weaviate/weaviate) option. As we've already looked at a cloud vector database, we'll try the self-hosted option here.\n\nFor this we will:\n- Set up a local deployment of Weaviate\n- Create indices in Weaviate\n- Store our data there\n- Fire some similarity search queries\n- Try a real use case"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 2f25fbd0-5d9b-4b0d-915c-8f4cde303639
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"2f25fbd0-5d9b-4b0d-915c-8f4cde303639\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ecef48c5-024b-4970-bd02-56a7cb90f695\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"83fcc0ee-9cc4-4d03-9f26-bda3c77705c1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a11644a5aa3252312c197afde75ef80ddd957da6aec0ccc5c714d1cbb9311bda\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7ecd2798-5ea5-4e6e-a7c0-3eda213ec0b0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"90fecac5f3411c5c078a789fd288daf8432efa3aa262124fce90eb352c3ffb0b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5907, \"end_char_idx\": 9346, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "ref_doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "chroma:document": "```python\n# Read vectors from strings back into a list\narticle_df['title_vector'] = article_df.title_vector.apply(literal_eval)\narticle_df['content_vector'] = article_df.content_vector.apply(literal_eval)\n\n# Set vector_id to be a string\narticle_df['vector_id'] = article_df['vector_id'].apply(str)\n```\n\n\n```python\narticle_df.info(show_counts=True)\n```\n\n    <class 'pandas.core.frame.DataFrame'>\n    RangeIndex: 25000 entries, 0 to 24999\n    Data columns (total 7 columns):\n     #   Column          Non-Null Count  Dtype \n    ---  ------          --------------  ----- \n     0   id              25000 non-null  int64 \n     1   url             25000 non-null  object\n     2   title           25000 non-null  object\n     3   text            25000 non-null  object\n     4   title_vector    25000 non-null  object\n     5   content_vector  25000 non-null  object\n     6   vector_id       25000 non-null  object\n    dtypes: int64(1), object(6)\n    memory usage: 1.3+ MB\n\n\n## Weaviate\n\nAnother vector database option we'll explore is **Weaviate**, which offers both a managed, [SaaS](https://console.weaviate.io/) option, as well as a self-hosted [open source](https://github.com/weaviate/weaviate) option. As we've already looked at a cloud vector database, we'll try the self-hosted option here.\n\nFor this we will:\n- Set up a local deployment of Weaviate\n- Create indices in Weaviate\n- Store our data there\n- Fire some similarity search queries\n- Try a real use case\n\n\n### Bring your own vectors approach\nIn this cookbook, we provide the data with already generated vectors. This is a good approach for scenarios, where your data is already vectorized.\n\n### Automated vectorization with OpenAI module\nFor scenarios, where your data is not vectorized yet, you can delegate the vectorization task with OpenAI to Weaviate.\nWeaviate offers a built-in module [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai), which takes care of the vectorization for you at:\n* import\n* for any CRUD operations\n* for semantic search\n\nCheck out the [Getting Started with Weaviate and OpenAI module cookbook](./weaviate/getting-started-with-weaviate-and-openai.ipynb) to learn step by step how to import and vectorize data in one step.\n\n### Setup\n\nTo run Weaviate locally, you'll need [Docker](https://www.docker.com/). Following the instructions contained in the Weaviate documentation [here](https://weaviate.io/developers/weaviate/installation/docker-compose), we created an example docker-compose.yml file in this repo saved at [./weaviate/docker-compose.yml](./weaviate/docker-compose.yml).\n\nAfter starting Docker, you can start Weaviate locally by navigating to the `examples/vector_databases/weaviate/` directory and running `docker-compose up -d`.\n\n#### SaaS\nAlternatively you can use [Weaviate Cloud Service](https://console.weaviate.io/) (WCS) to create a free Weaviate cluster.\n1. create a free account and/or login to [WCS](https://console.weaviate.io/)\n2. create a `Weaviate Cluster` with the following settings:\n    * Sandbox: `Sandbox Free`\n    * Weaviate Version: Use default (latest)\n    * OIDC Authentication: `Disabled`\n3. your instance should be ready in a minute or two\n4. make a note of the `Cluster Id`. The link will take you to the full path of your cluster (you will need it later to connect to it). It should be something like: `https://your-project-name-suffix.weaviate.network`"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 7ecd2798-5ea5-4e6e-a7c0-3eda213ec0b0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"7ecd2798-5ea5-4e6e-a7c0-3eda213ec0b0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ecef48c5-024b-4970-bd02-56a7cb90f695\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2f25fbd0-5d9b-4b0d-915c-8f4cde303639\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f779f991bb87dd122ed6185541715327cdd4094a4493ac98bc1847bec7429871\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dd540420-1137-471b-a2d0-352fc76a6f73\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"982b6e72b18ea116823a5652081d817fc7c69f77f70b1f9af7be82eb4780ea38\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9350, \"end_char_idx\": 11140, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "ref_doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "chroma:document": "```python\n# Option #1 - Self-hosted - Weaviate Open Source \nclient = weaviate.Client(\n    url=\"http://localhost:8080\",\n    additional_headers={\n        \"X-OpenAI-Api-Key\": os.getenv(\"OPENAI_API_KEY\")\n    }\n)\n```\n\n\n```python\n# Option #2 - SaaS - (Weaviate Cloud Service)\nclient = weaviate.Client(\n    url=\"https://your-wcs-instance-name.weaviate.network\",\n    additional_headers={\n        \"X-OpenAI-Api-Key\": os.getenv(\"OPENAI_API_KEY\")\n    }\n)\n```\n\n\n```python\nclient.is_ready()\n```\n\n### Index data\n\nIn Weaviate you create __schemas__ to capture each of the entities you will be searching. \n\nIn this case we'll create a schema called **Article** with the **title** vector from above included for us to search by.\n\nThe next few steps closely follow the documentation Weaviate provides [here](https://weaviate.io/developers/weaviate/quickstart).\n\n\n\n```python\n# Clear up the schema, so that we can recreate it\nclient.schema.delete_all()\nclient.schema.get()\n\n# Define the Schema object to use `text-embedding-3-small` on `title` and `content`, but skip it for `url`\narticle_schema = {\n    \"class\": \"Article\",\n    \"description\": \"A collection of articles\",\n    \"vectorizer\": \"text2vec-openai\",\n    \"moduleConfig\": {\n        \"text2vec-openai\": {\n          \"model\": \"ada\",\n          \"modelVersion\": \"002\",\n          \"type\": \"text\"\n        }\n    },\n    \"properties\": [{\n        \"name\": \"title\",\n        \"description\": \"Title of the article\",\n        \"dataType\": [\"string\"]\n    },\n    {\n        \"name\": \"content\",\n        \"description\": \"Contents of the article\",\n        \"dataType\": [\"text\"],\n        \"moduleConfig\": { \"text2vec-openai\": { \"skip\": True } }\n    }]\n}\n\n# add the Article schema\nclient.schema.create_class(article_schema)\n\n# get the schema to make sure it worked\nclient.schema.get()\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: dd540420-1137-471b-a2d0-352fc76a6f73
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"dd540420-1137-471b-a2d0-352fc76a6f73\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ecef48c5-024b-4970-bd02-56a7cb90f695\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7ecd2798-5ea5-4e6e-a7c0-3eda213ec0b0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9f45cae6bbc7e759a2bc076005dfe583d9c22ddc2f2ba7a5e03e010806085aaa\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0147c603-7d13-4716-ae65-94af41b4ec2d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e6ee35f80c0a64480f769fb7c0d160afc6cd88a0f192af479dec9b9d33e0cd37\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9564, \"end_char_idx\": 12819, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "ref_doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "chroma:document": "```python\n# Option #2 - SaaS - (Weaviate Cloud Service)\nclient = weaviate.Client(\n    url=\"https://your-wcs-instance-name.weaviate.network\",\n    additional_headers={\n        \"X-OpenAI-Api-Key\": os.getenv(\"OPENAI_API_KEY\")\n    }\n)\n```\n\n\n```python\nclient.is_ready()\n```\n\n### Index data\n\nIn Weaviate you create __schemas__ to capture each of the entities you will be searching. \n\nIn this case we'll create a schema called **Article** with the **title** vector from above included for us to search by.\n\nThe next few steps closely follow the documentation Weaviate provides [here](https://weaviate.io/developers/weaviate/quickstart).\n\n\n\n```python\n# Clear up the schema, so that we can recreate it\nclient.schema.delete_all()\nclient.schema.get()\n\n# Define the Schema object to use `text-embedding-3-small` on `title` and `content`, but skip it for `url`\narticle_schema = {\n    \"class\": \"Article\",\n    \"description\": \"A collection of articles\",\n    \"vectorizer\": \"text2vec-openai\",\n    \"moduleConfig\": {\n        \"text2vec-openai\": {\n          \"model\": \"ada\",\n          \"modelVersion\": \"002\",\n          \"type\": \"text\"\n        }\n    },\n    \"properties\": [{\n        \"name\": \"title\",\n        \"description\": \"Title of the article\",\n        \"dataType\": [\"string\"]\n    },\n    {\n        \"name\": \"content\",\n        \"description\": \"Contents of the article\",\n        \"dataType\": [\"text\"],\n        \"moduleConfig\": { \"text2vec-openai\": { \"skip\": True } }\n    }]\n}\n\n# add the Article schema\nclient.schema.create_class(article_schema)\n\n# get the schema to make sure it worked\nclient.schema.get()\n```\n\n\n\n\n    {'classes': [{'class': 'Article',\n       'description': 'A collection of articles',\n       'invertedIndexConfig': {'bm25': {'b': 0.75, 'k1': 1.2},\n        'cleanupIntervalSeconds': 60,\n        'stopwords': {'additions': None, 'preset': 'en', 'removals': None}},\n       'moduleConfig': {'text2vec-openai': {'model': 'ada',\n         'modelVersion': '002',\n         'type': 'text',\n         'vectorizeClassName': True}},\n       'properties': [{'dataType': ['string'],\n         'description': 'Title of the article',\n         'moduleConfig': {'text2vec-openai': {'skip': False,\n           'vectorizePropertyName': False}},\n         'name': 'title',\n         'tokenization': 'word'},\n        {'dataType': ['text'],\n         'description': 'Contents of the article',\n         'moduleConfig': {'text2vec-openai': {'skip': True,\n           'vectorizePropertyName': False}},\n         'name': 'content',\n         'tokenization': 'word'}],\n       'replicationConfig': {'factor': 1},\n       'shardingConfig': {'virtualPerPhysical': 128,\n        'desiredCount': 1,\n        'actualCount': 1,\n        'desiredVirtualCount': 128,\n        'actualVirtualCount': 128,\n        'key': '_id',\n        'strategy': 'hash',\n        'function': 'murmur3'},\n       'vectorIndexConfig': {'skip': False,\n        'cleanupIntervalSeconds': 300,\n        'maxConnections': 64,\n        'efConstruction': 128,\n        'ef': -1,\n        'dynamicEfMin': 100,\n        'dynamicEfMax': 500,\n        'dynamicEfFactor': 8,\n        'vectorCacheMaxObjects': 1000000000000,\n        'flatSearchCutoff': 40000,\n        'distance': 'cosine'},\n       'vectorIndexType': 'hnsw',\n       'vectorizer': 'text2vec-openai'}]}"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 0147c603-7d13-4716-ae65-94af41b4ec2d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"0147c603-7d13-4716-ae65-94af41b4ec2d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ecef48c5-024b-4970-bd02-56a7cb90f695\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dd540420-1137-471b-a2d0-352fc76a6f73\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"e30736ca499de12145dba1d2a701ae675399f0822f28ed30cdb00e7754731f67\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0b359bce-a9bc-41a5-aab2-6636dbc1e9ee\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b826d1f19ce073fb40a7dd4d7b173da3b4357cb1e4c3230cae13b71d465433f5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "ref_doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "chroma:document": "```python\n### Step 1 - configure Weaviate Batch, which optimizes CRUD operations in bulk\n# - starting batch size of 100\n# - dynamically increase/decrease based on performance\n# - add timeout retries if something goes wrong\n\nclient.batch.configure(\n    batch_size=100,\n    dynamic=True,\n    timeout_retries=3,\n)\n```\n\n\n\n\n    <weaviate.batch.crud_batch.Batch at 0x3f0ca0fa0>\n\n\n\n\n```python\n### Step 2 - import data\n\nprint(\"Uploading data with vectors to Article schema.\")\n\ncounter=0\n\nwith client.batch as batch:\n    for k,v in article_df.iterrows():\n        \n        # print update message every 100 objects        \n        if (counter %100 == 0):\n            print(f\"Import {counter} / {len(article_df)} \")\n        \n        properties = {\n            \"title\": v[\"title\"],\n            \"content\": v[\"text\"]\n        }\n        \n        vector = v[\"title_vector\"]\n        \n        batch.add_data_object(properties, \"Article\", None, vector)\n        counter = counter+1\n\nprint(f\"Importing ({len(article_df)}) Articles complete\")  \n```\n\n    Uploading data with vectors to Article schema.\n    Import 0 / 25000 \n    Import 100 / 25000 \n    Import 200 / 25000 \n    Import 300 / 25000 \n    Import 400 / 25000 \n    Import 500 / 25000 \n    Import 600 / 25000 \n    Import 700 / 25000 \n    Import 800 / 25000 \n    Import 900 / 25000 \n    Import 1000 / 25000 \n    Import 1100 / 25000 \n    Import 1200 / 25000 \n    Import 1300 / 25000 \n    Import 1400 / 25000 \n    Import 1500 / 25000 \n    Import 1600 / 25000 \n    Import 1700 / 25000 \n    Import 1800 / 25000 \n    Import 1900 / 25000 \n    Import 2000 / 25000 \n    Import 2100 / 25000 \n    Import 2200 / 25000 \n    Import 2300 / 25000 \n    Import 2400 / 25000 \n    Import 2500 / 25000 \n    Import 2600 / 25000 \n    Import 2700 / 25000 \n    Import 2800 / 25000 \n    Import 2900 / 25000 \n    Import 3000 / 25000 \n    Import 3100 / 25000 \n    Import 3200 / 25000 \n    Import 3300 / 25000 \n    Import 3400 / 25000 \n    Import 3500 / 25000 \n    Import 3600 / 25000 \n    Import 3700 / 25000 \n    Import 3800 / 25000 \n    Import 3900 / 25000 \n    Import 4000 / 25000 \n    Import 4100 / 25000 \n    Import 4200 /"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 0b359bce-a9bc-41a5-aab2-6636dbc1e9ee
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"0b359bce-a9bc-41a5-aab2-6636dbc1e9ee\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ecef48c5-024b-4970-bd02-56a7cb90f695\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0147c603-7d13-4716-ae65-94af41b4ec2d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"271697b691b536cf2b637b974e83f8fe5efec1df0923e46e23a840586a6cc2a6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1c0d5c2f-4353-407a-9858-11d6c9b3b483\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"817b9687fbb19a9f5a55424392c1379fb2b6b748f1b47e451a691cded3994200\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14121, \"end_char_idx\": 15714, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "ref_doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "chroma:document": "Import 900 / 25000 \n    Import 1000 / 25000 \n    Import 1100 / 25000 \n    Import 1200 / 25000 \n    Import 1300 / 25000 \n    Import 1400 / 25000 \n    Import 1500 / 25000 \n    Import 1600 / 25000 \n    Import 1700 / 25000 \n    Import 1800 / 25000 \n    Import 1900 / 25000 \n    Import 2000 / 25000 \n    Import 2100 / 25000 \n    Import 2200 / 25000 \n    Import 2300 / 25000 \n    Import 2400 / 25000 \n    Import 2500 / 25000 \n    Import 2600 / 25000 \n    Import 2700 / 25000 \n    Import 2800 / 25000 \n    Import 2900 / 25000 \n    Import 3000 / 25000 \n    Import 3100 / 25000 \n    Import 3200 / 25000 \n    Import 3300 / 25000 \n    Import 3400 / 25000 \n    Import 3500 / 25000 \n    Import 3600 / 25000 \n    Import 3700 / 25000 \n    Import 3800 / 25000 \n    Import 3900 / 25000 \n    Import 4000 / 25000 \n    Import 4100 / 25000 \n    Import 4200 / 25000 \n    Import 4300 / 25000 \n    Import 4400 / 25000 \n    Import 4500 / 25000 \n    Import 4600 / 25000 \n    Import 4700 / 25000 \n    Import 4800 / 25000 \n    Import 4900 / 25000 \n    Import 5000 / 25000 \n    Import 5100 / 25000 \n    Import 5200 / 25000 \n    Import 5300 / 25000 \n    Import 5400 / 25000 \n    Import 5500 / 25000 \n    Import 5600 / 25000 \n    Import 5700 / 25000 \n    Import 5800 / 25000 \n    Import 5900 / 25000 \n    Import 6000 / 25000 \n    Import 6100 / 25000 \n    Import 6200 / 25000 \n    Import 6300 / 25000 \n    Import 6400 / 25000 \n    Import 6500 / 25000 \n    Import 6600 / 25000 \n    Import 6700 / 25000 \n    Import 6800 / 25000 \n    Import 6900 / 25000 \n    Import 7000 / 25000 \n    Import 7100 / 25000 \n    Import 7200 / 25000"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 1c0d5c2f-4353-407a-9858-11d6c9b3b483
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"1c0d5c2f-4353-407a-9858-11d6c9b3b483\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ecef48c5-024b-4970-bd02-56a7cb90f695\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0b359bce-a9bc-41a5-aab2-6636dbc1e9ee\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"ba8d1b913c114074043bf0fc3d43e18c361163b2f8e8b53a651b7623b2da5096\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b7b139fc-71e9-4a5c-9e58-649ad8ca4664\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4d2127694d0da5e7bdd5b1f8131be3765976240e348bcd183cd3fc898e763d42\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14895, \"end_char_idx\": 16487, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "ref_doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "chroma:document": "Import 4000 / 25000 \n    Import 4100 / 25000 \n    Import 4200 / 25000 \n    Import 4300 / 25000 \n    Import 4400 / 25000 \n    Import 4500 / 25000 \n    Import 4600 / 25000 \n    Import 4700 / 25000 \n    Import 4800 / 25000 \n    Import 4900 / 25000 \n    Import 5000 / 25000 \n    Import 5100 / 25000 \n    Import 5200 / 25000 \n    Import 5300 / 25000 \n    Import 5400 / 25000 \n    Import 5500 / 25000 \n    Import 5600 / 25000 \n    Import 5700 / 25000 \n    Import 5800 / 25000 \n    Import 5900 / 25000 \n    Import 6000 / 25000 \n    Import 6100 / 25000 \n    Import 6200 / 25000 \n    Import 6300 / 25000 \n    Import 6400 / 25000 \n    Import 6500 / 25000 \n    Import 6600 / 25000 \n    Import 6700 / 25000 \n    Import 6800 / 25000 \n    Import 6900 / 25000 \n    Import 7000 / 25000 \n    Import 7100 / 25000 \n    Import 7200 / 25000 \n    Import 7300 / 25000 \n    Import 7400 / 25000 \n    Import 7500 / 25000 \n    Import 7600 / 25000 \n    Import 7700 / 25000 \n    Import 7800 / 25000 \n    Import 7900 / 25000 \n    Import 8000 / 25000 \n    Import 8100 / 25000 \n    Import 8200 / 25000 \n    Import 8300 / 25000 \n    Import 8400 / 25000 \n    Import 8500 / 25000 \n    Import 8600 / 25000 \n    Import 8700 / 25000 \n    Import 8800 / 25000 \n    Import 8900 / 25000 \n    Import 9000 / 25000 \n    Import 9100 / 25000 \n    Import 9200 / 25000 \n    Import 9300 / 25000 \n    Import 9400 / 25000 \n    Import 9500 / 25000 \n    Import 9600 / 25000 \n    Import 9700 / 25000 \n    Import 9800 / 25000 \n    Import 9900 / 25000 \n    Import 10000 / 25000 \n    Import 10100 / 25000 \n    Import 10200 / 25000 \n    Import 10300 /"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: b7b139fc-71e9-4a5c-9e58-649ad8ca4664
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"b7b139fc-71e9-4a5c-9e58-649ad8ca4664\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ecef48c5-024b-4970-bd02-56a7cb90f695\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1c0d5c2f-4353-407a-9858-11d6c9b3b483\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"81c436b5fc63bf39fb797bf9b534253189ac8076757f38595b5b23aa78d8cb5e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f78cea9a-a1f9-4ede-98dc-c05fa25968f3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"033d7a3b510026426048cd08515a88375fe8f2f51bb087c47a2fbccb5bf770f0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15652, \"end_char_idx\": 17273, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "ref_doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "chroma:document": "7000 / 25000 \n    Import 7100 / 25000 \n    Import 7200 / 25000 \n    Import 7300 / 25000 \n    Import 7400 / 25000 \n    Import 7500 / 25000 \n    Import 7600 / 25000 \n    Import 7700 / 25000 \n    Import 7800 / 25000 \n    Import 7900 / 25000 \n    Import 8000 / 25000 \n    Import 8100 / 25000 \n    Import 8200 / 25000 \n    Import 8300 / 25000 \n    Import 8400 / 25000 \n    Import 8500 / 25000 \n    Import 8600 / 25000 \n    Import 8700 / 25000 \n    Import 8800 / 25000 \n    Import 8900 / 25000 \n    Import 9000 / 25000 \n    Import 9100 / 25000 \n    Import 9200 / 25000 \n    Import 9300 / 25000 \n    Import 9400 / 25000 \n    Import 9500 / 25000 \n    Import 9600 / 25000 \n    Import 9700 / 25000 \n    Import 9800 / 25000 \n    Import 9900 / 25000 \n    Import 10000 / 25000 \n    Import 10100 / 25000 \n    Import 10200 / 25000 \n    Import 10300 / 25000 \n    Import 10400 / 25000 \n    Import 10500 / 25000 \n    Import 10600 / 25000 \n    Import 10700 / 25000 \n    Import 10800 / 25000 \n    Import 10900 / 25000 \n    Import 11000 / 25000 \n    Import 11100 / 25000 \n    Import 11200 / 25000 \n    Import 11300 / 25000 \n    Import 11400 / 25000 \n    Import 11500 / 25000 \n    Import 11600 / 25000 \n    Import 11700 / 25000 \n    Import 11800 / 25000 \n    Import 11900 / 25000 \n    Import 12000 / 25000 \n    Import 12100 / 25000 \n    Import 12200 / 25000 \n    Import 12300 / 25000 \n    Import 12400 / 25000 \n    Import 12500 / 25000 \n    Import 12600 / 25000 \n    Import 12700 / 25000 \n    Import 12800 / 25000 \n    Import 12900 / 25000 \n    Import 13000 / 25000 \n    Import 13100 / 25000 \n    Import 13200 / 25000 \n    Import 13300 / 25000"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: f78cea9a-a1f9-4ede-98dc-c05fa25968f3
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"f78cea9a-a1f9-4ede-98dc-c05fa25968f3\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ecef48c5-024b-4970-bd02-56a7cb90f695\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b7b139fc-71e9-4a5c-9e58-649ad8ca4664\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"621fa2aa09ee190dac52a45e9acc7b262e75d0d8ee167ab7dd212d7d303a20c9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"635e3432-689c-478e-9195-c7925a7910ad\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3d8b43043e604db0040ed442e3c16eea436617c0f84150730e58c41785e4191f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16421, \"end_char_idx\": 18073, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "ref_doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "chroma:document": "Import 10100 / 25000 \n    Import 10200 / 25000 \n    Import 10300 / 25000 \n    Import 10400 / 25000 \n    Import 10500 / 25000 \n    Import 10600 / 25000 \n    Import 10700 / 25000 \n    Import 10800 / 25000 \n    Import 10900 / 25000 \n    Import 11000 / 25000 \n    Import 11100 / 25000 \n    Import 11200 / 25000 \n    Import 11300 / 25000 \n    Import 11400 / 25000 \n    Import 11500 / 25000 \n    Import 11600 / 25000 \n    Import 11700 / 25000 \n    Import 11800 / 25000 \n    Import 11900 / 25000 \n    Import 12000 / 25000 \n    Import 12100 / 25000 \n    Import 12200 / 25000 \n    Import 12300 / 25000 \n    Import 12400 / 25000 \n    Import 12500 / 25000 \n    Import 12600 / 25000 \n    Import 12700 / 25000 \n    Import 12800 / 25000 \n    Import 12900 / 25000 \n    Import 13000 / 25000 \n    Import 13100 / 25000 \n    Import 13200 / 25000 \n    Import 13300 / 25000 \n    Import 13400 / 25000 \n    Import 13500 / 25000 \n    Import 13600 / 25000 \n    Import 13700 / 25000 \n    Import 13800 / 25000 \n    Import 13900 / 25000 \n    Import 14000 / 25000 \n    Import 14100 / 25000 \n    Import 14200 / 25000 \n    Import 14300 / 25000 \n    Import 14400 / 25000 \n    Import 14500 / 25000 \n    Import 14600 / 25000 \n    Import 14700 / 25000 \n    Import 14800 / 25000 \n    Import 14900 / 25000 \n    Import 15000 / 25000 \n    Import 15100 / 25000 \n    Import 15200 / 25000 \n    Import 15300 / 25000 \n    Import 15400 / 25000 \n    Import 15500 / 25000 \n    Import 15600 / 25000 \n    Import 15700 / 25000 \n    Import 15800 / 25000 \n    Import 15900 / 25000 \n    Import 16000 / 25000 \n    Import 16100 / 25000 \n    Import 16200 / 25000 \n    Import 16300 / 25000 \n    Import 16400 /"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 635e3432-689c-478e-9195-c7925a7910ad
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"635e3432-689c-478e-9195-c7925a7910ad\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ecef48c5-024b-4970-bd02-56a7cb90f695\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f78cea9a-a1f9-4ede-98dc-c05fa25968f3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"57d3473f8d99999e85bfa488103ac2d47cb7e6cfbe1e12574fcae90421bb3f65\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bfb3ddc0-dd4a-424a-8244-62970640de2a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0fb7cc44d0b07d1653fa1a91c95ed4363d079296d63dbc1f2268b9bd5f39e198\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17208, \"end_char_idx\": 18859, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "ref_doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "chroma:document": "13100 / 25000 \n    Import 13200 / 25000 \n    Import 13300 / 25000 \n    Import 13400 / 25000 \n    Import 13500 / 25000 \n    Import 13600 / 25000 \n    Import 13700 / 25000 \n    Import 13800 / 25000 \n    Import 13900 / 25000 \n    Import 14000 / 25000 \n    Import 14100 / 25000 \n    Import 14200 / 25000 \n    Import 14300 / 25000 \n    Import 14400 / 25000 \n    Import 14500 / 25000 \n    Import 14600 / 25000 \n    Import 14700 / 25000 \n    Import 14800 / 25000 \n    Import 14900 / 25000 \n    Import 15000 / 25000 \n    Import 15100 / 25000 \n    Import 15200 / 25000 \n    Import 15300 / 25000 \n    Import 15400 / 25000 \n    Import 15500 / 25000 \n    Import 15600 / 25000 \n    Import 15700 / 25000 \n    Import 15800 / 25000 \n    Import 15900 / 25000 \n    Import 16000 / 25000 \n    Import 16100 / 25000 \n    Import 16200 / 25000 \n    Import 16300 / 25000 \n    Import 16400 / 25000 \n    Import 16500 / 25000 \n    Import 16600 / 25000 \n    Import 16700 / 25000 \n    Import 16800 / 25000 \n    Import 16900 / 25000 \n    Import 17000 / 25000 \n    Import 17100 / 25000 \n    Import 17200 / 25000 \n    Import 17300 / 25000 \n    Import 17400 / 25000 \n    Import 17500 / 25000 \n    Import 17600 / 25000 \n    Import 17700 / 25000 \n    Import 17800 / 25000 \n    Import 17900 / 25000 \n    Import 18000 / 25000 \n    Import 18100 / 25000 \n    Import 18200 / 25000 \n    Import 18300 / 25000 \n    Import 18400 / 25000 \n    Import 18500 / 25000 \n    Import 18600 / 25000 \n    Import 18700 / 25000 \n    Import 18800 / 25000 \n    Import 18900 / 25000 \n    Import 19000 / 25000 \n    Import 19100 / 25000 \n    Import 19200 / 25000 \n    Import 19300 / 25000 \n    Import 19400 / 25000"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: bfb3ddc0-dd4a-424a-8244-62970640de2a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"bfb3ddc0-dd4a-424a-8244-62970640de2a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ecef48c5-024b-4970-bd02-56a7cb90f695\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"635e3432-689c-478e-9195-c7925a7910ad\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"ca51f04467adba54416ad3c9408e93bb0d4a68670b663aadeb447289c6fe5579\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bd015e80-525b-4ea2-a42e-20bd93442503\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a6e7db12c0c87c5baf1bbc57d8cf8993016903035c85c18fbf6369d1a533aa51\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18007, \"end_char_idx\": 19659, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "ref_doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "chroma:document": "Import 16200 / 25000 \n    Import 16300 / 25000 \n    Import 16400 / 25000 \n    Import 16500 / 25000 \n    Import 16600 / 25000 \n    Import 16700 / 25000 \n    Import 16800 / 25000 \n    Import 16900 / 25000 \n    Import 17000 / 25000 \n    Import 17100 / 25000 \n    Import 17200 / 25000 \n    Import 17300 / 25000 \n    Import 17400 / 25000 \n    Import 17500 / 25000 \n    Import 17600 / 25000 \n    Import 17700 / 25000 \n    Import 17800 / 25000 \n    Import 17900 / 25000 \n    Import 18000 / 25000 \n    Import 18100 / 25000 \n    Import 18200 / 25000 \n    Import 18300 / 25000 \n    Import 18400 / 25000 \n    Import 18500 / 25000 \n    Import 18600 / 25000 \n    Import 18700 / 25000 \n    Import 18800 / 25000 \n    Import 18900 / 25000 \n    Import 19000 / 25000 \n    Import 19100 / 25000 \n    Import 19200 / 25000 \n    Import 19300 / 25000 \n    Import 19400 / 25000 \n    Import 19500 / 25000 \n    Import 19600 / 25000 \n    Import 19700 / 25000 \n    Import 19800 / 25000 \n    Import 19900 / 25000 \n    Import 20000 / 25000 \n    Import 20100 / 25000 \n    Import 20200 / 25000 \n    Import 20300 / 25000 \n    Import 20400 / 25000 \n    Import 20500 / 25000 \n    Import 20600 / 25000 \n    Import 20700 / 25000 \n    Import 20800 / 25000 \n    Import 20900 / 25000 \n    Import 21000 / 25000 \n    Import 21100 / 25000 \n    Import 21200 / 25000 \n    Import 21300 / 25000 \n    Import 21400 / 25000 \n    Import 21500 / 25000 \n    Import 21600 / 25000 \n    Import 21700 / 25000 \n    Import 21800 / 25000 \n    Import 21900 / 25000 \n    Import 22000 / 25000 \n    Import 22100 / 25000 \n    Import 22200 / 25000 \n    Import 22300 / 25000 \n    Import 22400 / 25000 \n    Import 22500 /"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: bd015e80-525b-4ea2-a42e-20bd93442503
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"bd015e80-525b-4ea2-a42e-20bd93442503\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ecef48c5-024b-4970-bd02-56a7cb90f695\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bfb3ddc0-dd4a-424a-8244-62970640de2a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"cae7e4e2a6ad6c634a9f4689751390472ccf53177a5ccb0a53bd64ad9eae131a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c517f4aa-cb2e-4198-a500-71e144fc54fa\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"11ab5009488c061ec53d52b34bdfac3bb1ed7d14eccb9852acc725e337c4c2ee\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18794, \"end_char_idx\": 20330, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "ref_doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "chroma:document": "19200 / 25000 \n    Import 19300 / 25000 \n    Import 19400 / 25000 \n    Import 19500 / 25000 \n    Import 19600 / 25000 \n    Import 19700 / 25000 \n    Import 19800 / 25000 \n    Import 19900 / 25000 \n    Import 20000 / 25000 \n    Import 20100 / 25000 \n    Import 20200 / 25000 \n    Import 20300 / 25000 \n    Import 20400 / 25000 \n    Import 20500 / 25000 \n    Import 20600 / 25000 \n    Import 20700 / 25000 \n    Import 20800 / 25000 \n    Import 20900 / 25000 \n    Import 21000 / 25000 \n    Import 21100 / 25000 \n    Import 21200 / 25000 \n    Import 21300 / 25000 \n    Import 21400 / 25000 \n    Import 21500 / 25000 \n    Import 21600 / 25000 \n    Import 21700 / 25000 \n    Import 21800 / 25000 \n    Import 21900 / 25000 \n    Import 22000 / 25000 \n    Import 22100 / 25000 \n    Import 22200 / 25000 \n    Import 22300 / 25000 \n    Import 22400 / 25000 \n    Import 22500 / 25000 \n    Import 22600 / 25000 \n    Import 22700 / 25000 \n    Import 22800 / 25000 \n    Import 22900 / 25000 \n    Import 23000 / 25000 \n    Import 23100 / 25000 \n    Import 23200 / 25000 \n    Import 23300 / 25000 \n    Import 23400 / 25000 \n    Import 23500 / 25000 \n    Import 23600 / 25000 \n    Import 23700 / 25000 \n    Import 23800 / 25000 \n    Import 23900 / 25000 \n    Import 24000 / 25000 \n    Import 24100 / 25000 \n    Import 24200 / 25000 \n    Import 24300 / 25000 \n    Import 24400 / 25000 \n    Import 24500 / 25000 \n    Import 24600 / 25000 \n    Import 24700 / 25000 \n    Import 24800 / 25000 \n    Import 24900 / 25000 \n    Importing (25000) Articles complete"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: c517f4aa-cb2e-4198-a500-71e144fc54fa
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"c517f4aa-cb2e-4198-a500-71e144fc54fa\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ecef48c5-024b-4970-bd02-56a7cb90f695\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bd015e80-525b-4ea2-a42e-20bd93442503\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"625d61e05c335b7167075913662b13c61ef48de834d0d89f4f524763099d1f07\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e75977e7-5a02-4de3-83c8-65d341fa0f44\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bb61926ac4774cd1afc875939707c911be5d9e348f15974cd3999ad48fa7ef96\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19450, \"end_char_idx\": 21244, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "ref_doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "chroma:document": "/ 25000 \n    Import 21800 / 25000 \n    Import 21900 / 25000 \n    Import 22000 / 25000 \n    Import 22100 / 25000 \n    Import 22200 / 25000 \n    Import 22300 / 25000 \n    Import 22400 / 25000 \n    Import 22500 / 25000 \n    Import 22600 / 25000 \n    Import 22700 / 25000 \n    Import 22800 / 25000 \n    Import 22900 / 25000 \n    Import 23000 / 25000 \n    Import 23100 / 25000 \n    Import 23200 / 25000 \n    Import 23300 / 25000 \n    Import 23400 / 25000 \n    Import 23500 / 25000 \n    Import 23600 / 25000 \n    Import 23700 / 25000 \n    Import 23800 / 25000 \n    Import 23900 / 25000 \n    Import 24000 / 25000 \n    Import 24100 / 25000 \n    Import 24200 / 25000 \n    Import 24300 / 25000 \n    Import 24400 / 25000 \n    Import 24500 / 25000 \n    Import 24600 / 25000 \n    Import 24700 / 25000 \n    Import 24800 / 25000 \n    Import 24900 / 25000 \n    Importing (25000) Articles complete\n\n\n\n```python\n# Test that all data has loaded \u2013 get object count\nresult = (\n    client.query.aggregate(\"Article\")\n    .with_fields(\"meta { count }\")\n    .do()\n)\nprint(\"Object count: \", result[\"data\"][\"Aggregate\"][\"Article\"])\n```\n\n    Object count:  [{'meta': {'count': 25000}}]\n\n\n\n```python\n# Test one article has worked by checking one object\ntest_article = (\n    client.query\n    .get(\"Article\", [\"title\", \"content\", \"_additional {id}\"])\n    .with_limit(1)\n    .do()\n)[\"data\"][\"Get\"][\"Article\"][0]\n\nprint(test_article[\"_additional\"][\"id\"])\nprint(test_article[\"title\"])\nprint(test_article[\"content\"])\n```\n\n    000393f2-1182-4e3d-abcf-4217eda64be0\n    Lago d'Origlio\n    Lago d'Origlio is a lake in the municipality of Origlio, in Ticino, Switzerland.\n    \n    Lakes of Ticino\n\n\n### Search data\n\nAs above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: e75977e7-5a02-4de3-83c8-65d341fa0f44
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"e75977e7-5a02-4de3-83c8-65d341fa0f44\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ecef48c5-024b-4970-bd02-56a7cb90f695\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c517f4aa-cb2e-4198-a500-71e144fc54fa\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"09a3f50d4207592491d87f4a05a1e78015e0c33526c936b4f47f1ea9c8cd99da\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"97a93e0f-7579-4835-847c-f2a863e85d8a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"38ff07da86b966c01397abffc90605b7d2f0fd7a0fd9339a7963fa0a561b66e4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19983, \"end_char_idx\": 21840, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "ref_doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "chroma:document": "Import 23800 / 25000 \n    Import 23900 / 25000 \n    Import 24000 / 25000 \n    Import 24100 / 25000 \n    Import 24200 / 25000 \n    Import 24300 / 25000 \n    Import 24400 / 25000 \n    Import 24500 / 25000 \n    Import 24600 / 25000 \n    Import 24700 / 25000 \n    Import 24800 / 25000 \n    Import 24900 / 25000 \n    Importing (25000) Articles complete\n\n\n\n```python\n# Test that all data has loaded \u2013 get object count\nresult = (\n    client.query.aggregate(\"Article\")\n    .with_fields(\"meta { count }\")\n    .do()\n)\nprint(\"Object count: \", result[\"data\"][\"Aggregate\"][\"Article\"])\n```\n\n    Object count:  [{'meta': {'count': 25000}}]\n\n\n\n```python\n# Test one article has worked by checking one object\ntest_article = (\n    client.query\n    .get(\"Article\", [\"title\", \"content\", \"_additional {id}\"])\n    .with_limit(1)\n    .do()\n)[\"data\"][\"Get\"][\"Article\"][0]\n\nprint(test_article[\"_additional\"][\"id\"])\nprint(test_article[\"title\"])\nprint(test_article[\"content\"])\n```\n\n    000393f2-1182-4e3d-abcf-4217eda64be0\n    Lago d'Origlio\n    Lago d'Origlio is a lake in the municipality of Origlio, in Ticino, Switzerland.\n    \n    Lakes of Ticino\n\n\n### Search data\n\nAs above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors\n\n\n```python\ndef query_weaviate(query, collection_name, top_k=20):\n\n    # Creates embedding vector from user query\n    embedded_query = openai.Embedding.create(\n        input=query,\n        model=EMBEDDING_MODEL,\n    )[\"data\"][0]['embedding']\n    \n    near_vector = {\"vector\": embedded_query}\n\n    # Queries input schema with vectorised user query\n    query_result = (\n        client.query\n        .get(collection_name, [\"title\", \"content\", \"_additional {certainty distance}\"])\n        .with_near_vector(near_vector)\n        .with_limit(top_k)\n        .do()\n    )\n    \n    return query_result\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 97a93e0f-7579-4835-847c-f2a863e85d8a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"97a93e0f-7579-4835-847c-f2a863e85d8a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ecef48c5-024b-4970-bd02-56a7cb90f695\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e75977e7-5a02-4de3-83c8-65d341fa0f44\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"249bf8feeb3f9b6a3645561d9a212799cd37d268aa110036d8bee434f516a0e6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a3d74bd0-7401-4469-a7b3-f0ff39fbf01b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"250117ee8578b85026bd11c00c797f387ab9033d560b419116d20bdf2b90426f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 20295, \"end_char_idx\": 23310, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "ref_doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "chroma:document": "Importing (25000) Articles complete\n\n\n\n```python\n# Test that all data has loaded \u2013 get object count\nresult = (\n    client.query.aggregate(\"Article\")\n    .with_fields(\"meta { count }\")\n    .do()\n)\nprint(\"Object count: \", result[\"data\"][\"Aggregate\"][\"Article\"])\n```\n\n    Object count:  [{'meta': {'count': 25000}}]\n\n\n\n```python\n# Test one article has worked by checking one object\ntest_article = (\n    client.query\n    .get(\"Article\", [\"title\", \"content\", \"_additional {id}\"])\n    .with_limit(1)\n    .do()\n)[\"data\"][\"Get\"][\"Article\"][0]\n\nprint(test_article[\"_additional\"][\"id\"])\nprint(test_article[\"title\"])\nprint(test_article[\"content\"])\n```\n\n    000393f2-1182-4e3d-abcf-4217eda64be0\n    Lago d'Origlio\n    Lago d'Origlio is a lake in the municipality of Origlio, in Ticino, Switzerland.\n    \n    Lakes of Ticino\n\n\n### Search data\n\nAs above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors\n\n\n```python\ndef query_weaviate(query, collection_name, top_k=20):\n\n    # Creates embedding vector from user query\n    embedded_query = openai.Embedding.create(\n        input=query,\n        model=EMBEDDING_MODEL,\n    )[\"data\"][0]['embedding']\n    \n    near_vector = {\"vector\": embedded_query}\n\n    # Queries input schema with vectorised user query\n    query_result = (\n        client.query\n        .get(collection_name, [\"title\", \"content\", \"_additional {certainty distance}\"])\n        .with_near_vector(near_vector)\n        .with_limit(top_k)\n        .do()\n    )\n    \n    return query_result\n```\n\n\n```python\nquery_result = query_weaviate(\"modern art in Europe\", \"Article\")\ncounter = 0\nfor article in query_result[\"data\"][\"Get\"][\"Article\"]:\n    counter += 1\n    print(f\"{counter}. { article['title']} (Certainty: {round(article['_additional']['certainty'],3) }) (Distance: {round(article['_additional']['distance'],3) })\")\n```\n\n    1. Museum of Modern Art (Certainty: 0.938) (Distance: 0.125)\n    2. Western Europe (Certainty: 0.934) (Distance: 0.133)\n    3. Renaissance art (Certainty: 0.932) (Distance: 0.136)\n    4. Pop art (Certainty: 0.93) (Distance: 0.14)\n    5. Northern Europe (Certainty: 0.927) (Distance: 0.145)\n    6. Hellenistic art (Certainty: 0.926) (Distance: 0.147)\n    7. Modernist literature (Certainty: 0.924) (Distance: 0.153)\n    8. Art film (Certainty: 0.922) (Distance: 0.157)\n    9. Central Europe (Certainty: 0.921) (Distance: 0.157)\n    10. European (Certainty: 0.921) (Distance: 0.159)\n    11. Art (Certainty: 0.921) (Distance: 0.159)\n    12. Byzantine art (Certainty: 0.92) (Distance: 0.159)\n    13. Postmodernism (Certainty: 0.92) (Distance: 0.16)\n    14. Eastern Europe (Certainty: 0.92) (Distance: 0.161)\n    15. Europe (Certainty: 0.919) (Distance: 0.161)\n    16. Cubism (Certainty: 0.919) (Distance: 0.161)\n    17. Impressionism (Certainty: 0.919) (Distance: 0.162)\n    18. Bauhaus (Certainty: 0.919) (Distance: 0.162)\n    19. Expressionism (Certainty: 0.918) (Distance: 0.163)\n    20. Surrealism (Certainty: 0.918) (Distance: 0.163)"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: a3d74bd0-7401-4469-a7b3-f0ff39fbf01b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"a3d74bd0-7401-4469-a7b3-f0ff39fbf01b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ecef48c5-024b-4970-bd02-56a7cb90f695\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"97a93e0f-7579-4835-847c-f2a863e85d8a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b398480b99d7dfdcf56b5a7ed2ab3f7418706900a7b94535f22bc9cae3bfd82c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1426d632-43e8-4e2a-8687-337dd6e9887a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a1c9811b5079510a370a01716f8f65534cba6264be9dffb3c6b640ede9d73f23\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 23314, \"end_char_idx\": 25369, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "ref_doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "chroma:document": "```python\nquery_result = query_weaviate(\"Famous battles in Scottish history\", \"Article\")\ncounter = 0\nfor article in query_result[\"data\"][\"Get\"][\"Article\"]:\n    counter += 1\n    print(f\"{counter}. {article['title']} (Score: {round(article['_additional']['certainty'],3) })\")\n```\n\n    1. Historic Scotland (Score: 0.946)\n    2. First War of Scottish Independence (Score: 0.946)\n    3. Battle of Bannockburn (Score: 0.946)\n    4. Wars of Scottish Independence (Score: 0.944)\n    5. Second War of Scottish Independence (Score: 0.94)\n    6. List of Scottish monarchs (Score: 0.937)\n    7. Scottish Borders (Score: 0.932)\n    8. Braveheart (Score: 0.929)\n    9. John of Scotland (Score: 0.929)\n    10. Guardians of Scotland (Score: 0.926)\n    11. Holyrood Abbey (Score: 0.925)\n    12. Scottish (Score: 0.925)\n    13. Scots (Score: 0.925)\n    14. Robert I of Scotland (Score: 0.924)\n    15. Scottish people (Score: 0.924)\n    16. Edinburgh Castle (Score: 0.924)\n    17. Alexander I of Scotland (Score: 0.924)\n    18. Robert Burns (Score: 0.924)\n    19. Battle of Bosworth Field (Score: 0.922)\n    20. David II of Scotland (Score: 0.922)\n\n\n### Let Weaviate handle vector embeddings\n\nWeaviate has a [built-in module for OpenAI](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai), which takes care of the steps required to generate a vector embedding for your queries and any CRUD operations.\n\nThis allows you to run a vector query with the `with_near_text` filter, which uses your `OPEN_API_KEY`.\n\n\n```python\ndef near_text_weaviate(query, collection_name):\n    \n    nearText = {\n        \"concepts\": [query],\n        \"distance\": 0.7,\n    }\n\n    properties = [\n        \"title\", \"content\",\n        \"_additional {certainty distance}\"\n    ]\n\n    query_result = (\n        client.query\n        .get(collection_name, properties)\n        .with_near_text(nearText)\n        .with_limit(20)\n        .do()\n    )[\"data\"][\"Get\"][collection_name]\n    \n    print (f\"Objects returned: {len(query_result)}\")\n    \n    return query_result\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 1426d632-43e8-4e2a-8687-337dd6e9887a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"1426d632-43e8-4e2a-8687-337dd6e9887a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ecef48c5-024b-4970-bd02-56a7cb90f695\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a3d74bd0-7401-4469-a7b3-f0ff39fbf01b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bc2b5b771cefc2c8cd8e20f27a169c60ad479e462f932e40b9f33696bc15eacb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c84c6052-b04b-4376-a149-82a46519965c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4b1d951090cca7e1b4232cfac53d912cbda980def8226a40ef6711dc01592866\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 24446, \"end_char_idx\": 26841, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "ref_doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "chroma:document": "### Let Weaviate handle vector embeddings\n\nWeaviate has a [built-in module for OpenAI](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai), which takes care of the steps required to generate a vector embedding for your queries and any CRUD operations.\n\nThis allows you to run a vector query with the `with_near_text` filter, which uses your `OPEN_API_KEY`.\n\n\n```python\ndef near_text_weaviate(query, collection_name):\n    \n    nearText = {\n        \"concepts\": [query],\n        \"distance\": 0.7,\n    }\n\n    properties = [\n        \"title\", \"content\",\n        \"_additional {certainty distance}\"\n    ]\n\n    query_result = (\n        client.query\n        .get(collection_name, properties)\n        .with_near_text(nearText)\n        .with_limit(20)\n        .do()\n    )[\"data\"][\"Get\"][collection_name]\n    \n    print (f\"Objects returned: {len(query_result)}\")\n    \n    return query_result\n```\n\n\n```python\nquery_result = near_text_weaviate(\"modern art in Europe\",\"Article\")\ncounter = 0\nfor article in query_result:\n    counter += 1\n    print(f\"{counter}. { article['title']} (Certainty: {round(article['_additional']['certainty'],3) }) (Distance: {round(article['_additional']['distance'],3) })\")\n```\n\n    Objects returned: 20\n    1. Museum of Modern Art (Certainty: 0.938) (Distance: 0.125)\n    2. Western Europe (Certainty: 0.934) (Distance: 0.133)\n    3. Renaissance art (Certainty: 0.932) (Distance: 0.136)\n    4. Pop art (Certainty: 0.93) (Distance: 0.14)\n    5. Northern Europe (Certainty: 0.927) (Distance: 0.145)\n    6. Hellenistic art (Certainty: 0.926) (Distance: 0.147)\n    7. Modernist literature (Certainty: 0.923) (Distance: 0.153)\n    8. Art film (Certainty: 0.922) (Distance: 0.157)\n    9. Central Europe (Certainty: 0.921) (Distance: 0.157)\n    10. European (Certainty: 0.921) (Distance: 0.159)\n    11. Art (Certainty: 0.921) (Distance: 0.159)\n    12. Byzantine art (Certainty: 0.92) (Distance: 0.159)\n    13. Postmodernism (Certainty: 0.92) (Distance: 0.16)\n    14. Eastern Europe (Certainty: 0.92) (Distance: 0.161)\n    15. Europe (Certainty: 0.919) (Distance: 0.161)\n    16. Cubism (Certainty: 0.919) (Distance: 0.161)\n    17. Impressionism (Certainty: 0.919) (Distance: 0.162)\n    18. Bauhaus (Certainty: 0.919) (Distance: 0.162)\n    19. Surrealism (Certainty: 0.918) (Distance: 0.163)\n    20. Expressionism (Certainty: 0.918) (Distance: 0.163)"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: c84c6052-b04b-4376-a149-82a46519965c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb", "title": "Using Weaviate for Embeddings Search", "tokens": 8799, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"c84c6052-b04b-4376-a149-82a46519965c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ecef48c5-024b-4970-bd02-56a7cb90f695\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3e7701998a24d08c85b2ffd4d0e0d117ebe793597ce152c126f0fe2ff1e5d8fa\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1426d632-43e8-4e2a-8687-337dd6e9887a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/Using_Weaviate_for_embeddings_search.ipynb\", \"title\": \"Using Weaviate for Embeddings Search\", \"tokens\": 8799, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b20848e46e6e78314699670dde401e1e11e702c782d7557679e84f0123be667c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 26845, \"end_char_idx\": 28495, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "ref_doc_id": "ecef48c5-024b-4970-bd02-56a7cb90f695", "chroma:document": "```python\nquery_result = near_text_weaviate(\"Famous battles in Scottish history\",\"Article\")\ncounter = 0\nfor article in query_result:\n    counter += 1\n    print(f\"{counter}. { article['title']} (Certainty: {round(article['_additional']['certainty'],3) }) (Distance: {round(article['_additional']['distance'],3) })\")\n```\n\n    Objects returned: 20\n    1. Historic Scotland (Certainty: 0.946) (Distance: 0.107)\n    2. First War of Scottish Independence (Certainty: 0.946) (Distance: 0.108)\n    3. Battle of Bannockburn (Certainty: 0.946) (Distance: 0.109)\n    4. Wars of Scottish Independence (Certainty: 0.944) (Distance: 0.111)\n    5. Second War of Scottish Independence (Certainty: 0.94) (Distance: 0.121)\n    6. List of Scottish monarchs (Certainty: 0.937) (Distance: 0.127)\n    7. Scottish Borders (Certainty: 0.932) (Distance: 0.137)\n    8. Braveheart (Certainty: 0.929) (Distance: 0.141)\n    9. John of Scotland (Certainty: 0.929) (Distance: 0.142)\n    10. Guardians of Scotland (Certainty: 0.926) (Distance: 0.148)\n    11. Holyrood Abbey (Certainty: 0.925) (Distance: 0.15)\n    12. Scottish (Certainty: 0.925) (Distance: 0.15)\n    13. Scots (Certainty: 0.925) (Distance: 0.15)\n    14. Robert I of Scotland (Certainty: 0.924) (Distance: 0.151)\n    15. Scottish people (Certainty: 0.924) (Distance: 0.152)\n    16. Edinburgh Castle (Certainty: 0.924) (Distance: 0.153)\n    17. Alexander I of Scotland (Certainty: 0.924) (Distance: 0.153)\n    18. Robert Burns (Certainty: 0.924) (Distance: 0.153)\n    19. Battle of Bosworth Field (Certainty: 0.922) (Distance: 0.155)\n    20. David II of Scotland (Certainty: 0.922) (Distance: 0.157)\n\n\n\n```python\n\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 5f14be55-b52b-49b9-9796-95effcb7c007
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"5f14be55-b52b-49b9-9796-95effcb7c007\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"99590abc-7ffc-4059-a1c7-3465eea9bb5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bce57ee6c084b1c18fd858209606446bc92a66767f7298d29c766e10979f850d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6a008aeb-6ee1-437f-9609-d42f0419cc69\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1cedf7ad193670c958567e54a7c96fd27833e6fdc3c016f35ac09d9afa067293\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2380, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "doc_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "ref_doc_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "chroma:document": "# Question Answering in Weaviate with OpenAI Q&A module\n\nThis notebook is prepared for a scenario where:\n* Your data is not vectorized\n* You want to run Q&A ([learn more](https://weaviate.io/developers/weaviate/modules/reader-generator-modules/qna-openai)) on your data based on the [OpenAI completions](https://beta.openai.com/docs/api-reference/completions) endpoint.\n* You want to use Weaviate with the OpenAI module ([text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai)), to generate vector embeddings for you.\n\nThis notebook takes you through a simple flow to set up a Weaviate instance, connect to it (with OpenAI API key), configure data schema, import data (which will automatically generate vector embeddings for your data), and run question answering.\n\n## What is Weaviate\n\nWeaviate is an open-source vector search engine that stores data objects together with their vectors. This allows for combining vector search with structured filtering.\n\nWeaviate uses KNN algorithms to create an vector-optimized index, which allows your queries to run extremely fast. Learn more [here](https://weaviate.io/blog/why-is-vector-search-so-fast).\n\nWeaviate let you use your favorite ML-models, and scale seamlessly into billions of data objects.\n\n### Deployment options\n\nWhatever your scenario or production setup, Weaviate has an option for you. You can deploy Weaviate in the following setups:\n* Self-hosted \u2013 you can deploy Weaviate with docker locally, or any server you want.\n* SaaS \u2013 you can use [Weaviate Cloud Service (WCS)](https://console.weaviate.io/) to host your Weaviate instances.\n* Hybrid-SaaS \u2013 you can deploy Weaviate in your own private Cloud Service \n\n### Programming languages\n\nWeaviate offers four [client libraries](https://weaviate.io/developers/weaviate/client-libraries), which allow you to communicate from your apps:\n* [Python](https://weaviate.io/developers/weaviate/client-libraries/python)\n* [JavaScript](https://weaviate.io/developers/weaviate/client-libraries/javascript)\n* [Java](https://weaviate.io/developers/weaviate/client-libraries/java)\n* [Go](https://weaviate.io/developers/weaviate/client-libraries/go)\n\nAdditionally, Weaviate has a [REST layer](https://weaviate.io/developers/weaviate/api/rest/objects). Basically you can call Weaviate from any language that supports REST requests."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 6a008aeb-6ee1-437f-9609-d42f0419cc69
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"6a008aeb-6ee1-437f-9609-d42f0419cc69\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"99590abc-7ffc-4059-a1c7-3465eea9bb5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bce57ee6c084b1c18fd858209606446bc92a66767f7298d29c766e10979f850d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5f14be55-b52b-49b9-9796-95effcb7c007\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b00f052da40d4259422bac50fdc9af4c430b59ddfb7cd4333dda6495ec7105fb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"db2f00de-19ec-4277-a73c-a95a477af330\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"edf9f6dd3cbbe94a584e4dd8c5d77fb413406bea18b3d7afe040bd62b0577c42\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 822, \"end_char_idx\": 3997, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "doc_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "ref_doc_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "chroma:document": "## What is Weaviate\n\nWeaviate is an open-source vector search engine that stores data objects together with their vectors. This allows for combining vector search with structured filtering.\n\nWeaviate uses KNN algorithms to create an vector-optimized index, which allows your queries to run extremely fast. Learn more [here](https://weaviate.io/blog/why-is-vector-search-so-fast).\n\nWeaviate let you use your favorite ML-models, and scale seamlessly into billions of data objects.\n\n### Deployment options\n\nWhatever your scenario or production setup, Weaviate has an option for you. You can deploy Weaviate in the following setups:\n* Self-hosted \u2013 you can deploy Weaviate with docker locally, or any server you want.\n* SaaS \u2013 you can use [Weaviate Cloud Service (WCS)](https://console.weaviate.io/) to host your Weaviate instances.\n* Hybrid-SaaS \u2013 you can deploy Weaviate in your own private Cloud Service \n\n### Programming languages\n\nWeaviate offers four [client libraries](https://weaviate.io/developers/weaviate/client-libraries), which allow you to communicate from your apps:\n* [Python](https://weaviate.io/developers/weaviate/client-libraries/python)\n* [JavaScript](https://weaviate.io/developers/weaviate/client-libraries/javascript)\n* [Java](https://weaviate.io/developers/weaviate/client-libraries/java)\n* [Go](https://weaviate.io/developers/weaviate/client-libraries/go)\n\nAdditionally, Weaviate has a [REST layer](https://weaviate.io/developers/weaviate/api/rest/objects). Basically you can call Weaviate from any language that supports REST requests.\n\n## Demo Flow\nThe demo flow is:\n- **Prerequisites Setup**: Create a Weaviate instance and install required libraries\n- **Connect**: Connect to your Weaviate instance \n- **Schema Configuration**: Configure the schema of your data\n    - *Note*: Here we can define which OpenAI Embedding Model to use\n    - *Note*: Here we can configure which properties to index\n- **Import data**: Load a demo dataset and import it into Weaviate\n    - *Note*: The import process will automatically index your data - based on the configuration in the schema\n    - *Note*: You don't need to explicitly vectorize your data, Weaviate will communicate with OpenAI to do it for you\n- **Run Queries**: Query \n    - *Note*: You don't need to explicitly vectorize your queries, Weaviate will communicate with OpenAI to do it for you\n    - *Note*: The `qna-openai` module automatically communicates with the OpenAI completions endpoint\n\nOnce you've run through this notebook you should have a basic understanding of how to setup and use vector databases for question answering.\n\n## OpenAI Module in Weaviate\nAll Weaviate instances come equipped with the [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) and the [qna-openai](https://weaviate.io/developers/weaviate/modules/reader-generator-modules/qna-openai) modules.\n\nThe first module is responsible for handling vectorization at import (or any CRUD operations) and when you run a search query. The second module communicates with the OpenAI completions endpoint.\n\n### No need to manually vectorize data\nThis is great news for you."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: db2f00de-19ec-4277-a73c-a95a477af330
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"db2f00de-19ec-4277-a73c-a95a477af330\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"99590abc-7ffc-4059-a1c7-3465eea9bb5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bce57ee6c084b1c18fd858209606446bc92a66767f7298d29c766e10979f850d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6a008aeb-6ee1-437f-9609-d42f0419cc69\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"d2b67def3f2328706d8599bddb23231a1a707219dae6748e4f6b965e011930fe\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5fd9d549-527e-4705-8361-8f3405604ea0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"542895e94c9739af8d54a92c8b463901a65e9c5f6da37bdaeb93d209975e184e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2302, \"end_char_idx\": 5155, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "doc_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "ref_doc_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "chroma:document": "Basically you can call Weaviate from any language that supports REST requests.\n\n## Demo Flow\nThe demo flow is:\n- **Prerequisites Setup**: Create a Weaviate instance and install required libraries\n- **Connect**: Connect to your Weaviate instance \n- **Schema Configuration**: Configure the schema of your data\n    - *Note*: Here we can define which OpenAI Embedding Model to use\n    - *Note*: Here we can configure which properties to index\n- **Import data**: Load a demo dataset and import it into Weaviate\n    - *Note*: The import process will automatically index your data - based on the configuration in the schema\n    - *Note*: You don't need to explicitly vectorize your data, Weaviate will communicate with OpenAI to do it for you\n- **Run Queries**: Query \n    - *Note*: You don't need to explicitly vectorize your queries, Weaviate will communicate with OpenAI to do it for you\n    - *Note*: The `qna-openai` module automatically communicates with the OpenAI completions endpoint\n\nOnce you've run through this notebook you should have a basic understanding of how to setup and use vector databases for question answering.\n\n## OpenAI Module in Weaviate\nAll Weaviate instances come equipped with the [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) and the [qna-openai](https://weaviate.io/developers/weaviate/modules/reader-generator-modules/qna-openai) modules.\n\nThe first module is responsible for handling vectorization at import (or any CRUD operations) and when you run a search query. The second module communicates with the OpenAI completions endpoint.\n\n### No need to manually vectorize data\nThis is great news for you. With [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) you don't need to manually vectorize your data, as Weaviate will call OpenAI for you whenever necessary.\n\nAll you need to do is:\n1. provide your OpenAI API Key \u2013 when you connected to the Weaviate Client\n2. define which OpenAI vectorizer to use in your Schema\n\n## Prerequisites\n\nBefore we start this project, we need setup the following:\n\n* create a `Weaviate` instance\n* install libraries\n    * `weaviate-client`\n    * `datasets`\n    * `apache-beam`\n* get your [OpenAI API key](https://beta.openai.com/account/api-keys)\n\n===========================================================\n### Create a Weaviate instance\n\nTo create a Weaviate instance we have 2 options:\n\n1. (Recommended path) [Weaviate Cloud Service](https://console.weaviate.io/) \u2013 to host your Weaviate instance in the cloud. The free sandbox should be more than enough for this cookbook.\n2. Install and run Weaviate locally with Docker.\n\n#### Option 1 \u2013 WCS Installation Steps\n\nUse [Weaviate Cloud Service](https://console.weaviate.io/) (WCS) to create a free Weaviate cluster."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 5fd9d549-527e-4705-8361-8f3405604ea0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"5fd9d549-527e-4705-8361-8f3405604ea0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"99590abc-7ffc-4059-a1c7-3465eea9bb5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bce57ee6c084b1c18fd858209606446bc92a66767f7298d29c766e10979f850d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"db2f00de-19ec-4277-a73c-a95a477af330\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"66d90faf06126c6a292063eb8d8ba420f9e7e9501d54f4d44dd32f3dc597156e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"96ca7da2-3517-40e5-b39d-004daeaf8cf8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"edb5b569c70b90f9bd00e5eff3a7319fe0e77a79206bc15a24c687e3432c8db8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3734, \"end_char_idx\": 6810, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "doc_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "ref_doc_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "chroma:document": "The first module is responsible for handling vectorization at import (or any CRUD operations) and when you run a search query. The second module communicates with the OpenAI completions endpoint.\n\n### No need to manually vectorize data\nThis is great news for you. With [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) you don't need to manually vectorize your data, as Weaviate will call OpenAI for you whenever necessary.\n\nAll you need to do is:\n1. provide your OpenAI API Key \u2013 when you connected to the Weaviate Client\n2. define which OpenAI vectorizer to use in your Schema\n\n## Prerequisites\n\nBefore we start this project, we need setup the following:\n\n* create a `Weaviate` instance\n* install libraries\n    * `weaviate-client`\n    * `datasets`\n    * `apache-beam`\n* get your [OpenAI API key](https://beta.openai.com/account/api-keys)\n\n===========================================================\n### Create a Weaviate instance\n\nTo create a Weaviate instance we have 2 options:\n\n1. (Recommended path) [Weaviate Cloud Service](https://console.weaviate.io/) \u2013 to host your Weaviate instance in the cloud. The free sandbox should be more than enough for this cookbook.\n2. Install and run Weaviate locally with Docker.\n\n#### Option 1 \u2013 WCS Installation Steps\n\nUse [Weaviate Cloud Service](https://console.weaviate.io/) (WCS) to create a free Weaviate cluster.\n1. create a free account and/or login to [WCS](https://console.weaviate.io/)\n2. create a `Weaviate Cluster` with the following settings:\n    * Sandbox: `Sandbox Free`\n    * Weaviate Version: Use default (latest)\n    * OIDC Authentication: `Disabled`\n3. your instance should be ready in a minute or two\n4. make a note of the `Cluster Id`. The link will take you to the full path of your cluster (you will need it later to connect to it). It should be something like: `https://your-project-name.weaviate.network` \n\n#### Option 2 \u2013 local Weaviate instance with Docker\n\nInstall and run Weaviate locally with Docker.\n1. Download the [./docker-compose.yml](./docker-compose.yml) file\n2. Then open your terminal, navigate to where your docker-compose.yml file is located, and start docker with: `docker-compose up -d`\n3. Once this is ready, your instance should be available at [http://localhost:8080](http://localhost:8080)\n\nNote. To shut down your docker instance you can call: `docker-compose down`\n\n##### Learn more\nTo learn more, about using Weaviate with Docker see the [installation documentation](https://weaviate.io/developers/weaviate/installation/docker-compose).\n\n===========================================================    \n## Install required libraries\n\nBefore running this project make sure to have the following libraries:\n\n### Weaviate Python client\n\nThe [Weaviate Python client](https://weaviate.io/developers/weaviate/client-libraries/python) allows you to communicate with your Weaviate instance from your Python project.\n\n### datasets & apache-beam\n\nTo load sample data, you need the `datasets` library and its' dependency `apache-beam`."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 96ca7da2-3517-40e5-b39d-004daeaf8cf8
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"96ca7da2-3517-40e5-b39d-004daeaf8cf8\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"99590abc-7ffc-4059-a1c7-3465eea9bb5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bce57ee6c084b1c18fd858209606446bc92a66767f7298d29c766e10979f850d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5fd9d549-527e-4705-8361-8f3405604ea0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"d40b596d19276ad4b0d2ef723da4f1a2098882086974004a72971e52b6bc9392\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8f3a000e-40f1-4432-9163-5c18d7691ef1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c6071153842b32a8ff6bf387dde86cca1ac06f861683f8a8048c8964b5abf50f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5156, \"end_char_idx\": 7490, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "doc_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "ref_doc_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "chroma:document": "1. create a free account and/or login to [WCS](https://console.weaviate.io/)\n2. create a `Weaviate Cluster` with the following settings:\n    * Sandbox: `Sandbox Free`\n    * Weaviate Version: Use default (latest)\n    * OIDC Authentication: `Disabled`\n3. your instance should be ready in a minute or two\n4. make a note of the `Cluster Id`. The link will take you to the full path of your cluster (you will need it later to connect to it). It should be something like: `https://your-project-name.weaviate.network` \n\n#### Option 2 \u2013 local Weaviate instance with Docker\n\nInstall and run Weaviate locally with Docker.\n1. Download the [./docker-compose.yml](./docker-compose.yml) file\n2. Then open your terminal, navigate to where your docker-compose.yml file is located, and start docker with: `docker-compose up -d`\n3. Once this is ready, your instance should be available at [http://localhost:8080](http://localhost:8080)\n\nNote. To shut down your docker instance you can call: `docker-compose down`\n\n##### Learn more\nTo learn more, about using Weaviate with Docker see the [installation documentation](https://weaviate.io/developers/weaviate/installation/docker-compose).\n\n===========================================================    \n## Install required libraries\n\nBefore running this project make sure to have the following libraries:\n\n### Weaviate Python client\n\nThe [Weaviate Python client](https://weaviate.io/developers/weaviate/client-libraries/python) allows you to communicate with your Weaviate instance from your Python project.\n\n### datasets & apache-beam\n\nTo load sample data, you need the `datasets` library and its' dependency `apache-beam`.\n\n\n```python\n# Install the Weaviate client for Python\n!pip install weaviate-client>3.11.0\n\n# Install datasets and apache-beam to load the sample datasets\n!pip install datasets apache-beam\n```\n\n===========================================================\n## Prepare your OpenAI API key\n\nThe `OpenAI API key` is used for vectorization of your data at import, and for queries.\n\nIf you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n\nOnce you get your key, please add it to your environment variables as `OPENAI_API_KEY`.\n\n\n```python\n# Export OpenAI API Key\n!export OPENAI_API_KEY=\"your key\"\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 8f3a000e-40f1-4432-9163-5c18d7691ef1
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"8f3a000e-40f1-4432-9163-5c18d7691ef1\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"99590abc-7ffc-4059-a1c7-3465eea9bb5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bce57ee6c084b1c18fd858209606446bc92a66767f7298d29c766e10979f850d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"96ca7da2-3517-40e5-b39d-004daeaf8cf8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7f61e3e73ebc22b37062dd36f70fd276ecb31ebe751a712fe1b58ebe9354b700\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"74e9b8af-46f1-4b36-8454-558d169afc8b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"88f96e7784f267ea98a6b6819a100ebb82099924dc4490279e1f8d07c048f3f6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5768, \"end_char_idx\": 8400, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "doc_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "ref_doc_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "chroma:document": "1. Download the [./docker-compose.yml](./docker-compose.yml) file\n2. Then open your terminal, navigate to where your docker-compose.yml file is located, and start docker with: `docker-compose up -d`\n3. Once this is ready, your instance should be available at [http://localhost:8080](http://localhost:8080)\n\nNote. To shut down your docker instance you can call: `docker-compose down`\n\n##### Learn more\nTo learn more, about using Weaviate with Docker see the [installation documentation](https://weaviate.io/developers/weaviate/installation/docker-compose).\n\n===========================================================    \n## Install required libraries\n\nBefore running this project make sure to have the following libraries:\n\n### Weaviate Python client\n\nThe [Weaviate Python client](https://weaviate.io/developers/weaviate/client-libraries/python) allows you to communicate with your Weaviate instance from your Python project.\n\n### datasets & apache-beam\n\nTo load sample data, you need the `datasets` library and its' dependency `apache-beam`.\n\n\n```python\n# Install the Weaviate client for Python\n!pip install weaviate-client>3.11.0\n\n# Install datasets and apache-beam to load the sample datasets\n!pip install datasets apache-beam\n```\n\n===========================================================\n## Prepare your OpenAI API key\n\nThe `OpenAI API key` is used for vectorization of your data at import, and for queries.\n\nIf you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n\nOnce you get your key, please add it to your environment variables as `OPENAI_API_KEY`.\n\n\n```python\n# Export OpenAI API Key\n!export OPENAI_API_KEY=\"your key\"\n```\n\n\n```python\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\nimport os\n\n# Note. alternatively you can set a temporary env variable like this:\n# os.environ['OPENAI_API_KEY'] = 'your-key-goes-here'\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    print (\"OPENAI_API_KEY is ready\")\nelse:\n    print (\"OPENAI_API_KEY environment variable not found\")\n```\n\n## Connect to your Weaviate instance\n\nIn this section, we will:\n\n1. test env variable `OPENAI_API_KEY` \u2013 **make sure** you completed the step in [#Prepare-your-OpenAI-API-key](#Prepare-your-OpenAI-API-key)\n2. connect to your Weaviate your `OpenAI API Key`\n3. and test the client connection\n\n### The client \n\nAfter this step, the `client` object will be used to perform all Weaviate-related operations."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 74e9b8af-46f1-4b36-8454-558d169afc8b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"74e9b8af-46f1-4b36-8454-558d169afc8b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"99590abc-7ffc-4059-a1c7-3465eea9bb5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bce57ee6c084b1c18fd858209606446bc92a66767f7298d29c766e10979f850d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8f3a000e-40f1-4432-9163-5c18d7691ef1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bf8b9b161fe6b2a17113d853f6e624d1bc0ac810ab2c6d06578e33bf8f2d754e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c94705cf-1e81-4e92-89ca-88cb163d7f23\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4664c256fb8dd63a9972c509f70d3cd812cdfe63596fe35694a1b91fc0f9707c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6813, \"end_char_idx\": 9333, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "doc_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "ref_doc_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "chroma:document": "```python\n# Install the Weaviate client for Python\n!pip install weaviate-client>3.11.0\n\n# Install datasets and apache-beam to load the sample datasets\n!pip install datasets apache-beam\n```\n\n===========================================================\n## Prepare your OpenAI API key\n\nThe `OpenAI API key` is used for vectorization of your data at import, and for queries.\n\nIf you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n\nOnce you get your key, please add it to your environment variables as `OPENAI_API_KEY`.\n\n\n```python\n# Export OpenAI API Key\n!export OPENAI_API_KEY=\"your key\"\n```\n\n\n```python\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\nimport os\n\n# Note. alternatively you can set a temporary env variable like this:\n# os.environ['OPENAI_API_KEY'] = 'your-key-goes-here'\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    print (\"OPENAI_API_KEY is ready\")\nelse:\n    print (\"OPENAI_API_KEY environment variable not found\")\n```\n\n## Connect to your Weaviate instance\n\nIn this section, we will:\n\n1. test env variable `OPENAI_API_KEY` \u2013 **make sure** you completed the step in [#Prepare-your-OpenAI-API-key](#Prepare-your-OpenAI-API-key)\n2. connect to your Weaviate your `OpenAI API Key`\n3. and test the client connection\n\n### The client \n\nAfter this step, the `client` object will be used to perform all Weaviate-related operations.\n\n\n```python\nimport weaviate\nfrom datasets import load_dataset\nimport os\n\n# Connect to your Weaviate instance\nclient = weaviate.Client(\n    url=\"https://your-wcs-instance-name.weaviate.network/\",\n#   url=\"http://localhost:8080/\",\n    auth_client_secret=weaviate.auth.AuthApiKey(api_key=\"<YOUR-WEAVIATE-API-KEY>\"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)\n    additional_headers={\n        \"X-OpenAI-Api-Key\": os.getenv(\"OPENAI_API_KEY\")\n    }\n)\n\n# Check if your instance is live and ready\n# This should return `True`\nclient.is_ready()\n```\n\n# Schema\n\nIn this section, we will:\n1. configure the data schema for your data\n2. select OpenAI module\n\n> This is the second and final step, which requires OpenAI specific configuration.\n> After this step, the rest of instructions wlll only touch on Weaviate, as the OpenAI tasks will be handled automatically."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: c94705cf-1e81-4e92-89ca-88cb163d7f23
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"c94705cf-1e81-4e92-89ca-88cb163d7f23\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"99590abc-7ffc-4059-a1c7-3465eea9bb5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bce57ee6c084b1c18fd858209606446bc92a66767f7298d29c766e10979f850d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"74e9b8af-46f1-4b36-8454-558d169afc8b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a6b8983f7dc01bd13c03b2a31bf9295113b93f58ce3e33fd07f89c8bd668cf9d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f2239543-22b7-40f1-b169-5bd658798e3d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"83c5fd2f64583cc6965b52341845d5b99c7756e036529a54b66f8efb651a89b7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8403, \"end_char_idx\": 9933, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "doc_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "ref_doc_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "chroma:document": "```python\nimport weaviate\nfrom datasets import load_dataset\nimport os\n\n# Connect to your Weaviate instance\nclient = weaviate.Client(\n    url=\"https://your-wcs-instance-name.weaviate.network/\",\n#   url=\"http://localhost:8080/\",\n    auth_client_secret=weaviate.auth.AuthApiKey(api_key=\"<YOUR-WEAVIATE-API-KEY>\"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)\n    additional_headers={\n        \"X-OpenAI-Api-Key\": os.getenv(\"OPENAI_API_KEY\")\n    }\n)\n\n# Check if your instance is live and ready\n# This should return `True`\nclient.is_ready()\n```\n\n# Schema\n\nIn this section, we will:\n1. configure the data schema for your data\n2. select OpenAI module\n\n> This is the second and final step, which requires OpenAI specific configuration.\n> After this step, the rest of instructions wlll only touch on Weaviate, as the OpenAI tasks will be handled automatically.\n\n\n## What is a schema\n\nIn Weaviate you create __schemas__ to capture each of the entities you will be searching.\n\nA schema is how you tell Weaviate:\n* what embedding model should be used to vectorize the data\n* what your data is made of (property names and types)\n* which properties should be vectorized and indexed\n\nIn this cookbook we will use a dataset for `Articles`, which contains:\n* `title`\n* `content`\n* `url`\n\nWe want to vectorize `title` and `content`, but not the `url`.\n\nTo vectorize and query the data, we will use `text-embedding-3-small`. For Q&A we will use `gpt-3.5-turbo-instruct`."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: f2239543-22b7-40f1-b169-5bd658798e3d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"f2239543-22b7-40f1-b169-5bd658798e3d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"99590abc-7ffc-4059-a1c7-3465eea9bb5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bce57ee6c084b1c18fd858209606446bc92a66767f7298d29c766e10979f850d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c94705cf-1e81-4e92-89ca-88cb163d7f23\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"773ff49c660ab0d628bb6b1de03e0ea57fa40cc0cb27fc825ead0c010911a9f4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ed33eea6-7489-41cc-90f4-d09897bd5e07\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"65c244219ebf03383e791121f75bcbdf3b72a8343eea35f0ee347031a44f596e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8403, \"end_char_idx\": 11642, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "doc_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "ref_doc_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "chroma:document": "```python\nimport weaviate\nfrom datasets import load_dataset\nimport os\n\n# Connect to your Weaviate instance\nclient = weaviate.Client(\n    url=\"https://your-wcs-instance-name.weaviate.network/\",\n#   url=\"http://localhost:8080/\",\n    auth_client_secret=weaviate.auth.AuthApiKey(api_key=\"<YOUR-WEAVIATE-API-KEY>\"), # comment out this line if you are not using authentication for your Weaviate instance (i.e. for locally deployed instances)\n    additional_headers={\n        \"X-OpenAI-Api-Key\": os.getenv(\"OPENAI_API_KEY\")\n    }\n)\n\n# Check if your instance is live and ready\n# This should return `True`\nclient.is_ready()\n```\n\n# Schema\n\nIn this section, we will:\n1. configure the data schema for your data\n2. select OpenAI module\n\n> This is the second and final step, which requires OpenAI specific configuration.\n> After this step, the rest of instructions wlll only touch on Weaviate, as the OpenAI tasks will be handled automatically.\n\n\n## What is a schema\n\nIn Weaviate you create __schemas__ to capture each of the entities you will be searching.\n\nA schema is how you tell Weaviate:\n* what embedding model should be used to vectorize the data\n* what your data is made of (property names and types)\n* which properties should be vectorized and indexed\n\nIn this cookbook we will use a dataset for `Articles`, which contains:\n* `title`\n* `content`\n* `url`\n\nWe want to vectorize `title` and `content`, but not the `url`.\n\nTo vectorize and query the data, we will use `text-embedding-3-small`. For Q&A we will use `gpt-3.5-turbo-instruct`.\n\n\n```python\n# Clear up the schema, so that we can recreate it\nclient.schema.delete_all()\nclient.schema.get()\n\n# Define the Schema object to use `text-embedding-3-small` on `title` and `content`, but skip it for `url`\narticle_schema = {\n    \"class\": \"Article\",\n    \"description\": \"A collection of articles\",\n    \"vectorizer\": \"text2vec-openai\",\n    \"moduleConfig\": {\n        \"text2vec-openai\": {\n          \"model\": \"ada\",\n          \"modelVersion\": \"002\",\n          \"type\": \"text\"\n        }, \n        \"qna-openai\": {\n          \"model\": \"gpt-3.5-turbo-instruct\",\n          \"maxTokens\": 16,\n          \"temperature\": 0.0,\n          \"topP\": 1,\n          \"frequencyPenalty\": 0.0,\n          \"presencePenalty\": 0.0\n        }\n    },\n    \"properties\": [{\n        \"name\": \"title\",\n        \"description\": \"Title of the article\",\n        \"dataType\": [\"string\"]\n    },\n    {\n        \"name\": \"content\",\n        \"description\": \"Contents of the article\",\n        \"dataType\": [\"text\"]\n    },\n    {\n        \"name\": \"url\",\n        \"description\": \"URL to the article\",\n        \"dataType\": [\"string\"],\n        \"moduleConfig\": { \"text2vec-openai\": { \"skip\": True } }\n    }]\n}\n\n# add the Article schema\nclient.schema.create_class(article_schema)\n\n# get the schema to make sure it worked\nclient.schema.get()\n```\n\n## Import data\n\nIn this section we will:\n1. load the Simple Wikipedia dataset\n2. configure Weaviate Batch import (to make the import more efficient)\n3. import the data into Weaviate\n\n> Note: <br/>\n> Like mentioned before. We don't need to manually vectorize the data.<br/>\n> The [text2vec-openai](https://weaviate.io/developers/weaviate/modules/retriever-vectorizer-modules/text2vec-openai) module will take care of that."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: ed33eea6-7489-41cc-90f4-d09897bd5e07
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"ed33eea6-7489-41cc-90f4-d09897bd5e07\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"99590abc-7ffc-4059-a1c7-3465eea9bb5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bce57ee6c084b1c18fd858209606446bc92a66767f7298d29c766e10979f850d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f2239543-22b7-40f1-b169-5bd658798e3d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"dacddca736878c2d75b3705cb8c716b21cfd1e2b72f4111785b1133db707b174\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c8a5b3d3-16f9-46ac-b1b3-8f72277d9083\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"49887c2bf6e2a01509dc4d4ef70138967631275af3f7740204460169d0d90914\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11645, \"end_char_idx\": 14637, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "doc_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "ref_doc_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "chroma:document": "```python\n### STEP 1 - load the dataset\n\nfrom datasets import load_dataset\nfrom typing import List, Iterator\n\n# We'll use the datasets library to pull the Simple Wikipedia dataset for embedding\ndataset = list(load_dataset(\"wikipedia\", \"20220301.simple\")[\"train\"])\n\n# For testing, limited to 2.5k articles for demo purposes\ndataset = dataset[:2_500]\n\n# Limited to 25k articles for larger demo purposes\n# dataset = dataset[:25_000]\n\n# for free OpenAI acounts, you can use 50 objects\n# dataset = dataset[:50]\n```\n\n\n```python\n### Step 2 - configure Weaviate Batch, with\n# - starting batch size of 100\n# - dynamically increase/decrease based on performance\n# - add timeout retries if something goes wrong\n\nclient.batch.configure(\n    batch_size=10, \n    dynamic=True,\n    timeout_retries=3,\n#   callback=None,\n)\n```\n\n\n```python\n### Step 3 - import data\n\nprint(\"Importing Articles\")\n\ncounter=0\n\nwith client.batch as batch:\n    for article in dataset:\n        if (counter %10 == 0):\n            print(f\"Import {counter} / {len(dataset)} \")\n\n        properties = {\n            \"title\": article[\"title\"],\n            \"content\": article[\"text\"],\n            \"url\": article[\"url\"]\n        }\n        \n        batch.add_data_object(properties, \"Article\")\n        counter = counter+1\n\nprint(\"Importing Articles complete\")\n```\n\n\n```python\n# Test that all data has loaded \u2013 get object count\nresult = (\n    client.query.aggregate(\"Article\")\n    .with_fields(\"meta { count }\")\n    .do()\n)\nprint(\"Object count: \", result[\"data\"][\"Aggregate\"][\"Article\"], \"\\n\")\n```\n\n\n```python\n# Test one article has worked by checking one object\ntest_article = (\n    client.query\n    .get(\"Article\", [\"title\", \"url\", \"content\"])\n    .with_limit(1)\n    .do()\n)[\"data\"][\"Get\"][\"Article\"][0]\n\nprint(test_article['title'])\nprint(test_article['url'])\nprint(test_article['content'])\n```\n\n### Question Answering on the Data\n\nAs above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors\n\n\n```python\ndef qna(query, collection_name):\n    \n    properties = [\n        \"title\", \"content\", \"url\",\n        \"_additional { answer { hasAnswer property result startPosition endPosition } distance }\"\n    ]\n\n    ask = {\n        \"question\": query,\n        \"properties\": [\"content\"]\n    }\n\n    result = (\n        client.query\n        .get(collection_name, properties)\n        .with_ask(ask)\n        .with_limit(1)\n        .do()\n    )\n    \n    # Check for errors\n    if (\"errors\" in result):\n        print (\"\\033[91mYou probably have run out of OpenAI API calls for the current minute \u2013 the limit is set at 60 per minute.\")\n        raise Exception(result[\"errors\"][0]['message'])\n    \n    return result[\"data\"][\"Get\"][collection_name]\n```\n\n\n```python\nquery_result = qna(\"Did Alanis Morissette win a Grammy?\", \"Article\")\n\nfor i, article in enumerate(query_result):\n    print(f\"{i+1}. { article['_additional']['answer']['result']} (Distance: {round(article['_additional']['distance'],3) })\")\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: c8a5b3d3-16f9-46ac-b1b3-8f72277d9083
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb", "title": "Question Answering in Weaviate with OpenAI Q&A module", "tokens": 3664, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"c8a5b3d3-16f9-46ac-b1b3-8f72277d9083\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"99590abc-7ffc-4059-a1c7-3465eea9bb5b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bce57ee6c084b1c18fd858209606446bc92a66767f7298d29c766e10979f850d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ed33eea6-7489-41cc-90f4-d09897bd5e07\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/weaviate/question-answering-with-weaviate-and-openai.ipynb\", \"title\": \"Question Answering in Weaviate with OpenAI Q&A module\", \"tokens\": 3664, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"796eda1b591e267814ba27f72c0e01d35743b5d081e756ea8a3351c3cc088b7b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13192, \"end_char_idx\": 15233, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "doc_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "ref_doc_id": "99590abc-7ffc-4059-a1c7-3465eea9bb5b", "chroma:document": "```python\n# Test one article has worked by checking one object\ntest_article = (\n    client.query\n    .get(\"Article\", [\"title\", \"url\", \"content\"])\n    .with_limit(1)\n    .do()\n)[\"data\"][\"Get\"][\"Article\"][0]\n\nprint(test_article['title'])\nprint(test_article['url'])\nprint(test_article['content'])\n```\n\n### Question Answering on the Data\n\nAs above, we'll fire some queries at our new Index and get back results based on the closeness to our existing vectors\n\n\n```python\ndef qna(query, collection_name):\n    \n    properties = [\n        \"title\", \"content\", \"url\",\n        \"_additional { answer { hasAnswer property result startPosition endPosition } distance }\"\n    ]\n\n    ask = {\n        \"question\": query,\n        \"properties\": [\"content\"]\n    }\n\n    result = (\n        client.query\n        .get(collection_name, properties)\n        .with_ask(ask)\n        .with_limit(1)\n        .do()\n    )\n    \n    # Check for errors\n    if (\"errors\" in result):\n        print (\"\\033[91mYou probably have run out of OpenAI API calls for the current minute \u2013 the limit is set at 60 per minute.\")\n        raise Exception(result[\"errors\"][0]['message'])\n    \n    return result[\"data\"][\"Get\"][collection_name]\n```\n\n\n```python\nquery_result = qna(\"Did Alanis Morissette win a Grammy?\", \"Article\")\n\nfor i, article in enumerate(query_result):\n    print(f\"{i+1}. { article['_additional']['answer']['result']} (Distance: {round(article['_additional']['distance'],3) })\")\n```\n\n\n```python\nquery_result = qna(\"What is the capital of China?\", \"Article\")\n\nfor i, article in enumerate(query_result):\n    if article['_additional']['answer']['hasAnswer'] == False:\n      print('No answer found')\n    else:\n      print(f\"{i+1}. { article['_additional']['answer']['result']} (Distance: {round(article['_additional']['distance'],3) })\")\n```\n\nThanks for following along, you're now equipped to set up your own vector databases and use embeddings to do all kinds of cool things - enjoy! For more complex use cases please continue to work through other cookbook examples in this repo."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: bfd154fd-57cd-4620-a061-bdeb1a748be8
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb", "title": "Question Answering with LangChain, Deep Lake, & OpenAI", "tokens": 1929, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"bfd154fd-57cd-4620-a061-bdeb1a748be8\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb\", \"title\": \"Question Answering with LangChain, Deep Lake, & OpenAI\", \"tokens\": 1929, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bc204eab-7084-4001-939c-9b84cefb0dd4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb\", \"title\": \"Question Answering with LangChain, Deep Lake, & OpenAI\", \"tokens\": 1929, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"76569fcab9e7e34ca4979f5fbb29e72dcc69ff53c162fcfcd9d7425dcfe74460\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a15b03dd-1415-4c38-b30a-8e8e5e540663\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"665fc0e191604fb24e531a9081bb499ee93c0dfe8882853d9cca318743c8a157\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2582, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bc204eab-7084-4001-939c-9b84cefb0dd4", "doc_id": "bc204eab-7084-4001-939c-9b84cefb0dd4", "ref_doc_id": "bc204eab-7084-4001-939c-9b84cefb0dd4", "chroma:document": "# Question Answering with LangChain, Deep Lake, & OpenAI\n\nThis notebook shows how to implement a question answering system with LangChain, [Deep Lake](https://activeloop.ai/) as a vector store and OpenAI embeddings. We will take the following steps to achieve this:\n\n1. Load a Deep Lake text dataset\n2. Initialize a [Deep Lake vector store with LangChain](https://docs.activeloop.ai/tutorials/vector-store/deep-lake-vector-store-in-langchain)\n3. Add text to the vector store\n4. Run queries on the database\n5. Done!\n\nYou can also follow other tutorials such as question answering over any type of data (PDFs, json, csv, text): [chatting with any data](https://www.activeloop.ai/resources/data-chad-an-ai-app-with-lang-chain-deep-lake-to-chat-with-any-data/) stored in Deep Lake, [code understanding](https://www.activeloop.ai/resources/lang-chain-gpt-4-for-code-understanding-twitter-algorithm/), or [question answering over PDFs](https://www.activeloop.ai/resources/ultimate-guide-to-lang-chain-deep-lake-build-chat-gpt-to-answer-questions-on-your-financial-data/), or [recommending songs](https://www.activeloop.ai/resources/3-ways-to-build-a-recommendation-engine-for-songs-with-lang-chain/).\n\n## Install requirements\nLet's install the following packages.\n\n\n```python\n!pip install deeplake langchain openai tiktoken\n```\n\n## Authentication\nProvide your OpenAI API key here:\n\n\n```python\nimport getpass\nimport os\n\nos.environ['OPENAI_API_KEY'] = getpass.getpass()\n```\n\n    \u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\n\n\n## Load a Deep Lake text dataset\nWe will use a 20000 sample subset of the [cohere-wikipedia-22](https://app.activeloop.ai/davitbun/cohere-wikipedia-22) dataset for this example.\n\n\n```python\nimport deeplake\n\nds = deeplake.load(\"hub://activeloop/cohere-wikipedia-22-sample\")\nds.summary()\n```\n\n    \\\n\n    Opening dataset in read-only mode as you don't have write permissions.\n\n\n    -\n\n    This dataset can be visualized in Jupyter Notebook by ds.visualize() or at https://app.activeloop.ai/activeloop/cohere-wikipedia-22-sample\n    \n\n\n    |\n\n    hub://activeloop/cohere-wikipedia-22-sample loaded successfully.\n    \n    Dataset(path='hub://activeloop/cohere-wikipedia-22-sample', read_only=True, tensors=['ids', 'metadata', 'text'])\n    \n      tensor    htype     shape      dtype  compression\n     -------   -------   -------    -------  ------- \n       ids      text    (20000, 1)    str     None   \n     metadata   json    (20000, 1)    str     None   \n       text     text    (20000, 1)    str     None   \n\n\n    \n \n\n\n\nLet's take a look at a few samples:\n\n\n```python\nds[:3].text.data()[\"value\"]\n```"}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: a15b03dd-1415-4c38-b30a-8e8e5e540663
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb", "title": "Question Answering with LangChain, Deep Lake, & OpenAI", "tokens": 1929, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"a15b03dd-1415-4c38-b30a-8e8e5e540663\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb\", \"title\": \"Question Answering with LangChain, Deep Lake, & OpenAI\", \"tokens\": 1929, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bc204eab-7084-4001-939c-9b84cefb0dd4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb\", \"title\": \"Question Answering with LangChain, Deep Lake, & OpenAI\", \"tokens\": 1929, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"76569fcab9e7e34ca4979f5fbb29e72dcc69ff53c162fcfcd9d7425dcfe74460\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bfd154fd-57cd-4620-a061-bdeb1a748be8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb\", \"title\": \"Question Answering with LangChain, Deep Lake, & OpenAI\", \"tokens\": 1929, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"c6b5c44e116c47d8b67e904d4c15a1535a18a62061fb9321ba2c5fca966d3280\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6192ef69-2550-4bbe-baa2-c6c51f35a891\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1e87b575b3af86e056ade500aac8a05d17c75fc87052e2cc872f7722a79b02aa\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1260, \"end_char_idx\": 4053, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bc204eab-7084-4001-939c-9b84cefb0dd4", "doc_id": "bc204eab-7084-4001-939c-9b84cefb0dd4", "ref_doc_id": "bc204eab-7084-4001-939c-9b84cefb0dd4", "chroma:document": "```python\n!pip install deeplake langchain openai tiktoken\n```\n\n## Authentication\nProvide your OpenAI API key here:\n\n\n```python\nimport getpass\nimport os\n\nos.environ['OPENAI_API_KEY'] = getpass.getpass()\n```\n\n    \u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\n\n\n## Load a Deep Lake text dataset\nWe will use a 20000 sample subset of the [cohere-wikipedia-22](https://app.activeloop.ai/davitbun/cohere-wikipedia-22) dataset for this example.\n\n\n```python\nimport deeplake\n\nds = deeplake.load(\"hub://activeloop/cohere-wikipedia-22-sample\")\nds.summary()\n```\n\n    \\\n\n    Opening dataset in read-only mode as you don't have write permissions.\n\n\n    -\n\n    This dataset can be visualized in Jupyter Notebook by ds.visualize() or at https://app.activeloop.ai/activeloop/cohere-wikipedia-22-sample\n    \n\n\n    |\n\n    hub://activeloop/cohere-wikipedia-22-sample loaded successfully.\n    \n    Dataset(path='hub://activeloop/cohere-wikipedia-22-sample', read_only=True, tensors=['ids', 'metadata', 'text'])\n    \n      tensor    htype     shape      dtype  compression\n     -------   -------   -------    -------  ------- \n       ids      text    (20000, 1)    str     None   \n     metadata   json    (20000, 1)    str     None   \n       text     text    (20000, 1)    str     None   \n\n\n    \n \n\n\n\nLet's take a look at a few samples:\n\n\n```python\nds[:3].text.data()[\"value\"]\n```\n\n\n\n\n    ['The 24-hour clock is a way of telling the time in which the day runs from midnight to midnight and is divided into 24 hours, numbered from 0 to 23. It does not use a.m. or p.m. This system is also referred to (only in the US and the English speaking parts of Canada) as military time or (only in the United Kingdom and now very rarely) as continental time. In some parts of the world, it is called railway time. Also, the international standard notation of time (ISO 8601) is based on this format.',\n     'A time in the 24-hour clock is written in the form hours:minutes (for example, 01:23), or hours:minutes:seconds (01:23:45). Numbers under 10 have a zero in front (called a leading zero); e.g. 09:07. Under the 24-hour clock system, the day begins at midnight, 00:00, and the last minute of the day begins at 23:59 and ends at 24:00, which is identical to 00:00 of the following day. 12:00 can only be mid-day. Midnight is called 24:00 and is used to mean the end of the day and 00:00 is used to mean the beginning of the day. For example, you would say \"Tuesday at 24:00\" and \"Wednesday at 00:00\" to mean exactly the same time.',\n     'However, the US military prefers not to say 24:00 - they do not like to have two names for the same thing, so they always say \"23:59\", which is one minute before midnight.']\n\n\n\n## LangChain's Deep Lake vector store\nLet's define a `dataset_path`, this is where your Deep Lake vector store will house the text embeddings."}


created_at: 2024-07-28 17:45:51
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 6192ef69-2550-4bbe-baa2-c6c51f35a891
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb", "title": "Question Answering with LangChain, Deep Lake, & OpenAI", "tokens": 1929, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"6192ef69-2550-4bbe-baa2-c6c51f35a891\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb\", \"title\": \"Question Answering with LangChain, Deep Lake, & OpenAI\", \"tokens\": 1929, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bc204eab-7084-4001-939c-9b84cefb0dd4\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb\", \"title\": \"Question Answering with LangChain, Deep Lake, & OpenAI\", \"tokens\": 1929, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"76569fcab9e7e34ca4979f5fbb29e72dcc69ff53c162fcfcd9d7425dcfe74460\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a15b03dd-1415-4c38-b30a-8e8e5e540663\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/deeplake/deeplake_langchain_qa.ipynb\", \"title\": \"Question Answering with LangChain, Deep Lake, & OpenAI\", \"tokens\": 1929, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"147446e28dc2dda934924343163481d233b5ff132967a3a9c058449cd6eeb261\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"92c20786-3480-4fc5-9b05-4d935e29e4a5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4bdf07fab2d3c01768d1becc4b92455275dd6dfcf47ea16a1d27faff9254bc11\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2591, \"end_char_idx\": 5106, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bc204eab-7084-4001-939c-9b84cefb0dd4", "doc_id": "bc204eab-7084-4001-939c-9b84cefb0dd4", "ref_doc_id": "bc204eab-7084-4001-939c-9b84cefb0dd4", "chroma:document": "['The 24-hour clock is a way of telling the time in which the day runs from midnight to midnight and is divided into 24 hours, numbered from 0 to 23. It does not use a.m. or p.m. This system is also referred to (only in the US and the English speaking parts of Canada) as military time or (only in the United Kingdom and now very rarely) as continental time. In some parts of the world, it is called railway time. Also, the international standard notation of time (ISO 8601) is based on this format.',\n     'A time in the 24-hour clock is written in the form hours:minutes (for example, 01:23), or hours:minutes:seconds (01:23:45). Numbers under 10 have a zero in front (called a leading zero); e.g. 09:07. Under the 24-hour clock system, the day begins at midnight, 00:00, and the last minute of the day begins at 23:59 and ends at 24:00, which is identical to 00:00 of the following day. 12:00 can only be mid-day. Midnight is called 24:00 and is used to mean the end of the day and 00:00 is used to mean the beginning of the day. For example, you would say \"Tuesday at 24:00\" and \"Wednesday at 00:00\" to mean exactly the same time.',\n     'However, the US military prefers not to say 24:00 - they do not like to have two names for the same thing, so they always say \"23:59\", which is one minute before midnight.']\n\n\n\n## LangChain's Deep Lake vector store\nLet's define a `dataset_path`, this is where your Deep Lake vector store will house the text embeddings.\n\n\n```python\ndataset_path = 'wikipedia-embeddings-deeplake'\n```\n\nWe will setup OpenAI's `text-embedding-3-small` as our embedding function and initialize a Deep Lake vector store at `dataset_path`...\n\n\n```python\nfrom langchain.embeddings.openai import OpenAIEmbeddings\nfrom langchain.vectorstores import DeepLake\n\nembedding = OpenAIEmbeddings(model=\"text-embedding-3-small\")\ndb = DeepLake(dataset_path, embedding=embedding, overwrite=True)\n```\n\n    \n\n\n\n\n... and populate it with samples, one batch at a time, using the `add_texts` method.\n\n\n```python\nfrom tqdm.auto import tqdm\n\nbatch_size = 100\n\nnsamples = 10  # for testing. Replace with len(ds) to append everything\nfor i in tqdm(range(0, nsamples, batch_size)):\n    # find end of batch\n    i_end = min(nsamples, i + batch_size)\n\n    batch = ds[i:i_end]\n    id_batch = batch.ids.data()[\"value\"]\n    text_batch = batch.text.data()[\"value\"]\n    meta_batch = batch.metadata.data()[\"value\"]\n\n    db.add_texts(text_batch, metadatas=meta_batch, ids=id_batch)\n```\n\n\n      0%|          | 0/1 [00:00<?, ?it/s]"}


