# embeddings_queue

created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 821a352b-4613-4062-933a-39ea0ea10b12
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"821a352b-4613-4062-933a-39ea0ea10b12\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b5843b81-291b-40bc-b969-8753ef31e079\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e56ef2bc-a8ee-4934-b4ad-b45c2857ec9e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"1afbe544ed834c5a9480b442b36c737d26e2ffc587253647767748a8f67870ee\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2f99cc15-1539-4d7e-b1fa-eae44a9df176\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e9dcab3fc94bd79bc76d8419f0f6229bdb90ebe00c4b7b45ecd1a5172bdbd1c7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10875, \"end_char_idx\": 13801, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b5843b81-291b-40bc-b969-8753ef31e079", "doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "ref_doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "chroma:document": "```python\n# check out one of the texts we will use to create semantic embeddings\ndf[\"product_text\"][0]\n\n```\n\n\n\n\n    'name turtle check men navy blue shirt category apparel subcategory topwear color navy blue gender men'\n\n\n\n## Connect to Redis\n\nNow that we have our Redis database running, we can connect to it using the Redis-py client. We will use the default host and port for the Redis database which is `localhost:6379`.\n\n\n\n\n```python\nimport redis\nfrom redis.commands.search.indexDefinition import (\n    IndexDefinition,\n    IndexType\n)\nfrom redis.commands.search.query import Query\nfrom redis.commands.search.field import (\n    TagField,\n    NumericField,\n    TextField,\n    VectorField\n)\n\nREDIS_HOST =  \"localhost\"\nREDIS_PORT = 6379\nREDIS_PASSWORD = \"\" # default for passwordless Redis\n\n# Connect to Redis\nredis_client = redis.Redis(\n    host=REDIS_HOST,\n    port=REDIS_PORT,\n    password=REDIS_PASSWORD\n)\nredis_client.ping()\n\n```\n\n\n\n\n    True\n\n\n\n## Creating a Search Index in Redis\n\nThe below cells will show how to specify and create a search index in Redis. We will:\n\n1. Set some constants for defining our index like the distance metric and the index name\n2. Define the index schema with RediSearch fields\n3. Create the index\n\n\n```python\n# Constants\nINDEX_NAME = \"product_embeddings\"           # name of the search index\nPREFIX = \"doc\"                            # prefix for the document keys\nDISTANCE_METRIC = \"L2\"                # distance metric for the vectors (ex. COSINE, IP, L2)\nNUMBER_OF_VECTORS = len(df)\n\n```\n\n\n```python\n# Define RediSearch fields for each of the columns in the dataset\nname = TextField(name=\"productDisplayName\")\ncategory = TagField(name=\"masterCategory\")\narticleType = TagField(name=\"articleType\")\ngender = TagField(name=\"gender\")\nseason = TagField(name=\"season\")\nyear = NumericField(name=\"year\")\ntext_embedding = VectorField(\"product_vector\",\n    \"FLAT\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": 1536,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": NUMBER_OF_VECTORS,\n    }\n)\nfields = [name, category, articleType, gender, season, year, text_embedding]\n\n```\n\n\n```python\n# Check if index exists\ntry:\n    redis_client.ft(INDEX_NAME).info()\n    print(\"Index already exists\")\nexcept:\n    # Create RediSearch Index\n    redis_client.ft(INDEX_NAME).create_index(\n        fields = fields,\n        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n)\n\n```\n\n## Generate OpenAI Embeddings and Load Documents into the Index\n\nNow that we have a search index, we can load documents into it. We will use the dataframe containing the styles dataset loaded previously. In Redis, either the HASH or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The cells below will show how to get OpenAI embeddings for the different products and load documents into the index."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 2f99cc15-1539-4d7e-b1fa-eae44a9df176
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"2f99cc15-1539-4d7e-b1fa-eae44a9df176\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b5843b81-291b-40bc-b969-8753ef31e079\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"821a352b-4613-4062-933a-39ea0ea10b12\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"e8278f65a88c4513f9c039734366cf0f8c8577dbfb2f6ff4c395eeb7dc481a00\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"609bf07a-ecf3-47d9-8f04-a8dac51b515d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0cc7cfab9ef3a6ccc1ae9dd760ea15783bb28972036b85c80f10369ce4a5c891\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12113, \"end_char_idx\": 15336, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b5843b81-291b-40bc-b969-8753ef31e079", "doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "ref_doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "chroma:document": "```python\n# Constants\nINDEX_NAME = \"product_embeddings\"           # name of the search index\nPREFIX = \"doc\"                            # prefix for the document keys\nDISTANCE_METRIC = \"L2\"                # distance metric for the vectors (ex. COSINE, IP, L2)\nNUMBER_OF_VECTORS = len(df)\n\n```\n\n\n```python\n# Define RediSearch fields for each of the columns in the dataset\nname = TextField(name=\"productDisplayName\")\ncategory = TagField(name=\"masterCategory\")\narticleType = TagField(name=\"articleType\")\ngender = TagField(name=\"gender\")\nseason = TagField(name=\"season\")\nyear = NumericField(name=\"year\")\ntext_embedding = VectorField(\"product_vector\",\n    \"FLAT\", {\n        \"TYPE\": \"FLOAT32\",\n        \"DIM\": 1536,\n        \"DISTANCE_METRIC\": DISTANCE_METRIC,\n        \"INITIAL_CAP\": NUMBER_OF_VECTORS,\n    }\n)\nfields = [name, category, articleType, gender, season, year, text_embedding]\n\n```\n\n\n```python\n# Check if index exists\ntry:\n    redis_client.ft(INDEX_NAME).info()\n    print(\"Index already exists\")\nexcept:\n    # Create RediSearch Index\n    redis_client.ft(INDEX_NAME).create_index(\n        fields = fields,\n        definition = IndexDefinition(prefix=[PREFIX], index_type=IndexType.HASH)\n)\n\n```\n\n## Generate OpenAI Embeddings and Load Documents into the Index\n\nNow that we have a search index, we can load documents into it. We will use the dataframe containing the styles dataset loaded previously. In Redis, either the HASH or JSON (if using RedisJSON in addition to RediSearch) data types can be used to store documents. We will use the HASH data type in this example. The cells below will show how to get OpenAI embeddings for the different products and load documents into the index.\n\n\n```python\n# Use OpenAI get_embeddings batch requests to speed up embedding creation\ndef embeddings_batch_request(documents: pd.DataFrame):\n    records = documents.to_dict(\"records\")\n    print(\"Records to process: \", len(records))\n    product_vectors = []\n    docs = []\n    batchsize = 1000\n\n    for idx,doc in enumerate(records,start=1):\n        # create byte vectors\n        docs.append(doc[\"product_text\"])\n        if idx % batchsize == 0:\n            product_vectors += get_embeddings(docs, EMBEDDING_MODEL)\n            docs.clear()\n            print(\"Vectors processed \", len(product_vectors), end='\\r')\n    product_vectors += get_embeddings(docs, EMBEDDING_MODEL)\n    print(\"Vectors processed \", len(product_vectors), end='\\r')\n    return product_vectors\n\n```\n\n\n```python\ndef index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):\n    product_vectors = embeddings_batch_request(documents)\n    records = documents.to_dict(\"records\")\n    batchsize = 500\n\n    # Use Redis pipelines to batch calls and save on round trip network communication\n    pipe = client.pipeline()\n    for idx,doc in enumerate(records,start=1):\n        key = f\"{prefix}:{str(doc['product_id'])}\"\n\n        # create byte vectors\n        text_embedding = np.array((product_vectors[idx-1]), dtype=np.float32).tobytes()\n\n        # replace list of floats with byte vectors\n        doc[\"product_vector\"] = text_embedding\n\n        pipe.hset(key, mapping = doc)\n        if idx % batchsize == 0:\n            pipe.execute()\n    pipe.execute()\n\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 609bf07a-ecf3-47d9-8f04-a8dac51b515d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"609bf07a-ecf3-47d9-8f04-a8dac51b515d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b5843b81-291b-40bc-b969-8753ef31e079\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2f99cc15-1539-4d7e-b1fa-eae44a9df176\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"40bc1472f58d584aa6abb178eab99bf1c28ea2d78df3eef34313406bae03dd87\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1d5bd9c6-6df1-4b00-be90-e21cbc24efd5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"84210844c53e5d61d8ed50964d4be8ebf098a4d84e7d0e7bbbafc30037758387\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13804, \"end_char_idx\": 16042, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b5843b81-291b-40bc-b969-8753ef31e079", "doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "ref_doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "chroma:document": "```python\n# Use OpenAI get_embeddings batch requests to speed up embedding creation\ndef embeddings_batch_request(documents: pd.DataFrame):\n    records = documents.to_dict(\"records\")\n    print(\"Records to process: \", len(records))\n    product_vectors = []\n    docs = []\n    batchsize = 1000\n\n    for idx,doc in enumerate(records,start=1):\n        # create byte vectors\n        docs.append(doc[\"product_text\"])\n        if idx % batchsize == 0:\n            product_vectors += get_embeddings(docs, EMBEDDING_MODEL)\n            docs.clear()\n            print(\"Vectors processed \", len(product_vectors), end='\\r')\n    product_vectors += get_embeddings(docs, EMBEDDING_MODEL)\n    print(\"Vectors processed \", len(product_vectors), end='\\r')\n    return product_vectors\n\n```\n\n\n```python\ndef index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):\n    product_vectors = embeddings_batch_request(documents)\n    records = documents.to_dict(\"records\")\n    batchsize = 500\n\n    # Use Redis pipelines to batch calls and save on round trip network communication\n    pipe = client.pipeline()\n    for idx,doc in enumerate(records,start=1):\n        key = f\"{prefix}:{str(doc['product_id'])}\"\n\n        # create byte vectors\n        text_embedding = np.array((product_vectors[idx-1]), dtype=np.float32).tobytes()\n\n        # replace list of floats with byte vectors\n        doc[\"product_vector\"] = text_embedding\n\n        pipe.hset(key, mapping = doc)\n        if idx % batchsize == 0:\n            pipe.execute()\n    pipe.execute()\n\n```\n\n\n```python\n%%time\nindex_documents(redis_client, PREFIX, df)\nprint(f\"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}\")\n\n```\n\n    Records to process:  1978\n    Loaded 1978 documents in Redis search index with name: product_embeddings\n    CPU times: user 619 ms, sys: 78.9 ms, total: 698 ms\n    Wall time: 3.34 s\n\n\n## Simple Vector Search Queries with OpenAI Query Embeddings\n\nNow that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 1d5bd9c6-6df1-4b00-be90-e21cbc24efd5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"1d5bd9c6-6df1-4b00-be90-e21cbc24efd5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b5843b81-291b-40bc-b969-8753ef31e079\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"609bf07a-ecf3-47d9-8f04-a8dac51b515d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"48f85030f130ddfd49167074cb483835c5791c1fde7766fdad30bba72e99583a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c5e7253b-9db6-4fd5-a9ce-69d90550c1bc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"427afe109f8013c071a1ccf83d173d235e01dba5add7f8ea8e5834734dbe578c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14571, \"end_char_idx\": 17385, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b5843b81-291b-40bc-b969-8753ef31e079", "doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "ref_doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "chroma:document": "```python\ndef index_documents(client: redis.Redis, prefix: str, documents: pd.DataFrame):\n    product_vectors = embeddings_batch_request(documents)\n    records = documents.to_dict(\"records\")\n    batchsize = 500\n\n    # Use Redis pipelines to batch calls and save on round trip network communication\n    pipe = client.pipeline()\n    for idx,doc in enumerate(records,start=1):\n        key = f\"{prefix}:{str(doc['product_id'])}\"\n\n        # create byte vectors\n        text_embedding = np.array((product_vectors[idx-1]), dtype=np.float32).tobytes()\n\n        # replace list of floats with byte vectors\n        doc[\"product_vector\"] = text_embedding\n\n        pipe.hset(key, mapping = doc)\n        if idx % batchsize == 0:\n            pipe.execute()\n    pipe.execute()\n\n```\n\n\n```python\n%%time\nindex_documents(redis_client, PREFIX, df)\nprint(f\"Loaded {redis_client.info()['db0']['keys']} documents in Redis search index with name: {INDEX_NAME}\")\n\n```\n\n    Records to process:  1978\n    Loaded 1978 documents in Redis search index with name: product_embeddings\n    CPU times: user 619 ms, sys: 78.9 ms, total: 698 ms\n    Wall time: 3.34 s\n\n\n## Simple Vector Search Queries with OpenAI Query Embeddings\n\nNow that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database.\n\n\n```python\ndef search_redis(\n    redis_client: redis.Redis,\n    user_query: str,\n    index_name: str = \"product_embeddings\",\n    vector_field: str = \"product_vector\",\n    return_fields: list = [\"productDisplayName\", \"masterCategory\", \"gender\", \"season\", \"year\", \"vector_score\"],\n    hybrid_fields = \"*\",\n    k: int = 20,\n    print_results: bool = True,\n) -> List[dict]:\n\n    # Use OpenAI to create embedding vector from user query\n    embedded_query = openai.Embedding.create(input=user_query,\n                                            model=\"text-embedding-3-small\",\n                                            )[\"data\"][0]['embedding']\n\n    # Prepare the Query\n    base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'\n    query = (\n        Query(base_query)\n         .return_fields(*return_fields)\n         .sort_by(\"vector_score\")\n         .paging(0, k)\n         .dialect(2)\n    )\n    params_dict = {\"vector\": np.array(embedded_query).astype(dtype=np.float32).tobytes()}\n\n    # perform vector search\n    results = redis_client.ft(index_name).search(query, params_dict)\n    if print_results:\n        for i, product in enumerate(results.docs):\n            score = 1 - float(product.vector_score)\n            print(f\"{i}. {product.productDisplayName} (Score: {round(score ,3) })\")\n    return results.docs\n\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: c5e7253b-9db6-4fd5-a9ce-69d90550c1bc
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c5e7253b-9db6-4fd5-a9ce-69d90550c1bc\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b5843b81-291b-40bc-b969-8753ef31e079\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1d5bd9c6-6df1-4b00-be90-e21cbc24efd5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"e5f4b246adb17cb52327fd550023c0f81321cba47858ce209a16501ba1bdd521\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4fdbb533-646c-4e84-9cbc-0d142334c8f3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e98b9963d8996e68db99fea6328ba8f9af5d69fb5bd4690d91718a7e655aae6c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15702, \"end_char_idx\": 18333, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b5843b81-291b-40bc-b969-8753ef31e079", "doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "ref_doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "chroma:document": "## Simple Vector Search Queries with OpenAI Query Embeddings\n\nNow that we have a search index and documents loaded into it, we can run search queries. Below we will provide a function that will run a search query and return the results. Using this function we run a few queries that will show how you can utilize Redis as a vector database.\n\n\n```python\ndef search_redis(\n    redis_client: redis.Redis,\n    user_query: str,\n    index_name: str = \"product_embeddings\",\n    vector_field: str = \"product_vector\",\n    return_fields: list = [\"productDisplayName\", \"masterCategory\", \"gender\", \"season\", \"year\", \"vector_score\"],\n    hybrid_fields = \"*\",\n    k: int = 20,\n    print_results: bool = True,\n) -> List[dict]:\n\n    # Use OpenAI to create embedding vector from user query\n    embedded_query = openai.Embedding.create(input=user_query,\n                                            model=\"text-embedding-3-small\",\n                                            )[\"data\"][0]['embedding']\n\n    # Prepare the Query\n    base_query = f'{hybrid_fields}=>[KNN {k} @{vector_field} $vector AS vector_score]'\n    query = (\n        Query(base_query)\n         .return_fields(*return_fields)\n         .sort_by(\"vector_score\")\n         .paging(0, k)\n         .dialect(2)\n    )\n    params_dict = {\"vector\": np.array(embedded_query).astype(dtype=np.float32).tobytes()}\n\n    # perform vector search\n    results = redis_client.ft(index_name).search(query, params_dict)\n    if print_results:\n        for i, product in enumerate(results.docs):\n            score = 1 - float(product.vector_score)\n            print(f\"{i}. {product.productDisplayName} (Score: {round(score ,3) })\")\n    return results.docs\n\n```\n\n\n```python\n# Execute a simple vector search in Redis\nresults = search_redis(redis_client, 'man blue jeans', k=10)\n\n```\n\n    0. John Players Men Blue Jeans (Score: 0.791)\n    1. Lee Men Tino Blue Jeans (Score: 0.775)\n    2. Peter England Men Party Blue Jeans (Score: 0.763)\n    3. Lee Men Blue Chicago Fit Jeans (Score: 0.761)\n    4. Lee Men Blue Chicago Fit Jeans (Score: 0.761)\n    5. French Connection Men Blue Jeans (Score: 0.74)\n    6. Locomotive Men Washed Blue Jeans (Score: 0.739)\n    7. Locomotive Men Washed Blue Jeans (Score: 0.739)\n    8. Do U Speak Green Men Blue Shorts (Score: 0.736)\n    9. Palm Tree Kids Boy Washed Blue Jeans (Score: 0.732)\n\n\n## Hybrid Queries with Redis\n\nThe previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the example below, we will combine vector search with full text search."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 4fdbb533-646c-4e84-9cbc-0d142334c8f3
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"4fdbb533-646c-4e84-9cbc-0d142334c8f3\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b5843b81-291b-40bc-b969-8753ef31e079\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c5e7253b-9db6-4fd5-a9ce-69d90550c1bc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"cf746d4a575f77ef1cef83437989d7403667a80fb3c2e57ed2daca5e97bcc396\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f722227e-4dd0-4924-861b-aad660f2d925\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"78139ef8cfd93f539797b1e3f18d032888c1a0f0ac35a4e06eea55ce811870b3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17388, \"end_char_idx\": 20144, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b5843b81-291b-40bc-b969-8753ef31e079", "doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "ref_doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "chroma:document": "```python\n# Execute a simple vector search in Redis\nresults = search_redis(redis_client, 'man blue jeans', k=10)\n\n```\n\n    0. John Players Men Blue Jeans (Score: 0.791)\n    1. Lee Men Tino Blue Jeans (Score: 0.775)\n    2. Peter England Men Party Blue Jeans (Score: 0.763)\n    3. Lee Men Blue Chicago Fit Jeans (Score: 0.761)\n    4. Lee Men Blue Chicago Fit Jeans (Score: 0.761)\n    5. French Connection Men Blue Jeans (Score: 0.74)\n    6. Locomotive Men Washed Blue Jeans (Score: 0.739)\n    7. Locomotive Men Washed Blue Jeans (Score: 0.739)\n    8. Do U Speak Green Men Blue Shorts (Score: 0.736)\n    9. Palm Tree Kids Boy Washed Blue Jeans (Score: 0.732)\n\n\n## Hybrid Queries with Redis\n\nThe previous examples showed how run vector search queries with RediSearch. In this section, we will show how to combine vector search with other RediSearch fields for hybrid search. In the example below, we will combine vector search with full text search.\n\n\n```python\n# improve search quality by adding hybrid query for \"man blue jeans\" in the product vector combined with a phrase search for \"blue jeans\"\nresults = search_redis(redis_client,\n                       \"man blue jeans\",\n                       vector_field=\"product_vector\",\n                       k=10,\n                       hybrid_fields='@productDisplayName:\"blue jeans\"'\n                       )\n\n```\n\n    0. John Players Men Blue Jeans (Score: 0.791)\n    1. Lee Men Tino Blue Jeans (Score: 0.775)\n    2. Peter England Men Party Blue Jeans (Score: 0.763)\n    3. French Connection Men Blue Jeans (Score: 0.74)\n    4. Locomotive Men Washed Blue Jeans (Score: 0.739)\n    5. Locomotive Men Washed Blue Jeans (Score: 0.739)\n    6. Palm Tree Kids Boy Washed Blue Jeans (Score: 0.732)\n    7. Denizen Women Blue Jeans (Score: 0.725)\n    8. Jealous 21 Women Washed Blue Jeans (Score: 0.713)\n    9. Jealous 21 Women Washed Blue Jeans (Score: 0.713)\n\n\n\n```python\n# hybrid query for shirt in the product vector and only include results with the phrase \"slim fit\" in the title\nresults = search_redis(redis_client,\n                       \"shirt\",\n                       vector_field=\"product_vector\",\n                       k=10,\n                       hybrid_fields='@productDisplayName:\"slim fit\"'\n                       )\n\n```\n\n    0. Basics Men White Slim Fit Striped Shirt (Score: 0.633)\n    1. ADIDAS Men's Slim Fit White T-shirt (Score: 0.628)\n    2. Basics Men Blue Slim Fit Checked Shirt (Score: 0.627)\n    3. Basics Men Blue Slim Fit Checked Shirt (Score: 0.627)\n    4. Basics Men Red Slim Fit Checked Shirt (Score: 0.623)\n    5. Basics Men Navy Slim Fit Checked Shirt (Score: 0.613)\n    6. Lee Rinse Navy Blue Slim Fit Jeans (Score: 0.558)\n    7. Tokyo Talkies Women Navy Slim Fit Jeans (Score: 0.552)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: f722227e-4dd0-4924-861b-aad660f2d925
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"f722227e-4dd0-4924-861b-aad660f2d925\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b5843b81-291b-40bc-b969-8753ef31e079\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4fdbb533-646c-4e84-9cbc-0d142334c8f3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c51836e9e492cc2dcab5a96243cd2ce8bbe288e2176e587c036c7d82aeb615d3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"168e7d3e-48e9-4d5c-9299-34b9f52fe13f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6542bcbadd9014a90bbf608532b3c81be147d74bc42ba28447573347a75ab055\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19288, \"end_char_idx\": 21168, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b5843b81-291b-40bc-b969-8753ef31e079", "doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "ref_doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "chroma:document": "```python\n# hybrid query for shirt in the product vector and only include results with the phrase \"slim fit\" in the title\nresults = search_redis(redis_client,\n                       \"shirt\",\n                       vector_field=\"product_vector\",\n                       k=10,\n                       hybrid_fields='@productDisplayName:\"slim fit\"'\n                       )\n\n```\n\n    0. Basics Men White Slim Fit Striped Shirt (Score: 0.633)\n    1. ADIDAS Men's Slim Fit White T-shirt (Score: 0.628)\n    2. Basics Men Blue Slim Fit Checked Shirt (Score: 0.627)\n    3. Basics Men Blue Slim Fit Checked Shirt (Score: 0.627)\n    4. Basics Men Red Slim Fit Checked Shirt (Score: 0.623)\n    5. Basics Men Navy Slim Fit Checked Shirt (Score: 0.613)\n    6. Lee Rinse Navy Blue Slim Fit Jeans (Score: 0.558)\n    7. Tokyo Talkies Women Navy Slim Fit Jeans (Score: 0.552)\n\n\n\n```python\n# hybrid query for watch in the product vector and only include results with the tag \"Accessories\" in the masterCategory field\nresults = search_redis(redis_client,\n                       \"watch\",\n                       vector_field=\"product_vector\",\n                       k=10,\n                       hybrid_fields='@masterCategory:{Accessories}'\n                       )\n\n```\n\n    0. Titan Women Gold Watch (Score: 0.544)\n    1. Being Human Men Grey Dial Blue Strap Watch (Score: 0.544)\n    2. Police Men Black Dial Watch PL12170JSB (Score: 0.544)\n    3. Titan Men Black Watch (Score: 0.543)\n    4. Police Men Black Dial Chronograph Watch PL12777JS-02M (Score: 0.542)\n    5. CASIO Youth Series Digital Men Black Small Dial Digital Watch W-210-1CVDF I065 (Score: 0.542)\n    6. Titan Women Silver Watch (Score: 0.542)\n    7. Police Men Black Dial Watch PL12778MSU-61 (Score: 0.541)\n    8. Titan Raga Women Gold Watch (Score: 0.539)\n    9. ADIDAS Original Men Black Dial Chronograph Watch ADH2641 (Score: 0.539)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 168e7d3e-48e9-4d5c-9299-34b9f52fe13f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"168e7d3e-48e9-4d5c-9299-34b9f52fe13f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b5843b81-291b-40bc-b969-8753ef31e079\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f722227e-4dd0-4924-861b-aad660f2d925\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"713d92df7d088ee255358a31d994d5f66c8e34280a0486bae7f840591398c8f3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cd666e18-03d3-4299-bd7e-954c5ce35223\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"02816a573c7830234a50742354c3079de7e0a8d731e89b6d8555aa2988d7a767\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 20148, \"end_char_idx\": 22057, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b5843b81-291b-40bc-b969-8753ef31e079", "doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "ref_doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "chroma:document": "```python\n# hybrid query for watch in the product vector and only include results with the tag \"Accessories\" in the masterCategory field\nresults = search_redis(redis_client,\n                       \"watch\",\n                       vector_field=\"product_vector\",\n                       k=10,\n                       hybrid_fields='@masterCategory:{Accessories}'\n                       )\n\n```\n\n    0. Titan Women Gold Watch (Score: 0.544)\n    1. Being Human Men Grey Dial Blue Strap Watch (Score: 0.544)\n    2. Police Men Black Dial Watch PL12170JSB (Score: 0.544)\n    3. Titan Men Black Watch (Score: 0.543)\n    4. Police Men Black Dial Chronograph Watch PL12777JS-02M (Score: 0.542)\n    5. CASIO Youth Series Digital Men Black Small Dial Digital Watch W-210-1CVDF I065 (Score: 0.542)\n    6. Titan Women Silver Watch (Score: 0.542)\n    7. Police Men Black Dial Watch PL12778MSU-61 (Score: 0.541)\n    8. Titan Raga Women Gold Watch (Score: 0.539)\n    9. ADIDAS Original Men Black Dial Chronograph Watch ADH2641 (Score: 0.539)\n\n\n\n```python\n# hybrid query for sandals in the product vector and only include results within the 2011-2012 year range\nresults = search_redis(redis_client,\n                       \"sandals\",\n                       vector_field=\"product_vector\",\n                       k=10,\n                       hybrid_fields='@year:[2011 2012]'\n                       )\n\n```\n\n    0. Enroute Teens Orange Sandals (Score: 0.701)\n    1. Fila Men Camper Brown Sandals (Score: 0.692)\n    2. Clarks Men Black Leather Closed Sandals (Score: 0.691)\n    3. Coolers Men Black Sandals (Score: 0.69)\n    4. Coolers Men Black Sandals (Score: 0.69)\n    5. Enroute Teens Brown Sandals (Score: 0.69)\n    6. Crocs Dora Boots Pink Sandals (Score: 0.69)\n    7. Enroute Men Leather Black Sandals (Score: 0.685)\n    8. ADIDAS Men Navy Blue Benton Sandals (Score: 0.684)\n    9. Coolers Men Black Sports Sandals (Score: 0.684)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: cd666e18-03d3-4299-bd7e-954c5ce35223
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb", "title": "Running Hybrid VSS Queries with Redis and OpenAI", "tokens": 6310, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"cd666e18-03d3-4299-bd7e-954c5ce35223\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"b5843b81-291b-40bc-b969-8753ef31e079\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a8efdf414400e87ef0b9d84d3fb51f7269826d7c6cf54456d793754108c6e95f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"168e7d3e-48e9-4d5c-9299-34b9f52fe13f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redis-hybrid-query-examples.ipynb\", \"title\": \"Running Hybrid VSS Queries with Redis and OpenAI\", \"tokens\": 6310, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"d6d813c5fd8f9e2217fa334a0f9edc30614d28810ae15bee8a5538f4764658e8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21172, \"end_char_idx\": 23790, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "b5843b81-291b-40bc-b969-8753ef31e079", "doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "ref_doc_id": "b5843b81-291b-40bc-b969-8753ef31e079", "chroma:document": "```python\n# hybrid query for sandals in the product vector and only include results within the 2011-2012 year range\nresults = search_redis(redis_client,\n                       \"sandals\",\n                       vector_field=\"product_vector\",\n                       k=10,\n                       hybrid_fields='@year:[2011 2012]'\n                       )\n\n```\n\n    0. Enroute Teens Orange Sandals (Score: 0.701)\n    1. Fila Men Camper Brown Sandals (Score: 0.692)\n    2. Clarks Men Black Leather Closed Sandals (Score: 0.691)\n    3. Coolers Men Black Sandals (Score: 0.69)\n    4. Coolers Men Black Sandals (Score: 0.69)\n    5. Enroute Teens Brown Sandals (Score: 0.69)\n    6. Crocs Dora Boots Pink Sandals (Score: 0.69)\n    7. Enroute Men Leather Black Sandals (Score: 0.685)\n    8. ADIDAS Men Navy Blue Benton Sandals (Score: 0.684)\n    9. Coolers Men Black Sports Sandals (Score: 0.684)\n\n\n\n```python\n# hybrid query for sandals in the product vector and only include results within the 2011-2012 year range from the summer season\nresults = search_redis(redis_client,\n                       \"blue sandals\",\n                       vector_field=\"product_vector\",\n                       k=10,\n                       hybrid_fields='(@year:[2011 2012] @season:{Summer})'\n                       )\n\n```\n\n    0. ADIDAS Men Navy Blue Benton Sandals (Score: 0.691)\n    1. Enroute Teens Brown Sandals (Score: 0.681)\n    2. ADIDAS Women's Adi Groove Blue Flip Flop (Score: 0.672)\n    3. Enroute Women Turquoise Blue Flats (Score: 0.671)\n    4. Red Tape Men Black Sandals (Score: 0.67)\n    5. Enroute Teens Orange Sandals (Score: 0.661)\n    6. Vans Men Blue Era Scilla Plaid Shoes (Score: 0.658)\n    7. FILA Men Aruba Navy Blue Sandal (Score: 0.657)\n    8. Quiksilver Men Blue Flip Flops (Score: 0.656)\n    9. Reebok Men Navy Twist Sandals (Score: 0.656)\n\n\n\n```python\n# hybrid query for a brown belt filtering results by a year (NUMERIC) with a specific article types (TAG) and with a brand name (TEXT)\nresults = search_redis(redis_client,\n                       \"brown belt\",\n                       vector_field=\"product_vector\",\n                       k=10,\n                       hybrid_fields='(@year:[2012 2012] @articleType:{Shirts | Belts} @productDisplayName:\"Wrangler\")'\n                       )\n\n```\n\n    0. Wrangler Men Leather Brown Belt (Score: 0.67)\n    1. Wrangler Women Black Belt (Score: 0.639)\n    2. Wrangler Men Green Striped Shirt (Score: 0.575)\n    3. Wrangler Men Purple Striped Shirt (Score: 0.549)\n    4. Wrangler Men Griffith White Shirt (Score: 0.543)\n    5. Wrangler Women Stella Green Shirt (Score: 0.542)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 29496040-7325-4d75-a2b4-e932a53c236e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"29496040-7325-4d75-a2b4-e932a53c236e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6d3bac1e156fc9f35a5a0494df7d73ead8a1a3d7fd0e67ee56c3acbc3bb6a6fd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5f2b8edb-023c-484d-9dbb-f6d04314afcc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9577e2f6f7b9db4edb4ac6f9c29af58f8407f3420a92bfa98d400fc6e6a1e7eb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3175, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "doc_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "ref_doc_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "chroma:document": "# Redis as a Context Store with OpenAI Chat\nThis notebook demonstrates how to use Redis as high-speed context memory with ChatGPT.\n\n## Prerequisites\n* Redis instance with the Redis Search and Redis JSON modules\n* Redis-py client lib\n* OpenAI Python client lib\n* OpenAI API key\n\n## Installation\nInstall Python modules necessary for the examples.\n\n\n```python\n! pip install redis openai python-dotenv openai[datalib]\n```\n\n## OpenAI API Key\nCreate a .env file and add your OpenAI key to it\n\n\n```python\nOPENAI_API_KEY=your_key\n```\n\n## OpenAI Setup\nKey load + helper function for chat completion\n\n\n```python\nimport openai\nimport os\nfrom dotenv import load_dotenv\n\nload_dotenv()\nopenai.api_key = os.getenv(\"OPENAI_API_KEY\")\n\ndef get_completion(prompt, model=\"gpt-3.5-turbo\"):\n    messages = [{\"role\": \"user\", \"content\": prompt}]\n    response = openai.ChatCompletion.create(\n        model=model,\n        messages=messages,\n        temperature=0, \n    )\n    return response.choices[0].message[\"content\"]\n```\n\n## Experiment - Chat Completion on a Topic outside of the Model's Knowledge Cutoff Date\nGpt-3.5-turbo was trained on data up to Sep 2021.  Let's ask it a question about something that is beyond that date.  In this case, the FTX/Sam Bankman-Fried scandal.\n\n\n```python\nprompt = \"Is Sam Bankman-Fried's company, FTX, considered a well-managed company?\"\nresponse = get_completion(prompt)\nprint(response)\n```\n\n## Incomplete Information\nAn unfortunate behavior of these AI systems is the system will provide a confident-sounding response - even when the system is not confident with its result.  One way to mitigate this is prompt re-engineering, as seen below.\n\n\n```python\nprompt =\"Is Sam Bankman-Fried's company, FTX, considered a well-managed company?  If you don't know for certain, say unknown.\"\nresponse = get_completion(prompt)\nprint(response)\n```\n\n## Additional Context\nAnother way to combat incomplete information is to give the system more information such that it can make intelligent decisions vs guessing.  We'll use Redis as the source for that additional context.  We'll pull in business news articles from after the GPT knowledge cut-off date such that the system will have a better understanding of how FTX was actually managed. \n\n## Start the Redis Stack Docker container\n\n\n```python\n! docker compose up -d\n```\n\n## Connect Redis client\n\n\n```python\nfrom redis import from_url\n\nREDIS_URL = 'redis://localhost:6379'\nclient = from_url(REDIS_URL)\nclient.ping()\n```\n\n\n\n\n    True\n\n\n\n## Create Index\n[FT.CREATE](https://redis.io/commands/ft.create/)\n\n\n```python\nfrom redis.commands.search.field import TextField, VectorField\nfrom redis.commands.search.indexDefinition import IndexDefinition, IndexType\n\nschema = [ VectorField('$.vector', \n            \"FLAT\", \n            {   \"TYPE\": 'FLOAT32', \n                \"DIM\": 1536, \n                \"DISTANCE_METRIC\": \"COSINE\"\n            },  as_name='vector' ),\n            TextField('$.content', as_name='content')\n        ]\nidx_def = IndexDefinition(index_type=IndexType.JSON, prefix=['doc:'])\ntry: \n    client.ft('idx').dropindex()\nexcept:\n    pass\nclient.ft('idx').create_index(schema, definition=idx_def)\n```\n\n\n\n\n    b'OK'"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 5f2b8edb-023c-484d-9dbb-f6d04314afcc
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"5f2b8edb-023c-484d-9dbb-f6d04314afcc\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6d3bac1e156fc9f35a5a0494df7d73ead8a1a3d7fd0e67ee56c3acbc3bb6a6fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"29496040-7325-4d75-a2b4-e932a53c236e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"5a3b3720e58160761893cfb720af22c43f7f35ac661406fd3470cf8d8e5622dc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2e2087b1-a10c-4a0c-a002-ae1a6f514d74\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6f8b77ba77826d25bb17bac139a8a74e96a27b2ea3b399b86bb720d397d6930f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1658, \"end_char_idx\": 4564, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "doc_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "ref_doc_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "chroma:document": "```python\nprompt =\"Is Sam Bankman-Fried's company, FTX, considered a well-managed company?  If you don't know for certain, say unknown.\"\nresponse = get_completion(prompt)\nprint(response)\n```\n\n## Additional Context\nAnother way to combat incomplete information is to give the system more information such that it can make intelligent decisions vs guessing.  We'll use Redis as the source for that additional context.  We'll pull in business news articles from after the GPT knowledge cut-off date such that the system will have a better understanding of how FTX was actually managed. \n\n## Start the Redis Stack Docker container\n\n\n```python\n! docker compose up -d\n```\n\n## Connect Redis client\n\n\n```python\nfrom redis import from_url\n\nREDIS_URL = 'redis://localhost:6379'\nclient = from_url(REDIS_URL)\nclient.ping()\n```\n\n\n\n\n    True\n\n\n\n## Create Index\n[FT.CREATE](https://redis.io/commands/ft.create/)\n\n\n```python\nfrom redis.commands.search.field import TextField, VectorField\nfrom redis.commands.search.indexDefinition import IndexDefinition, IndexType\n\nschema = [ VectorField('$.vector', \n            \"FLAT\", \n            {   \"TYPE\": 'FLOAT32', \n                \"DIM\": 1536, \n                \"DISTANCE_METRIC\": \"COSINE\"\n            },  as_name='vector' ),\n            TextField('$.content', as_name='content')\n        ]\nidx_def = IndexDefinition(index_type=IndexType.JSON, prefix=['doc:'])\ntry: \n    client.ft('idx').dropindex()\nexcept:\n    pass\nclient.ft('idx').create_index(schema, definition=idx_def)\n```\n\n\n\n\n    b'OK'\n\n\n\n## Load Data Files into Redis as JSON Objects with Text and Vector Fields\n[Redis JSON](https://redis.io/docs/stack/json/)\n\n\n```python\nimport os\nimport openai\n\ndirectory = './assets/'\nmodel='text-embedding-3-small'\ni = 1\nfor file in os.listdir(directory):\n    with open(os.path.join(directory, file)) as f:\n        content = f.read()\n        vector = openai.Embedding.create(input = [content], model = model)['data'][0]['embedding']\n        client.json().set(f'doc:{i}', '$', {'content': content, 'vector': vector})\n    i += 1\n```\n\n## Embed the Question and Perform VSS to find the most relevant document\n[KNN Search](https://redis.io/docs/stack/search/reference/vectors/#knn-search)\n\n\n```python\nfrom redis.commands.search.query import Query\nimport numpy as np\n\nvec = np.array(openai.Embedding.create(input = [prompt], model = model)['data'][0]['embedding'], dtype=np.float32).tobytes()\nq = Query('*=>[KNN 1 @vector $query_vec AS vector_score]')\\\n    .sort_by('vector_score')\\\n    .return_fields('content')\\\n    .dialect(2)    \nparams = {\"query_vec\": vec}\n\ncontext = client.ft('idx').search(q, query_params=params).docs[0].content\nprint(context)\n```\n\n    Embattled Crypto Exchange FTX Files for Bankruptcy\n    \n    Nov. 11, 2022\n    On Monday, Sam Bankman-Fried, the chief executive of the cryptocurrency exchange FTX, took to Twitter to reassure his customers: \u201cFTX is fine,\u201d he wrote."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 2e2087b1-a10c-4a0c-a002-ae1a6f514d74
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"2e2087b1-a10c-4a0c-a002-ae1a6f514d74\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6d3bac1e156fc9f35a5a0494df7d73ead8a1a3d7fd0e67ee56c3acbc3bb6a6fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5f2b8edb-023c-484d-9dbb-f6d04314afcc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6451b0596e8c21f3360f5bb5f4fec5580b03019a334d4b4fb5491ed6d06a6a7e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a50ae065-0609-40ea-8610-d3df116fb916\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a70e33542e45a8ab605868decd6c71edd5f94f50b9e50309a56f0b591bcebcbd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3170, \"end_char_idx\": 6457, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "doc_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "ref_doc_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "chroma:document": "b'OK'\n\n\n\n## Load Data Files into Redis as JSON Objects with Text and Vector Fields\n[Redis JSON](https://redis.io/docs/stack/json/)\n\n\n```python\nimport os\nimport openai\n\ndirectory = './assets/'\nmodel='text-embedding-3-small'\ni = 1\nfor file in os.listdir(directory):\n    with open(os.path.join(directory, file)) as f:\n        content = f.read()\n        vector = openai.Embedding.create(input = [content], model = model)['data'][0]['embedding']\n        client.json().set(f'doc:{i}', '$', {'content': content, 'vector': vector})\n    i += 1\n```\n\n## Embed the Question and Perform VSS to find the most relevant document\n[KNN Search](https://redis.io/docs/stack/search/reference/vectors/#knn-search)\n\n\n```python\nfrom redis.commands.search.query import Query\nimport numpy as np\n\nvec = np.array(openai.Embedding.create(input = [prompt], model = model)['data'][0]['embedding'], dtype=np.float32).tobytes()\nq = Query('*=>[KNN 1 @vector $query_vec AS vector_score]')\\\n    .sort_by('vector_score')\\\n    .return_fields('content')\\\n    .dialect(2)    \nparams = {\"query_vec\": vec}\n\ncontext = client.ft('idx').search(q, query_params=params).docs[0].content\nprint(context)\n```\n\n    Embattled Crypto Exchange FTX Files for Bankruptcy\n    \n    Nov. 11, 2022\n    On Monday, Sam Bankman-Fried, the chief executive of the cryptocurrency exchange FTX, took to Twitter to reassure his customers: \u201cFTX is fine,\u201d he wrote. \u201cAssets are fine.\u201d\n    \n    On Friday, FTX announced that it was filing for bankruptcy, capping an extraordinary week of corporate drama that has upended crypto markets, sent shock waves through an industry struggling to gain mainstream credibility and sparked government investigations that could lead to more damaging revelations or even criminal charges.\n    \n    In a statement on Twitter, the company said that Mr. Bankman-Fried had resigned, with John J. Ray III, a corporate turnaround specialist, taking over as chief executive.\n    \n    The speed of FTX\u2019s downfall has left crypto insiders stunned. Just days ago, Mr. Bankman-Fried was considered one of the smartest leaders in the crypto industry, an influential figure in Washington who was lobbying to shape regulations. And FTX was widely viewed as one of the most stable and responsible companies in the freewheeling, loosely regulated crypto industry.\n    \n    \u201cHere we are, with one of the richest people in the world, his net worth dropping to zero, his business dropping to zero,\u201d said Jared Ellias, a bankruptcy professor at Harvard Law School. \u201cThe velocity of this failure is just unbelievable.\u201d\n    \n    Now, the bankruptcy has set up a rush among investors and customers to salvage funds from what remains of FTX. A surge of customers tried to withdraw funds from the platform this week, and the company couldn\u2019t meet the demand. The exchange owes as much as $8 billion, according to people familiar with its finances.\n    \n    FTX\u2019s collapse has destabilized the crypto industry, which was already reeling from a crash in the spring that drained $1 trillion from the market. The prices of the leading cryptocurrencies, Bitcoin and Ether, have plummeted. The crypto lender BlockFi, which was closely entangled with FTX, announced on Thursday that it was suspending operations as a result of FTX\u2019s collapse.\n    \n    Mr."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: a50ae065-0609-40ea-8610-d3df116fb916
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a50ae065-0609-40ea-8610-d3df116fb916\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6d3bac1e156fc9f35a5a0494df7d73ead8a1a3d7fd0e67ee56c3acbc3bb6a6fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2e2087b1-a10c-4a0c-a002-ae1a6f514d74\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"eb7ff24ccd44940a3d01f070b8d41e100da98198fc5c46dcbfb18dc4b3b7ea42\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0735e0ed-7e0a-48f7-bc04-399ef4ea6006\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b2153be666f5197b61503fa31fcdd4b832fd34f27f3c69be0b1bde62eaba416f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4565, \"end_char_idx\": 8177, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "doc_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "ref_doc_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "chroma:document": "\u201cAssets are fine.\u201d\n    \n    On Friday, FTX announced that it was filing for bankruptcy, capping an extraordinary week of corporate drama that has upended crypto markets, sent shock waves through an industry struggling to gain mainstream credibility and sparked government investigations that could lead to more damaging revelations or even criminal charges.\n    \n    In a statement on Twitter, the company said that Mr. Bankman-Fried had resigned, with John J. Ray III, a corporate turnaround specialist, taking over as chief executive.\n    \n    The speed of FTX\u2019s downfall has left crypto insiders stunned. Just days ago, Mr. Bankman-Fried was considered one of the smartest leaders in the crypto industry, an influential figure in Washington who was lobbying to shape regulations. And FTX was widely viewed as one of the most stable and responsible companies in the freewheeling, loosely regulated crypto industry.\n    \n    \u201cHere we are, with one of the richest people in the world, his net worth dropping to zero, his business dropping to zero,\u201d said Jared Ellias, a bankruptcy professor at Harvard Law School. \u201cThe velocity of this failure is just unbelievable.\u201d\n    \n    Now, the bankruptcy has set up a rush among investors and customers to salvage funds from what remains of FTX. A surge of customers tried to withdraw funds from the platform this week, and the company couldn\u2019t meet the demand. The exchange owes as much as $8 billion, according to people familiar with its finances.\n    \n    FTX\u2019s collapse has destabilized the crypto industry, which was already reeling from a crash in the spring that drained $1 trillion from the market. The prices of the leading cryptocurrencies, Bitcoin and Ether, have plummeted. The crypto lender BlockFi, which was closely entangled with FTX, announced on Thursday that it was suspending operations as a result of FTX\u2019s collapse.\n    \n    Mr. Bankman-Fried was backed by some of the highest-profile venture capital investors in Silicon Valley, including Sequoia Capital and Lightspeed Venture Partners. Some of those investors, facing questions about how closely they scrutinized FTX before they put money into it, have said that their nine-figure investments in the crypto exchange are now essentially worthless.\n    \n    The company\u2019s demise has also set off a reckoning over risky practices that have become pervasive in crypto, an industry that was founded partly as a corrective to the type of dangerous financial engineering that caused the 2008 economic crisis.\n    \n    \u201cI\u2019m really sorry, again, that we ended up here,\u201d Mr. Bankman-Fried said on Twitter on Friday. \u201cHopefully this can bring some amount of transparency, trust, and governance.\u201d\n    \n    The bankruptcy filing marks the start of what will probably be months or even years of legal fallout, as lawyers try to work out whether the exchange can ever continue to operate in some form and customers demand compensation. FTX is already the target of investigations by the Securities and Exchange Commission and the Justice Department, with investigators focused on whether the company improperly used customer funds to prop up Alameda Research, a trading firm that Mr. Bankman-Fried also founded.\n    \n    The bankruptcy filing included FTX, its U.S. arm and Alameda. According to a bare-bones legal filing in U.S. Bankruptcy Court in Delaware, FTX has assets valued between $10 billion and $50 billion, with the size of its liabilities in the same range. The company has more than 100,000 creditors, the filing said.\n    \n    The bankruptcy is a stunning fall from grace for the 30-year-old Mr."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 0735e0ed-7e0a-48f7-bc04-399ef4ea6006
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0735e0ed-7e0a-48f7-bc04-399ef4ea6006\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6d3bac1e156fc9f35a5a0494df7d73ead8a1a3d7fd0e67ee56c3acbc3bb6a6fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a50ae065-0609-40ea-8610-d3df116fb916\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c4c8b339bbc1e5b65179e10d1e4840c3827c0fd3ad3343412335d2e6fabc1d4f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5a5a074a-4e2e-4356-9fa5-f85f0c9e2e3b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"12e28056c6f8a916c4f0b7be06f8de47e4a3bcb67333d3d611275c27ff046d25\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6293, \"end_char_idx\": 9656, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "doc_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "ref_doc_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "chroma:document": "The crypto lender BlockFi, which was closely entangled with FTX, announced on Thursday that it was suspending operations as a result of FTX\u2019s collapse.\n    \n    Mr. Bankman-Fried was backed by some of the highest-profile venture capital investors in Silicon Valley, including Sequoia Capital and Lightspeed Venture Partners. Some of those investors, facing questions about how closely they scrutinized FTX before they put money into it, have said that their nine-figure investments in the crypto exchange are now essentially worthless.\n    \n    The company\u2019s demise has also set off a reckoning over risky practices that have become pervasive in crypto, an industry that was founded partly as a corrective to the type of dangerous financial engineering that caused the 2008 economic crisis.\n    \n    \u201cI\u2019m really sorry, again, that we ended up here,\u201d Mr. Bankman-Fried said on Twitter on Friday. \u201cHopefully this can bring some amount of transparency, trust, and governance.\u201d\n    \n    The bankruptcy filing marks the start of what will probably be months or even years of legal fallout, as lawyers try to work out whether the exchange can ever continue to operate in some form and customers demand compensation. FTX is already the target of investigations by the Securities and Exchange Commission and the Justice Department, with investigators focused on whether the company improperly used customer funds to prop up Alameda Research, a trading firm that Mr. Bankman-Fried also founded.\n    \n    The bankruptcy filing included FTX, its U.S. arm and Alameda. According to a bare-bones legal filing in U.S. Bankruptcy Court in Delaware, FTX has assets valued between $10 billion and $50 billion, with the size of its liabilities in the same range. The company has more than 100,000 creditors, the filing said.\n    \n    The bankruptcy is a stunning fall from grace for the 30-year-old Mr. Bankman-Fried, who cultivated a reputation as a boy genius with a host of endearing quirks, including a habit of sleeping on a beanbag at the office. At one point, he was one of the richest people in the industry, with an estimated fortune of $24 billion. He hobnobbed with actors, professional athletes and former world leaders.\n    \n    Mr. Bankman-Fried\u2019s crypto empire had an elaborate structure. The bankruptcy filing lists more than 130 corporate entities affiliated with FTX and Alameda. But as of June, FTX had only about 300 employees, a point of pride for Mr. Bankman-Fried, who said he had resisted calls from venture investors to hire more staff.\n    \n    \u201cWe told them additional employees added too quickly were net negative,\u201d Mr. Bankman-Fried said on Twitter in June. \u201cThey could take it or leave it.\u201d\n    \n    Unusually for a major start-up, none of FTX\u2019s investors had seats on the board, which instead consisted of Mr. Bankman-Fried, another FTX executive and a lawyer in Antigua and Barbuda.\n    \n    FTX and Alameda were based in the Bahamas, where Mr. Bankman-Fried and a small circle of top executives called most of the shots and lived together in a luxury resort. Officially, Alameda was run by Caroline Ellison, a former trader for the hedge fund Jane Street, but Mr. Bankman-Fried was heavily involved, contributing to the decision-making on big trades, according to a person familiar with the matter.\n    \n    In addition to Mr. Bankman-Fried and Ms."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 5a5a074a-4e2e-4356-9fa5-f85f0c9e2e3b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"5a5a074a-4e2e-4356-9fa5-f85f0c9e2e3b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6d3bac1e156fc9f35a5a0494df7d73ead8a1a3d7fd0e67ee56c3acbc3bb6a6fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0735e0ed-7e0a-48f7-bc04-399ef4ea6006\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"e2f0a939b9ab2e26d1d5aa1c4cddaf516e2c29675aedfbe82a8a1ae27427cde7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bd1752b7-7b38-4201-823f-8d1f6db8dde2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f0785849910c8ffbd3a0e71ac91ea9380511878168d243920912b8a520b60862\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8038, \"end_char_idx\": 11334, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "doc_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "ref_doc_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "chroma:document": "The company has more than 100,000 creditors, the filing said.\n    \n    The bankruptcy is a stunning fall from grace for the 30-year-old Mr. Bankman-Fried, who cultivated a reputation as a boy genius with a host of endearing quirks, including a habit of sleeping on a beanbag at the office. At one point, he was one of the richest people in the industry, with an estimated fortune of $24 billion. He hobnobbed with actors, professional athletes and former world leaders.\n    \n    Mr. Bankman-Fried\u2019s crypto empire had an elaborate structure. The bankruptcy filing lists more than 130 corporate entities affiliated with FTX and Alameda. But as of June, FTX had only about 300 employees, a point of pride for Mr. Bankman-Fried, who said he had resisted calls from venture investors to hire more staff.\n    \n    \u201cWe told them additional employees added too quickly were net negative,\u201d Mr. Bankman-Fried said on Twitter in June. \u201cThey could take it or leave it.\u201d\n    \n    Unusually for a major start-up, none of FTX\u2019s investors had seats on the board, which instead consisted of Mr. Bankman-Fried, another FTX executive and a lawyer in Antigua and Barbuda.\n    \n    FTX and Alameda were based in the Bahamas, where Mr. Bankman-Fried and a small circle of top executives called most of the shots and lived together in a luxury resort. Officially, Alameda was run by Caroline Ellison, a former trader for the hedge fund Jane Street, but Mr. Bankman-Fried was heavily involved, contributing to the decision-making on big trades, according to a person familiar with the matter.\n    \n    In addition to Mr. Bankman-Fried and Ms. Ellison, the circle of executives running FTX included Nishad Singh, FTX\u2019s director of engineering, and Gary Wang, the chief technology officer. Few others had visibility into how the company was run: When the firm collapsed this week, lower-ranking employees were left confused and blindsided, according to people familiar with the matter. Mr. Singh and Ms. Ellison did not respond to requests for comment; Mr. Wang could not immediately be reached.\n    \n    As a crypto exchange, FTX provided a marketplace for customers to buy, sell and store a wide range of digital currencies. Most of its revenue stemmed from a risky type of trade \u2014 in which crypto investors borrowed money to make huge bets on the future prices of cryptocurrencies \u2014 that remains illegal in the United States. But Mr. Bankman-Fried also ran a smaller U.S. affiliate that offered more basic trading options.\n    \n    Mr. Bankman-Fried\u2019s problems started over the weekend, when the chief executive of Binance, the largest crypto exchange, suggested publicly that FTX might be on shaky financial footing. A rush of customers tried to withdraw their crypto holdings from the platform, and FTX was unable to meet the demand.\n    \n    On Tuesday, Mr. Bankman-Fried said he had struck a deal to sell FTX to Binance. But after reviewing the company\u2019s financial documents, Binance\u2019s chief executive, Changpeng Zhao, pulled out of the agreement, leaving Mr. Bankman-Fried with limited options.\n    \n    In calls with investors and messages to employees this week, he apologized repeatedly and stressed that he was working hard to raise money and resolve the situation. But the hole was ultimately too big to fill."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: bd1752b7-7b38-4201-823f-8d1f6db8dde2
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"bd1752b7-7b38-4201-823f-8d1f6db8dde2\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6d3bac1e156fc9f35a5a0494df7d73ead8a1a3d7fd0e67ee56c3acbc3bb6a6fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5a5a074a-4e2e-4356-9fa5-f85f0c9e2e3b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6ba1f9af98be74e352933606270277132c8d94db6dd4a4db2fea3e61455ee66a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"939e3568-6777-4074-ba3a-b0461627f973\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9ca998b635bf1b5ffd80e15f3d78bca5560b3ef1b0c58578bf8f7095e46337a9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9616, \"end_char_idx\": 13136, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "doc_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "ref_doc_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "chroma:document": "In addition to Mr. Bankman-Fried and Ms. Ellison, the circle of executives running FTX included Nishad Singh, FTX\u2019s director of engineering, and Gary Wang, the chief technology officer. Few others had visibility into how the company was run: When the firm collapsed this week, lower-ranking employees were left confused and blindsided, according to people familiar with the matter. Mr. Singh and Ms. Ellison did not respond to requests for comment; Mr. Wang could not immediately be reached.\n    \n    As a crypto exchange, FTX provided a marketplace for customers to buy, sell and store a wide range of digital currencies. Most of its revenue stemmed from a risky type of trade \u2014 in which crypto investors borrowed money to make huge bets on the future prices of cryptocurrencies \u2014 that remains illegal in the United States. But Mr. Bankman-Fried also ran a smaller U.S. affiliate that offered more basic trading options.\n    \n    Mr. Bankman-Fried\u2019s problems started over the weekend, when the chief executive of Binance, the largest crypto exchange, suggested publicly that FTX might be on shaky financial footing. A rush of customers tried to withdraw their crypto holdings from the platform, and FTX was unable to meet the demand.\n    \n    On Tuesday, Mr. Bankman-Fried said he had struck a deal to sell FTX to Binance. But after reviewing the company\u2019s financial documents, Binance\u2019s chief executive, Changpeng Zhao, pulled out of the agreement, leaving Mr. Bankman-Fried with limited options.\n    \n    In calls with investors and messages to employees this week, he apologized repeatedly and stressed that he was working hard to raise money and resolve the situation. But the hole was ultimately too big to fill.\n    \n    FTX\u2019s bankruptcy is the latest \u2014 and by far the biggest \u2014 in a series of bankruptcies that have shaken the crypto world this year. After a market crash in the spring, two crypto lending companies, Celsius Network and Voyager Digital, filed for bankruptcy, kicking off months of legal maneuvering over how their remaining assets should be divided. In an ironic twist, FTX had recently won an auction to buy Voyager\u2019s remaining assets.\n    \n    As it enters its own bankruptcy process, FTX will be led by Mr. Ray, who has ample experience managing distressed situations. He helped manage Enron after the collapse of its business in an accounting fraud scandal in 2001. And he helped liquidate the trust of the subprime mortgage company ResCap after its 2012 bankruptcy.\n    \n    The bankruptcy proceedings may be only the beginning of Mr. Bankman-Fried\u2019s legal troubles. Federal investigators are examining the relationship between FTX and Alameda, and customers are likely to file lawsuits.\n    \n    Mr. Bankman-Fried\u2019s old allies have quickly abandoned him. On Thursday night, the team running the FTX Future Fund, a charitable group that Mr. Bankman-Fried bankrolled, announced that they were resigning.\n    \n    \u201cWe were shocked and immensely saddened to learn of the recent events at FTX,\u201d they wrote in a statement. \u201cWe have fundamental questions about the legitimacy and integrity of the business operations that were funding the FTX Foundation and the Future Fund.\u201d\n    \n    Not long ago, Mr. Bankman-Fried was performing a comedy routine onstage at a conference with Anthony Scaramucci, the former White House communications director and a business partner of FTX.\n    \n    \u201cI\u2019m disappointed,\u201d Mr. Scaramucci said in an interview on CNBC on Friday. \u201cDuped, I guess, is the right word.\u201d"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 939e3568-6777-4074-ba3a-b0461627f973
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb", "title": "Redis as a Context Store with OpenAI Chat", "tokens": 3100, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"939e3568-6777-4074-ba3a-b0461627f973\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6d3bac1e156fc9f35a5a0494df7d73ead8a1a3d7fd0e67ee56c3acbc3bb6a6fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bd1752b7-7b38-4201-823f-8d1f6db8dde2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisqna/redisqna.ipynb\", \"title\": \"Redis as a Context Store with OpenAI Chat\", \"tokens\": 3100, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"88ed8b948a6ed03c7080424b9b66bb197ca47a6a79ef45dc776e710068f284a1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11290, \"end_char_idx\": 14107, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "doc_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "ref_doc_id": "8b852d07-6c30-4ff7-b1ba-2143a8f1d8f2", "chroma:document": "But the hole was ultimately too big to fill.\n    \n    FTX\u2019s bankruptcy is the latest \u2014 and by far the biggest \u2014 in a series of bankruptcies that have shaken the crypto world this year. After a market crash in the spring, two crypto lending companies, Celsius Network and Voyager Digital, filed for bankruptcy, kicking off months of legal maneuvering over how their remaining assets should be divided. In an ironic twist, FTX had recently won an auction to buy Voyager\u2019s remaining assets.\n    \n    As it enters its own bankruptcy process, FTX will be led by Mr. Ray, who has ample experience managing distressed situations. He helped manage Enron after the collapse of its business in an accounting fraud scandal in 2001. And he helped liquidate the trust of the subprime mortgage company ResCap after its 2012 bankruptcy.\n    \n    The bankruptcy proceedings may be only the beginning of Mr. Bankman-Fried\u2019s legal troubles. Federal investigators are examining the relationship between FTX and Alameda, and customers are likely to file lawsuits.\n    \n    Mr. Bankman-Fried\u2019s old allies have quickly abandoned him. On Thursday night, the team running the FTX Future Fund, a charitable group that Mr. Bankman-Fried bankrolled, announced that they were resigning.\n    \n    \u201cWe were shocked and immensely saddened to learn of the recent events at FTX,\u201d they wrote in a statement. \u201cWe have fundamental questions about the legitimacy and integrity of the business operations that were funding the FTX Foundation and the Future Fund.\u201d\n    \n    Not long ago, Mr. Bankman-Fried was performing a comedy routine onstage at a conference with Anthony Scaramucci, the former White House communications director and a business partner of FTX.\n    \n    \u201cI\u2019m disappointed,\u201d Mr. Scaramucci said in an interview on CNBC on Friday. \u201cDuped, I guess, is the right word.\u201d\n    \n\n\n## Repeat the Question to OpenAI with context\nNow that we have relevant context, add that to the prompt to OpenAI and get a very different response.\n\n\n```python\nprompt = f\"\"\"\nUsing the information delimited by triple backticks, answer this question: Is Sam Bankman-Fried's company, FTX, considered a well-managed company?\n\nContext: ```{context}```\n\"\"\"\n\nresponse = get_completion(prompt)\nprint(response)\n```\n\n    No, Sam Bankman-Fried's company FTX is not considered a well-managed company as it has filed for bankruptcy and owes as much as $8 billion to its creditors. The collapse of FTX has destabilized the crypto industry, and the company is already the target of investigations by the Securities and Exchange Commission and the Justice Department. FTX was widely viewed as one of the most stable and responsible companies in the freewheeling, loosely regulated crypto industry, but its risky practices have become pervasive in crypto, leading to a reckoning."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 44253cf5-41e6-4829-83dd-469a8a7f8222
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"44253cf5-41e6-4829-83dd-469a8a7f8222\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"689cdec8-6d71-44ff-ace1-6265fe26cd02\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"99729068d07d382e306777f280f8426e203ffde162e1024fe16ecf6dc8d9ed3f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d5c02822-d5bb-432f-8612-540f19631ea1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8d2d0ca97243dde3ea8c9044ae310d031613be0ed046441b15d692edd9f715e9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 709, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "doc_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "ref_doc_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "chroma:document": "# Redis Vectors as JSON with OpenAI\nThis notebook expands on the other Redis OpenAI-cookbook examples with examples of how to use JSON with vectors.\n[Storing Vectors in JSON](https://redis.io/docs/stack/search/reference/vectors/#storing-vectors-in-json)\n\n## Prerequisites\n* Redis instance with the Redis Search and Redis JSON modules\n* Redis-py client lib\n* OpenAI API key\n\n## Installation\nInstall Python modules necessary for the examples.\n\n\n```python\n! pip install redis openai python-dotenv openai[datalib]\n```\n\n## OpenAI API Key\nCreate a .env file and add your OpenAI key to it\n\n\n```python\nOPENAI_API_KEY=your_key\n```\n\n## Create Text Vectors\nCreate embeddings (array of floats) of the news excerpts below."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: d5c02822-d5bb-432f-8612-540f19631ea1
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"d5c02822-d5bb-432f-8612-540f19631ea1\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"689cdec8-6d71-44ff-ace1-6265fe26cd02\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"99729068d07d382e306777f280f8426e203ffde162e1024fe16ecf6dc8d9ed3f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"44253cf5-41e6-4829-83dd-469a8a7f8222\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6a8155b7745299f7a5071e046cc4daa80a18e71223de5ddda7af3c6464321136\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0c3bf501-be71-4cc7-8871-6a69598eb0fa\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b52d340c57370ee932a3f11059eb95a1125a10f80eaf12ffe964ec2d6c67af04\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3176, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "doc_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "ref_doc_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "chroma:document": "# Redis Vectors as JSON with OpenAI\nThis notebook expands on the other Redis OpenAI-cookbook examples with examples of how to use JSON with vectors.\n[Storing Vectors in JSON](https://redis.io/docs/stack/search/reference/vectors/#storing-vectors-in-json)\n\n## Prerequisites\n* Redis instance with the Redis Search and Redis JSON modules\n* Redis-py client lib\n* OpenAI API key\n\n## Installation\nInstall Python modules necessary for the examples.\n\n\n```python\n! pip install redis openai python-dotenv openai[datalib]\n```\n\n## OpenAI API Key\nCreate a .env file and add your OpenAI key to it\n\n\n```python\nOPENAI_API_KEY=your_key\n```\n\n## Create Text Vectors\nCreate embeddings (array of floats) of the news excerpts below.\n\n\n```python\nimport openai\nimport os\nfrom dotenv import load_dotenv\n\nload_dotenv()\nopenai.api_key = os.getenv(\"OPENAI_API_KEY\")\n\ndef get_vector(text, model=\"text-embedding-3-small\"):\n    text = text.replace(\"\\n\", \" \")\n    return openai.Embedding.create(input = [text], model = model)['data'][0]['embedding']\n\ntext_1 = \"\"\"Japan narrowly escapes recession\n\nJapan's economy teetered on the brink of a technical recession in the three months to September, figures show.\n\nRevised figures indicated growth of just 0.1% - and a similar-sized contraction in the previous quarter. On an annual basis, the data suggests annual growth of just 0.2%, suggesting a much more hesitant recovery than had previously been thought. A common technical definition of a recession is two successive quarters of negative growth.\nThe government was keen to play down the worrying implications of the data. \"I maintain the view that Japan's economy remains in a minor adjustment phase in an upward climb, and we will monitor developments carefully,\" said economy minister Heizo Takenaka. But in the face of the strengthening yen making exports less competitive and indications of weakening economic conditions ahead, observers were less sanguine. \"It's painting a picture of a recovery... much patchier than previously thought,\" said Paul Sheard, economist at Lehman Brothers in Tokyo. Improvements in the job market apparently have yet to feed through to domestic demand, with private consumption up just 0.2% in the third quarter.\n\"\"\"\n\ntext_2 = \"\"\"Dibaba breaks 5,000m world record\n\nEthiopia's Tirunesh Dibaba set a new world record in winning the women's 5,000m at the Boston Indoor Games.\n\nDibaba won in 14 minutes 32.93 seconds to erase the previous world indoor mark of 14:39.29 set by another Ethiopian, Berhane Adera, in Stuttgart last year. But compatriot Kenenisa Bekele's record hopes were dashed when he miscounted his laps in the men's 3,000m and staged his sprint finish a lap too soon. Ireland's Alistair Cragg won in 7:39.89 as Bekele battled to second in 7:41.42. \"I didn't want to sit back and get out-kicked,\" said Cragg. \"So I kept on the pace. The plan was to go with 500m to go no matter what, but when Bekele made the mistake that was it. The race was mine.\" Sweden's Carolina Kluft, the Olympic heptathlon champion, and Slovenia's Jolanda Ceplak had winning performances, too. Kluft took the long jump at 6.63m, while Ceplak easily won the women's 800m in 2:01.52. \n\"\"\""}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 0c3bf501-be71-4cc7-8871-6a69598eb0fa
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0c3bf501-be71-4cc7-8871-6a69598eb0fa\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"689cdec8-6d71-44ff-ace1-6265fe26cd02\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"99729068d07d382e306777f280f8426e203ffde162e1024fe16ecf6dc8d9ed3f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d5c02822-d5bb-432f-8612-540f19631ea1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3ee45f7ea78cfc54c622379b96f056bbfa593a639ec7bd1c44e7fc7f520ec0f8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e9a87d8a-de9c-43c0-8fa4-aba635075ca8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8aa3fa81465b5cd8bfb62fd6cb659f9a29efc7516604b196434a77fe9e194f48\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3179, \"end_char_idx\": 6356, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "doc_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "ref_doc_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "chroma:document": "text_3 = \"\"\"Google's toolbar sparks concern\n\nSearch engine firm Google has released a trial tool which is concerning some net users because it directs people to pre-selected commercial websites.\n\nThe AutoLink feature comes with Google's latest toolbar and provides links in a webpage to Amazon.com if it finds a book's ISBN number on the site. It also links to Google's map service, if there is an address, or to car firm Carfax, if there is a licence plate. Google said the feature, available only in the US, \"adds useful links\". But some users are concerned that Google's dominant position in the search engine market place could mean it would be giving a competitive edge to firms like Amazon.\n\nAutoLink works by creating a link to a website based on information contained in a webpage - even if there is no link specified and whether or not the publisher of the page has given permission.\n\nIf a user clicks the AutoLink feature in the Google toolbar then a webpage with a book's unique ISBN number would link directly to Amazon's website. It could mean online libraries that list ISBN book numbers find they are directing users to Amazon.com whether they like it or not. Websites which have paid for advertising on their pages may also be directing people to rival services. Dan Gillmor, founder of Grassroots Media, which supports citizen-based media, said the tool was a \"bad idea, and an unfortunate move by a company that is looking to continue its hypergrowth\". In a statement Google said the feature was still only in beta, ie trial, stage and that the company welcomed feedback from users. It said: \"The user can choose never to click on the AutoLink button, and web pages she views will never be modified. \"In addition, the user can choose to disable the AutoLink feature entirely at any time.\"\n\nThe new tool has been compared to the Smart Tags feature from Microsoft by some users. It was widely criticised by net users and later dropped by Microsoft after concerns over trademark use were raised. Smart Tags allowed Microsoft to link any word on a web page to another site chosen by the company. Google said none of the companies which received AutoLinks had paid for the service. Some users said AutoLink would only be fair if websites had to sign up to allow the feature to work on their pages or if they received revenue for any \"click through\" to a commercial site. Cory Doctorow, European outreach coordinator for digital civil liberties group Electronic Fronter Foundation, said that Google should not be penalised for its market dominance. \"Of course Google should be allowed to direct people to whatever proxies it chooses. \"But as an end user I would want to know - 'Can I choose to use this service?, 'How much is Google being paid?', 'Can I substitute my own companies for the ones chosen by Google?'.\" Mr Doctorow said the only objection would be if users were forced into using AutoLink or \"tricked into using the service\".\n\"\"\"\n\ndoc_1 = {\"content\": text_1, \"vector\": get_vector(text_1)}\ndoc_2 = {\"content\": text_2, \"vector\": get_vector(text_2)}\ndoc_3 = {\"content\": text_3, \"vector\": get_vector(text_3)}\n\n```\n\n## Start the Redis Stack Docker container"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: e9a87d8a-de9c-43c0-8fa4-aba635075ca8
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e9a87d8a-de9c-43c0-8fa4-aba635075ca8\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"689cdec8-6d71-44ff-ace1-6265fe26cd02\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"99729068d07d382e306777f280f8426e203ffde162e1024fe16ecf6dc8d9ed3f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0c3bf501-be71-4cc7-8871-6a69598eb0fa\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"10469fa35bff15c4642d88ed5984e3709dde32dc8b5e687b00602a88724a932c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a6635bf3-50f0-4fc8-934f-d36ffaaa55d7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b9286c8669040e5a17b1e8e84fda5b717dfcc7b4eb9245ee5613eb8a88da4958\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6359, \"end_char_idx\": 8920, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "doc_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "ref_doc_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "chroma:document": "```python\n! docker compose up -d\n```\n\n    \u001b[1A\u001b[1B\u001b[0G\u001b[?25l[+] Running 0/0\n     \u283f Container redisjson-redis-1  Starting                                   \u001b[34m0.1s \u001b[0m\n    \u001b[?25h\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 0/1\n     \u283f Container redisjson-redis-1  Starting                                   \u001b[34m0.2s \u001b[0m\n    \u001b[?25h\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 0/1\n     \u283f Container redisjson-redis-1  Starting                                   \u001b[34m0.3s \u001b[0m\n    \u001b[?25h\u001b[1A\u001b[1A\u001b[0G\u001b[?25l[+] Running 0/1\n     \u283f Container redisjson-redis-1  Starting                                   \u001b[34m0.4s \u001b[0m\n    \u001b[?25h\u001b[1A\u001b[1A\u001b[0G\u001b[?25l\u001b[34m[+] Running 1/1\u001b[0m\n     \u001b[32m\u2714\u001b[0m Container redisjson-redis-1  \u001b[32mStarted\u001b[0m                                    \u001b[34m0.4s \u001b[0m\n    \u001b[?25h\n\n## Connect Redis client\n\n\n```python\nfrom redis import from_url\n\nREDIS_URL = 'redis://localhost:6379'\nclient = from_url(REDIS_URL)\nclient.ping()\n```\n\n\n\n\n    True\n\n\n\n## Create Index\n[FT.CREATE](https://redis.io/commands/ft.create/)\n\n\n```python\nfrom redis.commands.search.field import TextField, VectorField\nfrom redis.commands.search.indexDefinition import IndexDefinition, IndexType\n\nschema = [ VectorField('$.vector', \n            \"FLAT\", \n            {   \"TYPE\": 'FLOAT32', \n                \"DIM\": len(doc_1['vector']), \n                \"DISTANCE_METRIC\": \"COSINE\"\n            },  as_name='vector' ),\n            TextField('$.content', as_name='content')\n        ]\nidx_def = IndexDefinition(index_type=IndexType.JSON, prefix=['doc:'])\ntry: \n    client.ft('idx').dropindex()\nexcept:\n    pass\nclient.ft('idx').create_index(schema, definition=idx_def)\n```\n\n\n\n\n    b'OK'\n\n\n\n## Load Data into Redis as JSON objects\n[Redis JSON](https://redis.io/docs/stack/json/)\n\n\n```python\nclient.json().set('doc:1', '$', doc_1)\nclient.json().set('doc:2', '$', doc_2)\nclient.json().set('doc:3', '$', doc_3)\n```\n\n\n\n\n    True\n\n\n\n# Semantic Search\nGiven a sports-related article, search Redis via Vector Similarity Search (VSS) for similar articles.  \n[KNN Search](https://redis.io/docs/stack/search/reference/vectors/#knn-search)\n\n\n```python\nfrom redis.commands.search.query import Query\nimport numpy as np\n\ntext_4 = \"\"\"Radcliffe yet to answer GB call\n\nPaula Radcliffe has been granted extra time to decide whether to compete in the World Cross-Country Championships.\n\nThe 31-year-old is concerned the event, which starts on 19 March in France, could upset her preparations for the London Marathon on 17 April. \"There is no question that Paula would be a huge asset to the GB team,\" said Zara Hyde Peters of UK Athletics."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: a6635bf3-50f0-4fc8-934f-d36ffaaa55d7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a6635bf3-50f0-4fc8-934f-d36ffaaa55d7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"689cdec8-6d71-44ff-ace1-6265fe26cd02\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"99729068d07d382e306777f280f8426e203ffde162e1024fe16ecf6dc8d9ed3f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e9a87d8a-de9c-43c0-8fa4-aba635075ca8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8fd299fb1f0636a1758d69a5b3f4e143caac9560d8b6632210cd9ffd28a467db\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e2a738fa-afb3-4b30-8c5f-7318d745e30c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"431278fcc6f7763266c02451d5e8c6b681ddd24f0951f25d768679f2244321cb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7352, \"end_char_idx\": 10431, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "doc_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "ref_doc_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "chroma:document": "```python\nfrom redis.commands.search.field import TextField, VectorField\nfrom redis.commands.search.indexDefinition import IndexDefinition, IndexType\n\nschema = [ VectorField('$.vector', \n            \"FLAT\", \n            {   \"TYPE\": 'FLOAT32', \n                \"DIM\": len(doc_1['vector']), \n                \"DISTANCE_METRIC\": \"COSINE\"\n            },  as_name='vector' ),\n            TextField('$.content', as_name='content')\n        ]\nidx_def = IndexDefinition(index_type=IndexType.JSON, prefix=['doc:'])\ntry: \n    client.ft('idx').dropindex()\nexcept:\n    pass\nclient.ft('idx').create_index(schema, definition=idx_def)\n```\n\n\n\n\n    b'OK'\n\n\n\n## Load Data into Redis as JSON objects\n[Redis JSON](https://redis.io/docs/stack/json/)\n\n\n```python\nclient.json().set('doc:1', '$', doc_1)\nclient.json().set('doc:2', '$', doc_2)\nclient.json().set('doc:3', '$', doc_3)\n```\n\n\n\n\n    True\n\n\n\n# Semantic Search\nGiven a sports-related article, search Redis via Vector Similarity Search (VSS) for similar articles.  \n[KNN Search](https://redis.io/docs/stack/search/reference/vectors/#knn-search)\n\n\n```python\nfrom redis.commands.search.query import Query\nimport numpy as np\n\ntext_4 = \"\"\"Radcliffe yet to answer GB call\n\nPaula Radcliffe has been granted extra time to decide whether to compete in the World Cross-Country Championships.\n\nThe 31-year-old is concerned the event, which starts on 19 March in France, could upset her preparations for the London Marathon on 17 April. \"There is no question that Paula would be a huge asset to the GB team,\" said Zara Hyde Peters of UK Athletics. \"But she is working out whether she can accommodate the worlds without too much compromise in her marathon training.\" Radcliffe must make a decision by Tuesday - the deadline for team nominations. British team member Hayley Yelling said the team would understand if Radcliffe opted out of the event. \"It would be fantastic to have Paula in the team,\" said the European cross-country champion. \"But you have to remember that athletics is basically an individual sport and anything achieved for the team is a bonus. \"She is not messing us around. We all understand the problem.\" Radcliffe was world cross-country champion in 2001 and 2002 but missed last year's event because of injury. In her absence, the GB team won bronze in Brussels.\n\"\"\"\n\nvec = np.array(get_vector(text_4), dtype=np.float32).tobytes()\nq = Query('*=>[KNN 3 @vector $query_vec AS vector_score]')\\\n    .sort_by('vector_score')\\\n    .return_fields('vector_score', 'content')\\\n    .dialect(2)    \nparams = {\"query_vec\": vec}\n\nresults = client.ft('idx').search(q, query_params=params)\nfor doc in results.docs:\n    print(f\"distance:{round(float(doc['vector_score']),3)} content:{doc['content']}\\n\")\n\n```\n\n    distance:0.188 content:Dibaba breaks 5,000m world record\n    \n    Ethiopia's Tirunesh Dibaba set a new world record in winning the women's 5,000m at the Boston Indoor Games.\n    \n    Dibaba won in 14 minutes 32.93 seconds to erase the previous world indoor mark of 14:39.29 set by another Ethiopian, Berhane Adera, in Stuttgart last year."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: e2a738fa-afb3-4b30-8c5f-7318d745e30c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e2a738fa-afb3-4b30-8c5f-7318d745e30c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"689cdec8-6d71-44ff-ace1-6265fe26cd02\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"99729068d07d382e306777f280f8426e203ffde162e1024fe16ecf6dc8d9ed3f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a6635bf3-50f0-4fc8-934f-d36ffaaa55d7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"57b4cc0f16012c341723768f7c712d2abf80956c3ba593d954f49859643d3a7a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"032885cb-40fc-41b2-91aa-ad7c7dec7b53\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5559c858b990357d3a95ded10a53e69b4f160ff1ee6e0a09f415b29c5a2f59b6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8810, \"end_char_idx\": 11869, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "doc_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "ref_doc_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "chroma:document": "\"There is no question that Paula would be a huge asset to the GB team,\" said Zara Hyde Peters of UK Athletics. \"But she is working out whether she can accommodate the worlds without too much compromise in her marathon training.\" Radcliffe must make a decision by Tuesday - the deadline for team nominations. British team member Hayley Yelling said the team would understand if Radcliffe opted out of the event. \"It would be fantastic to have Paula in the team,\" said the European cross-country champion. \"But you have to remember that athletics is basically an individual sport and anything achieved for the team is a bonus. \"She is not messing us around. We all understand the problem.\" Radcliffe was world cross-country champion in 2001 and 2002 but missed last year's event because of injury. In her absence, the GB team won bronze in Brussels.\n\"\"\"\n\nvec = np.array(get_vector(text_4), dtype=np.float32).tobytes()\nq = Query('*=>[KNN 3 @vector $query_vec AS vector_score]')\\\n    .sort_by('vector_score')\\\n    .return_fields('vector_score', 'content')\\\n    .dialect(2)    \nparams = {\"query_vec\": vec}\n\nresults = client.ft('idx').search(q, query_params=params)\nfor doc in results.docs:\n    print(f\"distance:{round(float(doc['vector_score']),3)} content:{doc['content']}\\n\")\n\n```\n\n    distance:0.188 content:Dibaba breaks 5,000m world record\n    \n    Ethiopia's Tirunesh Dibaba set a new world record in winning the women's 5,000m at the Boston Indoor Games.\n    \n    Dibaba won in 14 minutes 32.93 seconds to erase the previous world indoor mark of 14:39.29 set by another Ethiopian, Berhane Adera, in Stuttgart last year. But compatriot Kenenisa Bekele's record hopes were dashed when he miscounted his laps in the men's 3,000m and staged his sprint finish a lap too soon. Ireland's Alistair Cragg won in 7:39.89 as Bekele battled to second in 7:41.42. \"I didn't want to sit back and get out-kicked,\" said Cragg. \"So I kept on the pace. The plan was to go with 500m to go no matter what, but when Bekele made the mistake that was it. The race was mine.\" Sweden's Carolina Kluft, the Olympic heptathlon champion, and Slovenia's Jolanda Ceplak had winning performances, too. Kluft took the long jump at 6.63m, while Ceplak easily won the women's 800m in 2:01.52. \n    \n    \n    distance:0.268 content:Japan narrowly escapes recession\n    \n    Japan's economy teetered on the brink of a technical recession in the three months to September, figures show.\n    \n    Revised figures indicated growth of just 0.1% - and a similar-sized contraction in the previous quarter. On an annual basis, the data suggests annual growth of just 0.2%, suggesting a much more hesitant recovery than had previously been thought. A common technical definition of a recession is two successive quarters of negative growth.\n    The government was keen to play down the worrying implications of the data. \"I maintain the view that Japan's economy remains in a minor adjustment phase in an upward climb, and we will monitor developments carefully,\" said economy minister Heizo Takenaka."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 032885cb-40fc-41b2-91aa-ad7c7dec7b53
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"032885cb-40fc-41b2-91aa-ad7c7dec7b53\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"689cdec8-6d71-44ff-ace1-6265fe26cd02\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"99729068d07d382e306777f280f8426e203ffde162e1024fe16ecf6dc8d9ed3f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e2a738fa-afb3-4b30-8c5f-7318d745e30c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"bc16df6507a18d68081acc7e900d1e96e7f4abdd6d732cd2abb867f17134d11c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1154c004-1d55-4f78-8e32-9a171e316dff\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f5fd67aa2975f011b683e2f53bde92842adbc5e18bdc32524915cf8a9b43883f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10432, \"end_char_idx\": 13842, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "doc_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "ref_doc_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "chroma:document": "But compatriot Kenenisa Bekele's record hopes were dashed when he miscounted his laps in the men's 3,000m and staged his sprint finish a lap too soon. Ireland's Alistair Cragg won in 7:39.89 as Bekele battled to second in 7:41.42. \"I didn't want to sit back and get out-kicked,\" said Cragg. \"So I kept on the pace. The plan was to go with 500m to go no matter what, but when Bekele made the mistake that was it. The race was mine.\" Sweden's Carolina Kluft, the Olympic heptathlon champion, and Slovenia's Jolanda Ceplak had winning performances, too. Kluft took the long jump at 6.63m, while Ceplak easily won the women's 800m in 2:01.52. \n    \n    \n    distance:0.268 content:Japan narrowly escapes recession\n    \n    Japan's economy teetered on the brink of a technical recession in the three months to September, figures show.\n    \n    Revised figures indicated growth of just 0.1% - and a similar-sized contraction in the previous quarter. On an annual basis, the data suggests annual growth of just 0.2%, suggesting a much more hesitant recovery than had previously been thought. A common technical definition of a recession is two successive quarters of negative growth.\n    The government was keen to play down the worrying implications of the data. \"I maintain the view that Japan's economy remains in a minor adjustment phase in an upward climb, and we will monitor developments carefully,\" said economy minister Heizo Takenaka. But in the face of the strengthening yen making exports less competitive and indications of weakening economic conditions ahead, observers were less sanguine. \"It's painting a picture of a recovery... much patchier than previously thought,\" said Paul Sheard, economist at Lehman Brothers in Tokyo. Improvements in the job market apparently have yet to feed through to domestic demand, with private consumption up just 0.2% in the third quarter.\n    \n    \n    distance:0.287 content:Google's toolbar sparks concern\n    \n    Search engine firm Google has released a trial tool which is concerning some net users because it directs people to pre-selected commercial websites.\n    \n    The AutoLink feature comes with Google's latest toolbar and provides links in a webpage to Amazon.com if it finds a book's ISBN number on the site. It also links to Google's map service, if there is an address, or to car firm Carfax, if there is a licence plate. Google said the feature, available only in the US, \"adds useful links\". But some users are concerned that Google's dominant position in the search engine market place could mean it would be giving a competitive edge to firms like Amazon.\n    \n    AutoLink works by creating a link to a website based on information contained in a webpage - even if there is no link specified and whether or not the publisher of the page has given permission.\n    \n    If a user clicks the AutoLink feature in the Google toolbar then a webpage with a book's unique ISBN number would link directly to Amazon's website. It could mean online libraries that list ISBN book numbers find they are directing users to Amazon.com whether they like it or not. Websites which have paid for advertising on their pages may also be directing people to rival services. Dan Gillmor, founder of Grassroots Media, which supports citizen-based media, said the tool was a \"bad idea, and an unfortunate move by a company that is looking to continue its hypergrowth\"."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 1154c004-1d55-4f78-8e32-9a171e316dff
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"1154c004-1d55-4f78-8e32-9a171e316dff\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"689cdec8-6d71-44ff-ace1-6265fe26cd02\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"99729068d07d382e306777f280f8426e203ffde162e1024fe16ecf6dc8d9ed3f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"032885cb-40fc-41b2-91aa-ad7c7dec7b53\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"d762a3230304081cc323a17581a7ad4edf344770d5366f40d6b7262f760f2d5b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9e36d66d-400a-4797-8d63-0ed1e299bc80\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"59d57b77dacb9f73dc95a23572e4f5d54c8d60d40435da540189d34bac7c85e3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12029, \"end_char_idx\": 15330, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "doc_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "ref_doc_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "chroma:document": "\"It's painting a picture of a recovery... much patchier than previously thought,\" said Paul Sheard, economist at Lehman Brothers in Tokyo. Improvements in the job market apparently have yet to feed through to domestic demand, with private consumption up just 0.2% in the third quarter.\n    \n    \n    distance:0.287 content:Google's toolbar sparks concern\n    \n    Search engine firm Google has released a trial tool which is concerning some net users because it directs people to pre-selected commercial websites.\n    \n    The AutoLink feature comes with Google's latest toolbar and provides links in a webpage to Amazon.com if it finds a book's ISBN number on the site. It also links to Google's map service, if there is an address, or to car firm Carfax, if there is a licence plate. Google said the feature, available only in the US, \"adds useful links\". But some users are concerned that Google's dominant position in the search engine market place could mean it would be giving a competitive edge to firms like Amazon.\n    \n    AutoLink works by creating a link to a website based on information contained in a webpage - even if there is no link specified and whether or not the publisher of the page has given permission.\n    \n    If a user clicks the AutoLink feature in the Google toolbar then a webpage with a book's unique ISBN number would link directly to Amazon's website. It could mean online libraries that list ISBN book numbers find they are directing users to Amazon.com whether they like it or not. Websites which have paid for advertising on their pages may also be directing people to rival services. Dan Gillmor, founder of Grassroots Media, which supports citizen-based media, said the tool was a \"bad idea, and an unfortunate move by a company that is looking to continue its hypergrowth\". In a statement Google said the feature was still only in beta, ie trial, stage and that the company welcomed feedback from users. It said: \"The user can choose never to click on the AutoLink button, and web pages she views will never be modified. \"In addition, the user can choose to disable the AutoLink feature entirely at any time.\"\n    \n    The new tool has been compared to the Smart Tags feature from Microsoft by some users. It was widely criticised by net users and later dropped by Microsoft after concerns over trademark use were raised. Smart Tags allowed Microsoft to link any word on a web page to another site chosen by the company. Google said none of the companies which received AutoLinks had paid for the service. Some users said AutoLink would only be fair if websites had to sign up to allow the feature to work on their pages or if they received revenue for any \"click through\" to a commercial site. Cory Doctorow, European outreach coordinator for digital civil liberties group Electronic Fronter Foundation, said that Google should not be penalised for its market dominance. \"Of course Google should be allowed to direct people to whatever proxies it chooses. \"But as an end user I would want to know - 'Can I choose to use this service?, 'How much is Google being paid?', 'Can I substitute my own companies for the ones chosen by Google?'.\" Mr Doctorow said the only objection would be if users were forced into using AutoLink or \"tricked into using the service\"."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 9e36d66d-400a-4797-8d63-0ed1e299bc80
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"9e36d66d-400a-4797-8d63-0ed1e299bc80\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"689cdec8-6d71-44ff-ace1-6265fe26cd02\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"99729068d07d382e306777f280f8426e203ffde162e1024fe16ecf6dc8d9ed3f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1154c004-1d55-4f78-8e32-9a171e316dff\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"28fb8baf1dfe08c713722cec1e8fa0ff5d6eb2746540db5b9990e7cd8471db95\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7a20b5e1-dd32-4e82-83b6-545f9bb9b0d4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"68b8a1c2f77389e0808b1e31504826bc9502b2ddd3dda8171adee4c77a9d1d76\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13547, \"end_char_idx\": 15715, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "doc_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "ref_doc_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "chroma:document": "Websites which have paid for advertising on their pages may also be directing people to rival services. Dan Gillmor, founder of Grassroots Media, which supports citizen-based media, said the tool was a \"bad idea, and an unfortunate move by a company that is looking to continue its hypergrowth\". In a statement Google said the feature was still only in beta, ie trial, stage and that the company welcomed feedback from users. It said: \"The user can choose never to click on the AutoLink button, and web pages she views will never be modified. \"In addition, the user can choose to disable the AutoLink feature entirely at any time.\"\n    \n    The new tool has been compared to the Smart Tags feature from Microsoft by some users. It was widely criticised by net users and later dropped by Microsoft after concerns over trademark use were raised. Smart Tags allowed Microsoft to link any word on a web page to another site chosen by the company. Google said none of the companies which received AutoLinks had paid for the service. Some users said AutoLink would only be fair if websites had to sign up to allow the feature to work on their pages or if they received revenue for any \"click through\" to a commercial site. Cory Doctorow, European outreach coordinator for digital civil liberties group Electronic Fronter Foundation, said that Google should not be penalised for its market dominance. \"Of course Google should be allowed to direct people to whatever proxies it chooses. \"But as an end user I would want to know - 'Can I choose to use this service?, 'How much is Google being paid?', 'Can I substitute my own companies for the ones chosen by Google?'.\" Mr Doctorow said the only objection would be if users were forced into using AutoLink or \"tricked into using the service\".\n    \n    \n\n\n## Hybrid Search\nUse a combination of full text search and VSS to find a matching article.  For this scenario, we filter on a full text search of the term 'recession' and then find the KNN articles.  In this case, business-related.  Reminder document #1 was about a recession in Japan.\n[Hybrid Queries](https://redis.io/docs/stack/search/reference/vectors/#hybrid-queries)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 7a20b5e1-dd32-4e82-83b6-545f9bb9b0d4
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb", "title": "Redis Vectors as JSON with OpenAI", "tokens": 4391, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"7a20b5e1-dd32-4e82-83b6-545f9bb9b0d4\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"689cdec8-6d71-44ff-ace1-6265fe26cd02\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"99729068d07d382e306777f280f8426e203ffde162e1024fe16ecf6dc8d9ed3f\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9e36d66d-400a-4797-8d63-0ed1e299bc80\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/redis/redisjson/redisjson.ipynb\", \"title\": \"Redis Vectors as JSON with OpenAI\", \"tokens\": 4391, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"0fc8f9c94e7ba5b44c6544dc9a67341e0bd3f820e1dcb615ed2a27e7470d700d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13973, \"end_char_idx\": 18915, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "doc_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "ref_doc_id": "689cdec8-6d71-44ff-ace1-6265fe26cd02", "chroma:document": "It said: \"The user can choose never to click on the AutoLink button, and web pages she views will never be modified. \"In addition, the user can choose to disable the AutoLink feature entirely at any time.\"\n    \n    The new tool has been compared to the Smart Tags feature from Microsoft by some users. It was widely criticised by net users and later dropped by Microsoft after concerns over trademark use were raised. Smart Tags allowed Microsoft to link any word on a web page to another site chosen by the company. Google said none of the companies which received AutoLinks had paid for the service. Some users said AutoLink would only be fair if websites had to sign up to allow the feature to work on their pages or if they received revenue for any \"click through\" to a commercial site. Cory Doctorow, European outreach coordinator for digital civil liberties group Electronic Fronter Foundation, said that Google should not be penalised for its market dominance. \"Of course Google should be allowed to direct people to whatever proxies it chooses. \"But as an end user I would want to know - 'Can I choose to use this service?, 'How much is Google being paid?', 'Can I substitute my own companies for the ones chosen by Google?'.\" Mr Doctorow said the only objection would be if users were forced into using AutoLink or \"tricked into using the service\".\n    \n    \n\n\n## Hybrid Search\nUse a combination of full text search and VSS to find a matching article.  For this scenario, we filter on a full text search of the term 'recession' and then find the KNN articles.  In this case, business-related.  Reminder document #1 was about a recession in Japan.\n[Hybrid Queries](https://redis.io/docs/stack/search/reference/vectors/#hybrid-queries)\n\n\n```python\ntext_5 = \"\"\"Ethiopia's crop production up 24%\n\nEthiopia produced 14.27 million tonnes of crops in 2004, 24% higher than in 2003 and 21% more than the average of the past five years, a report says.\n\nIn 2003, crop production totalled 11.49 million tonnes, the joint report from the Food and Agriculture Organisation and the World Food Programme said. Good rains, increased use of fertilizers and improved seeds contributed to the rise in production. Nevertheless, 2.2 million Ethiopians will still need emergency assistance.\n\nThe report calculated emergency food requirements for 2005 to be 387,500 tonnes. On top of that, 89,000 tonnes of fortified blended food and vegetable oil for \"targeted supplementary food distributions for a survival programme for children under five and pregnant and lactating women\" will be needed.\n\nIn eastern and southern Ethiopia, a prolonged drought has killed crops and drained wells. Last year, a total of 965,000 tonnes of food assistance was needed to help seven million Ethiopians. The Food and Agriculture Organisation (FAO) recommend that the food assistance is bought locally. \"Local purchase of cereals for food assistance programmes is recommended as far as possible, so as to assist domestic markets and farmers,\" said Henri Josserand, chief of FAO's Global Information and Early Warning System. Agriculture is the main economic activity in Ethiopia, representing 45% of gross domestic product. About 80% of Ethiopians depend directly or indirectly on agriculture.\n\"\"\"\n\nvec = np.array(get_vector(text_5), dtype=np.float32).tobytes()\nq = Query('@content:recession => [KNN 3 @vector $query_vec AS vector_score]')\\\n    .sort_by('vector_score')\\\n    .return_fields('vector_score', 'content')\\\n    .dialect(2)    \nparams = {\"query_vec\": vec}\n\nresults = client.ft('idx').search(q, query_params=params)\nfor doc in results.docs:\n    print(f\"distance:{round(float(doc['vector_score']),3)} content:{doc['content']}\\n\")\n```\n\n    distance:0.241 content:Japan narrowly escapes recession\n    \n    Japan's economy teetered on the brink of a technical recession in the three months to September, figures show.\n    \n    Revised figures indicated growth of just 0.1% - and a similar-sized contraction in the previous quarter. On an annual basis, the data suggests annual growth of just 0.2%, suggesting a much more hesitant recovery than had previously been thought. A common technical definition of a recession is two successive quarters of negative growth.\n    The government was keen to play down the worrying implications of the data. \"I maintain the view that Japan's economy remains in a minor adjustment phase in an upward climb, and we will monitor developments carefully,\" said economy minister Heizo Takenaka. But in the face of the strengthening yen making exports less competitive and indications of weakening economic conditions ahead, observers were less sanguine. \"It's painting a picture of a recovery... much patchier than previously thought,\" said Paul Sheard, economist at Lehman Brothers in Tokyo. Improvements in the job market apparently have yet to feed through to domestic demand, with private consumption up just 0.2% in the third quarter."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 35b72222-942c-43bc-9b4d-3ce76aca6c90
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"35b72222-942c-43bc-9b4d-3ce76aca6c90\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6acee902-5735-4d36-a385-6c034da1da8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e5a8951b-460f-4ad3-9a97-7e1bfdcac648\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2a91c59f48d3a558669bf051ea0cccd23a42c6f47c8e1eb52f8a859f4aaef26f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3594, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6acee902-5735-4d36-a385-6c034da1da8f", "doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "ref_doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "chroma:document": "# Using Typesense for Embeddings Search\n\nThis notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.\n\n### What is a Vector Database\n\nA vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.\n\n### Why use a Vector Database\n\nVector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.\n\n\n### Demo Flow\nThe demo flow is:\n- **Setup**: Import packages and set any required variables\n- **Load data**: Load a dataset and embed it using OpenAI embeddings\n- **Typesense**\n    - *Setup*: Set up the Typesense Python client. For more details go [here](https://typesense.org/docs/0.24.0/api/)\n    - *Index Data*: We'll create a collection and index it for both __titles__ and __content__.\n    - *Search Data*: Run a few example queries with various goals in mind.\n\nOnce you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.\n\n## Setup\n\nImport the required libraries and set the embedding model that we'd like to use.\n\n\n```python\n# We'll need to install the Typesense client\n!pip install typesense\n\n#Install wget to pull zip file\n!pip install wget\n```\n\n\n```python\nimport openai\n\nfrom typing import List, Iterator\nimport pandas as pd\nimport numpy as np\nimport os\nimport wget\nfrom ast import literal_eval\n\n# Typesense's client library for Python\nimport typesense\n\n# I've set this to our new embeddings model, this can be changed to the embedding model of your choice\nEMBEDDING_MODEL = \"text-embedding-3-small\"\n\n# Ignore unclosed SSL socket warnings - optional in case you get these errors\nimport warnings\n\nwarnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning) \n```\n\n## Load data\n\nIn this section we'll load embedded data that we've prepared previous to this session.\n\n\n```python\nembeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'\n\n# The file is ~700 MB so this will take some time\nwget.download(embeddings_url)\n```\n\n\n```python\nimport zipfile\nwith zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n    zip_ref.extractall(\"../data\")\n```\n\n\n```python\narticle_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: e5a8951b-460f-4ad3-9a97-7e1bfdcac648
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e5a8951b-460f-4ad3-9a97-7e1bfdcac648\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6acee902-5735-4d36-a385-6c034da1da8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"35b72222-942c-43bc-9b4d-3ce76aca6c90\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"1e7a8fe70fec5033f5fb1fa17044acd396515dbfbfd25c4bbb8cf7dc94ee28cf\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f0f769bb-6346-4d09-91dc-d41bbdefb70a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"55d75d734c6c465942a4bbe99c9e2550ca1dde90d5af183d906f3fc23ae149c3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2296, \"end_char_idx\": 4644, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6acee902-5735-4d36-a385-6c034da1da8f", "doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "ref_doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "chroma:document": "```python\n# We'll need to install the Typesense client\n!pip install typesense\n\n#Install wget to pull zip file\n!pip install wget\n```\n\n\n```python\nimport openai\n\nfrom typing import List, Iterator\nimport pandas as pd\nimport numpy as np\nimport os\nimport wget\nfrom ast import literal_eval\n\n# Typesense's client library for Python\nimport typesense\n\n# I've set this to our new embeddings model, this can be changed to the embedding model of your choice\nEMBEDDING_MODEL = \"text-embedding-3-small\"\n\n# Ignore unclosed SSL socket warnings - optional in case you get these errors\nimport warnings\n\nwarnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning) \n```\n\n## Load data\n\nIn this section we'll load embedded data that we've prepared previous to this session.\n\n\n```python\nembeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'\n\n# The file is ~700 MB so this will take some time\nwget.download(embeddings_url)\n```\n\n\n```python\nimport zipfile\nwith zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n    zip_ref.extractall(\"../data\")\n```\n\n\n```python\narticle_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')\n```\n\n\n```python\narticle_df.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>url</th>\n      <th>title</th>\n      <th>text</th>\n      <th>title_vector</th>\n      <th>content_vector</th>\n      <th>vector_id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>https://simple.wikipedia.org/wiki/April</td>\n      <td>April</td>\n      <td>April is the fourth month of the year in the J...</td>\n      <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n      <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>https://simple.wikipedia.org/wiki/August</td>\n      <td>August</td>\n      <td>August (Aug.)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: f0f769bb-6346-4d09-91dc-d41bbdefb70a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"f0f769bb-6346-4d09-91dc-d41bbdefb70a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6acee902-5735-4d36-a385-6c034da1da8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e5a8951b-460f-4ad3-9a97-7e1bfdcac648\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7cbf504b5e1f5bcc109b5f23a7a06b52775e22eeb1e548ca9bf07e707252fa46\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c2fdce29-3320-4d9d-9803-e60cf3c3a12e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0eb9f1fbcad7c87e0ae1cc80d55f22c888212d66f0901dbcd88725f848899c3c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3497, \"end_char_idx\": 5675, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6acee902-5735-4d36-a385-6c034da1da8f", "doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "ref_doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "chroma:document": "```python\narticle_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')\n```\n\n\n```python\narticle_df.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>url</th>\n      <th>title</th>\n      <th>text</th>\n      <th>title_vector</th>\n      <th>content_vector</th>\n      <th>vector_id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>https://simple.wikipedia.org/wiki/April</td>\n      <td>April</td>\n      <td>April is the fourth month of the year in the J...</td>\n      <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n      <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>https://simple.wikipedia.org/wiki/August</td>\n      <td>August</td>\n      <td>August (Aug.) is the eighth month of the year ...</td>\n      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>6</td>\n      <td>https://simple.wikipedia.org/wiki/Art</td>\n      <td>Art</td>\n      <td>Art is a creative activity that expresses imag...</td>\n      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>8</td>\n      <td>https://simple.wikipedia.org/wiki/A</td>\n      <td>A</td>\n      <td>A or a is the first letter of the English alph...</td>\n      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n      <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>9</td>\n      <td>https://simple.wikipedia.org/wiki/Air</td>\n      <td>Air</td>\n      <td>Air refers to the Earth's atmosphere."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: c2fdce29-3320-4d9d-9803-e60cf3c3a12e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c2fdce29-3320-4d9d-9803-e60cf3c3a12e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6acee902-5735-4d36-a385-6c034da1da8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f0f769bb-6346-4d09-91dc-d41bbdefb70a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3f9e1e2d7ec1d3927273924fe76b631250a45879929cf8e38a3c61766399c020\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8e0166d9-5243-4942-882e-6854a936ba4a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3f7b4d59d71a4aeb15f37ff473c5aaf54e96b32c0390919255591521f059a058\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4645, \"end_char_idx\": 6843, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6acee902-5735-4d36-a385-6c034da1da8f", "doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "ref_doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "chroma:document": "is the eighth month of the year ...</td>\n      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>6</td>\n      <td>https://simple.wikipedia.org/wiki/Art</td>\n      <td>Art</td>\n      <td>Art is a creative activity that expresses imag...</td>\n      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>8</td>\n      <td>https://simple.wikipedia.org/wiki/A</td>\n      <td>A</td>\n      <td>A or a is the first letter of the English alph...</td>\n      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n      <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>9</td>\n      <td>https://simple.wikipedia.org/wiki/Air</td>\n      <td>Air</td>\n      <td>Air refers to the Earth's atmosphere. Air is a...</td>\n      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n      <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n\n```python\n# Read vectors from strings back into a list\narticle_df['title_vector'] = article_df.title_vector.apply(literal_eval)\narticle_df['content_vector'] = article_df.content_vector.apply(literal_eval)\n\n# Set vector_id to be a string\narticle_df['vector_id'] = article_df['vector_id'].apply(str)\n```\n\n\n```python\narticle_df.info(show_counts=True)\n```\n\n    <class 'pandas.core.frame.DataFrame'>\n    RangeIndex: 25000 entries, 0 to 24999\n    Data columns (total 7 columns):\n     #   Column          Non-Null Count  Dtype \n    ---  ------          --------------  ----- \n     0   id              25000 non-null  int64 \n     1   url             25000 non-null  object\n     2   title           25000 non-null  object\n     3   text            25000 non-null  object\n     4   title_vector    25000 non-null  object\n     5   content_vector  25000 non-null  object\n     6   vector_id       25000 non-null  object\n    dtypes: int64(1), object(6)\n    memory usage: 1.3+ MB"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 8e0166d9-5243-4942-882e-6854a936ba4a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"8e0166d9-5243-4942-882e-6854a936ba4a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6acee902-5735-4d36-a385-6c034da1da8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c2fdce29-3320-4d9d-9803-e60cf3c3a12e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3df6f49c2f9b9707d27ab66c4db8b12984adf9523c41930532679c0bb71d7707\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"59c07658-6371-4227-8a1d-d8f23330c7eb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"084fafb325520eb33296139dcc48c0723a8178600ea83911ab4b1b154ceeaa1f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5676, \"end_char_idx\": 8300, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6acee902-5735-4d36-a385-6c034da1da8f", "doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "ref_doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "chroma:document": "Air is a...</td>\n      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n      <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n\n```python\n# Read vectors from strings back into a list\narticle_df['title_vector'] = article_df.title_vector.apply(literal_eval)\narticle_df['content_vector'] = article_df.content_vector.apply(literal_eval)\n\n# Set vector_id to be a string\narticle_df['vector_id'] = article_df['vector_id'].apply(str)\n```\n\n\n```python\narticle_df.info(show_counts=True)\n```\n\n    <class 'pandas.core.frame.DataFrame'>\n    RangeIndex: 25000 entries, 0 to 24999\n    Data columns (total 7 columns):\n     #   Column          Non-Null Count  Dtype \n    ---  ------          --------------  ----- \n     0   id              25000 non-null  int64 \n     1   url             25000 non-null  object\n     2   title           25000 non-null  object\n     3   text            25000 non-null  object\n     4   title_vector    25000 non-null  object\n     5   content_vector  25000 non-null  object\n     6   vector_id       25000 non-null  object\n    dtypes: int64(1), object(6)\n    memory usage: 1.3+ MB\n\n\n## Typesense\n\nThe next vector store we'll look at is [Typesense](https://typesense.org/), which is an open source, in-memory search engine, that you can either self-host or run on [Typesense Cloud](https://cloud.typesense.org).\n\nTypesense focuses on performance by storing the entire index in RAM (with a backup on disk) and also focuses on providing an out-of-the-box developer experience by simplifying available options and setting good defaults. It also lets you combine attribute-based filtering together with vector queries.\n\nFor this example, we will set up a local docker-based Typesense server, index our vectors in Typesense and then do some nearest-neighbor search queries. If you use Typesense Cloud, you can skip the docker setup part and just obtain the hostname and API keys from your cluster dashboard.\n\n### Setup\n\nTo run Typesense locally, you'll need [Docker](https://www.docker.com/). Following the instructions contained in the Typesense documentation [here](https://typesense.org/docs/guide/install-typesense.html#docker-compose), we created an example docker-compose.yml file in this repo saved at [./typesense/docker-compose.yml](./typesense/docker-compose.yml).\n\nAfter starting Docker, you can start Typesense locally by navigating to the `examples/vector_databases/typesense/` directory and running `docker-compose up -d`.\n\nThe default API key is set to `xyz` in the Docker compose file, and the default Typesense port to `8108`."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 59c07658-6371-4227-8a1d-d8f23330c7eb
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"59c07658-6371-4227-8a1d-d8f23330c7eb\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6acee902-5735-4d36-a385-6c034da1da8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8e0166d9-5243-4942-882e-6854a936ba4a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"095065432ef77c16bf4510c0bf368af330b12566e1aa669b7f1ba372cde99f2e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2dc2a2af-9daf-4f5a-b32d-16995662fc4f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"80fd9de12e06d8a76d5f1cc38bf576e9a5fb1604235eef258f985d538f3ec540\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6846, \"end_char_idx\": 8938, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6acee902-5735-4d36-a385-6c034da1da8f", "doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "ref_doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "chroma:document": "## Typesense\n\nThe next vector store we'll look at is [Typesense](https://typesense.org/), which is an open source, in-memory search engine, that you can either self-host or run on [Typesense Cloud](https://cloud.typesense.org).\n\nTypesense focuses on performance by storing the entire index in RAM (with a backup on disk) and also focuses on providing an out-of-the-box developer experience by simplifying available options and setting good defaults. It also lets you combine attribute-based filtering together with vector queries.\n\nFor this example, we will set up a local docker-based Typesense server, index our vectors in Typesense and then do some nearest-neighbor search queries. If you use Typesense Cloud, you can skip the docker setup part and just obtain the hostname and API keys from your cluster dashboard.\n\n### Setup\n\nTo run Typesense locally, you'll need [Docker](https://www.docker.com/). Following the instructions contained in the Typesense documentation [here](https://typesense.org/docs/guide/install-typesense.html#docker-compose), we created an example docker-compose.yml file in this repo saved at [./typesense/docker-compose.yml](./typesense/docker-compose.yml).\n\nAfter starting Docker, you can start Typesense locally by navigating to the `examples/vector_databases/typesense/` directory and running `docker-compose up -d`.\n\nThe default API key is set to `xyz` in the Docker compose file, and the default Typesense port to `8108`.\n\n\n```python\nimport typesense\n\ntypesense_client = \\\n    typesense.Client({\n        \"nodes\": [{\n            \"host\": \"localhost\",  # For Typesense Cloud use xxx.a1.typesense.net\n            \"port\": \"8108\",       # For Typesense Cloud use 443\n            \"protocol\": \"http\"    # For Typesense Cloud use https\n          }],\n          \"api_key\": \"xyz\",\n          \"connection_timeout_seconds\": 60\n        })\n```\n\n### Index data\n\nTo index vectors in Typesense, we'll first create a Collection (which is a collection of Documents) and turn on vector indexing for a particular field. You can even store multiple vector fields in a single document."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 2dc2a2af-9daf-4f5a-b32d-16995662fc4f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"2dc2a2af-9daf-4f5a-b32d-16995662fc4f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6acee902-5735-4d36-a385-6c034da1da8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"59c07658-6371-4227-8a1d-d8f23330c7eb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"75eef4d1175767d1efd2c59211e460eb191201dfdfc4c2c29c9a073a603209e5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"aaed3e50-f2f7-4820-80ae-0194f8232a06\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0440dbd67a67af32b5a21cba9b4f772e822c871315ff337767aee466213ca7bd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6acee902-5735-4d36-a385-6c034da1da8f", "doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "ref_doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "chroma:document": "```python\nimport typesense\n\ntypesense_client = \\\n    typesense.Client({\n        \"nodes\": [{\n            \"host\": \"localhost\",  # For Typesense Cloud use xxx.a1.typesense.net\n            \"port\": \"8108\",       # For Typesense Cloud use 443\n            \"protocol\": \"http\"    # For Typesense Cloud use https\n          }],\n          \"api_key\": \"xyz\",\n          \"connection_timeout_seconds\": 60\n        })\n```\n\n### Index data\n\nTo index vectors in Typesense, we'll first create a Collection (which is a collection of Documents) and turn on vector indexing for a particular field. You can even store multiple vector fields in a single document.\n\n\n```python\n# Delete existing collections if they already exist\ntry:\n    typesense_client.collections['wikipedia_articles'].delete()\nexcept Exception as e:\n    pass\n\n# Create a new collection\n\nschema = {\n    \"name\": \"wikipedia_articles\",\n    \"fields\": [\n        {\n            \"name\": \"content_vector\",\n            \"type\": \"float[]\",\n            \"num_dim\": len(article_df['content_vector'][0])\n        },\n        {\n            \"name\": \"title_vector\",\n            \"type\": \"float[]\",\n            \"num_dim\": len(article_df['title_vector'][0])\n        }\n    ]\n}\n\ncreate_response = typesense_client.collections.create(schema)\nprint(create_response)\n\nprint(\"Created new collection wikipedia-articles\")\n```\n\n    {'created_at': 1687165065, 'default_sorting_field': '', 'enable_nested_fields': False, 'fields': [{'facet': False, 'index': True, 'infix': False, 'locale': '', 'name': 'content_vector', 'num_dim': 1536, 'optional': False, 'sort': False, 'type': 'float[]'}, {'facet': False, 'index': True, 'infix': False, 'locale': '', 'name': 'title_vector', 'num_dim': 1536, 'optional': False, 'sort': False, 'type': 'float[]'}], 'name': 'wikipedia_articles', 'num_documents': 0, 'symbols_to_index': [], 'token_separators': []}\n    Created new collection wikipedia-articles\n\n\n\n```python\n# Upsert the vector data into the collection we just created\n#\n# Note: This can take a few minutes, especially if your on an M1 and running docker in an emulated mode\n\nprint(\"Indexing vectors in Typesense...\")\n\ndocument_counter = 0\ndocuments_batch = []\n\nfor k,v in article_df.iterrows():\n    # Create a document with the vector data\n\n    # Notice how you can add any fields that you haven't added to the schema to the document.\n    # These will be stored on disk and returned when the document is a hit.\n    # This is useful to store attributes required for display purposes.\n\n    document = {\n        \"title_vector\": v[\"title_vector\"],\n        \"content_vector\": v[\"content_vector\"],\n        \"title\": v[\"title\"],\n        \"content\": v[\"text\"],\n    }\n    documents_batch.append(document)\n    document_counter = document_counter + 1\n\n    # Upsert a batch of 100 documents\n    if document_counter % 100 == 0 or document_counter == len(article_df):\n        response = typesense_client.collections['wikipedia_articles'].documents.import_(documents_batch)\n        # print(response)\n\n        documents_batch = []\n        print(f\"Processed {document_counter} / {len(article_df)} \")\n\nprint(f\"Imported ({len(article_df)}) articles.\")\n```\n\n    Indexing vectors in Typesense.\n    Processed 100 /"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: aaed3e50-f2f7-4820-80ae-0194f8232a06
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"aaed3e50-f2f7-4820-80ae-0194f8232a06\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6acee902-5735-4d36-a385-6c034da1da8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2dc2a2af-9daf-4f5a-b32d-16995662fc4f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"2af19a722b4df3f4e84bfc2e2a21d1a21471de0b9d004488d9fc0460c6963135\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dd01b28c-d216-4989-b78f-e74d567e0994\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e740111a14d7ed1b2bcdfee2f41000d2cc098ccacef9758058b5b4c84f1a129e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6acee902-5735-4d36-a385-6c034da1da8f", "doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "ref_doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "chroma:document": "```python\n# Upsert the vector data into the collection we just created\n#\n# Note: This can take a few minutes, especially if your on an M1 and running docker in an emulated mode\n\nprint(\"Indexing vectors in Typesense...\")\n\ndocument_counter = 0\ndocuments_batch = []\n\nfor k,v in article_df.iterrows():\n    # Create a document with the vector data\n\n    # Notice how you can add any fields that you haven't added to the schema to the document.\n    # These will be stored on disk and returned when the document is a hit.\n    # This is useful to store attributes required for display purposes.\n\n    document = {\n        \"title_vector\": v[\"title_vector\"],\n        \"content_vector\": v[\"content_vector\"],\n        \"title\": v[\"title\"],\n        \"content\": v[\"text\"],\n    }\n    documents_batch.append(document)\n    document_counter = document_counter + 1\n\n    # Upsert a batch of 100 documents\n    if document_counter % 100 == 0 or document_counter == len(article_df):\n        response = typesense_client.collections['wikipedia_articles'].documents.import_(documents_batch)\n        # print(response)\n\n        documents_batch = []\n        print(f\"Processed {document_counter} / {len(article_df)} \")\n\nprint(f\"Imported ({len(article_df)}) articles.\")\n```\n\n    Indexing vectors in Typesense.\n    Processed 100 / 25000 \n    Processed 200 / 25000 \n    Processed 300 / 25000 \n    Processed 400 / 25000 \n    Processed 500 / 25000 \n    Processed 600 / 25000 \n    Processed 700 / 25000 \n    Processed 800 / 25000 \n    Processed 900 / 25000 \n    Processed 1000 / 25000 \n    Processed 1100 / 25000 \n    Processed 1200 / 25000 \n    Processed 1300 / 25000 \n    Processed 1400 / 25000 \n    Processed 1500 / 25000 \n    Processed 1600 / 25000 \n    Processed 1700 / 25000 \n    Processed 1800 / 25000 \n    Processed 1900 / 25000 \n    Processed 2000 / 25000 \n    Processed 2100 / 25000 \n    Processed 2200 / 25000 \n    Processed 2300 / 25000 \n    Processed 2400 / 25000 \n    Processed 2500 / 25000 \n    Processed 2600 / 25000 \n    Processed 2700 / 25000 \n    Processed 2800 / 25000 \n    Processed 2900 / 25000 \n    Processed 3000 / 25000 \n    Processed 3100 / 25000 \n    Processed 3200 / 25000 \n    Processed 3300 / 25000 \n    Processed 3400 / 25000 \n    Processed 3500 / 25000 \n    Processed 3600 / 25000 \n    Processed 3700 / 25000 \n    Processed 3800"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: dd01b28c-d216-4989-b78f-e74d567e0994
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"dd01b28c-d216-4989-b78f-e74d567e0994\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6acee902-5735-4d36-a385-6c034da1da8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"aaed3e50-f2f7-4820-80ae-0194f8232a06\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"361d6cacec6e6d602b69c234152d958d6d5c2f7cd7f707366f90fbbc068d3639\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"da930984-29a9-4c78-94e8-067f3d801bd8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a58bfc04366523ea0a6a10ba35571f4d56ffa547d9fd796e19d8980dfb948ee3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11662, \"end_char_idx\": 13317, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6acee902-5735-4d36-a385-6c034da1da8f", "doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "ref_doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "chroma:document": "25000 \n    Processed 800 / 25000 \n    Processed 900 / 25000 \n    Processed 1000 / 25000 \n    Processed 1100 / 25000 \n    Processed 1200 / 25000 \n    Processed 1300 / 25000 \n    Processed 1400 / 25000 \n    Processed 1500 / 25000 \n    Processed 1600 / 25000 \n    Processed 1700 / 25000 \n    Processed 1800 / 25000 \n    Processed 1900 / 25000 \n    Processed 2000 / 25000 \n    Processed 2100 / 25000 \n    Processed 2200 / 25000 \n    Processed 2300 / 25000 \n    Processed 2400 / 25000 \n    Processed 2500 / 25000 \n    Processed 2600 / 25000 \n    Processed 2700 / 25000 \n    Processed 2800 / 25000 \n    Processed 2900 / 25000 \n    Processed 3000 / 25000 \n    Processed 3100 / 25000 \n    Processed 3200 / 25000 \n    Processed 3300 / 25000 \n    Processed 3400 / 25000 \n    Processed 3500 / 25000 \n    Processed 3600 / 25000 \n    Processed 3700 / 25000 \n    Processed 3800 / 25000 \n    Processed 3900 / 25000 \n    Processed 4000 / 25000 \n    Processed 4100 / 25000 \n    Processed 4200 / 25000 \n    Processed 4300 / 25000 \n    Processed 4400 / 25000 \n    Processed 4500 / 25000 \n    Processed 4600 / 25000 \n    Processed 4700 / 25000 \n    Processed 4800 / 25000 \n    Processed 4900 / 25000 \n    Processed 5000 / 25000 \n    Processed 5100 / 25000 \n    Processed 5200 / 25000 \n    Processed 5300 / 25000 \n    Processed 5400 / 25000 \n    Processed 5500 / 25000 \n    Processed 5600 / 25000 \n    Processed 5700 / 25000 \n    Processed 5800 / 25000 \n    Processed 5900 / 25000 \n    Processed 6000 / 25000 \n    Processed 6100 / 25000 \n    Processed 6200 / 25000 \n    Processed 6300 / 25000 \n    Processed 6400 / 25000 \n    Processed 6500 / 25000 \n    Processed 6600 / 25000"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: da930984-29a9-4c78-94e8-067f3d801bd8
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"da930984-29a9-4c78-94e8-067f3d801bd8\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6acee902-5735-4d36-a385-6c034da1da8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dd01b28c-d216-4989-b78f-e74d567e0994\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"af8daee82685ddb510d1724708aae135081928047702e9a0e94ef53ff4d565e2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fb2fa127-058d-45c1-82aa-1e63990a5445\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"df9720db75276e6165c6e3b2dc655e31019e4ae142c7fab10afb43eebf320983\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12455, \"end_char_idx\": 14101, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6acee902-5735-4d36-a385-6c034da1da8f", "doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "ref_doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "chroma:document": "Processed 3600 / 25000 \n    Processed 3700 / 25000 \n    Processed 3800 / 25000 \n    Processed 3900 / 25000 \n    Processed 4000 / 25000 \n    Processed 4100 / 25000 \n    Processed 4200 / 25000 \n    Processed 4300 / 25000 \n    Processed 4400 / 25000 \n    Processed 4500 / 25000 \n    Processed 4600 / 25000 \n    Processed 4700 / 25000 \n    Processed 4800 / 25000 \n    Processed 4900 / 25000 \n    Processed 5000 / 25000 \n    Processed 5100 / 25000 \n    Processed 5200 / 25000 \n    Processed 5300 / 25000 \n    Processed 5400 / 25000 \n    Processed 5500 / 25000 \n    Processed 5600 / 25000 \n    Processed 5700 / 25000 \n    Processed 5800 / 25000 \n    Processed 5900 / 25000 \n    Processed 6000 / 25000 \n    Processed 6100 / 25000 \n    Processed 6200 / 25000 \n    Processed 6300 / 25000 \n    Processed 6400 / 25000 \n    Processed 6500 / 25000 \n    Processed 6600 / 25000 \n    Processed 6700 / 25000 \n    Processed 6800 / 25000 \n    Processed 6900 / 25000 \n    Processed 7000 / 25000 \n    Processed 7100 / 25000 \n    Processed 7200 / 25000 \n    Processed 7300 / 25000 \n    Processed 7400 / 25000 \n    Processed 7500 / 25000 \n    Processed 7600 / 25000 \n    Processed 7700 / 25000 \n    Processed 7800 / 25000 \n    Processed 7900 / 25000 \n    Processed 8000 / 25000 \n    Processed 8100 / 25000 \n    Processed 8200 / 25000 \n    Processed 8300 / 25000 \n    Processed 8400 / 25000 \n    Processed 8500 / 25000 \n    Processed 8600 / 25000 \n    Processed 8700 / 25000 \n    Processed 8800 / 25000 \n    Processed 8900 / 25000 \n    Processed 9000 / 25000 \n    Processed 9100 / 25000 \n    Processed 9200 / 25000 \n    Processed 9300 / 25000 \n    Processed 9400 / 25000"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: fb2fa127-058d-45c1-82aa-1e63990a5445
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"fb2fa127-058d-45c1-82aa-1e63990a5445\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6acee902-5735-4d36-a385-6c034da1da8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"da930984-29a9-4c78-94e8-067f3d801bd8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"150b855d002077b3c287feb3ead463edf90e3d927dfc2863986f4c9cd3186197\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ddae28dc-ca66-4161-81f3-ad503ba93b51\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7ec2973f64034579cebccea6270625b62f239a0041b894b76a87b39fc847afec\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13254, \"end_char_idx\": 14931, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6acee902-5735-4d36-a385-6c034da1da8f", "doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "ref_doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "chroma:document": "/ 25000 \n    Processed 6500 / 25000 \n    Processed 6600 / 25000 \n    Processed 6700 / 25000 \n    Processed 6800 / 25000 \n    Processed 6900 / 25000 \n    Processed 7000 / 25000 \n    Processed 7100 / 25000 \n    Processed 7200 / 25000 \n    Processed 7300 / 25000 \n    Processed 7400 / 25000 \n    Processed 7500 / 25000 \n    Processed 7600 / 25000 \n    Processed 7700 / 25000 \n    Processed 7800 / 25000 \n    Processed 7900 / 25000 \n    Processed 8000 / 25000 \n    Processed 8100 / 25000 \n    Processed 8200 / 25000 \n    Processed 8300 / 25000 \n    Processed 8400 / 25000 \n    Processed 8500 / 25000 \n    Processed 8600 / 25000 \n    Processed 8700 / 25000 \n    Processed 8800 / 25000 \n    Processed 8900 / 25000 \n    Processed 9000 / 25000 \n    Processed 9100 / 25000 \n    Processed 9200 / 25000 \n    Processed 9300 / 25000 \n    Processed 9400 / 25000 \n    Processed 9500 / 25000 \n    Processed 9600 / 25000 \n    Processed 9700 / 25000 \n    Processed 9800 / 25000 \n    Processed 9900 / 25000 \n    Processed 10000 / 25000 \n    Processed 10100 / 25000 \n    Processed 10200 / 25000 \n    Processed 10300 / 25000 \n    Processed 10400 / 25000 \n    Processed 10500 / 25000 \n    Processed 10600 / 25000 \n    Processed 10700 / 25000 \n    Processed 10800 / 25000 \n    Processed 10900 / 25000 \n    Processed 11000 / 25000 \n    Processed 11100 / 25000 \n    Processed 11200 / 25000 \n    Processed 11300 / 25000 \n    Processed 11400 / 25000 \n    Processed 11500 / 25000 \n    Processed 11600 / 25000 \n    Processed 11700 / 25000 \n    Processed 11800 / 25000 \n    Processed 11900 / 25000 \n    Processed 12000 / 25000 \n    Processed 12100 / 25000 \n    Processed 12200 / 25000 \n    Processed 12300 /"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: ddae28dc-ca66-4161-81f3-ad503ba93b51
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"ddae28dc-ca66-4161-81f3-ad503ba93b51\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6acee902-5735-4d36-a385-6c034da1da8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fb2fa127-058d-45c1-82aa-1e63990a5445\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a40a3d3d94810b91e2748ee90f0baea730d3490d11844a42c2acd63d774fb504\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f1f8608f-d7d7-4570-a501-4daa97c14c64\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c9c38900b3c582612849178156fa59e7f64e37aee204b59154ccea47d86e8556\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14051, \"end_char_idx\": 15749, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6acee902-5735-4d36-a385-6c034da1da8f", "doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "ref_doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "chroma:document": "Processed 9300 / 25000 \n    Processed 9400 / 25000 \n    Processed 9500 / 25000 \n    Processed 9600 / 25000 \n    Processed 9700 / 25000 \n    Processed 9800 / 25000 \n    Processed 9900 / 25000 \n    Processed 10000 / 25000 \n    Processed 10100 / 25000 \n    Processed 10200 / 25000 \n    Processed 10300 / 25000 \n    Processed 10400 / 25000 \n    Processed 10500 / 25000 \n    Processed 10600 / 25000 \n    Processed 10700 / 25000 \n    Processed 10800 / 25000 \n    Processed 10900 / 25000 \n    Processed 11000 / 25000 \n    Processed 11100 / 25000 \n    Processed 11200 / 25000 \n    Processed 11300 / 25000 \n    Processed 11400 / 25000 \n    Processed 11500 / 25000 \n    Processed 11600 / 25000 \n    Processed 11700 / 25000 \n    Processed 11800 / 25000 \n    Processed 11900 / 25000 \n    Processed 12000 / 25000 \n    Processed 12100 / 25000 \n    Processed 12200 / 25000 \n    Processed 12300 / 25000 \n    Processed 12400 / 25000 \n    Processed 12500 / 25000 \n    Processed 12600 / 25000 \n    Processed 12700 / 25000 \n    Processed 12800 / 25000 \n    Processed 12900 / 25000 \n    Processed 13000 / 25000 \n    Processed 13100 / 25000 \n    Processed 13200 / 25000 \n    Processed 13300 / 25000 \n    Processed 13400 / 25000 \n    Processed 13500 / 25000 \n    Processed 13600 / 25000 \n    Processed 13700 / 25000 \n    Processed 13800 / 25000 \n    Processed 13900 / 25000 \n    Processed 14000 / 25000 \n    Processed 14100 / 25000 \n    Processed 14200 / 25000 \n    Processed 14300 / 25000 \n    Processed 14400 / 25000 \n    Processed 14500 / 25000 \n    Processed 14600 / 25000 \n    Processed 14700 / 25000 \n    Processed 14800 / 25000 \n    Processed 14900 / 25000 \n    Processed 15000 / 25000 \n    Processed 15100 / 25000"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: f1f8608f-d7d7-4570-a501-4daa97c14c64
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"f1f8608f-d7d7-4570-a501-4daa97c14c64\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6acee902-5735-4d36-a385-6c034da1da8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ddae28dc-ca66-4161-81f3-ad503ba93b51\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"cd6352f6640982d0257b84b216ac72806e58f6f3c9d49fd7c3a82468689f40cd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"007d4924-5fb6-4ba0-83db-fadf2dbd407b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"545d1e91400799db133de10c1f6050cfdbfb7962dfa8ca4090dd406116618b5d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14856, \"end_char_idx\": 16561, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6acee902-5735-4d36-a385-6c034da1da8f", "doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "ref_doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "chroma:document": "Processed 12100 / 25000 \n    Processed 12200 / 25000 \n    Processed 12300 / 25000 \n    Processed 12400 / 25000 \n    Processed 12500 / 25000 \n    Processed 12600 / 25000 \n    Processed 12700 / 25000 \n    Processed 12800 / 25000 \n    Processed 12900 / 25000 \n    Processed 13000 / 25000 \n    Processed 13100 / 25000 \n    Processed 13200 / 25000 \n    Processed 13300 / 25000 \n    Processed 13400 / 25000 \n    Processed 13500 / 25000 \n    Processed 13600 / 25000 \n    Processed 13700 / 25000 \n    Processed 13800 / 25000 \n    Processed 13900 / 25000 \n    Processed 14000 / 25000 \n    Processed 14100 / 25000 \n    Processed 14200 / 25000 \n    Processed 14300 / 25000 \n    Processed 14400 / 25000 \n    Processed 14500 / 25000 \n    Processed 14600 / 25000 \n    Processed 14700 / 25000 \n    Processed 14800 / 25000 \n    Processed 14900 / 25000 \n    Processed 15000 / 25000 \n    Processed 15100 / 25000 \n    Processed 15200 / 25000 \n    Processed 15300 / 25000 \n    Processed 15400 / 25000 \n    Processed 15500 / 25000 \n    Processed 15600 / 25000 \n    Processed 15700 / 25000 \n    Processed 15800 / 25000 \n    Processed 15900 / 25000 \n    Processed 16000 / 25000 \n    Processed 16100 / 25000 \n    Processed 16200 / 25000 \n    Processed 16300 / 25000 \n    Processed 16400 / 25000 \n    Processed 16500 / 25000 \n    Processed 16600 / 25000 \n    Processed 16700 / 25000 \n    Processed 16800 / 25000 \n    Processed 16900 / 25000 \n    Processed 17000 / 25000 \n    Processed 17100 / 25000 \n    Processed 17200 / 25000 \n    Processed 17300 / 25000 \n    Processed 17400 / 25000 \n    Processed 17500 / 25000 \n    Processed 17600 / 25000 \n    Processed 17700 / 25000 \n    Processed 17800 / 25000 \n    Processed 17900 / 25000"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 007d4924-5fb6-4ba0-83db-fadf2dbd407b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"007d4924-5fb6-4ba0-83db-fadf2dbd407b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6acee902-5735-4d36-a385-6c034da1da8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f1f8608f-d7d7-4570-a501-4daa97c14c64\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"48096d27d6e9e012f6874217e189cba3068b9cf90f2fce4076ebdb0041a8f655\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c07dad06-9a25-4598-967c-5285eabf8f88\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"509bdeeb6a8ae6bbb811ab366799b3bfbe244e264bc29cbc96b3ac71a7e07667\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15684, \"end_char_idx\": 17396, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6acee902-5735-4d36-a385-6c034da1da8f", "doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "ref_doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "chroma:document": "/ 25000 \n    Processed 15000 / 25000 \n    Processed 15100 / 25000 \n    Processed 15200 / 25000 \n    Processed 15300 / 25000 \n    Processed 15400 / 25000 \n    Processed 15500 / 25000 \n    Processed 15600 / 25000 \n    Processed 15700 / 25000 \n    Processed 15800 / 25000 \n    Processed 15900 / 25000 \n    Processed 16000 / 25000 \n    Processed 16100 / 25000 \n    Processed 16200 / 25000 \n    Processed 16300 / 25000 \n    Processed 16400 / 25000 \n    Processed 16500 / 25000 \n    Processed 16600 / 25000 \n    Processed 16700 / 25000 \n    Processed 16800 / 25000 \n    Processed 16900 / 25000 \n    Processed 17000 / 25000 \n    Processed 17100 / 25000 \n    Processed 17200 / 25000 \n    Processed 17300 / 25000 \n    Processed 17400 / 25000 \n    Processed 17500 / 25000 \n    Processed 17600 / 25000 \n    Processed 17700 / 25000 \n    Processed 17800 / 25000 \n    Processed 17900 / 25000 \n    Processed 18000 / 25000 \n    Processed 18100 / 25000 \n    Processed 18200 / 25000 \n    Processed 18300 / 25000 \n    Processed 18400 / 25000 \n    Processed 18500 / 25000 \n    Processed 18600 / 25000 \n    Processed 18700 / 25000 \n    Processed 18800 / 25000 \n    Processed 18900 / 25000 \n    Processed 19000 / 25000 \n    Processed 19100 / 25000 \n    Processed 19200 / 25000 \n    Processed 19300 / 25000 \n    Processed 19400 / 25000 \n    Processed 19500 / 25000 \n    Processed 19600 / 25000 \n    Processed 19700 / 25000 \n    Processed 19800 / 25000 \n    Processed 19900 / 25000 \n    Processed 20000 / 25000 \n    Processed 20100 / 25000 \n    Processed 20200 / 25000 \n    Processed 20300 / 25000 \n    Processed 20400 / 25000 \n    Processed 20500 / 25000 \n    Processed 20600 / 25000 \n    Processed 20700 / 25000 \n    Processed 20800 /"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: c07dad06-9a25-4598-967c-5285eabf8f88
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c07dad06-9a25-4598-967c-5285eabf8f88\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6acee902-5735-4d36-a385-6c034da1da8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"007d4924-5fb6-4ba0-83db-fadf2dbd407b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"e5f97d5e2d8e62321234c756d53211078414744f4f91a0a2c3948a9319223b29\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7cebbc6c-2e6c-4d58-a64e-60e768f78b55\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4525b48a91c24c452758aa3fac7af83d08bd6a505c16513411d85cff30147fcb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16509, \"end_char_idx\": 18214, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6acee902-5735-4d36-a385-6c034da1da8f", "doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "ref_doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "chroma:document": "Processed 17800 / 25000 \n    Processed 17900 / 25000 \n    Processed 18000 / 25000 \n    Processed 18100 / 25000 \n    Processed 18200 / 25000 \n    Processed 18300 / 25000 \n    Processed 18400 / 25000 \n    Processed 18500 / 25000 \n    Processed 18600 / 25000 \n    Processed 18700 / 25000 \n    Processed 18800 / 25000 \n    Processed 18900 / 25000 \n    Processed 19000 / 25000 \n    Processed 19100 / 25000 \n    Processed 19200 / 25000 \n    Processed 19300 / 25000 \n    Processed 19400 / 25000 \n    Processed 19500 / 25000 \n    Processed 19600 / 25000 \n    Processed 19700 / 25000 \n    Processed 19800 / 25000 \n    Processed 19900 / 25000 \n    Processed 20000 / 25000 \n    Processed 20100 / 25000 \n    Processed 20200 / 25000 \n    Processed 20300 / 25000 \n    Processed 20400 / 25000 \n    Processed 20500 / 25000 \n    Processed 20600 / 25000 \n    Processed 20700 / 25000 \n    Processed 20800 / 25000 \n    Processed 20900 / 25000 \n    Processed 21000 / 25000 \n    Processed 21100 / 25000 \n    Processed 21200 / 25000 \n    Processed 21300 / 25000 \n    Processed 21400 / 25000 \n    Processed 21500 / 25000 \n    Processed 21600 / 25000 \n    Processed 21700 / 25000 \n    Processed 21800 / 25000 \n    Processed 21900 / 25000 \n    Processed 22000 / 25000 \n    Processed 22100 / 25000 \n    Processed 22200 / 25000 \n    Processed 22300 / 25000 \n    Processed 22400 / 25000 \n    Processed 22500 / 25000 \n    Processed 22600 / 25000 \n    Processed 22700 / 25000 \n    Processed 22800 / 25000 \n    Processed 22900 / 25000 \n    Processed 23000 / 25000 \n    Processed 23100 / 25000 \n    Processed 23200 / 25000 \n    Processed 23300 / 25000 \n    Processed 23400 / 25000 \n    Processed 23500 / 25000 \n    Processed 23600 / 25000"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 7cebbc6c-2e6c-4d58-a64e-60e768f78b55
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"7cebbc6c-2e6c-4d58-a64e-60e768f78b55\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6acee902-5735-4d36-a385-6c034da1da8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c07dad06-9a25-4598-967c-5285eabf8f88\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"aa292bcd6d33b61bb24ca33db76036221a6a50138f7e8386e199f83b847ed74b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"724f6b1f-8744-4c24-9484-9c6625ad2489\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a176813de6b675a350625d89711fa62b709a4be57050b2eb9023f7249b513b66\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17321, \"end_char_idx\": 19043, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6acee902-5735-4d36-a385-6c034da1da8f", "doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "ref_doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "chroma:document": "Processed 20600 / 25000 \n    Processed 20700 / 25000 \n    Processed 20800 / 25000 \n    Processed 20900 / 25000 \n    Processed 21000 / 25000 \n    Processed 21100 / 25000 \n    Processed 21200 / 25000 \n    Processed 21300 / 25000 \n    Processed 21400 / 25000 \n    Processed 21500 / 25000 \n    Processed 21600 / 25000 \n    Processed 21700 / 25000 \n    Processed 21800 / 25000 \n    Processed 21900 / 25000 \n    Processed 22000 / 25000 \n    Processed 22100 / 25000 \n    Processed 22200 / 25000 \n    Processed 22300 / 25000 \n    Processed 22400 / 25000 \n    Processed 22500 / 25000 \n    Processed 22600 / 25000 \n    Processed 22700 / 25000 \n    Processed 22800 / 25000 \n    Processed 22900 / 25000 \n    Processed 23000 / 25000 \n    Processed 23100 / 25000 \n    Processed 23200 / 25000 \n    Processed 23300 / 25000 \n    Processed 23400 / 25000 \n    Processed 23500 / 25000 \n    Processed 23600 / 25000 \n    Processed 23700 / 25000 \n    Processed 23800 / 25000 \n    Processed 23900 / 25000 \n    Processed 24000 / 25000 \n    Processed 24100 / 25000 \n    Processed 24200 / 25000 \n    Processed 24300 / 25000 \n    Processed 24400 / 25000 \n    Processed 24500 / 25000 \n    Processed 24600 / 25000 \n    Processed 24700 / 25000 \n    Processed 24800 / 25000 \n    Processed 24900 / 25000 \n    Processed 25000 / 25000 \n    Imported (25000) articles.\n\n\n\n```python\n# Check the number of documents imported\n\ncollection = typesense_client.collections['wikipedia_articles'].retrieve()\nprint(f'Collection has {collection[\"num_documents\"]} documents')\n```\n\n    Collection has 25000 documents\n\n\n### Search Data\n\nNow that we've imported the vectors into Typesense, we can do a nearest neighbor search on the `title_vector` or `content_vector` field."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 724f6b1f-8744-4c24-9484-9c6625ad2489
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"724f6b1f-8744-4c24-9484-9c6625ad2489\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6acee902-5735-4d36-a385-6c034da1da8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7cebbc6c-2e6c-4d58-a64e-60e768f78b55\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"effdb237750aeb38c749fc8b59c790738a3b26bfeb140c87d013527b1e4b88f3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"656c9378-b7cf-45f8-8c24-ddab8a2119ab\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f01b0cb75b86adc6763a042d104eda364466e791c8d4ac51908a45ebc5cea659\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17940, \"end_char_idx\": 19698, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6acee902-5735-4d36-a385-6c034da1da8f", "doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "ref_doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "chroma:document": "22700 / 25000 \n    Processed 22800 / 25000 \n    Processed 22900 / 25000 \n    Processed 23000 / 25000 \n    Processed 23100 / 25000 \n    Processed 23200 / 25000 \n    Processed 23300 / 25000 \n    Processed 23400 / 25000 \n    Processed 23500 / 25000 \n    Processed 23600 / 25000 \n    Processed 23700 / 25000 \n    Processed 23800 / 25000 \n    Processed 23900 / 25000 \n    Processed 24000 / 25000 \n    Processed 24100 / 25000 \n    Processed 24200 / 25000 \n    Processed 24300 / 25000 \n    Processed 24400 / 25000 \n    Processed 24500 / 25000 \n    Processed 24600 / 25000 \n    Processed 24700 / 25000 \n    Processed 24800 / 25000 \n    Processed 24900 / 25000 \n    Processed 25000 / 25000 \n    Imported (25000) articles.\n\n\n\n```python\n# Check the number of documents imported\n\ncollection = typesense_client.collections['wikipedia_articles'].retrieve()\nprint(f'Collection has {collection[\"num_documents\"]} documents')\n```\n\n    Collection has 25000 documents\n\n\n### Search Data\n\nNow that we've imported the vectors into Typesense, we can do a nearest neighbor search on the `title_vector` or `content_vector` field.\n\n\n```python\ndef query_typesense(query, field='title', top_k=20):\n\n    # Creates embedding vector from user query\n    openai.api_key = os.getenv(\"OPENAI_API_KEY\", \"sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\")\n    embedded_query = openai.Embedding.create(\n        input=query,\n        model=EMBEDDING_MODEL,\n    )['data'][0]['embedding']\n\n    typesense_results = typesense_client.multi_search.perform({\n        \"searches\": [{\n            \"q\": \"*\",\n            \"collection\": \"wikipedia_articles\",\n            \"vector_query\": f\"{field}_vector:([{','.join(str(v) for v in embedded_query)}], k:{top_k})\"\n        }]\n    }, {})\n\n    return typesense_results\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 656c9378-b7cf-45f8-8c24-ddab8a2119ab
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"656c9378-b7cf-45f8-8c24-ddab8a2119ab\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6acee902-5735-4d36-a385-6c034da1da8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"724f6b1f-8744-4c24-9484-9c6625ad2489\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7489e8e439de3c85d12f8a2ce33ef7d2df4157f3e82f56ace824a24de672aaac\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"74ccbca8-38cc-4d2a-ad42-f006e5f68707\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"214a5e4f55e039124db871c7a2355222d0b9e74e281662c658e97e9d8b6f1424\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18294, \"end_char_idx\": 20525, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6acee902-5735-4d36-a385-6c034da1da8f", "doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "ref_doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "chroma:document": "/ 25000 \n    Processed 24000 / 25000 \n    Processed 24100 / 25000 \n    Processed 24200 / 25000 \n    Processed 24300 / 25000 \n    Processed 24400 / 25000 \n    Processed 24500 / 25000 \n    Processed 24600 / 25000 \n    Processed 24700 / 25000 \n    Processed 24800 / 25000 \n    Processed 24900 / 25000 \n    Processed 25000 / 25000 \n    Imported (25000) articles.\n\n\n\n```python\n# Check the number of documents imported\n\ncollection = typesense_client.collections['wikipedia_articles'].retrieve()\nprint(f'Collection has {collection[\"num_documents\"]} documents')\n```\n\n    Collection has 25000 documents\n\n\n### Search Data\n\nNow that we've imported the vectors into Typesense, we can do a nearest neighbor search on the `title_vector` or `content_vector` field.\n\n\n```python\ndef query_typesense(query, field='title', top_k=20):\n\n    # Creates embedding vector from user query\n    openai.api_key = os.getenv(\"OPENAI_API_KEY\", \"sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\")\n    embedded_query = openai.Embedding.create(\n        input=query,\n        model=EMBEDDING_MODEL,\n    )['data'][0]['embedding']\n\n    typesense_results = typesense_client.multi_search.perform({\n        \"searches\": [{\n            \"q\": \"*\",\n            \"collection\": \"wikipedia_articles\",\n            \"vector_query\": f\"{field}_vector:([{','.join(str(v) for v in embedded_query)}], k:{top_k})\"\n        }]\n    }, {})\n\n    return typesense_results\n```\n\n\n```python\nquery_results = query_typesense('modern art in Europe', 'title')\n\nfor i, hit in enumerate(query_results['results'][0]['hits']):\n    document = hit[\"document\"]\n    vector_distance = hit[\"vector_distance\"]\n    print(f'{i + 1}. {document[\"title\"]} (Distance: {vector_distance})')\n```\n\n    1. Museum of Modern Art (Distance: 0.12482291460037231)\n    2. Western Europe (Distance: 0.13255876302719116)\n    3. Renaissance art (Distance: 0.13584274053573608)\n    4. Pop art (Distance: 0.1396539807319641)\n    5. Northern Europe (Distance: 0.14534103870391846)\n    6. Hellenistic art (Distance: 0.1472070813179016)\n    7. Modernist literature (Distance: 0.15296930074691772)\n    8. Art film (Distance: 0.1567266583442688)\n    9. Central Europe (Distance: 0.15741699934005737)\n    10. European (Distance: 0.1585891842842102)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 74ccbca8-38cc-4d2a-ad42-f006e5f68707
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb", "title": "Using Typesense for Embeddings Search", "tokens": 6500, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"74ccbca8-38cc-4d2a-ad42-f006e5f68707\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"6acee902-5735-4d36-a385-6c034da1da8f\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ce997efdc8ef99168d3a268a64673416c2f022f364ddaac342e3e5ba038733fd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"656c9378-b7cf-45f8-8c24-ddab8a2119ab\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/Using_Typesense_for_embeddings_search.ipynb\", \"title\": \"Using Typesense for Embeddings Search\", \"tokens\": 6500, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"cd3d35f1c44e2e4a13567e5c62061314ac4b87a80b6315c7bfcb3118899e0fb4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19701, \"end_char_idx\": 21632, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "6acee902-5735-4d36-a385-6c034da1da8f", "doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "ref_doc_id": "6acee902-5735-4d36-a385-6c034da1da8f", "chroma:document": "```python\nquery_results = query_typesense('modern art in Europe', 'title')\n\nfor i, hit in enumerate(query_results['results'][0]['hits']):\n    document = hit[\"document\"]\n    vector_distance = hit[\"vector_distance\"]\n    print(f'{i + 1}. {document[\"title\"]} (Distance: {vector_distance})')\n```\n\n    1. Museum of Modern Art (Distance: 0.12482291460037231)\n    2. Western Europe (Distance: 0.13255876302719116)\n    3. Renaissance art (Distance: 0.13584274053573608)\n    4. Pop art (Distance: 0.1396539807319641)\n    5. Northern Europe (Distance: 0.14534103870391846)\n    6. Hellenistic art (Distance: 0.1472070813179016)\n    7. Modernist literature (Distance: 0.15296930074691772)\n    8. Art film (Distance: 0.1567266583442688)\n    9. Central Europe (Distance: 0.15741699934005737)\n    10. European (Distance: 0.1585891842842102)\n\n\n\n```python\nquery_results = query_typesense('Famous battles in Scottish history', 'content')\n\nfor i, hit in enumerate(query_results['results'][0]['hits']):\n    document = hit[\"document\"]\n    vector_distance = hit[\"vector_distance\"]\n    print(f'{i + 1}. {document[\"title\"]} (Distance: {vector_distance})')\n```\n\n    1. Battle of Bannockburn (Distance: 0.1306111216545105)\n    2. Wars of Scottish Independence (Distance: 0.1384994387626648)\n    3. 1651 (Distance: 0.14744246006011963)\n    4. First War of Scottish Independence (Distance: 0.15033596754074097)\n    5. Robert I of Scotland (Distance: 0.15376019477844238)\n    6. 841 (Distance: 0.15609073638916016)\n    7. 1716 (Distance: 0.15615153312683105)\n    8. 1314 (Distance: 0.16280347108840942)\n    9. 1263 (Distance: 0.16361045837402344)\n    10. William Wallace (Distance: 0.16464537382125854)\n\n\nThanks for following along, you're now equipped to set up your own vector databases and use embeddings to do all kinds of cool things - enjoy! For more complex use cases please continue to work through other cookbook examples in this repo.\n\n\n```python\n\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 29b10c7f-a84f-4c5f-8c5c-3626b7298b3a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/README.ipynb", "title": "Typesense", "tokens": 509, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"29b10c7f-a84f-4c5f-8c5c-3626b7298b3a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/README.ipynb\", \"title\": \"Typesense\", \"tokens\": 509, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"ae5b61aa-f57c-4a17-9ce4-836bf0f4faf6\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/typesense/README.ipynb\", \"title\": \"Typesense\", \"tokens\": 509, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"cabd82417c9062e166e7e1b5f4cdc773b78af104d9b13438a188224c9c882026\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2378, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "ae5b61aa-f57c-4a17-9ce4-836bf0f4faf6", "doc_id": "ae5b61aa-f57c-4a17-9ce4-836bf0f4faf6", "ref_doc_id": "ae5b61aa-f57c-4a17-9ce4-836bf0f4faf6", "chroma:document": "# Typesense\n\nTypesense is an open source, in-memory search engine, that you can either [self-host](https://typesense.org/docs/guide/install-typesense.html#option-2-local-machine-self-hosting) or run on [Typesense Cloud](https://cloud.typesense.org/).\n\n## Why Typesense?\n\nTypesense focuses on performance by storing the entire index in RAM (with a backup on disk) and also focuses on providing an out-of-the-box developer experience by simplifying available options and setting good defaults. \n\nIt also lets you combine attribute-based filtering together with vector queries, to fetch the most relevant documents.\n\n### Other features\n\nBesides vector storage and search, Typesense also offers the following features:\n\n- Typo Tolerance: Handles typographical errors elegantly, out-of-the-box.\n- Tunable Ranking: Easy to tailor your search results to perfection.\n- Sorting: Dynamically sort results based on a particular field at query time (helpful for features like \"Sort by Price (asc)\").\n- Faceting & Filtering: Drill down and refine results.\n- Grouping & Distinct: Group similar results together to show more variety.\n- Federated Search: Search across multiple collections (indices) in a single HTTP request.\n- Scoped API Keys: Generate API keys that only allow access to certain records, for multi-tenant applications.\n- Synonyms: Define words as equivalents of each other, so searching for a word will also return results for the synonyms defined.\n- Curation & Merchandizing: Boost particular records to a fixed position in the search results, to feature them.\n- Raft-based Clustering: Set up a distributed cluster that is highly available.\n- Seamless Version Upgrades: As new versions of Typesense come out, upgrading is as simple as swapping out the binary and restarting Typesense.\n- No Runtime Dependencies: Typesense is a single binary that you can run locally or in production with a single command.\n\n## How To\n\n- To learn more about how to use Typesense with OpenAI embeddings, see the notebook here for an example: [examples/vector_databases/Using_vector_databases_for_embeddings_search.ipynb](/examples/vector_databases/Using_vector_databases_for_embeddings_search.ipynb)\n- To learn more about Typesense's vector search feature, read the docs here: [https://typesense.org/docs/0.24.1/api/vector-search.html](https://typesense.org/docs/0.24.1/api/vector-search.html)."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 969c38fb-dbc1-43ad-baa6-dafa27c5c807
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"969c38fb-dbc1-43ad-baa6-dafa27c5c807\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"47d7866a-4d5f-4bdd-aca4-334d43bea246\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a149b2bcb8a9412e4fb840ac9a4816d84ad786e046df3166e10f07d6151243f6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"770009e3-6b87-4627-8052-c35758efff91\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b817823958446adb3e1a30a3aa0d120d661aa758da39a3f0de871d6dbcf57a41\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3428, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "doc_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "ref_doc_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "chroma:document": "# Azure AI Search as a vector database for OpenAI embeddings\n\nThis notebook provides step by step instuctions on using Azure AI Search (f.k.a Azure Cognitive Search) as a vector database with OpenAI embeddings. Azure AI Search is a cloud search service that gives developers infrastructure, APIs, and tools for building a rich search experience over private, heterogeneous content in web, mobile, and enterprise applications.\n\n## Prerequistites:\nFor the purposes of this exercise you must have the following:\n- [Azure AI Search Service](https://learn.microsoft.com/azure/search/)\n- [OpenAI Key](https://platform.openai.com/account/api-keys) or [Azure OpenAI credentials](https://learn.microsoft.com/azure/cognitive-services/openai/)\n\n\n```python\n! pip install wget\n! pip install azure-search-documents \n! pip install azure-identity\n! pip install openai\n```\n\n## Import required libraries\n\n\n```python\nimport json  \nimport wget\nimport pandas as pd\nimport zipfile\nfrom openai import AzureOpenAI\nfrom azure.identity import DefaultAzureCredential, get_bearer_token_provider\nfrom azure.core.credentials import AzureKeyCredential  \nfrom azure.search.documents import SearchClient, SearchIndexingBufferedSender  \nfrom azure.search.documents.indexes import SearchIndexClient  \nfrom azure.search.documents.models import (\n    QueryAnswerType,\n    QueryCaptionType,\n    QueryType,\n    VectorizedQuery,\n)\nfrom azure.search.documents.indexes.models import (\n    HnswAlgorithmConfiguration,\n    HnswParameters,\n    SearchField,\n    SearchableField,\n    SearchFieldDataType,\n    SearchIndex,\n    SemanticConfiguration,\n    SemanticField,\n    SemanticPrioritizedFields,\n    SemanticSearch,\n    SimpleField,\n    VectorSearch,\n    VectorSearchAlgorithmKind,\n    VectorSearchAlgorithmMetric,\n    VectorSearchProfile,\n)\n\n```\n\n## Configure OpenAI settings\n\nThis section guides you through setting up authentication for Azure OpenAI, allowing you to securely interact with the service using either Azure Active Directory (AAD) or an API key. Before proceeding, ensure you have your Azure OpenAI endpoint and credentials ready. For detailed instructions on setting up AAD with Azure OpenAI, refer to the [official documentation](https://learn.microsoft.com/azure/ai-services/openai/how-to/managed-identity).\n\n\n\n```python\nendpoint: str = \"YOUR_AZURE_OPENAI_ENDPOINT\"\napi_key: str = \"YOUR_AZURE_OPENAI_KEY\"\napi_version: str = \"2023-05-15\"\ndeployment = \"YOUR_AZURE_OPENAI_DEPLOYMENT_NAME\"\ncredential = DefaultAzureCredential()\ntoken_provider = get_bearer_token_provider(\n    credential, \"https://cognitiveservices.azure.com/.default\"\n)\n\n# Set this flag to True if you are using Azure Active Directory\nuse_aad_for_aoai = True \n\nif use_aad_for_aoai:\n    # Use Azure Active Directory (AAD) authentication\n    client = AzureOpenAI(\n        azure_endpoint=endpoint,\n        api_version=api_version,\n        azure_ad_token_provider=token_provider,\n    )\nelse:\n    # Use API key authentication\n    client = AzureOpenAI(\n        api_key=api_key,\n        api_version=api_version,\n        azure_endpoint=endpoint,\n    )\n```\n\n## Configure Azure AI Search Vector Store settings\nThis section explains how to set up the Azure AI Search client for integrating with the Vector Store feature. You can locate your Azure AI Search service details in the Azure Portal or programmatically via the [Search Management SDK](https://learn.microsoft.com/rest/api/searchmanagement/)."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 770009e3-6b87-4627-8052-c35758efff91
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"770009e3-6b87-4627-8052-c35758efff91\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"47d7866a-4d5f-4bdd-aca4-334d43bea246\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a149b2bcb8a9412e4fb840ac9a4816d84ad786e046df3166e10f07d6151243f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"969c38fb-dbc1-43ad-baa6-dafa27c5c807\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a0495584532b8c8cc3652afe3479ce4777d3494cb5eb96f1b7821a4fa12279eb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"aa743891-29c9-4ac2-979a-0461f1029a8d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a9196c8859376eee01a2bfb36455618ef0e4678c2323a7d483b62efc6426c836\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2286, \"end_char_idx\": 4949, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "doc_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "ref_doc_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "chroma:document": "```python\nendpoint: str = \"YOUR_AZURE_OPENAI_ENDPOINT\"\napi_key: str = \"YOUR_AZURE_OPENAI_KEY\"\napi_version: str = \"2023-05-15\"\ndeployment = \"YOUR_AZURE_OPENAI_DEPLOYMENT_NAME\"\ncredential = DefaultAzureCredential()\ntoken_provider = get_bearer_token_provider(\n    credential, \"https://cognitiveservices.azure.com/.default\"\n)\n\n# Set this flag to True if you are using Azure Active Directory\nuse_aad_for_aoai = True \n\nif use_aad_for_aoai:\n    # Use Azure Active Directory (AAD) authentication\n    client = AzureOpenAI(\n        azure_endpoint=endpoint,\n        api_version=api_version,\n        azure_ad_token_provider=token_provider,\n    )\nelse:\n    # Use API key authentication\n    client = AzureOpenAI(\n        api_key=api_key,\n        api_version=api_version,\n        azure_endpoint=endpoint,\n    )\n```\n\n## Configure Azure AI Search Vector Store settings\nThis section explains how to set up the Azure AI Search client for integrating with the Vector Store feature. You can locate your Azure AI Search service details in the Azure Portal or programmatically via the [Search Management SDK](https://learn.microsoft.com/rest/api/searchmanagement/).\n\n\n\n```python\n# Configuration\nsearch_service_endpoint: str = \"YOUR_AZURE_SEARCH_ENDPOINT\"\nsearch_service_api_key: str = \"YOUR_AZURE_SEARCH_ADMIN_KEY\"\nindex_name: str = \"azure-ai-search-openai-cookbook-demo\"\n\n# Set this flag to True if you are using Azure Active Directory\nuse_aad_for_search = True  \n\nif use_aad_for_search:\n    # Use Azure Active Directory (AAD) authentication\n    credential = DefaultAzureCredential()\nelse:\n    # Use API key authentication\n    credential = AzureKeyCredential(search_service_api_key)\n\n# Initialize the SearchClient with the selected authentication method\nsearch_client = SearchClient(\n    endpoint=search_service_endpoint, index_name=index_name, credential=credential\n)\n```\n\n## Load data\n\n\n\n```python\nembeddings_url = \"https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip\"\n\n# The file is ~700 MB so this will take some time\nwget.download(embeddings_url)\n```\n\n\n\n\n    'vector_database_wikipedia_articles_embedded.zip'\n\n\n\n\n```python\nwith zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\", \"r\") as zip_ref:\n    zip_ref.extractall(\"../../data\")\n```\n\n\n```python\narticle_df = pd.read_csv(\"../../data/vector_database_wikipedia_articles_embedded.csv\")\n\n# Read vectors from strings back into a list using json.loads\narticle_df[\"title_vector\"] = article_df.title_vector.apply(json.loads)\narticle_df[\"content_vector\"] = article_df.content_vector.apply(json.loads)\narticle_df[\"vector_id\"] = article_df[\"vector_id\"].apply(str)\narticle_df.head()\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: aa743891-29c9-4ac2-979a-0461f1029a8d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"aa743891-29c9-4ac2-979a-0461f1029a8d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"47d7866a-4d5f-4bdd-aca4-334d43bea246\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a149b2bcb8a9412e4fb840ac9a4816d84ad786e046df3166e10f07d6151243f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"770009e3-6b87-4627-8052-c35758efff91\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"5df93f1a61aa9d49129d165fa7281cb4bfd4289deda2ff02d3f9e93e70ddd088\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a1244fbd-6f82-4cd5-ba56-dbabfa6870e0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8cd0a9c9c8bed21afa4c25e5abe679fb75283551f60ad360179fa48e2e466a3e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3432, \"end_char_idx\": 5965, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "doc_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "ref_doc_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "chroma:document": "```python\n# Configuration\nsearch_service_endpoint: str = \"YOUR_AZURE_SEARCH_ENDPOINT\"\nsearch_service_api_key: str = \"YOUR_AZURE_SEARCH_ADMIN_KEY\"\nindex_name: str = \"azure-ai-search-openai-cookbook-demo\"\n\n# Set this flag to True if you are using Azure Active Directory\nuse_aad_for_search = True  \n\nif use_aad_for_search:\n    # Use Azure Active Directory (AAD) authentication\n    credential = DefaultAzureCredential()\nelse:\n    # Use API key authentication\n    credential = AzureKeyCredential(search_service_api_key)\n\n# Initialize the SearchClient with the selected authentication method\nsearch_client = SearchClient(\n    endpoint=search_service_endpoint, index_name=index_name, credential=credential\n)\n```\n\n## Load data\n\n\n\n```python\nembeddings_url = \"https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip\"\n\n# The file is ~700 MB so this will take some time\nwget.download(embeddings_url)\n```\n\n\n\n\n    'vector_database_wikipedia_articles_embedded.zip'\n\n\n\n\n```python\nwith zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\", \"r\") as zip_ref:\n    zip_ref.extractall(\"../../data\")\n```\n\n\n```python\narticle_df = pd.read_csv(\"../../data/vector_database_wikipedia_articles_embedded.csv\")\n\n# Read vectors from strings back into a list using json.loads\narticle_df[\"title_vector\"] = article_df.title_vector.apply(json.loads)\narticle_df[\"content_vector\"] = article_df.content_vector.apply(json.loads)\narticle_df[\"vector_id\"] = article_df[\"vector_id\"].apply(str)\narticle_df.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>url</th>\n      <th>title</th>\n      <th>text</th>\n      <th>title_vector</th>\n      <th>content_vector</th>\n      <th>vector_id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>https://simple.wikipedia.org/wiki/April</td>\n      <td>April</td>\n      <td>April is the fourth month of the year in the J...</td>\n      <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n      <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>https://simple.wikipedia.org/wiki/August</td>\n      <td>August</td>\n      <td>August (Aug.)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: a1244fbd-6f82-4cd5-ba56-dbabfa6870e0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a1244fbd-6f82-4cd5-ba56-dbabfa6870e0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"47d7866a-4d5f-4bdd-aca4-334d43bea246\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a149b2bcb8a9412e4fb840ac9a4816d84ad786e046df3166e10f07d6151243f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"aa743891-29c9-4ac2-979a-0461f1029a8d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7d71a4e84805b2135a35b1ce67df039b459273ea612813ef437995b9cde44f69\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"de998d98-3681-434d-9e88-a2a3a356dfcd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d4235367260c1a0275a057bb0bbf3eefa97b9b7dc3c0634b8fbec9670ab8f4de\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4954, \"end_char_idx\": 6996, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "doc_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "ref_doc_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "chroma:document": "<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>url</th>\n      <th>title</th>\n      <th>text</th>\n      <th>title_vector</th>\n      <th>content_vector</th>\n      <th>vector_id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>https://simple.wikipedia.org/wiki/April</td>\n      <td>April</td>\n      <td>April is the fourth month of the year in the J...</td>\n      <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n      <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>https://simple.wikipedia.org/wiki/August</td>\n      <td>August</td>\n      <td>August (Aug.) is the eighth month of the year ...</td>\n      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>6</td>\n      <td>https://simple.wikipedia.org/wiki/Art</td>\n      <td>Art</td>\n      <td>Art is a creative activity that expresses imag...</td>\n      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>8</td>\n      <td>https://simple.wikipedia.org/wiki/A</td>\n      <td>A</td>\n      <td>A or a is the first letter of the English alph...</td>\n      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n      <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>9</td>\n      <td>https://simple.wikipedia.org/wiki/Air</td>\n      <td>Air</td>\n      <td>Air refers to the Earth's atmosphere."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: de998d98-3681-434d-9e88-a2a3a356dfcd
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"de998d98-3681-434d-9e88-a2a3a356dfcd\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"47d7866a-4d5f-4bdd-aca4-334d43bea246\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a149b2bcb8a9412e4fb840ac9a4816d84ad786e046df3166e10f07d6151243f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a1244fbd-6f82-4cd5-ba56-dbabfa6870e0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"95df80a6c297e820f4fc8e9181b6f2cbae6a308cd502368a612e437f6c97c571\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"17cb20f4-8b6a-4797-b83a-96bf076ab042\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a51cc74e1e1a0eda1ed3d3ed049f385ed1e816d85f75d70300952e5de30d28f1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5966, \"end_char_idx\": 7656, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "doc_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "ref_doc_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "chroma:document": "is the eighth month of the year ...</td>\n      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>6</td>\n      <td>https://simple.wikipedia.org/wiki/Art</td>\n      <td>Art</td>\n      <td>Art is a creative activity that expresses imag...</td>\n      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>8</td>\n      <td>https://simple.wikipedia.org/wiki/A</td>\n      <td>A</td>\n      <td>A or a is the first letter of the English alph...</td>\n      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n      <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>9</td>\n      <td>https://simple.wikipedia.org/wiki/Air</td>\n      <td>Air</td>\n      <td>Air refers to the Earth's atmosphere. Air is a...</td>\n      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n      <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n##  Create an index\nThis code snippet demonstrates how to define and create a search index using the `SearchIndexClient` from the Azure AI Search Python SDK. The index incorporates both vector search and semantic ranker capabilities. For more details, visit our documentation on how to [Create a Vector Index](https://learn.microsoft.com/azure/search/vector-search-how-to-create-index?.tabs=config-2023-11-01%2Crest-2023-11-01%2Cpush%2Cportal-check-index)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 17cb20f4-8b6a-4797-b83a-96bf076ab042
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"17cb20f4-8b6a-4797-b83a-96bf076ab042\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"47d7866a-4d5f-4bdd-aca4-334d43bea246\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a149b2bcb8a9412e4fb840ac9a4816d84ad786e046df3166e10f07d6151243f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"de998d98-3681-434d-9e88-a2a3a356dfcd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c156617449bee787255a645055e79057b3f06647139910cd9d394ed021c95df6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d04dc1cc-0089-49f6-b416-efb39034192d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ce12792f3674b96822509b0107a8c99b7aa015b4808bf0de4ebfc23c55112dab\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6997, \"end_char_idx\": 10470, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "doc_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "ref_doc_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "chroma:document": "Air is a...</td>\n      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n      <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n##  Create an index\nThis code snippet demonstrates how to define and create a search index using the `SearchIndexClient` from the Azure AI Search Python SDK. The index incorporates both vector search and semantic ranker capabilities. For more details, visit our documentation on how to [Create a Vector Index](https://learn.microsoft.com/azure/search/vector-search-how-to-create-index?.tabs=config-2023-11-01%2Crest-2023-11-01%2Cpush%2Cportal-check-index)\n\n\n```python\n# Initialize the SearchIndexClient\nindex_client = SearchIndexClient(\n    endpoint=search_service_endpoint, credential=credential\n)\n\n# Define the fields for the index\nfields = [\n    SimpleField(name=\"id\", type=SearchFieldDataType.String),\n    SimpleField(name=\"vector_id\", type=SearchFieldDataType.String, key=True),\n    SimpleField(name=\"url\", type=SearchFieldDataType.String),\n    SearchableField(name=\"title\", type=SearchFieldDataType.String),\n    SearchableField(name=\"text\", type=SearchFieldDataType.String),\n    SearchField(\n        name=\"title_vector\",\n        type=SearchFieldDataType.Collection(SearchFieldDataType.Single),\n        vector_search_dimensions=1536,\n        vector_search_profile_name=\"my-vector-config\",\n    ),\n    SearchField(\n        name=\"content_vector\",\n        type=SearchFieldDataType.Collection(SearchFieldDataType.Single),\n        vector_search_dimensions=1536,\n        vector_search_profile_name=\"my-vector-config\",\n    ),\n]\n\n# Configure the vector search configuration\nvector_search = VectorSearch(\n    algorithms=[\n        HnswAlgorithmConfiguration(\n            name=\"my-hnsw\",\n            kind=VectorSearchAlgorithmKind.HNSW,\n            parameters=HnswParameters(\n                m=4,\n                ef_construction=400,\n                ef_search=500,\n                metric=VectorSearchAlgorithmMetric.COSINE,\n            ),\n        )\n    ],\n    profiles=[\n        VectorSearchProfile(\n            name=\"my-vector-config\",\n            algorithm_configuration_name=\"my-hnsw\",\n        )\n    ],\n)\n\n# Configure the semantic search configuration\nsemantic_search = SemanticSearch(\n    configurations=[\n        SemanticConfiguration(\n            name=\"my-semantic-config\",\n            prioritized_fields=SemanticPrioritizedFields(\n                title_field=SemanticField(field_name=\"title\"),\n                keywords_fields=[SemanticField(field_name=\"url\")],\n                content_fields=[SemanticField(field_name=\"text\")],\n            ),\n        )\n    ]\n)\n\n# Create the search index with the vector search and semantic search configurations\nindex = SearchIndex(\n    name=index_name,\n    fields=fields,\n    vector_search=vector_search,\n    semantic_search=semantic_search,\n)\n\n# Create or update the index\nresult = index_client.create_or_update_index(index)\nprint(f\"{result.name} created\")\n```\n\n    azure-ai-search-openai-cookbook-demo created\n\n\n## Uploading Data to Azure AI Search Index\n\nThe following code snippet outlines the process of uploading a batch of documents\u2014specifically, Wikipedia articles with pre-computed embeddings\u2014from a pandas DataFrame to an Azure AI Search index. For a detailed guide on data import strategies and best practices, refer to  [Data Import in Azure AI Search](https://learn.microsoft.com/azure/search/search-what-is-data-import)."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: d04dc1cc-0089-49f6-b416-efb39034192d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"d04dc1cc-0089-49f6-b416-efb39034192d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"47d7866a-4d5f-4bdd-aca4-334d43bea246\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a149b2bcb8a9412e4fb840ac9a4816d84ad786e046df3166e10f07d6151243f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"17cb20f4-8b6a-4797-b83a-96bf076ab042\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"03b7ab787cfad6f649111629c971675d3224fffaf25bd96d224302327d8a74b7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"70f36228-1886-4bab-ab73-b97d05f6db36\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9cb511fe0a0b3f02dbf8d7e68cba010bc0c634c7a5c8c9b9f666f38155cb5622\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10050, \"end_char_idx\": 13371, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "doc_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "ref_doc_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "chroma:document": "## Uploading Data to Azure AI Search Index\n\nThe following code snippet outlines the process of uploading a batch of documents\u2014specifically, Wikipedia articles with pre-computed embeddings\u2014from a pandas DataFrame to an Azure AI Search index. For a detailed guide on data import strategies and best practices, refer to  [Data Import in Azure AI Search](https://learn.microsoft.com/azure/search/search-what-is-data-import).\n\n\n\n```python\nfrom azure.core.exceptions import HttpResponseError\n\n# Convert the 'id' and 'vector_id' columns to string so one of them can serve as our key field\narticle_df[\"id\"] = article_df[\"id\"].astype(str)\narticle_df[\"vector_id\"] = article_df[\"vector_id\"].astype(str)\n# Convert the DataFrame to a list of dictionaries\ndocuments = article_df.to_dict(orient=\"records\")\n\n# Create a SearchIndexingBufferedSender\nbatch_client = SearchIndexingBufferedSender(\n    search_service_endpoint, index_name, credential\n)\n\ntry:\n    # Add upload actions for all documents in a single call\n    batch_client.upload_documents(documents=documents)\n\n    # Manually flush to send any remaining documents in the buffer\n    batch_client.flush()\nexcept HttpResponseError as e:\n    print(f\"An error occurred: {e}\")\nfinally:\n    # Clean up resources\n    batch_client.close()\n\nprint(f\"Uploaded {len(documents)} documents in total\")\n```\n\n    Uploaded 25000 documents in total\n\n\nIf your dataset didn't already contain pre-computed embeddings, you can create embeddings by using the below function using the `openai` python library. You'll also notice the same function and model are being used to generate query embeddings for performing vector searches.\n\n\n```python\n# Example function to generate document embedding\ndef generate_embeddings(text, model):\n    # Generate embeddings for the provided text using the specified model\n    embeddings_response = client.embeddings.create(model=model, input=text)\n    # Extract the embedding data from the response\n    embedding = embeddings_response.data[0].embedding\n    return embedding\n\n\nfirst_document_content = documents[0][\"text\"]\nprint(f\"Content: {first_document_content[:100]}\")\n\ncontent_vector = generate_embeddings(first_document_content, deployment)\nprint(\"Content vector generated\")\n```\n\n    Content: April is the fourth month of the year in the Julian and Gregorian calendars, and comes between March\n    Content vector generated\n\n\n## Perform a vector similarity search\n\n\n```python\n# Pure Vector Search\nquery = \"modern art in Europe\"\n  \nsearch_client = SearchClient(search_service_endpoint, index_name, credential)  \nvector_query = VectorizedQuery(vector=generate_embeddings(query, deployment), k_nearest_neighbors=3, fields=\"content_vector\")\n  \nresults = search_client.search(  \n    search_text=None,  \n    vector_queries= [vector_query], \n    select=[\"title\", \"text\", \"url\"] \n)\n  \nfor result in results:  \n    print(f\"Title: {result['title']}\")  \n    print(f\"Score: {result['@search.score']}\")  \n    print(f\"URL: {result['url']}\\n\")  \n```\n\n    Title: Documenta\n    Score: 0.8599451\n    URL: https://simple.wikipedia.org/wiki/Documenta\n    \n    Title: Museum of Modern Art\n    Score: 0.85260946\n    URL: https://simple.wikipedia.org/wiki/Museum%20of%20Modern%20Art\n    \n    Title: Expressionism\n    Score: 0.852354\n    URL: https://simple.wikipedia.org/wiki/Expressionism"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 70f36228-1886-4bab-ab73-b97d05f6db36
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"70f36228-1886-4bab-ab73-b97d05f6db36\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"47d7866a-4d5f-4bdd-aca4-334d43bea246\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a149b2bcb8a9412e4fb840ac9a4816d84ad786e046df3166e10f07d6151243f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d04dc1cc-0089-49f6-b416-efb39034192d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6a0ab6871fa106a7321877786a6d37a115c5840c40c9b49f3a4eca90d599076c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3f2450a4-470a-4334-bad8-89c30bcb808a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bbb1dc309c39ee2dae8ac5640e358fddb5885afe9e676a6bff92031cab38570d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11701, \"end_char_idx\": 14778, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "doc_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "ref_doc_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "chroma:document": "```python\n# Example function to generate document embedding\ndef generate_embeddings(text, model):\n    # Generate embeddings for the provided text using the specified model\n    embeddings_response = client.embeddings.create(model=model, input=text)\n    # Extract the embedding data from the response\n    embedding = embeddings_response.data[0].embedding\n    return embedding\n\n\nfirst_document_content = documents[0][\"text\"]\nprint(f\"Content: {first_document_content[:100]}\")\n\ncontent_vector = generate_embeddings(first_document_content, deployment)\nprint(\"Content vector generated\")\n```\n\n    Content: April is the fourth month of the year in the Julian and Gregorian calendars, and comes between March\n    Content vector generated\n\n\n## Perform a vector similarity search\n\n\n```python\n# Pure Vector Search\nquery = \"modern art in Europe\"\n  \nsearch_client = SearchClient(search_service_endpoint, index_name, credential)  \nvector_query = VectorizedQuery(vector=generate_embeddings(query, deployment), k_nearest_neighbors=3, fields=\"content_vector\")\n  \nresults = search_client.search(  \n    search_text=None,  \n    vector_queries= [vector_query], \n    select=[\"title\", \"text\", \"url\"] \n)\n  \nfor result in results:  \n    print(f\"Title: {result['title']}\")  \n    print(f\"Score: {result['@search.score']}\")  \n    print(f\"URL: {result['url']}\\n\")  \n```\n\n    Title: Documenta\n    Score: 0.8599451\n    URL: https://simple.wikipedia.org/wiki/Documenta\n    \n    Title: Museum of Modern Art\n    Score: 0.85260946\n    URL: https://simple.wikipedia.org/wiki/Museum%20of%20Modern%20Art\n    \n    Title: Expressionism\n    Score: 0.852354\n    URL: https://simple.wikipedia.org/wiki/Expressionism\n    \n\n\n## Perform a Hybrid Search\nHybrid search combines the capabilities of traditional keyword-based search with vector-based similarity search to provide more relevant and contextual results. This approach is particularly useful when dealing with complex queries that benefit from understanding the semantic meaning behind the text.\n\nThe provided code snippet demonstrates how to execute a hybrid search query:\n\n\n```python\n# Hybrid Search\nquery = \"Famous battles in Scottish history\"  \n  \nsearch_client = SearchClient(search_service_endpoint, index_name, credential)  \nvector_query = VectorizedQuery(vector=generate_embeddings(query, deployment), k_nearest_neighbors=3, fields=\"content_vector\")\n  \nresults = search_client.search(  \n    search_text=query,  \n    vector_queries= [vector_query], \n    select=[\"title\", \"text\", \"url\"],\n    top=3\n)\n  \nfor result in results:  \n    print(f\"Title: {result['title']}\")  \n    print(f\"Score: {result['@search.score']}\")  \n    print(f\"URL: {result['url']}\\n\")  \n```\n\n    Title: Wars of Scottish Independence\n    Score: 0.03306011110544205\n    URL: https://simple.wikipedia.org/wiki/Wars%20of%20Scottish%20Independence\n    \n    Title: Battle of Bannockburn\n    Score: 0.022253260016441345\n    URL: https://simple.wikipedia.org/wiki/Battle%20of%20Bannockburn\n    \n    Title: Scottish\n    Score: 0.016393441706895828\n    URL: https://simple.wikipedia.org/wiki/Scottish"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 3f2450a4-470a-4334-bad8-89c30bcb808a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"3f2450a4-470a-4334-bad8-89c30bcb808a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"47d7866a-4d5f-4bdd-aca4-334d43bea246\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a149b2bcb8a9412e4fb840ac9a4816d84ad786e046df3166e10f07d6151243f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"70f36228-1886-4bab-ab73-b97d05f6db36\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"fea9944f7fdf015042220da0e8062e8e4f475f8fed28639e106149fcee3e7835\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f9c3a84c-1a28-4978-bb81-4e8c8e286034\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"35286841c342fb1f74eddd0c4edcbab0ee1b65805bc4297f96ef8a15bb0e7d6b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13379, \"end_char_idx\": 15097, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "doc_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "ref_doc_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "chroma:document": "## Perform a Hybrid Search\nHybrid search combines the capabilities of traditional keyword-based search with vector-based similarity search to provide more relevant and contextual results. This approach is particularly useful when dealing with complex queries that benefit from understanding the semantic meaning behind the text.\n\nThe provided code snippet demonstrates how to execute a hybrid search query:\n\n\n```python\n# Hybrid Search\nquery = \"Famous battles in Scottish history\"  \n  \nsearch_client = SearchClient(search_service_endpoint, index_name, credential)  \nvector_query = VectorizedQuery(vector=generate_embeddings(query, deployment), k_nearest_neighbors=3, fields=\"content_vector\")\n  \nresults = search_client.search(  \n    search_text=query,  \n    vector_queries= [vector_query], \n    select=[\"title\", \"text\", \"url\"],\n    top=3\n)\n  \nfor result in results:  \n    print(f\"Title: {result['title']}\")  \n    print(f\"Score: {result['@search.score']}\")  \n    print(f\"URL: {result['url']}\\n\")  \n```\n\n    Title: Wars of Scottish Independence\n    Score: 0.03306011110544205\n    URL: https://simple.wikipedia.org/wiki/Wars%20of%20Scottish%20Independence\n    \n    Title: Battle of Bannockburn\n    Score: 0.022253260016441345\n    URL: https://simple.wikipedia.org/wiki/Battle%20of%20Bannockburn\n    \n    Title: Scottish\n    Score: 0.016393441706895828\n    URL: https://simple.wikipedia.org/wiki/Scottish\n    \n\n\n## Perform a Hybrid Search with Reranking (powered by Bing)\n[Semantic ranker](https://learn.microsoft.com/azure/search/semantic-search-overview) measurably improves search relevance by using language understanding to rerank search results. Additionally, you can get extractive captions, answers, and highlights."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: f9c3a84c-1a28-4978-bb81-4e8c8e286034
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb", "title": "Azure AI Search as a vector database for OpenAI embeddings", "tokens": 4317, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"f9c3a84c-1a28-4978-bb81-4e8c8e286034\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"47d7866a-4d5f-4bdd-aca4-334d43bea246\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a149b2bcb8a9412e4fb840ac9a4816d84ad786e046df3166e10f07d6151243f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3f2450a4-470a-4334-bad8-89c30bcb808a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/azuresearch/Getting_started_with_azure_ai_search_and_openai.ipynb\", \"title\": \"Azure AI Search as a vector database for OpenAI embeddings\", \"tokens\": 4317, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c0b33f287fba3580a7aaf8c113c5ea41a5d6de93f66fdfd9e24a03d19213caa4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13788, \"end_char_idx\": 18220, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "doc_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "ref_doc_id": "47d7866a-4d5f-4bdd-aca4-334d43bea246", "chroma:document": "```python\n# Hybrid Search\nquery = \"Famous battles in Scottish history\"  \n  \nsearch_client = SearchClient(search_service_endpoint, index_name, credential)  \nvector_query = VectorizedQuery(vector=generate_embeddings(query, deployment), k_nearest_neighbors=3, fields=\"content_vector\")\n  \nresults = search_client.search(  \n    search_text=query,  \n    vector_queries= [vector_query], \n    select=[\"title\", \"text\", \"url\"],\n    top=3\n)\n  \nfor result in results:  \n    print(f\"Title: {result['title']}\")  \n    print(f\"Score: {result['@search.score']}\")  \n    print(f\"URL: {result['url']}\\n\")  \n```\n\n    Title: Wars of Scottish Independence\n    Score: 0.03306011110544205\n    URL: https://simple.wikipedia.org/wiki/Wars%20of%20Scottish%20Independence\n    \n    Title: Battle of Bannockburn\n    Score: 0.022253260016441345\n    URL: https://simple.wikipedia.org/wiki/Battle%20of%20Bannockburn\n    \n    Title: Scottish\n    Score: 0.016393441706895828\n    URL: https://simple.wikipedia.org/wiki/Scottish\n    \n\n\n## Perform a Hybrid Search with Reranking (powered by Bing)\n[Semantic ranker](https://learn.microsoft.com/azure/search/semantic-search-overview) measurably improves search relevance by using language understanding to rerank search results. Additionally, you can get extractive captions, answers, and highlights. \n\n\n```python\n# Semantic Hybrid Search\nquery = \"What were the key technological advancements during the Industrial Revolution?\"\n\nsearch_client = SearchClient(search_service_endpoint, index_name, credential)\nvector_query = VectorizedQuery(\n    vector=generate_embeddings(query, deployment),\n    k_nearest_neighbors=3,\n    fields=\"content_vector\",\n)\n\nresults = search_client.search(\n    search_text=query,\n    vector_queries=[vector_query],\n    select=[\"title\", \"text\", \"url\"],\n    query_type=QueryType.SEMANTIC,\n    semantic_configuration_name=\"my-semantic-config\",\n    query_caption=QueryCaptionType.EXTRACTIVE,\n    query_answer=QueryAnswerType.EXTRACTIVE,\n    top=3,\n)\n\nsemantic_answers = results.get_answers()\nfor answer in semantic_answers:\n    if answer.highlights:\n        print(f\"Semantic Answer: {answer.highlights}\")\n    else:\n        print(f\"Semantic Answer: {answer.text}\")\n    print(f\"Semantic Answer Score: {answer.score}\\n\")\n\nfor result in results:\n    print(f\"Title: {result['title']}\")\n    print(f\"Reranker Score: {result['@search.reranker_score']}\")\n    print(f\"URL: {result['url']}\")\n    captions = result[\"@search.captions\"]\n    if captions:\n        caption = captions[0]\n        if caption.highlights:\n            print(f\"Caption: {caption.highlights}\\n\")\n        else:\n            print(f\"Caption: {caption.text}\\n\")\n```\n\n    Semantic Answer: Advancements  During the industrial revolution, new technology brought many changes. For example:<em>   Canals</em> were built to allow heavy goods to be moved easily where they were needed. The steam engine became the main source of power. It replaced horses and human labor. Cheap iron and steel became mass-produced.\n    Semantic Answer Score: 0.90478515625\n    \n    Title: Industrial Revolution\n    Reranker Score: 3.408700942993164\n    URL: https://simple.wikipedia.org/wiki/Industrial%20Revolution\n    Caption: Advancements  During the industrial revolution, new technology brought many changes. For example:   Canals were built to allow heavy goods to be moved easily where they were needed. The steam engine became the main source of power. It replaced horses and human labor. Cheap iron and steel became mass-produced.\n    \n    Title: Printing\n    Reranker Score: 1.603400707244873\n    URL: https://simple.wikipedia.org/wiki/Printing\n    Caption: Machines to speed printing, cheaper paper, automatic stitching and binding all arrived in the 19th century during the industrial revolution. What had once been done by a few men by hand was now done by limited companies on huge machines. The result was much lower prices, and a much wider readership.\n    \n    Title: Industrialisation\n    Reranker Score: 1.3238357305526733\n    URL: https://simple.wikipedia.org/wiki/Industrialisation\n    Caption: <em>Industrialisation</em> (or<em> industrialization)</em> is a process that happens in countries when they start to use machines to do work that was once done by people.<em> Industrialisation changes</em> the things people do.<em> Industrialisation</em> caused towns to grow larger. Many people left farming to take higher paid jobs in factories in towns."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 39b6653f-2233-40f4-a164-6fc939ddeda9
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"39b6653f-2233-40f4-a164-6fc939ddeda9\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb\", \"title\": \"Using PolarDB-PG as a vector database for OpenAI embeddings\", \"tokens\": 2748, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"aa34d306-c403-425b-b95b-45841d5f1e32\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb\", \"title\": \"Using PolarDB-PG as a vector database for OpenAI embeddings\", \"tokens\": 2748, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7ea9039a22b6e6ab7acdec7882243aa4ae9839a86a9764f9c68acccd1014b060\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fce88700-0d92-4838-98d8-0e450d1ecea2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"357aabe7ee275efe4192419d0085587472be4045ce60ac2fb8c1b9bd971909d2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3468, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "aa34d306-c403-425b-b95b-45841d5f1e32", "doc_id": "aa34d306-c403-425b-b95b-45841d5f1e32", "ref_doc_id": "aa34d306-c403-425b-b95b-45841d5f1e32", "chroma:document": "# Using PolarDB-PG as a vector database for OpenAI embeddings\n\nThis notebook guides you step by step on using PolarDB-PG as a vector database for OpenAI embeddings.\n\nThis notebook presents an end-to-end process of:\n1. Using precomputed embeddings created by OpenAI API.\n2. Storing the embeddings in a cloud instance of PolarDB-PG.\n3. Converting raw text query to an embedding with OpenAI API.\n4. Using PolarDB-PG to perform the nearest neighbour search in the created collection.\n\n### What is PolarDB-PG\n\n[PolarDB-PG](https://www.alibabacloud.com/help/en/polardb/latest/what-is-polardb-2) is a high-performance vector database that adopts a read-write separation architecture. It is a cloud-native database managed by Alibaba Cloud, 100% compatible with PostgreSQL, and highly compatible with Oracle syntax. It supports processing massive vector data storage and queries, and greatly improves the efficiency of vector calculations through optimization of underlying execution algorithms, providing users with fast, elastic, high-performance, massive storage, and secure and reliable vector database services. Additionally, PolarDB-PG also supports multi-dimensional and multi-modal spatiotemporal information engines and geographic information engines.At the same time, PolarDB-PG is equipped with complete OLAP functionality and service level agreements, which has been recognized and used by many users;\n\n\n\n\n\n\n\n### Deployment options\n\n- Using [PolarDB-PG Cloud Vector Database](https://www.alibabacloud.com/product/polardb-for-postgresql). [Click here](https://www.alibabacloud.com/product/polardb-for-postgresql?spm=a3c0i.147400.6791778070.243.9f204881g5cjpP) to fast deploy it.\n\n## Prerequisites\n\nFor the purposes of this exercise we need to prepare a couple of things:\n\n1. PolarDB-PG cloud server instance.\n2. The 'psycopg2' library to interact with the vector database. Any other postgresql client library is ok.\n3. An [OpenAI API key](https://beta.openai.com/account/api-keys).\n\nWe might validate if the server was launched successfully by running a simple curl command:\n\n### Install requirements\n\nThis notebook obviously requires the `openai` and `psycopg2` packages, but there are also some other additional libraries we will use. The following command installs them all:\n\n\n\n```python\n! pip install openai psycopg2 pandas wget\n```\n\nPrepare your OpenAI API key\nThe OpenAI API key is used for vectorization of the documents and queries.\n\nIf you don't have an OpenAI API key, you can get one from https://beta.openai.com/account/api-keys.\n\nOnce you get your key, please add it to your environment variables as OPENAI_API_KEY.\n\nIf you have any doubts about setting the API key through environment variables, please refer to [Best Practices for API Key Safety](https://help.openai.com/en/articles/5112595-best-practices-for-api-key-safety).\n\n\n```python\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    print(\"OPENAI_API_KEY is ready\")\nelse:\n    print(\"OPENAI_API_KEY environment variable not found\")\n```\n\n    OPENAI_API_KEY is ready\n\n\n## Connect to PolarDB\nFirst add it to your environment variables. or you can just change the \"psycopg2.connect\" parameters below\n\nConnecting to a running instance of PolarDB server is easy with the official Python library:"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: fce88700-0d92-4838-98d8-0e450d1ecea2
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"fce88700-0d92-4838-98d8-0e450d1ecea2\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb\", \"title\": \"Using PolarDB-PG as a vector database for OpenAI embeddings\", \"tokens\": 2748, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"aa34d306-c403-425b-b95b-45841d5f1e32\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb\", \"title\": \"Using PolarDB-PG as a vector database for OpenAI embeddings\", \"tokens\": 2748, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7ea9039a22b6e6ab7acdec7882243aa4ae9839a86a9764f9c68acccd1014b060\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"39b6653f-2233-40f4-a164-6fc939ddeda9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb\", \"title\": \"Using PolarDB-PG as a vector database for OpenAI embeddings\", \"tokens\": 2748, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ee697728ae22c33584f898502e330983241dab4bb4109fd83aa5ce355cc5e008\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e82ff232-3a28-4823-8fdb-58140635222b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"eb8d4ce2a9e537423ed98f7afabb29a18a9add9fe31825a69ca22d9ffcd99ead\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2284, \"end_char_idx\": 5584, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "aa34d306-c403-425b-b95b-45841d5f1e32", "doc_id": "aa34d306-c403-425b-b95b-45841d5f1e32", "ref_doc_id": "aa34d306-c403-425b-b95b-45841d5f1e32", "chroma:document": "```python\n! pip install openai psycopg2 pandas wget\n```\n\nPrepare your OpenAI API key\nThe OpenAI API key is used for vectorization of the documents and queries.\n\nIf you don't have an OpenAI API key, you can get one from https://beta.openai.com/account/api-keys.\n\nOnce you get your key, please add it to your environment variables as OPENAI_API_KEY.\n\nIf you have any doubts about setting the API key through environment variables, please refer to [Best Practices for API Key Safety](https://help.openai.com/en/articles/5112595-best-practices-for-api-key-safety).\n\n\n```python\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    print(\"OPENAI_API_KEY is ready\")\nelse:\n    print(\"OPENAI_API_KEY environment variable not found\")\n```\n\n    OPENAI_API_KEY is ready\n\n\n## Connect to PolarDB\nFirst add it to your environment variables. or you can just change the \"psycopg2.connect\" parameters below\n\nConnecting to a running instance of PolarDB server is easy with the official Python library:\n\n\n```python\nimport os\nimport psycopg2\n\n# Note. alternatively you can set a temporary env variable like this:\n# os.environ[\"PGHOST\"] = \"your_host\"\n# os.environ[\"PGPORT\"] \"5432\"),\n# os.environ[\"PGDATABASE\"] \"postgres\"),\n# os.environ[\"PGUSER\"] \"user\"),\n# os.environ[\"PGPASSWORD\"] \"password\"),\n\nconnection = psycopg2.connect(\n    host=os.environ.get(\"PGHOST\", \"localhost\"),\n    port=os.environ.get(\"PGPORT\", \"5432\"),\n    database=os.environ.get(\"PGDATABASE\", \"postgres\"),\n    user=os.environ.get(\"PGUSER\", \"user\"),\n    password=os.environ.get(\"PGPASSWORD\", \"password\")\n)\n\n# Create a new cursor object\ncursor = connection.cursor()\n```\n\nWe can test the connection by running any available method:\n\n\n```python\n# Execute a simple query to test the connection\ncursor.execute(\"SELECT 1;\")\nresult = cursor.fetchone()\n\n# Check the query result\nif result == (1,):\n    print(\"Connection successful!\")\nelse:\n    print(\"Connection failed.\")\n```\n\n    Connection successful!\n\n\n\n```python\nimport wget\n\nembeddings_url = \"https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip\"\n\n# The file is ~700 MB so this will take some time\nwget.download(embeddings_url)\n```\n\n\n\n\n    'vector_database_wikipedia_articles_embedded.zip'\n\n\n\nThe downloaded file has to be then extracted:\n\n\n```python\nimport zipfile\nimport os\nimport re\nimport tempfile\n\ncurrent_directory = os.getcwd()\nzip_file_path = os.path.join(current_directory, \"vector_database_wikipedia_articles_embedded.zip\")\noutput_directory = os.path.join(current_directory, \"../../data\")\n\nwith zipfile.ZipFile(zip_file_path, \"r\") as zip_ref:\n    zip_ref.extractall(output_directory)\n\n\n# check the csv file exist\nfile_name = \"vector_database_wikipedia_articles_embedded.csv\"\ndata_directory = os.path.join(current_directory, \"../../data\")\nfile_path = os.path.join(data_directory, file_name)\n\n\nif os.path.exists(file_path):\n    print(f\"The file {file_name} exists in the data directory.\")\nelse:\n    print(f\"The file {file_name} does not exist in the data directory.\")\n```\n\n    The file vector_database_wikipedia_articles_embedded.csv exists in the data directory."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: e82ff232-3a28-4823-8fdb-58140635222b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e82ff232-3a28-4823-8fdb-58140635222b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb\", \"title\": \"Using PolarDB-PG as a vector database for OpenAI embeddings\", \"tokens\": 2748, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"aa34d306-c403-425b-b95b-45841d5f1e32\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb\", \"title\": \"Using PolarDB-PG as a vector database for OpenAI embeddings\", \"tokens\": 2748, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7ea9039a22b6e6ab7acdec7882243aa4ae9839a86a9764f9c68acccd1014b060\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fce88700-0d92-4838-98d8-0e450d1ecea2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb\", \"title\": \"Using PolarDB-PG as a vector database for OpenAI embeddings\", \"tokens\": 2748, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"b41d8cd51e96dd025aab755547320317cb1087201db2b065f2230e76d269ee39\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d0ca58a2-42d1-441d-94f9-cbad1b7afef4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6e4f36f0512d624a799fb7e9d5169aa3dbb52592b6ca5380a7f650bec67db57f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4162, \"end_char_idx\": 6832, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "aa34d306-c403-425b-b95b-45841d5f1e32", "doc_id": "aa34d306-c403-425b-b95b-45841d5f1e32", "ref_doc_id": "aa34d306-c403-425b-b95b-45841d5f1e32", "chroma:document": "```python\n# Execute a simple query to test the connection\ncursor.execute(\"SELECT 1;\")\nresult = cursor.fetchone()\n\n# Check the query result\nif result == (1,):\n    print(\"Connection successful!\")\nelse:\n    print(\"Connection failed.\")\n```\n\n    Connection successful!\n\n\n\n```python\nimport wget\n\nembeddings_url = \"https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip\"\n\n# The file is ~700 MB so this will take some time\nwget.download(embeddings_url)\n```\n\n\n\n\n    'vector_database_wikipedia_articles_embedded.zip'\n\n\n\nThe downloaded file has to be then extracted:\n\n\n```python\nimport zipfile\nimport os\nimport re\nimport tempfile\n\ncurrent_directory = os.getcwd()\nzip_file_path = os.path.join(current_directory, \"vector_database_wikipedia_articles_embedded.zip\")\noutput_directory = os.path.join(current_directory, \"../../data\")\n\nwith zipfile.ZipFile(zip_file_path, \"r\") as zip_ref:\n    zip_ref.extractall(output_directory)\n\n\n# check the csv file exist\nfile_name = \"vector_database_wikipedia_articles_embedded.csv\"\ndata_directory = os.path.join(current_directory, \"../../data\")\nfile_path = os.path.join(data_directory, file_name)\n\n\nif os.path.exists(file_path):\n    print(f\"The file {file_name} exists in the data directory.\")\nelse:\n    print(f\"The file {file_name} does not exist in the data directory.\")\n```\n\n    The file vector_database_wikipedia_articles_embedded.csv exists in the data directory.\n\n\n## Index data\n\nPolarDB stores data in __relation__ where each object is described by at least one vector. Our relation will be called **articles** and each object will be described by both **title** and **content** vectors. \n\nWe will start with creating a relation and create a vector index on both **title** and **content**, and then we will fill it with our precomputed embeddings.\n\n\n```python\ncreate_table_sql = '''\nCREATE TABLE IF NOT EXISTS public.articles (\n    id INTEGER NOT NULL,\n    url TEXT,\n    title TEXT,\n    content TEXT,\n    title_vector vector(1536),\n    content_vector vector(1536),\n    vector_id INTEGER\n);\n\nALTER TABLE public.articles ADD PRIMARY KEY (id);\n'''\n\n# SQL statement for creating indexes\ncreate_indexes_sql = '''\nCREATE INDEX ON public.articles USING ivfflat (content_vector) WITH (lists = 1000);\n\nCREATE INDEX ON public.articles USING ivfflat (title_vector) WITH (lists = 1000);\n'''\n\n# Execute the SQL statements\ncursor.execute(create_table_sql)\ncursor.execute(create_indexes_sql)\n\n# Commit the changes\nconnection.commit()\n```\n\n## Load data\n\nIn this section we are going to load the data prepared previous to this session, so you don't have to recompute the embeddings of Wikipedia articles with your own credits."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: d0ca58a2-42d1-441d-94f9-cbad1b7afef4
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"d0ca58a2-42d1-441d-94f9-cbad1b7afef4\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb\", \"title\": \"Using PolarDB-PG as a vector database for OpenAI embeddings\", \"tokens\": 2748, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"aa34d306-c403-425b-b95b-45841d5f1e32\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb\", \"title\": \"Using PolarDB-PG as a vector database for OpenAI embeddings\", \"tokens\": 2748, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7ea9039a22b6e6ab7acdec7882243aa4ae9839a86a9764f9c68acccd1014b060\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e82ff232-3a28-4823-8fdb-58140635222b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb\", \"title\": \"Using PolarDB-PG as a vector database for OpenAI embeddings\", \"tokens\": 2748, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"de67d0c6c5a0a3b5f0ea49646cc328e3cec97bd4196be54936a7e5018bcf3064\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"baa0db2c-c764-46e3-99c2-cf4ec0dac98f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f706c861f786060476c68eb7e51f1a6fb1b683ec10e3695acce7fef2f191d7b1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5109, \"end_char_idx\": 8230, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "aa34d306-c403-425b-b95b-45841d5f1e32", "doc_id": "aa34d306-c403-425b-b95b-45841d5f1e32", "ref_doc_id": "aa34d306-c403-425b-b95b-45841d5f1e32", "chroma:document": "# check the csv file exist\nfile_name = \"vector_database_wikipedia_articles_embedded.csv\"\ndata_directory = os.path.join(current_directory, \"../../data\")\nfile_path = os.path.join(data_directory, file_name)\n\n\nif os.path.exists(file_path):\n    print(f\"The file {file_name} exists in the data directory.\")\nelse:\n    print(f\"The file {file_name} does not exist in the data directory.\")\n```\n\n    The file vector_database_wikipedia_articles_embedded.csv exists in the data directory.\n\n\n## Index data\n\nPolarDB stores data in __relation__ where each object is described by at least one vector. Our relation will be called **articles** and each object will be described by both **title** and **content** vectors. \n\nWe will start with creating a relation and create a vector index on both **title** and **content**, and then we will fill it with our precomputed embeddings.\n\n\n```python\ncreate_table_sql = '''\nCREATE TABLE IF NOT EXISTS public.articles (\n    id INTEGER NOT NULL,\n    url TEXT,\n    title TEXT,\n    content TEXT,\n    title_vector vector(1536),\n    content_vector vector(1536),\n    vector_id INTEGER\n);\n\nALTER TABLE public.articles ADD PRIMARY KEY (id);\n'''\n\n# SQL statement for creating indexes\ncreate_indexes_sql = '''\nCREATE INDEX ON public.articles USING ivfflat (content_vector) WITH (lists = 1000);\n\nCREATE INDEX ON public.articles USING ivfflat (title_vector) WITH (lists = 1000);\n'''\n\n# Execute the SQL statements\ncursor.execute(create_table_sql)\ncursor.execute(create_indexes_sql)\n\n# Commit the changes\nconnection.commit()\n```\n\n## Load data\n\nIn this section we are going to load the data prepared previous to this session, so you don't have to recompute the embeddings of Wikipedia articles with your own credits.\n\n\n```python\nimport io\n\n# Path to your local CSV file\ncsv_file_path = '../../data/vector_database_wikipedia_articles_embedded.csv'\n\n# Define a generator function to process the file line by line\ndef process_file(file_path):\n    with open(file_path, 'r') as file:\n        for line in file:\n            yield line\n\n# Create a StringIO object to store the modified lines\nmodified_lines = io.StringIO(''.join(list(process_file(csv_file_path))))\n\n# Create the COPY command for the copy_expert method\ncopy_command = '''\nCOPY public.articles (id, url, title, content, title_vector, content_vector, vector_id)\nFROM STDIN WITH (FORMAT CSV, HEADER true, DELIMITER ',');\n'''\n\n# Execute the COPY command using the copy_expert method\ncursor.copy_expert(copy_command, modified_lines)\n\n# Commit the changes\nconnection.commit()\n```\n\n\n```python\n# Check the collection size to make sure all the points have been stored\ncount_sql = \"\"\"select count(*) from public.articles;\"\"\"\ncursor.execute(count_sql)\nresult = cursor.fetchone()\nprint(f\"Count:{result[0]}\")\n```\n\n    Count:25000\n\n\n## Search data\n\nOnce the data is put into Qdrant we will start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search. Since the precomputed embeddings were created with `text-embedding-3-small` OpenAI model we also have to use it during search."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: baa0db2c-c764-46e3-99c2-cf4ec0dac98f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"baa0db2c-c764-46e3-99c2-cf4ec0dac98f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb\", \"title\": \"Using PolarDB-PG as a vector database for OpenAI embeddings\", \"tokens\": 2748, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"aa34d306-c403-425b-b95b-45841d5f1e32\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb\", \"title\": \"Using PolarDB-PG as a vector database for OpenAI embeddings\", \"tokens\": 2748, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7ea9039a22b6e6ab7acdec7882243aa4ae9839a86a9764f9c68acccd1014b060\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d0ca58a2-42d1-441d-94f9-cbad1b7afef4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb\", \"title\": \"Using PolarDB-PG as a vector database for OpenAI embeddings\", \"tokens\": 2748, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"9693089b14ea0eab20f334aa877784ba035948f7ed9b3a7e42c2e75b767e65ff\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d09516de-c4fc-4924-b323-1613e7273940\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bc9b4d11117499d2f634b6dcc04e14259d7684f9aec7f2138c743cce142eccd4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6835, \"end_char_idx\": 9025, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "aa34d306-c403-425b-b95b-45841d5f1e32", "doc_id": "aa34d306-c403-425b-b95b-45841d5f1e32", "ref_doc_id": "aa34d306-c403-425b-b95b-45841d5f1e32", "chroma:document": "```python\nimport io\n\n# Path to your local CSV file\ncsv_file_path = '../../data/vector_database_wikipedia_articles_embedded.csv'\n\n# Define a generator function to process the file line by line\ndef process_file(file_path):\n    with open(file_path, 'r') as file:\n        for line in file:\n            yield line\n\n# Create a StringIO object to store the modified lines\nmodified_lines = io.StringIO(''.join(list(process_file(csv_file_path))))\n\n# Create the COPY command for the copy_expert method\ncopy_command = '''\nCOPY public.articles (id, url, title, content, title_vector, content_vector, vector_id)\nFROM STDIN WITH (FORMAT CSV, HEADER true, DELIMITER ',');\n'''\n\n# Execute the COPY command using the copy_expert method\ncursor.copy_expert(copy_command, modified_lines)\n\n# Commit the changes\nconnection.commit()\n```\n\n\n```python\n# Check the collection size to make sure all the points have been stored\ncount_sql = \"\"\"select count(*) from public.articles;\"\"\"\ncursor.execute(count_sql)\nresult = cursor.fetchone()\nprint(f\"Count:{result[0]}\")\n```\n\n    Count:25000\n\n\n## Search data\n\nOnce the data is put into Qdrant we will start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search. Since the precomputed embeddings were created with `text-embedding-3-small` OpenAI model we also have to use it during search.\n\n\n```python\ndef query_polardb(query, collection_name, vector_name=\"title_vector\", top_k=20):\n\n    # Creates embedding vector from user query\n    embedded_query = openai.Embedding.create(\n        input=query,\n        model=\"text-embedding-3-small\",\n    )[\"data\"][0][\"embedding\"]\n\n    # Convert the embedded_query to PostgreSQL compatible format\n    embedded_query_pg = \"[\" + \",\".join(map(str, embedded_query)) + \"]\"\n\n    # Create SQL query\n    query_sql = f\"\"\"\n    SELECT id, url, title, l2_distance({vector_name},'{embedded_query_pg}'::VECTOR(1536)) AS similarity\n    FROM {collection_name}\n    ORDER BY {vector_name} <-> '{embedded_query_pg}'::VECTOR(1536)\n    LIMIT {top_k};\n    \"\"\"\n    # Execute the query\n    cursor.execute(query_sql)\n    results = cursor.fetchall()\n\n    return results\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: d09516de-c4fc-4924-b323-1613e7273940
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"d09516de-c4fc-4924-b323-1613e7273940\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb\", \"title\": \"Using PolarDB-PG as a vector database for OpenAI embeddings\", \"tokens\": 2748, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"aa34d306-c403-425b-b95b-45841d5f1e32\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb\", \"title\": \"Using PolarDB-PG as a vector database for OpenAI embeddings\", \"tokens\": 2748, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7ea9039a22b6e6ab7acdec7882243aa4ae9839a86a9764f9c68acccd1014b060\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"baa0db2c-c764-46e3-99c2-cf4ec0dac98f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb\", \"title\": \"Using PolarDB-PG as a vector database for OpenAI embeddings\", \"tokens\": 2748, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"4365797689ab3ee99a565a6a8c0741a017d90ec5d1f07b6dcd9a559d2ae6c653\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dfbd0dcc-fbbe-4888-9f1e-009eef6f0f46\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3e415c5c34d49f9e4ff77a6eb2a8d4d18805c54291a7e2021f3e34fee063d0e7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7650, \"end_char_idx\": 9938, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "aa34d306-c403-425b-b95b-45841d5f1e32", "doc_id": "aa34d306-c403-425b-b95b-45841d5f1e32", "ref_doc_id": "aa34d306-c403-425b-b95b-45841d5f1e32", "chroma:document": "```python\n# Check the collection size to make sure all the points have been stored\ncount_sql = \"\"\"select count(*) from public.articles;\"\"\"\ncursor.execute(count_sql)\nresult = cursor.fetchone()\nprint(f\"Count:{result[0]}\")\n```\n\n    Count:25000\n\n\n## Search data\n\nOnce the data is put into Qdrant we will start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search. Since the precomputed embeddings were created with `text-embedding-3-small` OpenAI model we also have to use it during search.\n\n\n```python\ndef query_polardb(query, collection_name, vector_name=\"title_vector\", top_k=20):\n\n    # Creates embedding vector from user query\n    embedded_query = openai.Embedding.create(\n        input=query,\n        model=\"text-embedding-3-small\",\n    )[\"data\"][0][\"embedding\"]\n\n    # Convert the embedded_query to PostgreSQL compatible format\n    embedded_query_pg = \"[\" + \",\".join(map(str, embedded_query)) + \"]\"\n\n    # Create SQL query\n    query_sql = f\"\"\"\n    SELECT id, url, title, l2_distance({vector_name},'{embedded_query_pg}'::VECTOR(1536)) AS similarity\n    FROM {collection_name}\n    ORDER BY {vector_name} <-> '{embedded_query_pg}'::VECTOR(1536)\n    LIMIT {top_k};\n    \"\"\"\n    # Execute the query\n    cursor.execute(query_sql)\n    results = cursor.fetchall()\n\n    return results\n```\n\n\n```python\nimport openai\n\nquery_results = query_polardb(\"modern art in Europe\", \"Articles\")\nfor i, result in enumerate(query_results):\n    print(f\"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})\")\n```\n\n    1. Museum of Modern Art (Score: 0.5)\n    2. Western Europe (Score: 0.485)\n    3. Renaissance art (Score: 0.479)\n    4. Pop art (Score: 0.472)\n    5. Northern Europe (Score: 0.461)\n    6. Hellenistic art (Score: 0.457)\n    7. Modernist literature (Score: 0.447)\n    8. Art film (Score: 0.44)\n    9. Central Europe (Score: 0.439)\n    10. European (Score: 0.437)\n    11. Art (Score: 0.437)\n    12. Byzantine art (Score: 0.436)\n    13. Postmodernism (Score: 0.434)\n    14. Eastern Europe (Score: 0.433)\n    15. Europe (Score: 0.433)\n    16. Cubism (Score: 0.432)\n    17. Impressionism (Score: 0.432)\n    18. Bauhaus (Score: 0.431)\n    19. Surrealism (Score: 0.429)\n    20. Expressionism (Score: 0.429)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: dfbd0dcc-fbbe-4888-9f1e-009eef6f0f46
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb", "title": "Using PolarDB-PG as a vector database for OpenAI embeddings", "tokens": 2748, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"dfbd0dcc-fbbe-4888-9f1e-009eef6f0f46\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb\", \"title\": \"Using PolarDB-PG as a vector database for OpenAI embeddings\", \"tokens\": 2748, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"aa34d306-c403-425b-b95b-45841d5f1e32\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb\", \"title\": \"Using PolarDB-PG as a vector database for OpenAI embeddings\", \"tokens\": 2748, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7ea9039a22b6e6ab7acdec7882243aa4ae9839a86a9764f9c68acccd1014b060\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d09516de-c4fc-4924-b323-1613e7273940\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/PolarDB/Getting_started_with_PolarDB_and_OpenAI.ipynb\", \"title\": \"Using PolarDB-PG as a vector database for OpenAI embeddings\", \"tokens\": 2748, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"66c1d4fb958cbf6ba1017769611b9426b807a3aafe532545a729e0f28fa01217\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9028, \"end_char_idx\": 10891, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "aa34d306-c403-425b-b95b-45841d5f1e32", "doc_id": "aa34d306-c403-425b-b95b-45841d5f1e32", "ref_doc_id": "aa34d306-c403-425b-b95b-45841d5f1e32", "chroma:document": "```python\nimport openai\n\nquery_results = query_polardb(\"modern art in Europe\", \"Articles\")\nfor i, result in enumerate(query_results):\n    print(f\"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})\")\n```\n\n    1. Museum of Modern Art (Score: 0.5)\n    2. Western Europe (Score: 0.485)\n    3. Renaissance art (Score: 0.479)\n    4. Pop art (Score: 0.472)\n    5. Northern Europe (Score: 0.461)\n    6. Hellenistic art (Score: 0.457)\n    7. Modernist literature (Score: 0.447)\n    8. Art film (Score: 0.44)\n    9. Central Europe (Score: 0.439)\n    10. European (Score: 0.437)\n    11. Art (Score: 0.437)\n    12. Byzantine art (Score: 0.436)\n    13. Postmodernism (Score: 0.434)\n    14. Eastern Europe (Score: 0.433)\n    15. Europe (Score: 0.433)\n    16. Cubism (Score: 0.432)\n    17. Impressionism (Score: 0.432)\n    18. Bauhaus (Score: 0.431)\n    19. Surrealism (Score: 0.429)\n    20. Expressionism (Score: 0.429)\n\n\n\n```python\n# This time we'll query using content vector\nquery_results = query_polardb(\"Famous battles in Scottish history\", \"Articles\", \"content_vector\")\nfor i, result in enumerate(query_results):\n    print(f\"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})\")\n```\n\n    1. Battle of Bannockburn (Score: 0.489)\n    2. Wars of Scottish Independence (Score: 0.474)\n    3. 1651 (Score: 0.457)\n    4. First War of Scottish Independence (Score: 0.452)\n    5. Robert I of Scotland (Score: 0.445)\n    6. 841 (Score: 0.441)\n    7. 1716 (Score: 0.441)\n    8. 1314 (Score: 0.429)\n    9. 1263 (Score: 0.428)\n    10. William Wallace (Score: 0.426)\n    11. Stirling (Score: 0.419)\n    12. 1306 (Score: 0.419)\n    13. 1746 (Score: 0.418)\n    14. 1040s (Score: 0.414)\n    15. 1106 (Score: 0.412)\n    16. 1304 (Score: 0.411)\n    17. David II of Scotland (Score: 0.408)\n    18. Braveheart (Score: 0.407)\n    19. 1124 (Score: 0.406)\n    20. July 27 (Score: 0.405)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: e258eb03-3941-4756-84c1-1c93e7418b3d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/README.ipynb", "title": "MongoDB Atlas Vector Search", "tokens": 157, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e258eb03-3941-4756-84c1-1c93e7418b3d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/README.ipynb\", \"title\": \"MongoDB Atlas Vector Search\", \"tokens\": 157, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"222e2fb5-a6f0-4e25-8e5e-2329111a8aa2\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/README.ipynb\", \"title\": \"MongoDB Atlas Vector Search\", \"tokens\": 157, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"b9912da6f660b8ae862199a6e74542351ee2c71f608e425011fc8711b00619b6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 843, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "222e2fb5-a6f0-4e25-8e5e-2329111a8aa2", "doc_id": "222e2fb5-a6f0-4e25-8e5e-2329111a8aa2", "ref_doc_id": "222e2fb5-a6f0-4e25-8e5e-2329111a8aa2", "chroma:document": "# MongoDB Atlas Vector Search\n\n\n[Atlas Vector Search](https://www.mongodb.com/products/platform/atlas-vector-search) is a fully managed service that simplifies the process of effectively indexing high-dimensional vector data within MongoDB and being able to perform fast vector similarity searches. With Atlas Vector Search, you can use MongoDB as a standalone vector database for a new project or augment your existing MongoDB collections with vector search functionality. With Atlas Vector Search, you can use the powerful capabilities of vector search in any major public cloud (AWS, Azure, GCP) and achieve massive scalability and data security out of the box while being enterprise-ready with provisions like FedRamp, SoC2 compliance.\n\nDocumentation - [link](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-overview/)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 0bc18f70-942c-46cd-af9e-f99bdcf16467
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0bc18f70-942c-46cd-af9e-f99bdcf16467\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb\", \"title\": \"Step 1: Setup the environment\", \"tokens\": 2902, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c1bf1351-a47d-48e1-89e4-d12554457cab\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb\", \"title\": \"Step 1: Setup the environment\", \"tokens\": 2902, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8f788f5473198dcc1da342627aa5ab25abc285731f79fe25751d58da8e39a855\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"aa5d654c-87c8-4087-a6b5-046ed02ec5c6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bf834f704d62afb2fe7e52d066cf8100ddb46d3a676d5263c89b50c754e9b754\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1876, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "doc_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "ref_doc_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "chroma:document": "This notebook demonstrates how to  build a semantic search application  using OpenAI and [MongoDB Atlas vector search](https://www.mongodb.com/products/platform/atlas-vector-search)\n\n\n```python\n!pip install pymongo openai\n```\n\n    Collecting pymongo\n      Downloading pymongo-4.6.0-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (677 kB)\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m677.1/677.1 kB\u001b[0m \u001b[31m10.3 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25hCollecting openai\n      Downloading openai-1.3.3-py3-none-any.whl (220 kB)\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m220.3/220.3 kB\u001b[0m \u001b[31m24.4 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25hCollecting dnspython<3.0.0,>=1.16.0 (from pymongo)\n      Downloading dnspython-2.4.2-py3-none-any.whl (300 kB)\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m300.4/300.4 kB\u001b[0m \u001b[31m29.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25hRequirement already satisfied: anyio<4,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from openai) (3.7.1)\n    Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai) (1.7.0)\n    Collecting httpx<1,>=0.23.0 (from openai)\n      Downloading httpx-0.25.1-py3-none-any.whl (75 kB)\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m75.0/75.0 kB\u001b[0m \u001b[31m9.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25hRequirement already satisfied: pydantic<3,>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from openai) (1.10.13)\n    Requirement already satisfied: tqdm>4 in /usr/local/lib/python3.10/dist-packages (from openai) (4.66.1)\n    Requirement already satisfied: typing-extensions<5,>=4.5 in /usr/local/lib/python3.10/dist-packages (from openai) (4.5.0)\n    Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/dist-packages (from anyio<4,>=3.5.0->openai) (3."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: aa5d654c-87c8-4087-a6b5-046ed02ec5c6
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"aa5d654c-87c8-4087-a6b5-046ed02ec5c6\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb\", \"title\": \"Step 1: Setup the environment\", \"tokens\": 2902, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c1bf1351-a47d-48e1-89e4-d12554457cab\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb\", \"title\": \"Step 1: Setup the environment\", \"tokens\": 2902, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8f788f5473198dcc1da342627aa5ab25abc285731f79fe25751d58da8e39a855\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0bc18f70-942c-46cd-af9e-f99bdcf16467\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb\", \"title\": \"Step 1: Setup the environment\", \"tokens\": 2902, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8a6240bc55436b94946f28f7945e01b4c01faa42edc639d5445e6b99626d7b42\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"291c4855-6f93-4d38-9694-942ca5f30d50\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ffbaea1a2dffc3cf0494ba5f6589dc9bf4e8e0d451fd7b014fe17c0bc4dd9d0c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 882, \"end_char_idx\": 2734, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "doc_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "ref_doc_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "chroma:document": "4/300.4 kB\u001b[0m \u001b[31m29.0 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25hRequirement already satisfied: anyio<4,>=3.5.0 in /usr/local/lib/python3.10/dist-packages (from openai) (3.7.1)\n    Requirement already satisfied: distro<2,>=1.7.0 in /usr/lib/python3/dist-packages (from openai) (1.7.0)\n    Collecting httpx<1,>=0.23.0 (from openai)\n      Downloading httpx-0.25.1-py3-none-any.whl (75 kB)\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m75.0/75.0 kB\u001b[0m \u001b[31m9.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25hRequirement already satisfied: pydantic<3,>=1.9.0 in /usr/local/lib/python3.10/dist-packages (from openai) (1.10.13)\n    Requirement already satisfied: tqdm>4 in /usr/local/lib/python3.10/dist-packages (from openai) (4.66.1)\n    Requirement already satisfied: typing-extensions<5,>=4.5 in /usr/local/lib/python3.10/dist-packages (from openai) (4.5.0)\n    Requirement already satisfied: idna>=2.8 in /usr/local/lib/python3.10/dist-packages (from anyio<4,>=3.5.0->openai) (3.4)\n    Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.10/dist-packages (from anyio<4,>=3.5.0->openai) (1.3.0)\n    Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<4,>=3.5.0->openai) (1.1.3)\n    Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->openai) (2023.7.22)\n    Collecting httpcore (from httpx<1,>=0.23.0->openai)\n      Downloading httpcore-1.0.2-py3-none-any.whl (76 kB)\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m76.9/76.9 kB\u001b[0m \u001b[31m7.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25hCollecting h11<0.15,>=0.13 (from httpcore->httpx<1,>=0.23.0->openai)\n      Downloading h11-0.14.0-py3-none-any.whl (58 kB)\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m6."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 291c4855-6f93-4d38-9694-942ca5f30d50
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"291c4855-6f93-4d38-9694-942ca5f30d50\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb\", \"title\": \"Step 1: Setup the environment\", \"tokens\": 2902, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c1bf1351-a47d-48e1-89e4-d12554457cab\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb\", \"title\": \"Step 1: Setup the environment\", \"tokens\": 2902, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8f788f5473198dcc1da342627aa5ab25abc285731f79fe25751d58da8e39a855\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"aa5d654c-87c8-4087-a6b5-046ed02ec5c6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb\", \"title\": \"Step 1: Setup the environment\", \"tokens\": 2902, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6f9b81568bc6f2a0d0a6928ead925b3201833dcd10e0a72f6e981805010df246\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"81f45211-0cbe-477e-ac88-68f5dac67233\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"94e807b7219fe8c59ad1f9b8cae8ef0cb093a36f7270ee06453b7131a0139df3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1797, \"end_char_idx\": 3281, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "doc_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "ref_doc_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "chroma:document": "8 in /usr/local/lib/python3.10/dist-packages (from anyio<4,>=3.5.0->openai) (3.4)\n    Requirement already satisfied: sniffio>=1.1 in /usr/local/lib/python3.10/dist-packages (from anyio<4,>=3.5.0->openai) (1.3.0)\n    Requirement already satisfied: exceptiongroup in /usr/local/lib/python3.10/dist-packages (from anyio<4,>=3.5.0->openai) (1.1.3)\n    Requirement already satisfied: certifi in /usr/local/lib/python3.10/dist-packages (from httpx<1,>=0.23.0->openai) (2023.7.22)\n    Collecting httpcore (from httpx<1,>=0.23.0->openai)\n      Downloading httpcore-1.0.2-py3-none-any.whl (76 kB)\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m76.9/76.9 kB\u001b[0m \u001b[31m7.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25hCollecting h11<0.15,>=0.13 (from httpcore->httpx<1,>=0.23.0->openai)\n      Downloading h11-0.14.0-py3-none-any.whl (58 kB)\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m6.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25hInstalling collected packages: h11, dnspython, pymongo, httpcore, httpx, openai\n    \u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n    llmx 0.0.15a0 requires cohere, which is not installed.\n    llmx 0.0.15a0 requires tiktoken, which is not installed.\u001b[0m\u001b[31m\n    \u001b[0mSuccessfully installed dnspython-2.4.2 h11-0.14.0 httpcore-1.0.2 httpx-0.25.1 openai-1.3.3 pymongo-4.6.0"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 81f45211-0cbe-477e-ac88-68f5dac67233
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"81f45211-0cbe-477e-ac88-68f5dac67233\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb\", \"title\": \"Step 1: Setup the environment\", \"tokens\": 2902, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c1bf1351-a47d-48e1-89e4-d12554457cab\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb\", \"title\": \"Step 1: Setup the environment\", \"tokens\": 2902, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8f788f5473198dcc1da342627aa5ab25abc285731f79fe25751d58da8e39a855\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"291c4855-6f93-4d38-9694-942ca5f30d50\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb\", \"title\": \"Step 1: Setup the environment\", \"tokens\": 2902, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"9c4d9c8cac9f58b91d5b220bc14d673f540648c5e3c146cf0e7b00ac85d83d30\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a77c510b-581d-468a-bdf6-8bdeb90c7cd3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"02278226c2c323bbf32f73259683f52f9b87c1351ac311f682742132edcd9d89\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2356, \"end_char_idx\": 4821, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "doc_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "ref_doc_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "chroma:document": "0.2-py3-none-any.whl (76 kB)\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m76.9/76.9 kB\u001b[0m \u001b[31m7.9 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25hCollecting h11<0.15,>=0.13 (from httpcore->httpx<1,>=0.23.0->openai)\n      Downloading h11-0.14.0-py3-none-any.whl (58 kB)\n    \u001b[2K     \u001b[90m\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u2501\u001b[0m \u001b[32m58.3/58.3 kB\u001b[0m \u001b[31m6.8 MB/s\u001b[0m eta \u001b[36m0:00:00\u001b[0m\n    \u001b[?25hInstalling collected packages: h11, dnspython, pymongo, httpcore, httpx, openai\n    \u001b[31mERROR: pip's dependency resolver does not currently take into account all the packages that are installed. This behaviour is the source of the following dependency conflicts.\n    llmx 0.0.15a0 requires cohere, which is not installed.\n    llmx 0.0.15a0 requires tiktoken, which is not installed.\u001b[0m\u001b[31m\n    \u001b[0mSuccessfully installed dnspython-2.4.2 h11-0.14.0 httpcore-1.0.2 httpx-0.25.1 openai-1.3.3 pymongo-4.6.0\n\n\n# Step 1: Setup the environment\n\nThere are 2 pre-requisites for this:\n\n1.   **MongoDB Atlas cluster**: To create a forever free MongoDB Atlas cluster, first, you need to create a MongoDB Atlas account if you don't already have one. Visit the [MongoDB Atlas website](https://www.mongodb.com/atlas/database) and click on \u201cRegister.\u201d Visit the [MongoDB Atlas](https://account.mongodb.com/account/login) dashboard and set up your cluster. In order to take advantage of the `$vectorSearch` operator in an aggregation pipeline, you need to run MongoDB Atlas 6.0.11 or higher. This tutorial can be built using a free cluster. When you\u2019re setting up your deployment, you\u2019ll be prompted to set up a database user and rules for your network connection. Please ensure you save your username and password somewhere safe and have the correct IP address rules in place so your cluster can connect properly. If you need more help getting started, check out our [tutorial on MongoDB Atlas](https://www.mongodb.com/basics/mongodb-atlas-tutorial).\n\n2. **OpenAI API key**   To create your OpenAI key, you'll need to create an account. Once you have that, visit the [OpenAI platform](https://platform.openai.com/). Click on your profile icon in the top right of the screen to get the dropdown menu and select \u201cView API keys\u201d.\n\n\n\n```python\nimport getpass\n\nMONGODB_ATLAS_CLUSTER_URI = getpass.getpass(\"MongoDB Atlas Cluster URI:\")\nOPENAI_API_KEY = getpass.getpass(\"OpenAI API Key:\")\n\n```\n\n    MongoDB Atlas Cluster URI:\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\n    OpenAI API Key:\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: a77c510b-581d-468a-bdf6-8bdeb90c7cd3
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a77c510b-581d-468a-bdf6-8bdeb90c7cd3\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb\", \"title\": \"Step 1: Setup the environment\", \"tokens\": 2902, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c1bf1351-a47d-48e1-89e4-d12554457cab\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb\", \"title\": \"Step 1: Setup the environment\", \"tokens\": 2902, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8f788f5473198dcc1da342627aa5ab25abc285731f79fe25751d58da8e39a855\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"81f45211-0cbe-477e-ac88-68f5dac67233\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb\", \"title\": \"Step 1: Setup the environment\", \"tokens\": 2902, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"97a183732ffcc783448d5e9c595f68f2b6fb325b8b8a6a2b16354fd95c2cb22c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e1cd89fa-710a-4ca3-ad92-f911e16d0d5f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"571a5955e7443e30e668be4a9be947c18d75d86dee8f72d6952262232532546f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3284, \"end_char_idx\": 6448, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "doc_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "ref_doc_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "chroma:document": "# Step 1: Setup the environment\n\nThere are 2 pre-requisites for this:\n\n1.   **MongoDB Atlas cluster**: To create a forever free MongoDB Atlas cluster, first, you need to create a MongoDB Atlas account if you don't already have one. Visit the [MongoDB Atlas website](https://www.mongodb.com/atlas/database) and click on \u201cRegister.\u201d Visit the [MongoDB Atlas](https://account.mongodb.com/account/login) dashboard and set up your cluster. In order to take advantage of the `$vectorSearch` operator in an aggregation pipeline, you need to run MongoDB Atlas 6.0.11 or higher. This tutorial can be built using a free cluster. When you\u2019re setting up your deployment, you\u2019ll be prompted to set up a database user and rules for your network connection. Please ensure you save your username and password somewhere safe and have the correct IP address rules in place so your cluster can connect properly. If you need more help getting started, check out our [tutorial on MongoDB Atlas](https://www.mongodb.com/basics/mongodb-atlas-tutorial).\n\n2. **OpenAI API key**   To create your OpenAI key, you'll need to create an account. Once you have that, visit the [OpenAI platform](https://platform.openai.com/). Click on your profile icon in the top right of the screen to get the dropdown menu and select \u201cView API keys\u201d.\n\n\n\n```python\nimport getpass\n\nMONGODB_ATLAS_CLUSTER_URI = getpass.getpass(\"MongoDB Atlas Cluster URI:\")\nOPENAI_API_KEY = getpass.getpass(\"OpenAI API Key:\")\n\n```\n\n    MongoDB Atlas Cluster URI:\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\n    OpenAI API Key:\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\n\n\nNote: After executing the step above you will be prompted to enter the credentials.\n\nFor this tutorial, we will be using the\n[MongoDB sample dataset](https://www.mongodb.com/docs/atlas/sample-data/). Load the sample dataset using the Atlas UI. We'll be using the \u201csample_mflix\u201d database, which contains a \u201cmovies\u201d collection where each document contains fields like title, plot, genres, cast, directors, etc.\n\n\n\n```python\nimport openai\nimport pymongo\n\nclient = pymongo.MongoClient(MONGODB_ATLAS_CLUSTER_URI)\ndb = client.sample_mflix\ncollection = db.movies\n\nopenai.api_key = OPENAI_API_KEY\n```\n\n\n```python\nATLAS_VECTOR_SEARCH_INDEX_NAME = \"default\"\nEMBEDDING_FIELD_NAME = \"embedding_openai_nov19_23\"\n```\n\n# Step 2: Setup embeddings generation function\n\n\n```python\nmodel = \"text-embedding-3-small\"\ndef generate_embedding(text: str) -> list[float]:\n    return openai.embeddings.create(input = [text], model=model).data[0].embedding\n\n```\n\n# Step 3: Create and store embeddings\n\nEach document in the sample dataset sample_mflix.movies corresponds to a movie; we will execute an operation to create a vector embedding for the data in the \"plot\" field and store it in the database. Creating vector embeddings using OpenAI embeddings endpoint is necessary for performing a similarity search based on intent.\n\n\n```python\nfrom pymongo import ReplaceOne\n\n# Update the collection with the embeddings\nrequests = []\n\nfor doc in collection.find({'plot':{\"$exists\": True}}).limit(500):\n  doc[EMBEDDING_FIELD_NAME] = generate_embedding(doc['plot'])\n  requests.append(ReplaceOne({'_id': doc['_id']}, doc))\n\ncollection.bulk_write(requests)\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: e1cd89fa-710a-4ca3-ad92-f911e16d0d5f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e1cd89fa-710a-4ca3-ad92-f911e16d0d5f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb\", \"title\": \"Step 1: Setup the environment\", \"tokens\": 2902, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c1bf1351-a47d-48e1-89e4-d12554457cab\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb\", \"title\": \"Step 1: Setup the environment\", \"tokens\": 2902, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8f788f5473198dcc1da342627aa5ab25abc285731f79fe25751d58da8e39a855\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a77c510b-581d-468a-bdf6-8bdeb90c7cd3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb\", \"title\": \"Step 1: Setup the environment\", \"tokens\": 2902, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"53b9a6793dc309aee07f5155d030b210e046c947dc4f63f3875c9766f7a5c5cb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3c5fbbaf-35f8-4cc9-88cb-b6e639d20e8c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3a34d797ad92aaceb42b14a8efcfb4685b5ab57ac3a23603b92b063227964f51\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4824, \"end_char_idx\": 7459, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "doc_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "ref_doc_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "chroma:document": "Note: After executing the step above you will be prompted to enter the credentials.\n\nFor this tutorial, we will be using the\n[MongoDB sample dataset](https://www.mongodb.com/docs/atlas/sample-data/). Load the sample dataset using the Atlas UI. We'll be using the \u201csample_mflix\u201d database, which contains a \u201cmovies\u201d collection where each document contains fields like title, plot, genres, cast, directors, etc.\n\n\n\n```python\nimport openai\nimport pymongo\n\nclient = pymongo.MongoClient(MONGODB_ATLAS_CLUSTER_URI)\ndb = client.sample_mflix\ncollection = db.movies\n\nopenai.api_key = OPENAI_API_KEY\n```\n\n\n```python\nATLAS_VECTOR_SEARCH_INDEX_NAME = \"default\"\nEMBEDDING_FIELD_NAME = \"embedding_openai_nov19_23\"\n```\n\n# Step 2: Setup embeddings generation function\n\n\n```python\nmodel = \"text-embedding-3-small\"\ndef generate_embedding(text: str) -> list[float]:\n    return openai.embeddings.create(input = [text], model=model).data[0].embedding\n\n```\n\n# Step 3: Create and store embeddings\n\nEach document in the sample dataset sample_mflix.movies corresponds to a movie; we will execute an operation to create a vector embedding for the data in the \"plot\" field and store it in the database. Creating vector embeddings using OpenAI embeddings endpoint is necessary for performing a similarity search based on intent.\n\n\n```python\nfrom pymongo import ReplaceOne\n\n# Update the collection with the embeddings\nrequests = []\n\nfor doc in collection.find({'plot':{\"$exists\": True}}).limit(500):\n  doc[EMBEDDING_FIELD_NAME] = generate_embedding(doc['plot'])\n  requests.append(ReplaceOne({'_id': doc['_id']}, doc))\n\ncollection.bulk_write(requests)\n```\n\n\n\n\n    BulkWriteResult({'writeErrors': [], 'writeConcernErrors': [], 'nInserted': 0, 'nUpserted': 0, 'nMatched': 50, 'nModified': 50, 'nRemoved': 0, 'upserted': []}, acknowledged=True)\n\n\n\nAfter executing the above, the documents in \"movies\" collection will contain an additional field of \"embedding\", as defined by the `EMBEDDDING_FIELD_NAME` variable, apart from already existing fields like title, plot, genres, cast, directors, etc.\n\nNote: We are restricting this to just 500 documents in the interest of time. If you want to do this over the entire dataset of 23,000+ documents in our sample_mflix database, it will take a little while. Alternatively, you can use the [sample_mflix.embedded_movies collection](https://www.mongodb.com/docs/atlas/sample-data/sample-mflix/#sample_mflix.embedded_movies) which includes a pre-populated `plot_embedding` field that contains embeddings created using OpenAI's `text-embedding-3-small` embedding model that you can use with the Atlas Search vector search feature."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 3c5fbbaf-35f8-4cc9-88cb-b6e639d20e8c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"3c5fbbaf-35f8-4cc9-88cb-b6e639d20e8c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb\", \"title\": \"Step 1: Setup the environment\", \"tokens\": 2902, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c1bf1351-a47d-48e1-89e4-d12554457cab\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb\", \"title\": \"Step 1: Setup the environment\", \"tokens\": 2902, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8f788f5473198dcc1da342627aa5ab25abc285731f79fe25751d58da8e39a855\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e1cd89fa-710a-4ca3-ad92-f911e16d0d5f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb\", \"title\": \"Step 1: Setup the environment\", \"tokens\": 2902, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7858d3b2ce3cc2720174f4ee178e2c98aff7f9eaa69f97e55a7222c0c3ddad38\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"51ef8167-a673-4f3f-bfa9-f73ddb944996\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4e17cf03984c26731caab4e18a9b46fef2c7c804125d480a06f5b764ad35110a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6126, \"end_char_idx\": 9246, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "doc_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "ref_doc_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "chroma:document": "```python\nfrom pymongo import ReplaceOne\n\n# Update the collection with the embeddings\nrequests = []\n\nfor doc in collection.find({'plot':{\"$exists\": True}}).limit(500):\n  doc[EMBEDDING_FIELD_NAME] = generate_embedding(doc['plot'])\n  requests.append(ReplaceOne({'_id': doc['_id']}, doc))\n\ncollection.bulk_write(requests)\n```\n\n\n\n\n    BulkWriteResult({'writeErrors': [], 'writeConcernErrors': [], 'nInserted': 0, 'nUpserted': 0, 'nMatched': 50, 'nModified': 50, 'nRemoved': 0, 'upserted': []}, acknowledged=True)\n\n\n\nAfter executing the above, the documents in \"movies\" collection will contain an additional field of \"embedding\", as defined by the `EMBEDDDING_FIELD_NAME` variable, apart from already existing fields like title, plot, genres, cast, directors, etc.\n\nNote: We are restricting this to just 500 documents in the interest of time. If you want to do this over the entire dataset of 23,000+ documents in our sample_mflix database, it will take a little while. Alternatively, you can use the [sample_mflix.embedded_movies collection](https://www.mongodb.com/docs/atlas/sample-data/sample-mflix/#sample_mflix.embedded_movies) which includes a pre-populated `plot_embedding` field that contains embeddings created using OpenAI's `text-embedding-3-small` embedding model that you can use with the Atlas Search vector search feature.\n\n\n\n\n# Step 4: Create a vector search index\n\nWe will create Atlas Vector Search Index on this collection which will allow us to perform the Approximate KNN search, which powers the semantic search.\nWe will cover 2 ways to create this index - Atlas UI and using MongoDB python driver.\n\n(Optional) [Documentation: Create a Vector Search Index ](https://www.mongodb.com/docs/atlas/atlas-search/field-types/knn-vector/)\n\nNow head over to [Atlas UI](cloud.mongodb.com) and create an Atlas Vector Search index using the steps descibed [here](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-tutorial/#create-the-atlas-vector-search-index).  The 'dimensions' field with value 1536, corresponds to openAI text-embedding-ada002.\n\nUse the definition given below in the JSON editor on the Atlas UI.\n\n```\n{\n  \"mappings\": {\n    \"dynamic\": true,\n    \"fields\": {\n      \"embedding\": {\n        \"dimensions\": 1536,\n        \"similarity\": \"dotProduct\",\n        \"type\": \"knnVector\"\n      }\n    }\n  }\n}\n```\n\n(Optional) Alternatively, we can use [pymongo driver to create these vector search indexes programatically](https://pymongo.readthedocs.io/en/stable/api/pymongo/collection.html#pymongo.collection.Collection.create_search_index)\nThe python command given in the cell below will create the index (this only works for the most recent version of the Python Driver for MongoDB and MongoDB server version 7.0+ Atlas cluster).\n\n\n```python\ncollection.create_search_index(\n    {\"definition\":\n        {\"mappings\": {\"dynamic\": True, \"fields\": {\n            EMBEDDING_FIELD_NAME : {\n                \"dimensions\": 1536,\n                \"similarity\": \"dotProduct\",\n                \"type\": \"knnVector\"\n                }}}},\n     \"name\": ATLAS_VECTOR_SEARCH_INDEX_NAME\n    }\n)\n```\n\n\n\n\n    'default'"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 51ef8167-a673-4f3f-bfa9-f73ddb944996
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb", "title": "Step 1: Setup the environment", "tokens": 2902, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"51ef8167-a673-4f3f-bfa9-f73ddb944996\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb\", \"title\": \"Step 1: Setup the environment\", \"tokens\": 2902, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"c1bf1351-a47d-48e1-89e4-d12554457cab\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb\", \"title\": \"Step 1: Setup the environment\", \"tokens\": 2902, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8f788f5473198dcc1da342627aa5ab25abc285731f79fe25751d58da8e39a855\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3c5fbbaf-35f8-4cc9-88cb-b6e639d20e8c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/mongodb_atlas/semantic_search_using_mongodb_atlas_vector_search.ipynb\", \"title\": \"Step 1: Setup the environment\", \"tokens\": 2902, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ab3f3c0edcf7acaddc171cacfc25251ccc741ec5a3b886f609d24cbd92fec52f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8886, \"end_char_idx\": 10153, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "doc_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "ref_doc_id": "c1bf1351-a47d-48e1-89e4-d12554457cab", "chroma:document": "```python\ncollection.create_search_index(\n    {\"definition\":\n        {\"mappings\": {\"dynamic\": True, \"fields\": {\n            EMBEDDING_FIELD_NAME : {\n                \"dimensions\": 1536,\n                \"similarity\": \"dotProduct\",\n                \"type\": \"knnVector\"\n                }}}},\n     \"name\": ATLAS_VECTOR_SEARCH_INDEX_NAME\n    }\n)\n```\n\n\n\n\n    'default'\n\n\n\n# Step 5: Query your data\n\nThe results for the query here finds movies which have semantically similar plots to the text captured in the query string, rather than being based on the keyword search.\n\n(Optional) [Documentation: Run Vector Search Queries](https://www.mongodb.com/docs/atlas/atlas-vector-search/vector-search-stage/)\n\n\n```python\n\ndef query_results(query, k):\n  results = collection.aggregate([\n    {\n        '$vectorSearch': {\n            \"index\": ATLAS_VECTOR_SEARCH_INDEX_NAME,\n            \"path\": EMBEDDING_FIELD_NAME,\n            \"queryVector\": generate_embedding(query),\n            \"numCandidates\": 50,\n            \"limit\": 5,\n        }\n    }\n    ])\n  return results\n```\n\n\n```python\nquery=\"imaginary characters from outerspace at war with earthlings\"\nmovies = query_results(query, 5)\n\nfor movie in movies:\n    print(f'Movie Name: {movie[\"title\"]},\\nMovie Plot: {movie[\"plot\"]}\\n')\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 8c08307f-3fc1-4a56-bb79-4bec5aa06e05
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/README.ipynb", "title": "Supabase Vector Database", "tokens": 485, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"8c08307f-3fc1-4a56-bb79-4bec5aa06e05\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/README.ipynb\", \"title\": \"Supabase Vector Database\", \"tokens\": 485, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"d26fddf9-8a9b-4dd2-80f1-b2ad07d27f80\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/README.ipynb\", \"title\": \"Supabase Vector Database\", \"tokens\": 485, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"4ff93df9f93e39a08ec3f10972e80cc13ac6edeb5c465c982b134b3f5df9383e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2108, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "d26fddf9-8a9b-4dd2-80f1-b2ad07d27f80", "doc_id": "d26fddf9-8a9b-4dd2-80f1-b2ad07d27f80", "ref_doc_id": "d26fddf9-8a9b-4dd2-80f1-b2ad07d27f80", "chroma:document": "# Supabase Vector Database\n\n[Supabase](https://supabase.com/docs) is an open-source Firebase alternative built on top of [Postgres](https://en.wikipedia.org/wiki/PostgreSQL), a production-grade SQL database.\n\n[Supabase Vector](https://supabase.com/docs/guides/ai) is a vector toolkit built on [pgvector](https://github.com/pgvector/pgvector), a Postgres extension that allows you to store your embeddings inside the same database that holds the rest of your application data. When combined with pgvector's indexing algorithms, vector search remains [fast at large scales](https://supabase.com/blog/increase-performance-pgvector-hnsw).\n\nSupabase adds an ecosystem of services and tools on top of Postgres that makes app development as quick as possible, including:\n\n- [Auto-generated REST APIs](https://supabase.com/docs/guides/api)\n- [Auto-generated GraphQL APIs](https://supabase.com/docs/guides/graphql)\n- [Realtime APIs](https://supabase.com/docs/guides/realtime)\n- [Authentication](https://supabase.com/docs/guides/auth)\n- [File storage](https://supabase.com/docs/guides/storage)\n- [Edge functions](https://supabase.com/docs/guides/functions)\n\nWe can use these services alongside pgvector to store and query embeddings within Postgres.\n\n## OpenAI Cookbook Examples\n\nBelow are guides and resources that walk you through how to use OpenAI embedding models with Supabase Vector.\n\n| Guide                                    | Description                                                |\n| ---------------------------------------- | ---------------------------------------------------------- |\n| [Semantic search](./semantic-search.mdx) | Store, index, and query embeddings at scale using pgvector |\n\n## Additional resources\n\n- [Vector columns](https://supabase.com/docs/guides/ai/vector-columns)\n- [Vector indexes](https://supabase.com/docs/guides/ai/vector-indexes)\n- [RAG with permissions](https://supabase.com/docs/guides/ai/rag-with-permissions)\n- [Going to production](https://supabase.com/docs/guides/ai/going-to-prod)\n- [Deciding on compute](https://supabase.com/docs/guides/ai/choosing-compute-addon)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: a3694015-9ceb-4d1a-b8b7-0f6003d4960d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb", "title": "Semantic search using Supabase Vector", "tokens": 2931, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a3694015-9ceb-4d1a-b8b7-0f6003d4960d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb\", \"title\": \"Semantic search using Supabase Vector\", \"tokens\": 2931, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"194d89f3-3014-4f51-b486-e26acb62c304\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb\", \"title\": \"Semantic search using Supabase Vector\", \"tokens\": 2931, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a2bad350f3b11c3364f449d8ace39db42dfdfcb0d328c863612badce410f5ef6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"085d703f-8fcc-4c85-b930-a686f26c60f4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"274e5dfc41f35f0b36786cb64a0a123a242ab8e22fbc13569da7e0aa96d3a065\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3306, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "194d89f3-3014-4f51-b486-e26acb62c304", "doc_id": "194d89f3-3014-4f51-b486-e26acb62c304", "ref_doc_id": "194d89f3-3014-4f51-b486-e26acb62c304", "chroma:document": "# Semantic search using Supabase Vector\n\nThe purpose of this guide is to demonstrate how to store OpenAI embeddings in [Supabase Vector](https://supabase.com/docs/guides/ai) (Postgres + pgvector) for the purposes of semantic search.\n\n[Supabase](https://supabase.com/docs) is an open-source Firebase alternative built on top of [Postgres](https://en.wikipedia.org/wiki/PostgreSQL), a production-grade SQL database. Since Supabase Vector is built on [pgvector](https://github.com/pgvector/pgvector), you can store your embeddings within the same database that holds the rest of your application data. When combined with pgvector's indexing algorithms, vector search remains [fast at large scales](https://supabase.com/blog/increase-performance-pgvector-hnsw).\n\nSupabase adds an ecosystem of services and tools to make app development as quick as possible (such as an [auto-generated REST API](https://postgrest.org/)). We'll use these services to store and query embeddings within Postgres.\n\nThis guide covers:\n\n1. [Setting up your database](#setup-database)\n2. [Creating a SQL table](#create-a-vector-table) that can store vector data\n3. [Generating OpenAI embeddings](#generate-openai-embeddings) using OpenAI's JavaScript client\n4. [Storing the embeddings](#store-embeddings-in-database) in your SQL table using the Supabase JavaScript client\n5. [Performing semantic search](#semantic-search) over the embeddings using a Postgres function and the Supabase JavaScript client\n\n## Setup database\n\nFirst head over to https://database.new to provision your Supabase database. This will create a Postgres database on the Supabase cloud platform. Alternatively, you can follow the [local development](https://supabase.com/docs/guides/cli/getting-started) options if you prefer to run your database locally using Docker.\n\nIn the studio, jump to the [SQL editor](https://supabase.com/dashboard/project/_/sql/new) and execute the following SQL to enable pgvector:\n\n```sql\n-- Enable the pgvector extension\ncreate extension if not exists vector;\n```\n\n> In a production application, the best practice is to use [database migrations](https://supabase.com/docs/guides/cli/local-development#database-migrations) so that all SQL operations are managed within source control. To keep things simple in this guide, we'll execute queries directly in the SQL Editor. If you are building a production app, feel free to move these into a database migration.\n\n## Create a vector table\n\nNext we'll create a table to store documents and embeddings. In the SQL Editor, run:\n\n```sql\ncreate table documents (\n  id bigint primary key generated always as identity,\n  content text not null,\n  embedding vector (1536) not null\n);\n```\n\nSince Supabase is built on Postgres, we're just using regular SQL here. You can modify this table however you like to better fit your application. If you have existing database tables, you can simply add a new `vector` column to the appropriate table.\n\nThe important piece to understand is the `vector` data type, which is a new data type that became available when we enabled the pgvector extension earlier. The size of the vector (1536 here) represents the number of dimensions in the embedding. Since we're using OpenAI's `text-embedding-3-small` model in this example, we set the vector size to 1536."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 085d703f-8fcc-4c85-b930-a686f26c60f4
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb", "title": "Semantic search using Supabase Vector", "tokens": 2931, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"085d703f-8fcc-4c85-b930-a686f26c60f4\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb\", \"title\": \"Semantic search using Supabase Vector\", \"tokens\": 2931, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"194d89f3-3014-4f51-b486-e26acb62c304\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb\", \"title\": \"Semantic search using Supabase Vector\", \"tokens\": 2931, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a2bad350f3b11c3364f449d8ace39db42dfdfcb0d328c863612badce410f5ef6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a3694015-9ceb-4d1a-b8b7-0f6003d4960d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/supabase/semantic-search.ipynb\", \"title\": \"Semantic search using Supabase Vector\", \"tokens\": 2931, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c36c841f0aa71e620852839c85103d1321127727010c64b472f781405ead2fd4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e6c7b816-6a7f-4515-8bef-a139ae15db5f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7d979b28daafe7b9225568861afd644a52d9f7d3e724cdb07ed822d282d9efdc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1572, \"end_char_idx\": 4865, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "194d89f3-3014-4f51-b486-e26acb62c304", "doc_id": "194d89f3-3014-4f51-b486-e26acb62c304", "ref_doc_id": "194d89f3-3014-4f51-b486-e26acb62c304", "chroma:document": "This will create a Postgres database on the Supabase cloud platform. Alternatively, you can follow the [local development](https://supabase.com/docs/guides/cli/getting-started) options if you prefer to run your database locally using Docker.\n\nIn the studio, jump to the [SQL editor](https://supabase.com/dashboard/project/_/sql/new) and execute the following SQL to enable pgvector:\n\n```sql\n-- Enable the pgvector extension\ncreate extension if not exists vector;\n```\n\n> In a production application, the best practice is to use [database migrations](https://supabase.com/docs/guides/cli/local-development#database-migrations) so that all SQL operations are managed within source control. To keep things simple in this guide, we'll execute queries directly in the SQL Editor. If you are building a production app, feel free to move these into a database migration.\n\n## Create a vector table\n\nNext we'll create a table to store documents and embeddings. In the SQL Editor, run:\n\n```sql\ncreate table documents (\n  id bigint primary key generated always as identity,\n  content text not null,\n  embedding vector (1536) not null\n);\n```\n\nSince Supabase is built on Postgres, we're just using regular SQL here. You can modify this table however you like to better fit your application. If you have existing database tables, you can simply add a new `vector` column to the appropriate table.\n\nThe important piece to understand is the `vector` data type, which is a new data type that became available when we enabled the pgvector extension earlier. The size of the vector (1536 here) represents the number of dimensions in the embedding. Since we're using OpenAI's `text-embedding-3-small` model in this example, we set the vector size to 1536.\n\nLet's go ahead and create a vector index on this table so that future queries remain performant as the table grows:\n\n```sql\ncreate index on documents using hnsw (embedding vector_ip_ops);\n```\n\nThis index uses the [HNSW](https://supabase.com/docs/guides/ai/vector-indexes/hnsw-indexes) algorithm to index vectors stored in the `embedding` column, and specifically when using the inner product operator (`<#>`). We'll explain more about this operator later when we implement our match function.\n\nLet's also follow security best practices by enabling row level security on the table:\n\n```sql\nalter table documents enable row level security;\n```\n\nThis will prevent unauthorized access to this table through the auto-generated REST API (more on this shortly).\n\n## Generate OpenAI embeddings\n\nThis guide uses JavaScript to generate embeddings, but you can easily modify it to use any [language supported by OpenAI](https://platform.openai.com/docs/libraries).\n\nIf you are using JavaScript, feel free to use whichever server-side JavaScript runtime that you prefer (Node.js, Deno, Supabase Edge Functions).\n\nIf you're using Node.js, first install `openai` as a dependency:\n\n```shell\nnpm install openai\n```\n\nthen import it:\n\n```js\nimport OpenAI from \"openai\";\n```\n\nIf you're using Deno or Supabase Edge Functions, you can import `openai` directly from a URL:\n\n```js\nimport OpenAI from \"https://esm.sh/openai@4\";\n```\n\n> In this example we import from https://esm.sh which is a CDN that automatically fetches the respective NPM module for you and serves it over HTTP."}


