# embeddings_queue

created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 05c31335-5189-419f-9ecf-89042ef1c9cc
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"05c31335-5189-419f-9ecf-89042ef1c9cc\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4a45ef53-aefd-4743-897a-f9a419dd67e7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"7be14d3a937caed52cba6198905d299ebc14fe0ebd31255a016d88f94f6386ac\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"af9f8bfb-1a44-49e6-a430-823e328b4402\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0ea5a0a0365b421b510fde794bf46d94ab27e5befdb44b3c48202f7e84e917fa\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 52537, \"end_char_idx\": 55995, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "To use the LLMChain, first create a prompt template. from langchain.prompts import PromptTemplate from langchain.llms import OpenAI llm = OpenAI(temperature=0.9) prompt = PromptTemplate( input_variables=[\"product\"], template=\"What is a good\n    \n    ---\n    \n    chain first uses a LLM to construct the url to hit, then makes that request with the Requests wrapper, and finally runs that result through the language model again in order to product a natural language response. Example Notebook LLMBash Chain Links Used: BashProcess, LLMChain Notes: This chain takes user input (a question), uses an LLM chain to convert it to a bash command to run in the terminal, and then returns that as the result. Example Notebook LLMChecker Chain Links Used: LLMChain Notes: This chain takes user input (a question), uses an LLM chain to answer that question, and then uses other LLMChains to self-check that answer. Example Notebook LLMRequests Chain Links Used: Requests, LLMChain Notes: This chain takes a URL and other inputs, uses Requests to get the data at that URL, and then passes that along with the other inputs into an LLMChain to generate a response. The example included shows how to ask a question to Google - it firsts constructs a Google url, then fetches the data there, then passes that data + the original question into an LLMChain to get an answer. Example Notebook Moderation Chain Links Used: LLMChain, ModerationChain Notes: This chain shows how to use OpenAI\u2019s content\n    \n    ---\n    \n    Prompts: This includes prompt management, prompt optimization, and prompt serialization. LLMs: This includes a generic interface for all LLMs, and common utilities for working with LLMs. Document Loaders: This includes a standard interface for loading documents, as well as specific integrations to all types of text data sources. Utils: Language models are often more powerful when interacting with other sources of knowledge or computation. This can include Python REPLs, embeddings, search engines, and more. LangChain provides a large collection of common utils to use in your application. Chains: Chains go beyond just a single LLM call, and are sequences of calls (whether to an LLM or a different utility). LangChain provides a standard interface for chains, lots of integrations with other tools, and end-to-end chains for common applications. Indexes: Language models are often more powerful when combined with your own text data - this module covers best practices for doing exactly that. Agents: Agents involve an LLM making decisions about which Actions to take, taking that Action, seeing an Observation, and repeating that until done. LangChain provides a standard interface for agents, a selection of agents to choose from, and examples of end to end agents. Memory: Memory is the\n    \n    -----\n    \n    how do I use the LLMChain in LangChain?\n\n\nNow we ask the question:\n\n\n```python\n# system message to 'prime' the model\nprimer = f\"\"\"You are Q&A bot. A highly intelligent system that answers\nuser questions based on the information provided by the user above\neach question. If the information can not be found in the information\nprovided by the user you truthfully say \"I don't know\".\n\"\"\"\n\nres = openai.ChatCompletion.create(\n    model=\"gpt-4\",\n    messages=[\n        {\"role\": \"system\", \"content\": primer},\n        {\"role\": \"user\", \"content\": augmented_query}\n    ]\n)\n```\n\nTo display this response nicely, we will display it in markdown."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: af9f8bfb-1a44-49e6-a430-823e328b4402
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"af9f8bfb-1a44-49e6-a430-823e328b4402\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"05c31335-5189-419f-9ecf-89042ef1c9cc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"cf49c30205ec3da8523e7893fbf5f3cbdeacd2192daadab2e043a49c6bb2950b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"10e4d349-705a-44ce-b9aa-234eb4598f7d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"30f9cd2e071e6db21cb6da1f34cbc02c2a9d36734194c203353d0e07de610424\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 54229, \"end_char_idx\": 57341, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "Document Loaders: This includes a standard interface for loading documents, as well as specific integrations to all types of text data sources. Utils: Language models are often more powerful when interacting with other sources of knowledge or computation. This can include Python REPLs, embeddings, search engines, and more. LangChain provides a large collection of common utils to use in your application. Chains: Chains go beyond just a single LLM call, and are sequences of calls (whether to an LLM or a different utility). LangChain provides a standard interface for chains, lots of integrations with other tools, and end-to-end chains for common applications. Indexes: Language models are often more powerful when combined with your own text data - this module covers best practices for doing exactly that. Agents: Agents involve an LLM making decisions about which Actions to take, taking that Action, seeing an Observation, and repeating that until done. LangChain provides a standard interface for agents, a selection of agents to choose from, and examples of end to end agents. Memory: Memory is the\n    \n    -----\n    \n    how do I use the LLMChain in LangChain?\n\n\nNow we ask the question:\n\n\n```python\n# system message to 'prime' the model\nprimer = f\"\"\"You are Q&A bot. A highly intelligent system that answers\nuser questions based on the information provided by the user above\neach question. If the information can not be found in the information\nprovided by the user you truthfully say \"I don't know\".\n\"\"\"\n\nres = openai.ChatCompletion.create(\n    model=\"gpt-4\",\n    messages=[\n        {\"role\": \"system\", \"content\": primer},\n        {\"role\": \"user\", \"content\": augmented_query}\n    ]\n)\n```\n\nTo display this response nicely, we will display it in markdown.\n\n\n```python\nfrom IPython.display import Markdown\n\ndisplay(Markdown(res['choices'][0]['message']['content']))\n```\n\n\nTo use the LLMChain in LangChain, follow these steps:\n\n1. Import the necessary classes:\n```python\nfrom langchain.prompts import PromptTemplate\nfrom langchain.llms import OpenAI\nfrom langchain.chains import LLMChain\n```\n\n2. Create an instance of the LLM and set the configuration options:\n```python\nllm = OpenAI(temperature=0.9)\n```\n\n3. Create a PromptTemplate instance with the input variables and the template:\n```python\nprompt = PromptTemplate(\n    input_variables=[\"product\"],\n    template=\"What is a good product for {product}?\",\n)\n```\n\n4. Create an LLMChain instance by passing the LLM and PromptTemplate instances:\n```python\nllm_chain = LLMChain(llm=llm, prompt_template=prompt)\n```\n\n5. Run the LLMChain with user input:\n```python\nresponse = llm_chain.run({\"product\": \"software development\"})\n```\n\n6. Access the generated response:\n```python\ngenerated_text = response[\"generated_text\"]\n```\n\nIn this example, the LLMChain is used to generate a response by passing through the user input and formatting it using the prompt template. The response is then obtained from the LLM instance (in this case, OpenAI), and the generated text can be accessed from the response dictionary.\n\n\nLet's compare this to a non-augmented query..."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 10e4d349-705a-44ce-b9aa-234eb4598f7d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb", "title": "Retrieval Augmentation for GPT-4 using Pinecone", "tokens": 13195, "retrieve_doc": false, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"10e4d349-705a-44ce-b9aa-234eb4598f7d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"bf0ed612-cb81-4483-9b6b-98bc5789893c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"1010f8a021f460f2df4bd535d1d88b705bc2cc1d299dcb514c40a159745fc229\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"af9f8bfb-1a44-49e6-a430-823e328b4402\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/pinecone/GPT4_Retrieval_Augmentation.ipynb\", \"title\": \"Retrieval Augmentation for GPT-4 using Pinecone\", \"tokens\": 13195, \"retrieve_doc\": false, \"source\": \"openai_cookbooks\"}, \"hash\": \"29c29cfea2e8a3ffcdca4197d075a07beeeb856e668c4e849d68721d3006e726\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 55998, \"end_char_idx\": 58764, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "ref_doc_id": "bf0ed612-cb81-4483-9b6b-98bc5789893c", "chroma:document": "```python\nfrom IPython.display import Markdown\n\ndisplay(Markdown(res['choices'][0]['message']['content']))\n```\n\n\nTo use the LLMChain in LangChain, follow these steps:\n\n1. Import the necessary classes:\n```python\nfrom langchain.prompts import PromptTemplate\nfrom langchain.llms import OpenAI\nfrom langchain.chains import LLMChain\n```\n\n2. Create an instance of the LLM and set the configuration options:\n```python\nllm = OpenAI(temperature=0.9)\n```\n\n3. Create a PromptTemplate instance with the input variables and the template:\n```python\nprompt = PromptTemplate(\n    input_variables=[\"product\"],\n    template=\"What is a good product for {product}?\",\n)\n```\n\n4. Create an LLMChain instance by passing the LLM and PromptTemplate instances:\n```python\nllm_chain = LLMChain(llm=llm, prompt_template=prompt)\n```\n\n5. Run the LLMChain with user input:\n```python\nresponse = llm_chain.run({\"product\": \"software development\"})\n```\n\n6. Access the generated response:\n```python\ngenerated_text = response[\"generated_text\"]\n```\n\nIn this example, the LLMChain is used to generate a response by passing through the user input and formatting it using the prompt template. The response is then obtained from the LLM instance (in this case, OpenAI), and the generated text can be accessed from the response dictionary.\n\n\nLet's compare this to a non-augmented query...\n\n\n```python\nres = openai.ChatCompletion.create(\n    model=\"gpt-4\",\n    messages=[\n        {\"role\": \"system\", \"content\": primer},\n        {\"role\": \"user\", \"content\": query}\n    ]\n)\ndisplay(Markdown(res['choices'][0]['message']['content']))\n```\n\n\nI don't know.\n\n\nIf we drop the `\"I don't know\"` part of the `primer`?\n\n\n```python\nres = openai.ChatCompletion.create(\n    model=\"gpt-4\",\n    messages=[\n        {\"role\": \"system\", \"content\": \"You are Q&A bot. A highly intelligent system that answers user questions\"},\n        {\"role\": \"user\", \"content\": query}\n    ]\n)\ndisplay(Markdown(res['choices'][0]['message']['content']))\n```\n\n\nLangChain hasn't provided any public documentation on LLMChain, nor is there a known technology called LLMChain in their library. To better assist you, please provide more information or context about LLMChain and LangChain.\n\nMeanwhile, if you are referring to LangChain, a blockchain-based decentralized AI language model, you can start by visiting their official website (if they have one), exploring their available resources, such as documentation and tutorials, and following any instructions on setting up their technology.\n\nIf you are looking for help with a specific language chain or model in natural language processing, consider rephrasing your question to provide more accurate information or visit relevant resources like GPT-3 or other NLP-related documentation.\n\n\n\n```python\n\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: fbde881e-3823-4a6a-ba88-9365ba233303
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb", "title": "Vector similarity search using Neon Postgres", "tokens": 2947, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"fbde881e-3823-4a6a-ba88-9365ba233303\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7a1fe4be-5d9b-4f6b-866f-66a9470b5927\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3be94c682f3a332b91a21c63fd7e215ba9994445c22203fccf81f907a42dce64\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b81f0d49-f44a-4fed-a642-bdd41bfab093\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"59274334f782b21279e197a473a8d338d5e490e566c81b3a04c4dfd4d69fffd2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3092, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "doc_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "ref_doc_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "chroma:document": "# Vector similarity search using Neon Postgres\n\nThis notebook guides you through using [Neon Serverless Postgres](https://neon.tech/) as a vector database for OpenAI embeddings. It demonstrates how to:\n\n1. Use embeddings created by OpenAI API.\n2. Store embeddings in a Neon Serverless Postgres database.\n3. Convert a raw text query to an embedding with OpenAI API.\n4. Use Neon with the `pgvector` extension to perform vector similarity search.\n\n## Prerequisites\n\nBefore you begin, ensure that you have the following:\n\n1. A Neon Postgres database. You can create an account and set up a project with a ready-to-use `neondb` database in a few simple steps. For instructions, see [Sign up](https://neon.tech/docs/get-started-with-neon/signing-up) and [Create your first project](https://neon.tech/docs/get-started-with-neon/setting-up-a-project).\n2. A connection string for your Neon database. You can copy it from the **Connection Details** widget on the Neon **Dashboard**. See [Connect from any application](https://neon.tech/docs/connect/connect-from-any-app).\n3. The `pgvector` extension. Install the extension in Neon by running `CREATE EXTENSION vector;`. For instructions, see [Enable the pgvector extension](https://neon.tech/docs/extensions/pgvector#enable-the-pgvector-extension). \n4. Your [OpenAI API key](https://platform.openai.com/account/api-keys).\n5. Python and `pip`.\n\n### Install required modules\n\nThis notebook requires the `openai`, `psycopg2`, `pandas`, `wget`, and `python-dotenv` packages. You can install them with `pip`:\n\n\n\n```python\n! pip install openai psycopg2 pandas wget python-dotenv\n```\n\n### Prepare your OpenAI API key\n\nAn OpenAI API key is required to generate vectors for documents and queries.\n\nIf you do not have an OpenAI API key, obtain one from https://platform.openai.com/account/api-keys.\n\nAdd the OpenAI API key as an operating system environment variable or provide it for the session when prompted. If you define an environment variable, name the variable `OPENAI_API_KEY`.\n\nFor information about configuring your OpenAI API key as an environment variable, refer to [Best Practices for API Key Safety](https://help.openai.com/en/articles/5112595-best-practices-for-api-key-safety).\n\n### Test your OpenAPI key\n\n\n```python\n# Test to ensure that your OpenAI API key is defined as an environment variable or provide it when prompted\n# If you run this notebook locally, you may have to reload the terminal and the notebook to make the environment available\n\nimport os\nfrom getpass import getpass\n\n# Check if OPENAI_API_KEY is set as an environment variable\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    print(\"Your OPENAI_API_KEY is ready\")\nelse:\n    # If not, prompt for it\n    api_key = getpass(\"Enter your OPENAI_API_KEY: \")\n    if api_key:\n        print(\"Your OPENAI_API_KEY is now available for this session\")\n        # Optionally, you can set it as an environment variable for the current session\n        os.environ[\"OPENAI_API_KEY\"] = api_key\n    else:\n        print(\"You did not enter your OPENAI_API_KEY\")\n```\n\n    Your OPENAI_API_KEY is ready"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: b81f0d49-f44a-4fed-a642-bdd41bfab093
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb", "title": "Vector similarity search using Neon Postgres", "tokens": 2947, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"b81f0d49-f44a-4fed-a642-bdd41bfab093\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7a1fe4be-5d9b-4f6b-866f-66a9470b5927\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3be94c682f3a332b91a21c63fd7e215ba9994445c22203fccf81f907a42dce64\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fbde881e-3823-4a6a-ba88-9365ba233303\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8f07d5d2dda35c1e7c51a0317d4d6cef20970e9736d96340bf319529be4795dc\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"609b6107-a0a3-4872-ac7f-fab695118aa5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b4ba84e4484f66b4b6be27e57e0f02709791c8205cc53d90741d0e567e3a54de\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1547, \"end_char_idx\": 4870, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "doc_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "ref_doc_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "chroma:document": "```python\n! pip install openai psycopg2 pandas wget python-dotenv\n```\n\n### Prepare your OpenAI API key\n\nAn OpenAI API key is required to generate vectors for documents and queries.\n\nIf you do not have an OpenAI API key, obtain one from https://platform.openai.com/account/api-keys.\n\nAdd the OpenAI API key as an operating system environment variable or provide it for the session when prompted. If you define an environment variable, name the variable `OPENAI_API_KEY`.\n\nFor information about configuring your OpenAI API key as an environment variable, refer to [Best Practices for API Key Safety](https://help.openai.com/en/articles/5112595-best-practices-for-api-key-safety).\n\n### Test your OpenAPI key\n\n\n```python\n# Test to ensure that your OpenAI API key is defined as an environment variable or provide it when prompted\n# If you run this notebook locally, you may have to reload the terminal and the notebook to make the environment available\n\nimport os\nfrom getpass import getpass\n\n# Check if OPENAI_API_KEY is set as an environment variable\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    print(\"Your OPENAI_API_KEY is ready\")\nelse:\n    # If not, prompt for it\n    api_key = getpass(\"Enter your OPENAI_API_KEY: \")\n    if api_key:\n        print(\"Your OPENAI_API_KEY is now available for this session\")\n        # Optionally, you can set it as an environment variable for the current session\n        os.environ[\"OPENAI_API_KEY\"] = api_key\n    else:\n        print(\"You did not enter your OPENAI_API_KEY\")\n```\n\n    Your OPENAI_API_KEY is ready\n\n\n## Connect to your Neon database\n\nProvide your Neon database connection string below or define it in an `.env` file using a `DATABASE_URL` variable. For information about obtaining a Neon connection string, see [Connect from any application](https://neon.tech/docs/connect/connect-from-any-app).\n\n\n```python\nimport os\nimport psycopg2\nfrom dotenv import load_dotenv\n\n# Load environment variables from .env file\nload_dotenv()\n\n# The connection string can be provided directly here.\n# Replace the next line with Your Neon connection string.\nconnection_string = \"postgres://<user>:<password>@<hostname>/<dbname>\"\n\n# If connection_string is not directly provided above, \n# then check if DATABASE_URL is set in the environment or .env.\nif not connection_string:\n    connection_string = os.environ.get(\"DATABASE_URL\")\n\n    # If neither method provides a connection string, raise an error.\n    if not connection_string:\n        raise ValueError(\"Please provide a valid connection string either in the code or in the .env file as DATABASE_URL.\")\n\n# Connect using the connection string\nconnection = psycopg2.connect(connection_string)\n\n# Create a new cursor object\ncursor = connection.cursor()\n```\n\nTest the connection to your database:\n\n\n```python\n# Execute this query to test the database connection\ncursor.execute(\"SELECT 1;\")\nresult = cursor.fetchone()\n\n# Check the query result\nif result == (1,):\n    print(\"Your database connection was successful!\")\nelse:\n    print(\"Your connection failed.\")\n```\n\n    Your database connection was successful!\n\n\nThis guide uses pre-computed Wikipedia article embeddings available in the OpenAI Cookbook `examples` directory so that you do not have to compute embeddings with your own OpenAI credits. \n\nImport the pre-computed embeddings zip file:"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 609b6107-a0a3-4872-ac7f-fab695118aa5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb", "title": "Vector similarity search using Neon Postgres", "tokens": 2947, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"609b6107-a0a3-4872-ac7f-fab695118aa5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7a1fe4be-5d9b-4f6b-866f-66a9470b5927\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3be94c682f3a332b91a21c63fd7e215ba9994445c22203fccf81f907a42dce64\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b81f0d49-f44a-4fed-a642-bdd41bfab093\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"d5166e3ab929d2afb8755b10db3fe304982b2cfa139fd3285572cae820da2b0d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"98cf62b4-c974-4b90-804f-65ee5a08984e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1a4c4196141553d253293ffa37a627777c517529d7ab413c91fc7988bba0e329\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3095, \"end_char_idx\": 6311, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "doc_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "ref_doc_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "chroma:document": "## Connect to your Neon database\n\nProvide your Neon database connection string below or define it in an `.env` file using a `DATABASE_URL` variable. For information about obtaining a Neon connection string, see [Connect from any application](https://neon.tech/docs/connect/connect-from-any-app).\n\n\n```python\nimport os\nimport psycopg2\nfrom dotenv import load_dotenv\n\n# Load environment variables from .env file\nload_dotenv()\n\n# The connection string can be provided directly here.\n# Replace the next line with Your Neon connection string.\nconnection_string = \"postgres://<user>:<password>@<hostname>/<dbname>\"\n\n# If connection_string is not directly provided above, \n# then check if DATABASE_URL is set in the environment or .env.\nif not connection_string:\n    connection_string = os.environ.get(\"DATABASE_URL\")\n\n    # If neither method provides a connection string, raise an error.\n    if not connection_string:\n        raise ValueError(\"Please provide a valid connection string either in the code or in the .env file as DATABASE_URL.\")\n\n# Connect using the connection string\nconnection = psycopg2.connect(connection_string)\n\n# Create a new cursor object\ncursor = connection.cursor()\n```\n\nTest the connection to your database:\n\n\n```python\n# Execute this query to test the database connection\ncursor.execute(\"SELECT 1;\")\nresult = cursor.fetchone()\n\n# Check the query result\nif result == (1,):\n    print(\"Your database connection was successful!\")\nelse:\n    print(\"Your connection failed.\")\n```\n\n    Your database connection was successful!\n\n\nThis guide uses pre-computed Wikipedia article embeddings available in the OpenAI Cookbook `examples` directory so that you do not have to compute embeddings with your own OpenAI credits. \n\nImport the pre-computed embeddings zip file:\n\n\n```python\nimport wget\n\nembeddings_url = \"https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip\"\n\n# The file is ~700 MB. Importing it will take several minutes.\nwget.download(embeddings_url)\n```\n\n\n\n\n    'vector_database_wikipedia_articles_embedded.zip'\n\n\n\nExtract the downloaded zip file:\n\n\n```python\nimport zipfile\nimport os\nimport re\nimport tempfile\n\ncurrent_directory = os.getcwd()\nzip_file_path = os.path.join(current_directory, \"vector_database_wikipedia_articles_embedded.zip\")\noutput_directory = os.path.join(current_directory, \"../../data\")\n\nwith zipfile.ZipFile(zip_file_path, \"r\") as zip_ref:\n    zip_ref.extractall(output_directory)\n\n\n# Check to see if the csv file was extracted\nfile_name = \"vector_database_wikipedia_articles_embedded.csv\"\ndata_directory = os.path.join(current_directory, \"../../data\")\nfile_path = os.path.join(data_directory, file_name)\n\n\nif os.path.exists(file_path):\n    print(f\"The csv file {file_name} exists in the data directory.\")\nelse:\n    print(f\"The csv file {file_name} does not exist in the data directory.\")\n```\n\n    The file vector_database_wikipedia_articles_embedded.csv exists in the data directory.\n\n\n## Create a table and add indexes for your vector embeddings\n\nThe vector table created in your database is called **articles**. Each object has **title** and **content** vectors. \n\nAn index is defined on both the **title** and **content** vector columns."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 98cf62b4-c974-4b90-804f-65ee5a08984e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb", "title": "Vector similarity search using Neon Postgres", "tokens": 2947, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"98cf62b4-c974-4b90-804f-65ee5a08984e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7a1fe4be-5d9b-4f6b-866f-66a9470b5927\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3be94c682f3a332b91a21c63fd7e215ba9994445c22203fccf81f907a42dce64\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"609b6107-a0a3-4872-ac7f-fab695118aa5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"2ca7f01aa25914a2be44cd7912e7dd5d80e1720b8858e0c1da90e049d930224d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6ed64f1f-b25f-4569-93a3-72eecda53e37\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"68f394f4291179c0aadde84501df7f2ae2a8d46b2a05de1e6c5049847c26ad85\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4636, \"end_char_idx\": 8065, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "doc_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "ref_doc_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "chroma:document": "This guide uses pre-computed Wikipedia article embeddings available in the OpenAI Cookbook `examples` directory so that you do not have to compute embeddings with your own OpenAI credits. \n\nImport the pre-computed embeddings zip file:\n\n\n```python\nimport wget\n\nembeddings_url = \"https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip\"\n\n# The file is ~700 MB. Importing it will take several minutes.\nwget.download(embeddings_url)\n```\n\n\n\n\n    'vector_database_wikipedia_articles_embedded.zip'\n\n\n\nExtract the downloaded zip file:\n\n\n```python\nimport zipfile\nimport os\nimport re\nimport tempfile\n\ncurrent_directory = os.getcwd()\nzip_file_path = os.path.join(current_directory, \"vector_database_wikipedia_articles_embedded.zip\")\noutput_directory = os.path.join(current_directory, \"../../data\")\n\nwith zipfile.ZipFile(zip_file_path, \"r\") as zip_ref:\n    zip_ref.extractall(output_directory)\n\n\n# Check to see if the csv file was extracted\nfile_name = \"vector_database_wikipedia_articles_embedded.csv\"\ndata_directory = os.path.join(current_directory, \"../../data\")\nfile_path = os.path.join(data_directory, file_name)\n\n\nif os.path.exists(file_path):\n    print(f\"The csv file {file_name} exists in the data directory.\")\nelse:\n    print(f\"The csv file {file_name} does not exist in the data directory.\")\n```\n\n    The file vector_database_wikipedia_articles_embedded.csv exists in the data directory.\n\n\n## Create a table and add indexes for your vector embeddings\n\nThe vector table created in your database is called **articles**. Each object has **title** and **content** vectors. \n\nAn index is defined on both the **title** and **content** vector columns.\n\n\n```python\ncreate_table_sql = '''\nCREATE TABLE IF NOT EXISTS public.articles (\n    id INTEGER NOT NULL,\n    url TEXT,\n    title TEXT,\n    content TEXT,\n    title_vector vector(1536),\n    content_vector vector(1536),\n    vector_id INTEGER\n);\n\nALTER TABLE public.articles ADD PRIMARY KEY (id);\n'''\n\n# SQL statement for creating indexes\ncreate_indexes_sql = '''\nCREATE INDEX ON public.articles USING ivfflat (content_vector) WITH (lists = 1000);\n\nCREATE INDEX ON public.articles USING ivfflat (title_vector) WITH (lists = 1000);\n'''\n\n# Execute the SQL statements\ncursor.execute(create_table_sql)\ncursor.execute(create_indexes_sql)\n\n# Commit the changes\nconnection.commit()\n```\n\n## Load the data\n\nLoad the pre-computed vector data into your `articles` table from the `.csv` file. There are 25000 records, so expect the operation to take several minutes.\n\n\n```python\nimport io\n\n# Path to your local CSV file\ncsv_file_path = '../../data/vector_database_wikipedia_articles_embedded.csv'\n\n# Define a generator function to process the csv file\ndef process_file(file_path):\n    with open(file_path, 'r', encoding='utf-8') as file:\n        for line in file:\n            yield line\n\n# Create a StringIO object to store the modified lines\nmodified_lines = io.StringIO(''.join(list(process_file(csv_file_path))))\n\n# Create the COPY command for copy_expert\ncopy_command = '''\nCOPY public.articles (id, url, title, content, title_vector, content_vector, vector_id)\nFROM STDIN WITH (FORMAT CSV, HEADER true, DELIMITER ',');\n'''\n\n# Execute the COPY command using copy_expert\ncursor.copy_expert(copy_command, modified_lines)\n\n# Commit the changes\nconnection.commit()\n```\n\nCheck the number of records to ensure the data has been been loaded. There should be 25000 records."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 6ed64f1f-b25f-4569-93a3-72eecda53e37
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb", "title": "Vector similarity search using Neon Postgres", "tokens": 2947, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"6ed64f1f-b25f-4569-93a3-72eecda53e37\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7a1fe4be-5d9b-4f6b-866f-66a9470b5927\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3be94c682f3a332b91a21c63fd7e215ba9994445c22203fccf81f907a42dce64\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"98cf62b4-c974-4b90-804f-65ee5a08984e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"30ecde2b292e0d985fc46b74791fd7494e0734a0ba015db2fcd454248cd2e56e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a1cb9c8f-1842-4113-9b13-0e1f1947cb4d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8e9be04a9b45ed2ad19b08b614cb789a4979286c28fe9e98d54ced49a84e3c85\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6314, \"end_char_idx\": 8914, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "doc_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "ref_doc_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "chroma:document": "```python\ncreate_table_sql = '''\nCREATE TABLE IF NOT EXISTS public.articles (\n    id INTEGER NOT NULL,\n    url TEXT,\n    title TEXT,\n    content TEXT,\n    title_vector vector(1536),\n    content_vector vector(1536),\n    vector_id INTEGER\n);\n\nALTER TABLE public.articles ADD PRIMARY KEY (id);\n'''\n\n# SQL statement for creating indexes\ncreate_indexes_sql = '''\nCREATE INDEX ON public.articles USING ivfflat (content_vector) WITH (lists = 1000);\n\nCREATE INDEX ON public.articles USING ivfflat (title_vector) WITH (lists = 1000);\n'''\n\n# Execute the SQL statements\ncursor.execute(create_table_sql)\ncursor.execute(create_indexes_sql)\n\n# Commit the changes\nconnection.commit()\n```\n\n## Load the data\n\nLoad the pre-computed vector data into your `articles` table from the `.csv` file. There are 25000 records, so expect the operation to take several minutes.\n\n\n```python\nimport io\n\n# Path to your local CSV file\ncsv_file_path = '../../data/vector_database_wikipedia_articles_embedded.csv'\n\n# Define a generator function to process the csv file\ndef process_file(file_path):\n    with open(file_path, 'r', encoding='utf-8') as file:\n        for line in file:\n            yield line\n\n# Create a StringIO object to store the modified lines\nmodified_lines = io.StringIO(''.join(list(process_file(csv_file_path))))\n\n# Create the COPY command for copy_expert\ncopy_command = '''\nCOPY public.articles (id, url, title, content, title_vector, content_vector, vector_id)\nFROM STDIN WITH (FORMAT CSV, HEADER true, DELIMITER ',');\n'''\n\n# Execute the COPY command using copy_expert\ncursor.copy_expert(copy_command, modified_lines)\n\n# Commit the changes\nconnection.commit()\n```\n\nCheck the number of records to ensure the data has been been loaded. There should be 25000 records.\n\n\n```python\n# Check the size of the data\ncount_sql = \"\"\"select count(*) from public.articles;\"\"\"\ncursor.execute(count_sql)\nresult = cursor.fetchone()\nprint(f\"Count:{result[0]}\")\n```\n\n    Count:25000\n\n\n## Search your data\n\nAfter the data is stored in your Neon database, you can query the data for nearest neighbors. \n\nStart by defining the `query_neon` function, which is executed when you run the vector similarity search. The function creates an embedding based on the user's query, prepares the SQL query, and runs the SQL query with the embedding. The pre-computed embeddings that you loaded into your database were created with `text-embedding-3-small` OpenAI model, so you must use the same model to create an embedding for the similarity search.\n\nA `vector_name` parameter is provided that allows you to search based on \"title\" or \"content\"."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: a1cb9c8f-1842-4113-9b13-0e1f1947cb4d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb", "title": "Vector similarity search using Neon Postgres", "tokens": 2947, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a1cb9c8f-1842-4113-9b13-0e1f1947cb4d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7a1fe4be-5d9b-4f6b-866f-66a9470b5927\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3be94c682f3a332b91a21c63fd7e215ba9994445c22203fccf81f907a42dce64\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6ed64f1f-b25f-4569-93a3-72eecda53e37\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7ce91047831847e6a1c0b763c64e890bb84a11b6de8718e482318effced8535c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"171b5023-56a1-4cb2-9abd-38ea8258cec2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"39662add4fd6dd4ac0f012934d2367131f42a71ae125074d41f1561915995d47\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7165, \"end_char_idx\": 9777, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "doc_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "ref_doc_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "chroma:document": "```python\nimport io\n\n# Path to your local CSV file\ncsv_file_path = '../../data/vector_database_wikipedia_articles_embedded.csv'\n\n# Define a generator function to process the csv file\ndef process_file(file_path):\n    with open(file_path, 'r', encoding='utf-8') as file:\n        for line in file:\n            yield line\n\n# Create a StringIO object to store the modified lines\nmodified_lines = io.StringIO(''.join(list(process_file(csv_file_path))))\n\n# Create the COPY command for copy_expert\ncopy_command = '''\nCOPY public.articles (id, url, title, content, title_vector, content_vector, vector_id)\nFROM STDIN WITH (FORMAT CSV, HEADER true, DELIMITER ',');\n'''\n\n# Execute the COPY command using copy_expert\ncursor.copy_expert(copy_command, modified_lines)\n\n# Commit the changes\nconnection.commit()\n```\n\nCheck the number of records to ensure the data has been been loaded. There should be 25000 records.\n\n\n```python\n# Check the size of the data\ncount_sql = \"\"\"select count(*) from public.articles;\"\"\"\ncursor.execute(count_sql)\nresult = cursor.fetchone()\nprint(f\"Count:{result[0]}\")\n```\n\n    Count:25000\n\n\n## Search your data\n\nAfter the data is stored in your Neon database, you can query the data for nearest neighbors. \n\nStart by defining the `query_neon` function, which is executed when you run the vector similarity search. The function creates an embedding based on the user's query, prepares the SQL query, and runs the SQL query with the embedding. The pre-computed embeddings that you loaded into your database were created with `text-embedding-3-small` OpenAI model, so you must use the same model to create an embedding for the similarity search.\n\nA `vector_name` parameter is provided that allows you to search based on \"title\" or \"content\".\n\n\n```python\ndef query_neon(query, collection_name, vector_name=\"title_vector\", top_k=20):\n\n    # Create an embedding vector from the user query\n    embedded_query = openai.Embedding.create(\n        input=query,\n        model=\"text-embedding-3-small\",\n    )[\"data\"][0][\"embedding\"]\n\n    # Convert the embedded_query to PostgreSQL compatible format\n    embedded_query_pg = \"[\" + \",\".join(map(str, embedded_query)) + \"]\"\n\n    # Create the SQL query\n    query_sql = f\"\"\"\n    SELECT id, url, title, l2_distance({vector_name},'{embedded_query_pg}'::VECTOR(1536)) AS similarity\n    FROM {collection_name}\n    ORDER BY {vector_name} <-> '{embedded_query_pg}'::VECTOR(1536)\n    LIMIT {top_k};\n    \"\"\"\n    # Execute the query\n    cursor.execute(query_sql)\n    results = cursor.fetchall()\n\n    return results\n```\n\nRun a similarity search based on `title_vector` embeddings:"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 171b5023-56a1-4cb2-9abd-38ea8258cec2
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb", "title": "Vector similarity search using Neon Postgres", "tokens": 2947, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"171b5023-56a1-4cb2-9abd-38ea8258cec2\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7a1fe4be-5d9b-4f6b-866f-66a9470b5927\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3be94c682f3a332b91a21c63fd7e215ba9994445c22203fccf81f907a42dce64\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a1cb9c8f-1842-4113-9b13-0e1f1947cb4d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"dd993023170041653128e1317a01757af1bde7a7360f3dcfeb81e4e1bd21feb9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"551691e0-ba9d-4bd1-9fe2-4b14c41d176a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"09563670940806d3127e841b07d877ac4df00d9595880483fa895ac33e09c604\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8068, \"end_char_idx\": 10776, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "doc_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "ref_doc_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "chroma:document": "```python\n# Check the size of the data\ncount_sql = \"\"\"select count(*) from public.articles;\"\"\"\ncursor.execute(count_sql)\nresult = cursor.fetchone()\nprint(f\"Count:{result[0]}\")\n```\n\n    Count:25000\n\n\n## Search your data\n\nAfter the data is stored in your Neon database, you can query the data for nearest neighbors. \n\nStart by defining the `query_neon` function, which is executed when you run the vector similarity search. The function creates an embedding based on the user's query, prepares the SQL query, and runs the SQL query with the embedding. The pre-computed embeddings that you loaded into your database were created with `text-embedding-3-small` OpenAI model, so you must use the same model to create an embedding for the similarity search.\n\nA `vector_name` parameter is provided that allows you to search based on \"title\" or \"content\".\n\n\n```python\ndef query_neon(query, collection_name, vector_name=\"title_vector\", top_k=20):\n\n    # Create an embedding vector from the user query\n    embedded_query = openai.Embedding.create(\n        input=query,\n        model=\"text-embedding-3-small\",\n    )[\"data\"][0][\"embedding\"]\n\n    # Convert the embedded_query to PostgreSQL compatible format\n    embedded_query_pg = \"[\" + \",\".join(map(str, embedded_query)) + \"]\"\n\n    # Create the SQL query\n    query_sql = f\"\"\"\n    SELECT id, url, title, l2_distance({vector_name},'{embedded_query_pg}'::VECTOR(1536)) AS similarity\n    FROM {collection_name}\n    ORDER BY {vector_name} <-> '{embedded_query_pg}'::VECTOR(1536)\n    LIMIT {top_k};\n    \"\"\"\n    # Execute the query\n    cursor.execute(query_sql)\n    results = cursor.fetchall()\n\n    return results\n```\n\nRun a similarity search based on `title_vector` embeddings:\n\n\n```python\n# Query based on `title_vector` embeddings\nimport openai\n\nquery_results = query_neon(\"Greek mythology\", \"Articles\")\nfor i, result in enumerate(query_results):\n    print(f\"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})\")\n```\n\n    1. Greek mythology (Score: 0.998)\n    2. Roman mythology (Score: 0.7)\n    3. Greek underworld (Score: 0.637)\n    4. Mythology (Score: 0.635)\n    5. Classical mythology (Score: 0.629)\n    6. Japanese mythology (Score: 0.615)\n    7. Norse mythology (Score: 0.569)\n    8. Greek language (Score: 0.566)\n    9. Zeus (Score: 0.534)\n    10. List of mythologies (Score: 0.531)\n    11. Jupiter (mythology) (Score: 0.53)\n    12. Greek (Score: 0.53)\n    13. Gaia (mythology) (Score: 0.526)\n    14. Titan (mythology) (Score: 0.522)\n    15. Mercury (mythology) (Score: 0.521)\n    16. Ancient Greece (Score: 0.52)\n    17. Greek alphabet (Score: 0.52)\n    18. Venus (mythology) (Score: 0.515)\n    19. Pluto (mythology) (Score: 0.515)\n    20. Athena (Score: 0.514)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 551691e0-ba9d-4bd1-9fe2-4b14c41d176a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb", "title": "Vector similarity search using Neon Postgres", "tokens": 2947, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"551691e0-ba9d-4bd1-9fe2-4b14c41d176a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7a1fe4be-5d9b-4f6b-866f-66a9470b5927\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3be94c682f3a332b91a21c63fd7e215ba9994445c22203fccf81f907a42dce64\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"171b5023-56a1-4cb2-9abd-38ea8258cec2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"d6c9aa61b2300b92b01f629b9fac67e2a69930870a4cf7f23eff0c89f0b2fb36\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b7d531d4-5ef0-4c03-8412-aa9fb2d768cb\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3ebfe531ac4285d22117a4e8d1249a83ec790bdbe515938312a8f6a9c9d15ac5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9780, \"end_char_idx\": 10840, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "doc_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "ref_doc_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "chroma:document": "```python\n# Query based on `title_vector` embeddings\nimport openai\n\nquery_results = query_neon(\"Greek mythology\", \"Articles\")\nfor i, result in enumerate(query_results):\n    print(f\"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})\")\n```\n\n    1. Greek mythology (Score: 0.998)\n    2. Roman mythology (Score: 0.7)\n    3. Greek underworld (Score: 0.637)\n    4. Mythology (Score: 0.635)\n    5. Classical mythology (Score: 0.629)\n    6. Japanese mythology (Score: 0.615)\n    7. Norse mythology (Score: 0.569)\n    8. Greek language (Score: 0.566)\n    9. Zeus (Score: 0.534)\n    10. List of mythologies (Score: 0.531)\n    11. Jupiter (mythology) (Score: 0.53)\n    12. Greek (Score: 0.53)\n    13. Gaia (mythology) (Score: 0.526)\n    14. Titan (mythology) (Score: 0.522)\n    15. Mercury (mythology) (Score: 0.521)\n    16. Ancient Greece (Score: 0.52)\n    17. Greek alphabet (Score: 0.52)\n    18. Venus (mythology) (Score: 0.515)\n    19. Pluto (mythology) (Score: 0.515)\n    20. Athena (Score: 0.514)\n\n\nRun a similarity search based on `content_vector` embeddings:"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: b7d531d4-5ef0-4c03-8412-aa9fb2d768cb
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb", "title": "Vector similarity search using Neon Postgres", "tokens": 2947, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"b7d531d4-5ef0-4c03-8412-aa9fb2d768cb\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7a1fe4be-5d9b-4f6b-866f-66a9470b5927\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3be94c682f3a332b91a21c63fd7e215ba9994445c22203fccf81f907a42dce64\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"551691e0-ba9d-4bd1-9fe2-4b14c41d176a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/neon-postgres-vector-search-pgvector.ipynb\", \"title\": \"Vector similarity search using Neon Postgres\", \"tokens\": 2947, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"fc882413a7f0609b9737ef81ef6799efe5bb97abbda23aa903adc037dbf08350\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9780, \"end_char_idx\": 11791, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "doc_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "ref_doc_id": "7a1fe4be-5d9b-4f6b-866f-66a9470b5927", "chroma:document": "```python\n# Query based on `title_vector` embeddings\nimport openai\n\nquery_results = query_neon(\"Greek mythology\", \"Articles\")\nfor i, result in enumerate(query_results):\n    print(f\"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})\")\n```\n\n    1. Greek mythology (Score: 0.998)\n    2. Roman mythology (Score: 0.7)\n    3. Greek underworld (Score: 0.637)\n    4. Mythology (Score: 0.635)\n    5. Classical mythology (Score: 0.629)\n    6. Japanese mythology (Score: 0.615)\n    7. Norse mythology (Score: 0.569)\n    8. Greek language (Score: 0.566)\n    9. Zeus (Score: 0.534)\n    10. List of mythologies (Score: 0.531)\n    11. Jupiter (mythology) (Score: 0.53)\n    12. Greek (Score: 0.53)\n    13. Gaia (mythology) (Score: 0.526)\n    14. Titan (mythology) (Score: 0.522)\n    15. Mercury (mythology) (Score: 0.521)\n    16. Ancient Greece (Score: 0.52)\n    17. Greek alphabet (Score: 0.52)\n    18. Venus (mythology) (Score: 0.515)\n    19. Pluto (mythology) (Score: 0.515)\n    20. Athena (Score: 0.514)\n\n\nRun a similarity search based on `content_vector` embeddings:\n\n\n```python\n# Query based on `content_vector` embeddings\nquery_results = query_neon(\"Famous battles in Greek history\", \"Articles\", \"content_vector\")\nfor i, result in enumerate(query_results):\n    print(f\"{i + 1}. {result[2]} (Score: {round(1 - result[3], 3)})\")\n```\n\n    1. 222 BC (Score: 0.489)\n    2. Trojan War (Score: 0.458)\n    3. Peloponnesian War (Score: 0.456)\n    4. History of the Peloponnesian War (Score: 0.449)\n    5. 430 BC (Score: 0.441)\n    6. 168 BC (Score: 0.436)\n    7. Ancient Greece (Score: 0.429)\n    8. Classical Athens (Score: 0.428)\n    9. 499 BC (Score: 0.427)\n    10. Leonidas I (Score: 0.426)\n    11. Battle (Score: 0.421)\n    12. Greek War of Independence (Score: 0.421)\n    13. Menelaus (Score: 0.419)\n    14. Thebes, Greece (Score: 0.417)\n    15. Patroclus (Score: 0.417)\n    16. 427 BC (Score: 0.416)\n    17. 429 BC (Score: 0.413)\n    18. August 2 (Score: 0.412)\n    19. Ionia (Score: 0.411)\n    20. 323 (Score: 0.409)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 957075f3-e3ac-4f8f-81c6-b77f3001db0d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/README.ipynb", "title": "What is Neon?", "tokens": 667, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"957075f3-e3ac-4f8f-81c6-b77f3001db0d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/README.ipynb\", \"title\": \"What is Neon?\", \"tokens\": 667, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"366f0d2c-7a91-4646-a221-0b8de4aeb09a\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/neon/README.ipynb\", \"title\": \"What is Neon?\", \"tokens\": 667, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a08561bdb0259adc50c15e4d4b8468098d850acee49dce8aee6ee3ee94c1be44\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3046, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "366f0d2c-7a91-4646-a221-0b8de4aeb09a", "doc_id": "366f0d2c-7a91-4646-a221-0b8de4aeb09a", "ref_doc_id": "366f0d2c-7a91-4646-a221-0b8de4aeb09a", "chroma:document": "# What is Neon?\n\n[Neon](https://neon.tech/) is Serverless Postgres built for the cloud. Neon separates compute and storage to offer modern developer features such as autoscaling, database branching, scale-to-zero, and more.\n\n## Vector search\n\nNeon supports vector search using the [pgvector](https://neon.tech/docs/extensions/pgvector) open-source PostgreSQL extension, which enables Postgres as a vector database for storing and querying embeddings.\n\n## OpenAI cookbook notebook\n\nCheck out the notebook in this repo for working with Neon Serverless Postgres as your vector database.\n\n### Semantic search using Neon Postgres with pgvector and OpenAI\n\nIn this notebook you will learn how to:\n\n1. Use embeddings created by OpenAI API\n2. Store embeddings in a Neon Serverless Postgres database\n3. Convert a raw text query to an embedding with OpenAI API\n4. Use Neon with the `pgvector` extension to perform vector similarity search\n\n## Scaling Support\n\nNeon enables you to scale your AI applications with the following features:\n\n- [Autoscaling](https://neon.tech/docs/introduction/read-replicas): If your AI application experiences heavy load during certain hours of the day or at different times, Neon can automatically scale compute resources without manual intervention. During periods of inactivity, Neon is able to scale to zero.\n- [Instant read replicas](https://neon.tech/docs/introduction/read-replicas): Neon supports instant read replicas, which are independent read-only compute instances designed to perform read operations on the same data as your read-write computes. With read replicas, you can offload reads from your read-write compute instance to a dedicated read-only compute instance for your AI application.\n- [The Neon serverless driver](https://neon.tech/docs/serverless/serverless-driver): Neon supports a low-latency serverless PostgreSQL driver for JavaScript and TypeScript applications that allows you to query data from serverless and edge environments, making it possible to achieve sub-10ms queries.\n\n## More Examples\n\n- [Build an AI-powered semantic search application](https://github.com/neondatabase/yc-idea-matcher) - Submit a startup idea and get a list of similar ideas that YCombinator has invested in before\n- [Build an AI-powered chatbot](https://github.com/neondatabase/ask-neon) - A Postgres Q&A chatbot that uses Postgres as a vector database\n- [Vercel Postgres pgvector Starter](https://vercel.com/templates/next.js/postgres-pgvector) - Vector similarity search with Vercel Postgres (powered by Neon)\n\n## Additional Resources\n\n- [Building AI applications with Neon](https://neon.tech/ai)\n- [Neon AI & embeddings documentation](https://neon.tech/docs/ai/ai-intro)\n- [Building an AI-powered Chatbot using Vercel, OpenAI, and Postgres](neon.tech/blog/building-an-ai-powered-chatbot-using-vercel-openai-and-postgres)\n- [Web-based AI SQL Playground and connecting to Postgres from the browser](https://neon.tech/blog/postgres-ai-playground)\n- [pgvector GitHub repository](https://github.com/pgvector/pgvector)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 7aff98fb-51aa-4b60-875a-faebba42a2c6
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Qdrant and OpenAI", "tokens": 3804, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"7aff98fb-51aa-4b60-875a-faebba42a2c6\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7bf1b729-d3e9-4629-a1ec-e84a15c9756c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6ffba8f46e266148633227faf1ec78c9b968b615a7c3cfab385f4740105c85ff\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"43e8d6fd-3054-4505-a002-faed45c9678e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6e4f80d4b416bbc3d69a9c1cca689e805b5c73b5f62bb6fe4b88bab3fe449992\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2968, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "doc_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "ref_doc_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "chroma:document": "# Question Answering with Langchain, Qdrant and OpenAI\n\nThis notebook presents how to implement a Question Answering system with Langchain, Qdrant as a knowledge based and OpenAI embeddings. If you are not familiar with Qdrant, it's better to check out the [Getting_started_with_Qdrant_and_OpenAI.ipynb](Getting_started_with_Qdrant_and_OpenAI.ipynb) notebook.\n\nThis notebook presents an end-to-end process of:\n1. Calculating the embeddings with OpenAI API.\n2. Storing the embeddings in a local instance of Qdrant to build a knowledge base.\n3. Converting raw text query to an embedding with OpenAI API.\n4. Using Qdrant to perform the nearest neighbour search in the created collection to find some context.\n5. Asking LLM to find the answer in a given context.\n\nAll the steps will be simplified to calling some corresponding Langchain methods.\n\n## Prerequisites\n\nFor the purposes of this exercise we need to prepare a couple of things:\n\n1. Qdrant server instance. In our case a local Docker container.\n2. The [qdrant-client](https://github.com/qdrant/qdrant_client) library to interact with the vector database.\n3. [Langchain](https://github.com/hwchase17/langchain) as a framework.\n3. An [OpenAI API key](https://beta.openai.com/account/api-keys).\n\n### Start Qdrant server\n\nWe're going to use a local Qdrant instance running in a Docker container. The easiest way to launch it is to use the attached [docker-compose.yaml] file and run the following command:\n\n\n```python\n! docker-compose up -d\n```\n\n    Starting qdrant_qdrant_1 ... \n    \u001b[1Bting qdrant_qdrant_1 ... \u001b[32mdone\u001b[0m\n\nWe might validate if the server was launched successfully by running a simple curl command:\n\n\n```python\n! curl http://localhost:6333\n```\n\n    {\"title\":\"qdrant - vector search engine\",\"version\":\"1.0.1\"}\n\n### Install requirements\n\nThis notebook obviously requires the `openai`, `langchain` and `qdrant-client` packages.\n\n\n\n```python\n! pip install openai qdrant-client \"langchain==0.0.100\" wget\n```\n\n### Prepare your OpenAI API key\n\nThe OpenAI API key is used for vectorization of the documents and queries.\n\nIf you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n\nOnce you get your key, please add it to your environment variables as `OPENAI_API_KEY` by running following command:\n\n\n```python\n! export OPENAI_API_KEY=\"your API key\"\n```\n\n\n```python\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\nimport os\n\n# Note. alternatively you can set a temporary env variable like this:\n# os.environ[\"OPENAI_API_KEY\"] = \"sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\"\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    print(\"OPENAI_API_KEY is ready\")\nelse:\n    print(\"OPENAI_API_KEY environment variable not found\")\n```\n\n    OPENAI_API_KEY is ready"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 43e8d6fd-3054-4505-a002-faed45c9678e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Qdrant and OpenAI", "tokens": 3804, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"43e8d6fd-3054-4505-a002-faed45c9678e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7bf1b729-d3e9-4629-a1ec-e84a15c9756c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6ffba8f46e266148633227faf1ec78c9b968b615a7c3cfab385f4740105c85ff\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7aff98fb-51aa-4b60-875a-faebba42a2c6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"20f1f40d4b0bbf33296a59bfef04729739ad6d843d59e86e32766ed63f8136ea\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d2e0236a-e310-4735-8ac5-501e89960a19\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"30e6c5d295957291869f0f904fe996b870949dc58fb92525dc7f224752626f42\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1459, \"end_char_idx\": 4474, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "doc_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "ref_doc_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "chroma:document": "```python\n! docker-compose up -d\n```\n\n    Starting qdrant_qdrant_1 ... \n    \u001b[1Bting qdrant_qdrant_1 ... \u001b[32mdone\u001b[0m\n\nWe might validate if the server was launched successfully by running a simple curl command:\n\n\n```python\n! curl http://localhost:6333\n```\n\n    {\"title\":\"qdrant - vector search engine\",\"version\":\"1.0.1\"}\n\n### Install requirements\n\nThis notebook obviously requires the `openai`, `langchain` and `qdrant-client` packages.\n\n\n\n```python\n! pip install openai qdrant-client \"langchain==0.0.100\" wget\n```\n\n### Prepare your OpenAI API key\n\nThe OpenAI API key is used for vectorization of the documents and queries.\n\nIf you don't have an OpenAI API key, you can get one from [https://beta.openai.com/account/api-keys](https://beta.openai.com/account/api-keys).\n\nOnce you get your key, please add it to your environment variables as `OPENAI_API_KEY` by running following command:\n\n\n```python\n! export OPENAI_API_KEY=\"your API key\"\n```\n\n\n```python\n# Test that your OpenAI API key is correctly set as an environment variable\n# Note. if you run this notebook locally, you will need to reload your terminal and the notebook for the env variables to be live.\nimport os\n\n# Note. alternatively you can set a temporary env variable like this:\n# os.environ[\"OPENAI_API_KEY\"] = \"sk-xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx\"\n\nif os.getenv(\"OPENAI_API_KEY\") is not None:\n    print(\"OPENAI_API_KEY is ready\")\nelse:\n    print(\"OPENAI_API_KEY environment variable not found\")\n```\n\n    OPENAI_API_KEY is ready\n\n\n## Load data\n\nIn this section we are going to load the data containing some natural questions and answers to them. All the data will be used to create a Langchain application with Qdrant being the knowledge base.\n\n\n```python\nimport wget\n\n# All the examples come from https://ai.google.com/research/NaturalQuestions\n# This is a sample of the training set that we download and extract for some\n# further processing.\nwget.download(\"https://storage.googleapis.com/dataset-natural-questions/questions.json\")\nwget.download(\"https://storage.googleapis.com/dataset-natural-questions/answers.json\")\n```\n\n    100% [..............................................................................] 95372 / 95372\n\n\n\n\n    'answers.json'\n\n\n\n\n```python\nimport json\n\nwith open(\"questions.json\", \"r\") as fp:\n    questions = json.load(fp)\n\nwith open(\"answers.json\", \"r\") as fp:\n    answers = json.load(fp)\n```\n\n\n```python\nprint(questions[0])\n```\n\n    when is the last episode of season 8 of the walking dead\n\n\n\n```python\nprint(answers[0])\n```\n\n    No . overall No. in season Title Directed by Written by Original air date U.S. viewers ( millions ) 100 `` Mercy '' Greg Nicotero Scott M. Gimple October 22 , 2017 ( 2017 - 10 - 22 ) 11.44 Rick , Maggie , and Ezekiel rally their communities together to take down Negan . Gregory attempts to have the Hilltop residents side with Negan , but they all firmly stand behind Maggie . The group attacks the Sanctuary , taking down its fences and flooding the compound with walkers ."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: d2e0236a-e310-4735-8ac5-501e89960a19
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Qdrant and OpenAI", "tokens": 3804, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"d2e0236a-e310-4735-8ac5-501e89960a19\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7bf1b729-d3e9-4629-a1ec-e84a15c9756c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6ffba8f46e266148633227faf1ec78c9b968b615a7c3cfab385f4740105c85ff\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"43e8d6fd-3054-4505-a002-faed45c9678e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"b15361f221b5a19192a6dcef3ec8890b1724213e38300656cf563517ce9c503f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3a06aa29-87f0-43bc-845a-4c2a1a936567\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b35332f55e097d4f1aa8a07acc11ca5866970b94d13a31b95b2a973ffbe3cc73\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2971, \"end_char_idx\": 6111, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "doc_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "ref_doc_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "chroma:document": "## Load data\n\nIn this section we are going to load the data containing some natural questions and answers to them. All the data will be used to create a Langchain application with Qdrant being the knowledge base.\n\n\n```python\nimport wget\n\n# All the examples come from https://ai.google.com/research/NaturalQuestions\n# This is a sample of the training set that we download and extract for some\n# further processing.\nwget.download(\"https://storage.googleapis.com/dataset-natural-questions/questions.json\")\nwget.download(\"https://storage.googleapis.com/dataset-natural-questions/answers.json\")\n```\n\n    100% [..............................................................................] 95372 / 95372\n\n\n\n\n    'answers.json'\n\n\n\n\n```python\nimport json\n\nwith open(\"questions.json\", \"r\") as fp:\n    questions = json.load(fp)\n\nwith open(\"answers.json\", \"r\") as fp:\n    answers = json.load(fp)\n```\n\n\n```python\nprint(questions[0])\n```\n\n    when is the last episode of season 8 of the walking dead\n\n\n\n```python\nprint(answers[0])\n```\n\n    No . overall No. in season Title Directed by Written by Original air date U.S. viewers ( millions ) 100 `` Mercy '' Greg Nicotero Scott M. Gimple October 22 , 2017 ( 2017 - 10 - 22 ) 11.44 Rick , Maggie , and Ezekiel rally their communities together to take down Negan . Gregory attempts to have the Hilltop residents side with Negan , but they all firmly stand behind Maggie . The group attacks the Sanctuary , taking down its fences and flooding the compound with walkers . With the Sanctuary defaced , everyone leaves except Gabriel , who reluctantly stays to save Gregory , but is left behind when Gregory abandons him . Surrounded by walkers , Gabriel hides in a trailer , where he is trapped inside with Negan . 101 `` The Damned '' Rosemary Rodriguez Matthew Negrete & Channing Powell October 29 , 2017 ( 2017 - 10 - 29 ) 8.92 Rick 's forces split into separate parties to attack several of the Saviors ' outposts , during which many members of the group are killed ; Eric is critically injured and rushed away by Aaron . Jesus stops Tara and Morgan from executing a group of surrendered Saviors . While clearing an outpost with Daryl , Rick is confronted and held at gunpoint by Morales , a survivor he met in the initial Atlanta camp , who is now with the Saviors . 102 `` Monsters '' Greg Nicotero Matthew Negrete & Channing Powell November 5 , 2017 ( 2017 - 11 - 05 ) 8.52 Daryl finds Morales threatening Rick and kills him ; the duo then pursue a group of Saviors who are transporting weapons to another outpost . Gregory returns to Hilltop , and after a heated argument , Maggie ultimately allows him back in the community . Eric dies from his injuries , leaving Aaron distraught . Despite Tara and Morgan 's objections , Jesus leads the group of surrendered Saviors to Hilltop . Ezekiel 's group attacks another Savior compound , during which several Kingdommers are shot while protecting Ezekiel . 103 `` Some Guy '' Dan Liu David Leslie Johnson November 12 , 2017 ( 2017 - 11 - 12 ) 8.69 Ezekiel 's group is overwhelmed by the Saviors , who kill all of them except for Ezekiel himself and Jerry ."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 3a06aa29-87f0-43bc-845a-4c2a1a936567
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Qdrant and OpenAI", "tokens": 3804, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"3a06aa29-87f0-43bc-845a-4c2a1a936567\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7bf1b729-d3e9-4629-a1ec-e84a15c9756c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6ffba8f46e266148633227faf1ec78c9b968b615a7c3cfab385f4740105c85ff\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d2e0236a-e310-4735-8ac5-501e89960a19\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"7f9a79f427935cfc7b47fef72763843db0454fe0e16eff471cf7cd4fea8db90b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"213f7a71-ad4d-41ba-ac76-7458588afd0f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2fd05f792f59a9591da45a93998efb4f4b6fa61d55f24622e819493ad35fb3e8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4475, \"end_char_idx\": 7638, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "doc_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "ref_doc_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "chroma:document": "With the Sanctuary defaced , everyone leaves except Gabriel , who reluctantly stays to save Gregory , but is left behind when Gregory abandons him . Surrounded by walkers , Gabriel hides in a trailer , where he is trapped inside with Negan . 101 `` The Damned '' Rosemary Rodriguez Matthew Negrete & Channing Powell October 29 , 2017 ( 2017 - 10 - 29 ) 8.92 Rick 's forces split into separate parties to attack several of the Saviors ' outposts , during which many members of the group are killed ; Eric is critically injured and rushed away by Aaron . Jesus stops Tara and Morgan from executing a group of surrendered Saviors . While clearing an outpost with Daryl , Rick is confronted and held at gunpoint by Morales , a survivor he met in the initial Atlanta camp , who is now with the Saviors . 102 `` Monsters '' Greg Nicotero Matthew Negrete & Channing Powell November 5 , 2017 ( 2017 - 11 - 05 ) 8.52 Daryl finds Morales threatening Rick and kills him ; the duo then pursue a group of Saviors who are transporting weapons to another outpost . Gregory returns to Hilltop , and after a heated argument , Maggie ultimately allows him back in the community . Eric dies from his injuries , leaving Aaron distraught . Despite Tara and Morgan 's objections , Jesus leads the group of surrendered Saviors to Hilltop . Ezekiel 's group attacks another Savior compound , during which several Kingdommers are shot while protecting Ezekiel . 103 `` Some Guy '' Dan Liu David Leslie Johnson November 12 , 2017 ( 2017 - 11 - 12 ) 8.69 Ezekiel 's group is overwhelmed by the Saviors , who kill all of them except for Ezekiel himself and Jerry . Carol clears the inside of the compound , killing all but two Saviors , who almost escape but are eventually caught by Rick and Daryl . En route to the Kingdom , Ezekiel , Jerry , and Carol are surrounded by walkers , but Shiva sacrifices herself to save them . The trio returns to the Kingdom , where Ezekiel 's confidence in himself as a leader has diminished . 104 5 `` The Big Scary U '' Michael E. Satrazemis Story by : Scott M. Gimple & David Leslie Johnson & Angela Kang Teleplay by : David Leslie Johnson & Angela Kang November 19 , 2017 ( 2017 - 11 - 19 ) 7.85 After confessing their sins to each other , Gabriel and Negan manage to escape from the trailer . Simon and the other lieutenants grow suspicious of each other , knowing that Rick 's forces must have inside information . The workers in the Sanctuary become increasingly frustrated with their living conditions , and a riot nearly ensues , until Negan returns and restores order . Gabriel is locked in a cell , where Eugene discovers him sick and suffering . Meanwhile , Rick and Daryl argue over how to take out the Saviors , leading Daryl to abandon Rick . 105 6 `` The King , the Widow , and Rick '' John Polson Angela Kang & Corey Reed November 26 , 2017 ( 2017 - 11 - 26 ) 8.28 Rick visits Jadis in hopes of convincing her to turn against Negan ; Jadis refuses , and locks Rick in a shipping container . Carl encounters Siddiq in the woods and recruits him to Alexandria . Daryl and Tara plot to deviate from Rick 's plans by destroying the Sanctuary ."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 213f7a71-ad4d-41ba-ac76-7458588afd0f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Qdrant and OpenAI", "tokens": 3804, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"213f7a71-ad4d-41ba-ac76-7458588afd0f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7bf1b729-d3e9-4629-a1ec-e84a15c9756c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6ffba8f46e266148633227faf1ec78c9b968b615a7c3cfab385f4740105c85ff\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3a06aa29-87f0-43bc-845a-4c2a1a936567\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"418c83666e85d5b7534c7a92a499d65eb989dfe3ea8d9e240590807d6e8f8cb1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8512b94e-b5ba-450a-a939-0e8766163933\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d47d6bb939b32c2a91b07eefa9126c0fb0f63470ba2b8e0d1fd4b5e2c7dca1cc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6112, \"end_char_idx\": 9294, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "doc_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "ref_doc_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "chroma:document": "Carol clears the inside of the compound , killing all but two Saviors , who almost escape but are eventually caught by Rick and Daryl . En route to the Kingdom , Ezekiel , Jerry , and Carol are surrounded by walkers , but Shiva sacrifices herself to save them . The trio returns to the Kingdom , where Ezekiel 's confidence in himself as a leader has diminished . 104 5 `` The Big Scary U '' Michael E. Satrazemis Story by : Scott M. Gimple & David Leslie Johnson & Angela Kang Teleplay by : David Leslie Johnson & Angela Kang November 19 , 2017 ( 2017 - 11 - 19 ) 7.85 After confessing their sins to each other , Gabriel and Negan manage to escape from the trailer . Simon and the other lieutenants grow suspicious of each other , knowing that Rick 's forces must have inside information . The workers in the Sanctuary become increasingly frustrated with their living conditions , and a riot nearly ensues , until Negan returns and restores order . Gabriel is locked in a cell , where Eugene discovers him sick and suffering . Meanwhile , Rick and Daryl argue over how to take out the Saviors , leading Daryl to abandon Rick . 105 6 `` The King , the Widow , and Rick '' John Polson Angela Kang & Corey Reed November 26 , 2017 ( 2017 - 11 - 26 ) 8.28 Rick visits Jadis in hopes of convincing her to turn against Negan ; Jadis refuses , and locks Rick in a shipping container . Carl encounters Siddiq in the woods and recruits him to Alexandria . Daryl and Tara plot to deviate from Rick 's plans by destroying the Sanctuary . Ezekiel isolates himself at the Kingdom , where Carol tries to encourage him to be the leader his people need . Maggie has the group of captured Saviors placed in a holding area and forces Gregory to join them as punishment for betraying Hilltop . 106 7 `` Time for After '' Larry Teng Matthew Negrete & Corey Reed December 3 , 2017 ( 2017 - 12 - 03 ) 7.47 After learning of Dwight 's association with Rick 's group , Eugene affirms his loyalty to Negan and outlines a plan to get rid of the walkers surrounding the Sanctuary . With help from Morgan and Tara , Daryl drives a truck through the Sanctuary 's walls , flooding its interior with walkers , killing many Saviors . Rick finally convinces Jadis and the Scavengers to align with him , and they plan to force the Saviors to surrender . However , when they arrive at the Sanctuary , Rick is horrified to see the breached walls and no sign of the walker herd . 107 8 `` How It 's Gotta Be '' Michael E. Satrazemis David Leslie Johnson & Angela Kang December 10 , 2017 ( 2017 - 12 - 10 ) 7.89 Eugene 's plan allows the Saviors to escape , and separately , the Saviors waylay the Alexandria , Hilltop , and Kingdom forces . The Scavengers abandon Rick , after which he returns to Alexandria . Ezekiel ensures that the Kingdom residents are able to escape before locking himself in the community with the Saviors . Eugene aids Gabriel and Doctor Carson in escaping the Sanctuary in order to ease his conscience . Negan attacks Alexandria , but Carl devises a plan to allow the Alexandria residents to escape into the sewers . Carl reveals he was bitten by a walker while escorting Siddiq to Alexandria ."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 8512b94e-b5ba-450a-a939-0e8766163933
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Qdrant and OpenAI", "tokens": 3804, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"8512b94e-b5ba-450a-a939-0e8766163933\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7bf1b729-d3e9-4629-a1ec-e84a15c9756c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6ffba8f46e266148633227faf1ec78c9b968b615a7c3cfab385f4740105c85ff\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"213f7a71-ad4d-41ba-ac76-7458588afd0f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"91e6877ecca7b8b40915a06e82cbb78583b3cba851e9db44b5d681e3f6c6b9ff\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4372c82e-b504-4f71-aa41-43a98d090025\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4b16789cc35b73c72ee4ef2fefa7b4ee55bb163511a815198e8d154f152909bf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7639, \"end_char_idx\": 10642, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "doc_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "ref_doc_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "chroma:document": "Ezekiel isolates himself at the Kingdom , where Carol tries to encourage him to be the leader his people need . Maggie has the group of captured Saviors placed in a holding area and forces Gregory to join them as punishment for betraying Hilltop . 106 7 `` Time for After '' Larry Teng Matthew Negrete & Corey Reed December 3 , 2017 ( 2017 - 12 - 03 ) 7.47 After learning of Dwight 's association with Rick 's group , Eugene affirms his loyalty to Negan and outlines a plan to get rid of the walkers surrounding the Sanctuary . With help from Morgan and Tara , Daryl drives a truck through the Sanctuary 's walls , flooding its interior with walkers , killing many Saviors . Rick finally convinces Jadis and the Scavengers to align with him , and they plan to force the Saviors to surrender . However , when they arrive at the Sanctuary , Rick is horrified to see the breached walls and no sign of the walker herd . 107 8 `` How It 's Gotta Be '' Michael E. Satrazemis David Leslie Johnson & Angela Kang December 10 , 2017 ( 2017 - 12 - 10 ) 7.89 Eugene 's plan allows the Saviors to escape , and separately , the Saviors waylay the Alexandria , Hilltop , and Kingdom forces . The Scavengers abandon Rick , after which he returns to Alexandria . Ezekiel ensures that the Kingdom residents are able to escape before locking himself in the community with the Saviors . Eugene aids Gabriel and Doctor Carson in escaping the Sanctuary in order to ease his conscience . Negan attacks Alexandria , but Carl devises a plan to allow the Alexandria residents to escape into the sewers . Carl reveals he was bitten by a walker while escorting Siddiq to Alexandria . 108 9 `` Honor '' Greg Nicotero Matthew Negrete & Channing Powell February 25 , 2018 ( 2018 - 02 - 25 ) 8.28 After the Saviors leave Alexandria , the survivors make for the Hilltop while Rick and Michonne stay behind to say their final goodbyes to a dying Carl , who pleads with Rick to build a better future alongside the Saviors before killing himself . In the Kingdom , Morgan and Carol launch a rescue mission for Ezekiel . Although they are successful and retake the Kingdom , the Saviors ' lieutenant Gavin is killed by Benjamin 's vengeful brother Henry . 109 10 `` The Lost and the Plunderers '' TBA TBA March 4 , 2018 ( 2018 - 03 - 04 ) TBD 110 11 `` Dead or Alive Or '' TBA TBA March 11 , 2018 ( 2018 - 03 - 11 ) TBD 111 12 `` The Key '' TBA TBA March 18 , 2018 ( 2018 - 03 - 18 ) TBD\n\n\n## Chain definition\n\nLangchain is already integrated with Qdrant and performs all the indexing for given list of documents. In our case we are going to store the set of answers we have.\n\n\n```python\nfrom langchain.vectorstores import Qdrant\nfrom langchain.embeddings import OpenAIEmbeddings\nfrom langchain import VectorDBQA, OpenAI\n\nembeddings = OpenAIEmbeddings()\ndoc_store = Qdrant.from_texts(\n    answers, embeddings, host=\"localhost\" \n)\n```\n\nAt this stage all the possible answers are already stored in Qdrant, so we can define the whole QA chain."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 4372c82e-b504-4f71-aa41-43a98d090025
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Qdrant and OpenAI", "tokens": 3804, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"4372c82e-b504-4f71-aa41-43a98d090025\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7bf1b729-d3e9-4629-a1ec-e84a15c9756c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6ffba8f46e266148633227faf1ec78c9b968b615a7c3cfab385f4740105c85ff\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8512b94e-b5ba-450a-a939-0e8766163933\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8382126d532011cea12c7c0414568c78b79aa0b76e94117680b08f8c5f709e7e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c84401ca-1d2b-449b-b7d8-2f3d441dcb4c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ced1a2c2ed984140d13c0a99963e6a245bd052ba7c456422800649e7529cd17e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9217, \"end_char_idx\": 11400, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "doc_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "ref_doc_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "chroma:document": "Carl reveals he was bitten by a walker while escorting Siddiq to Alexandria . 108 9 `` Honor '' Greg Nicotero Matthew Negrete & Channing Powell February 25 , 2018 ( 2018 - 02 - 25 ) 8.28 After the Saviors leave Alexandria , the survivors make for the Hilltop while Rick and Michonne stay behind to say their final goodbyes to a dying Carl , who pleads with Rick to build a better future alongside the Saviors before killing himself . In the Kingdom , Morgan and Carol launch a rescue mission for Ezekiel . Although they are successful and retake the Kingdom , the Saviors ' lieutenant Gavin is killed by Benjamin 's vengeful brother Henry . 109 10 `` The Lost and the Plunderers '' TBA TBA March 4 , 2018 ( 2018 - 03 - 04 ) TBD 110 11 `` Dead or Alive Or '' TBA TBA March 11 , 2018 ( 2018 - 03 - 11 ) TBD 111 12 `` The Key '' TBA TBA March 18 , 2018 ( 2018 - 03 - 18 ) TBD\n\n\n## Chain definition\n\nLangchain is already integrated with Qdrant and performs all the indexing for given list of documents. In our case we are going to store the set of answers we have.\n\n\n```python\nfrom langchain.vectorstores import Qdrant\nfrom langchain.embeddings import OpenAIEmbeddings\nfrom langchain import VectorDBQA, OpenAI\n\nembeddings = OpenAIEmbeddings()\ndoc_store = Qdrant.from_texts(\n    answers, embeddings, host=\"localhost\" \n)\n```\n\nAt this stage all the possible answers are already stored in Qdrant, so we can define the whole QA chain.\n\n\n```python\nllm = OpenAI()\nqa = VectorDBQA.from_chain_type(\n    llm=llm, \n    chain_type=\"stuff\", \n    vectorstore=doc_store,\n    return_source_documents=False,\n)\n```\n\n## Search data\n\nOnce the data is put into Qdrant we can start asking some questions. A question will be automatically vectorized by OpenAI model, and the created vector will be used to find some possibly matching answers in Qdrant. Once retrieved, the most similar answers will be incorporated into the prompt sent to OpenAI Large Language Model. The communication between all the services is shown on a graph:\n\n![](https://qdrant.tech/articles_data/langchain-integration/flow-diagram.png)\n\n\n\n```python\nimport random\n\nrandom.seed(52)\nselected_questions = random.choices(questions, k=5)\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: c84401ca-1d2b-449b-b7d8-2f3d441dcb4c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Qdrant and OpenAI", "tokens": 3804, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c84401ca-1d2b-449b-b7d8-2f3d441dcb4c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7bf1b729-d3e9-4629-a1ec-e84a15c9756c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6ffba8f46e266148633227faf1ec78c9b968b615a7c3cfab385f4740105c85ff\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4372c82e-b504-4f71-aa41-43a98d090025\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"782a8abe0933a3a913ceee02209172a5c655409efec187db6281c731ad9f6d46\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3f3b51c5-c86e-44b8-8fe7-ec0490d7c609\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"90b1c9b433ebdd985f3bd3cf588d8179ba401b34dcf06dd9a1a22a6dc44b4c64\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10092, \"end_char_idx\": 13313, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "doc_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "ref_doc_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "chroma:document": "## Chain definition\n\nLangchain is already integrated with Qdrant and performs all the indexing for given list of documents. In our case we are going to store the set of answers we have.\n\n\n```python\nfrom langchain.vectorstores import Qdrant\nfrom langchain.embeddings import OpenAIEmbeddings\nfrom langchain import VectorDBQA, OpenAI\n\nembeddings = OpenAIEmbeddings()\ndoc_store = Qdrant.from_texts(\n    answers, embeddings, host=\"localhost\" \n)\n```\n\nAt this stage all the possible answers are already stored in Qdrant, so we can define the whole QA chain.\n\n\n```python\nllm = OpenAI()\nqa = VectorDBQA.from_chain_type(\n    llm=llm, \n    chain_type=\"stuff\", \n    vectorstore=doc_store,\n    return_source_documents=False,\n)\n```\n\n## Search data\n\nOnce the data is put into Qdrant we can start asking some questions. A question will be automatically vectorized by OpenAI model, and the created vector will be used to find some possibly matching answers in Qdrant. Once retrieved, the most similar answers will be incorporated into the prompt sent to OpenAI Large Language Model. The communication between all the services is shown on a graph:\n\n![](https://qdrant.tech/articles_data/langchain-integration/flow-diagram.png)\n\n\n\n```python\nimport random\n\nrandom.seed(52)\nselected_questions = random.choices(questions, k=5)\n```\n\n\n```python\nfor question in selected_questions:\n    print(\">\", question)\n    print(qa.run(question), end=\"\\n\\n\")\n```\n\n    > where do frankenstein and the monster first meet\n     Victor and the Creature first meet in the mountains.\n    \n    > who are the actors in fast and furious\n     The actors in the Fast and Furious films are Vin Diesel, Paul Walker, Michelle Rodriguez, Jordana Brewster, Tyrese Gibson, Ludacris, Lucas Black, Sung Kang, Gal Gadot, Dwayne Johnson, Matt Schulze, Chad Lindberg, Johnny Strong, Eva Mendes, Devon Aoki, Nathalie Kelley, Bow Wow, Tego Calder\u00f3n, Don Omar, Elsa Pataky, Kurt Russell, Nathalie Emmanuel, Scott Eastwood, Noel Gugliemi, Ja Rule, Thom Barry, Ted Levine, Minka Kelly, James Remar, Amaury Nolasco, Michael Ealy, MC Jin, Brian Goodman, Lynda Boyd, Jason Tobin, Neela, Liza Lapira, Alimi Ballard, Yorgo Constantine, Geoff Meed, Jeimy Osorio, Max William Crane, Charlie & Miller Kimsey, Eden Estrella, Romeo Santos, John Brotherton, Helen Mirren, Celestino Cornielle, Janmarco Santiago, Carlos De La Hoz, James Ayoub, Rick Yune, Cole Hauser, Brian Tee, John Ortiz, Luke Evans, Jason Statham, Charlize Theron, Reggie Lee, Mo Gallini, Roberto Sanchez, Leonardo\n    \n    > properties of red black tree in data structure\n     Red black trees are a type of binary tree with a special set of properties. Each node is either red or black, the root is black, and if a node is red, then both its children are black. Every path from a given node to any of its descendant NIL nodes contains the same number of black nodes. The number of black nodes from the root to a node is the node's black depth, and the uniform number of black nodes in all paths from root to the leaves is called the black-height of the red-black tree.\n    \n    > who designed the national coat of arms of south africa\n     Iaan Bekker\n    \n    > caravaggio's death of the virgin pamela askew\n     I don't know."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 3f3b51c5-c86e-44b8-8fe7-ec0490d7c609
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb", "title": "Question Answering with Langchain, Qdrant and OpenAI", "tokens": 3804, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"3f3b51c5-c86e-44b8-8fe7-ec0490d7c609\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7bf1b729-d3e9-4629-a1ec-e84a15c9756c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6ffba8f46e266148633227faf1ec78c9b968b615a7c3cfab385f4740105c85ff\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c84401ca-1d2b-449b-b7d8-2f3d441dcb4c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/QA_with_Langchain_Qdrant_and_OpenAI.ipynb\", \"title\": \"Question Answering with Langchain, Qdrant and OpenAI\", \"tokens\": 3804, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"690ab512f3ba0bcfe9148b1bb028dbe6c25756cb0a75e524716d5659addd03eb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13321, \"end_char_idx\": 15908, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "doc_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "ref_doc_id": "7bf1b729-d3e9-4629-a1ec-e84a15c9756c", "chroma:document": "### Custom prompt templates\n\nThe `stuff` chain type in Langchain uses a specific prompt with question and context documents incorporated. This is what the default prompt looks like:\n\n```text\nUse the following pieces of context to answer the question at the end. If you don't know the answer, just say that you don't know, don't try to make up an answer.\n{context}\nQuestion: {question}\nHelpful Answer:\n```\n\nWe can, however, provide our prompt template and change the behaviour of the OpenAI LLM, while still using the `stuff` chain type. It is important to keep `{context}` and `{question}` as placeholders.\n\n#### Experimenting with custom prompts\n\nWe can try using a different prompt template, so the model:\n1. Responds with a single-sentence answer if it knows it.\n2. Suggests a random song title if it doesn't know the answer to our question.\n\n\n```python\nfrom langchain.prompts import PromptTemplate\n```\n\n\n```python\ncustom_prompt = \"\"\"\nUse the following pieces of context to answer the question at the end. Please provide\na short single-sentence summary answer only. If you don't know the answer or if it's \nnot present in given context, don't try to make up an answer, but suggest me a random \nunrelated song title I could listen to. \nContext: {context}\nQuestion: {question}\nHelpful Answer:\n\"\"\"\n```\n\n\n```python\ncustom_prompt_template = PromptTemplate(\n    template=custom_prompt, input_variables=[\"context\", \"question\"]\n)\n```\n\n\n```python\ncustom_qa = VectorDBQA.from_chain_type(\n    llm=llm, \n    chain_type=\"stuff\", \n    vectorstore=doc_store,\n    return_source_documents=False,\n    chain_type_kwargs={\"prompt\": custom_prompt_template},\n)\n```\n\n\n```python\nrandom.seed(41)\nfor question in random.choices(questions, k=5):\n    print(\">\", question)\n    print(custom_qa.run(question), end=\"\\n\\n\")\n```\n\n    > what was uncle jesse's original last name on full house\n    Uncle Jesse's original last name on Full House was Cochran.\n    \n    > when did the volcano erupt in indonesia 2018\n    No volcanic eruption is mentioned in the given context. Suggested Song: \"Ring of Fire\" by Johnny Cash.\n    \n    > what does a dualist way of thinking mean\n    Dualist way of thinking means that the mind and body are separate entities, with the mind being a non-physical substance.\n    \n    > the first civil service commission in india was set up on the basis of recommendation of\n    The first Civil Service Commission in India was not set up on the basis of a recommendation.\n    \n    > how old do you have to be to get a tattoo in utah\n    In Utah, you must be at least 18 years old to get a tattoo."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: dfff671a-7d8c-4a53-b65e-06871062d79c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb", "title": "Using Qdrant for Embeddings Search", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"dfff671a-7d8c-4a53-b65e-06871062d79c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4c7aafd6-3704-425b-af10-41f8c20b95bb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"368a0b23a141651962d829ce5d9cde0d2bec897f204a469e2bbbcd633b8ef4bd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"42f99026-9f0c-41b3-ae93-2ef20c519fad\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"406b371b81037794911c525539b6cbb09fdd41e3aa5e617d72e210e2012f59bc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3579, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "doc_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "ref_doc_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "chroma:document": "# Using Qdrant for Embeddings Search\n\nThis notebook takes you through a simple flow to download some data, embed it, and then index and search it using a selection of vector databases. This is a common requirement for customers who want to store and search our embeddings with their own data in a secure environment to support production use cases such as chatbots, topic modelling and more.\n\n### What is a Vector Database\n\nA vector database is a database made to store, manage and search embedding vectors. The use of embeddings to encode unstructured data (text, audio, video and more) as vectors for consumption by machine-learning models has exploded in recent years, due to the increasing effectiveness of AI in solving use cases involving natural language, image recognition and other unstructured forms of data. Vector databases have emerged as an effective solution for enterprises to deliver and scale these use cases.\n\n### Why use a Vector Database\n\nVector databases enable enterprises to take many of the embeddings use cases we've shared in this repo (question and answering, chatbot and recommendation services, for example), and make use of them in a secure, scalable environment. Many of our customers make embeddings solve their problems at small scale but performance and security hold them back from going into production - we see vector databases as a key component in solving that, and in this guide we'll walk through the basics of embedding text data, storing it in a vector database and using it for semantic search.\n\n\n### Demo Flow\nThe demo flow is:\n- **Setup**: Import packages and set any required variables\n- **Load data**: Load a dataset and embed it using OpenAI embeddings\n- **Qdrant**\n    - *Setup*: Here we'll set up the Python client for Qdrant. For more details go [here](https://github.com/qdrant/qdrant_client)\n    - *Index Data*: We'll create a collection with vectors for __titles__ and __content__\n    - *Search Data*: We'll run a few searches to confirm it works\n\nOnce you've run through this notebook you should have a basic understanding of how to setup and use vector databases, and can move on to more complex use cases making use of our embeddings.\n\n## Setup\n\nImport the required libraries and set the embedding model that we'd like to use.\n\n\n```python\n# We'll need to install Qdrant client\n!pip install qdrant-client\n```\n\n\n```python\nimport openai\nimport pandas as pd\nfrom ast import literal_eval\nimport qdrant_client # Qdrant's client library for Python\n\n# This can be changed to the embedding model of your choice. Make sure its the same model that is used for generating embeddings\nEMBEDDING_MODEL = \"text-embedding-ada-002\"\n\n# Ignore unclosed SSL socket warnings - optional in case you get these errors\nimport warnings\n\nwarnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning) \n```\n\n## Load data\n\nIn this section we'll load embedded data that we've prepared previous to this session.\n\n\n```python\nimport requests\n\nembeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'\n\n# The file is ~700 MB so this will take some time\nresponse = requests.get(embeddings_url, verify=True)  # Set verify=False to bypass SSL verification\nwith open('vector_database_wikipedia_articles_embedded.zip', 'wb') as file:\n    file.write(response.content)\n```\n\n\n```python\nimport zipfile\nwith zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n    zip_ref.extractall(\"../data\")\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 42f99026-9f0c-41b3-ae93-2ef20c519fad
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb", "title": "Using Qdrant for Embeddings Search", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"42f99026-9f0c-41b3-ae93-2ef20c519fad\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4c7aafd6-3704-425b-af10-41f8c20b95bb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"368a0b23a141651962d829ce5d9cde0d2bec897f204a469e2bbbcd633b8ef4bd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dfff671a-7d8c-4a53-b65e-06871062d79c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c8518b32e27c0abbd797f45d9d3eb341f99c1439090565461a23eb7550c698a0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"948f32d7-385c-475f-a3b0-cd04c8fe5b4f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"55d75d734c6c465942a4bbe99c9e2550ca1dde90d5af183d906f3fc23ae149c3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2288, \"end_char_idx\": 4729, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "doc_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "ref_doc_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "chroma:document": "```python\n# We'll need to install Qdrant client\n!pip install qdrant-client\n```\n\n\n```python\nimport openai\nimport pandas as pd\nfrom ast import literal_eval\nimport qdrant_client # Qdrant's client library for Python\n\n# This can be changed to the embedding model of your choice. Make sure its the same model that is used for generating embeddings\nEMBEDDING_MODEL = \"text-embedding-ada-002\"\n\n# Ignore unclosed SSL socket warnings - optional in case you get these errors\nimport warnings\n\nwarnings.filterwarnings(action=\"ignore\", message=\"unclosed\", category=ResourceWarning)\nwarnings.filterwarnings(\"ignore\", category=DeprecationWarning) \n```\n\n## Load data\n\nIn this section we'll load embedded data that we've prepared previous to this session.\n\n\n```python\nimport requests\n\nembeddings_url = 'https://cdn.openai.com/API/examples/data/vector_database_wikipedia_articles_embedded.zip'\n\n# The file is ~700 MB so this will take some time\nresponse = requests.get(embeddings_url, verify=True)  # Set verify=False to bypass SSL verification\nwith open('vector_database_wikipedia_articles_embedded.zip', 'wb') as file:\n    file.write(response.content)\n```\n\n\n```python\nimport zipfile\nwith zipfile.ZipFile(\"vector_database_wikipedia_articles_embedded.zip\",\"r\") as zip_ref:\n    zip_ref.extractall(\"../data\")\n```\n\n\n```python\narticle_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')\n```\n\n\n```python\narticle_df.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>url</th>\n      <th>title</th>\n      <th>text</th>\n      <th>title_vector</th>\n      <th>content_vector</th>\n      <th>vector_id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>https://simple.wikipedia.org/wiki/April</td>\n      <td>April</td>\n      <td>April is the fourth month of the year in the J...</td>\n      <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n      <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>https://simple.wikipedia.org/wiki/August</td>\n      <td>August</td>\n      <td>August (Aug.)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 948f32d7-385c-475f-a3b0-cd04c8fe5b4f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb", "title": "Using Qdrant for Embeddings Search", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"948f32d7-385c-475f-a3b0-cd04c8fe5b4f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4c7aafd6-3704-425b-af10-41f8c20b95bb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"368a0b23a141651962d829ce5d9cde0d2bec897f204a469e2bbbcd633b8ef4bd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"42f99026-9f0c-41b3-ae93-2ef20c519fad\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c554cf9ce9eb7cca865c248b7c26a0418ac6dee451d62af0665dbca9fd42a7ad\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bb39b733-f88f-45bf-811f-7a7eca3551e6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0eb9f1fbcad7c87e0ae1cc80d55f22c888212d66f0901dbcd88725f848899c3c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3582, \"end_char_idx\": 5760, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "doc_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "ref_doc_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "chroma:document": "```python\narticle_df = pd.read_csv('../data/vector_database_wikipedia_articles_embedded.csv')\n```\n\n\n```python\narticle_df.head()\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>id</th>\n      <th>url</th>\n      <th>title</th>\n      <th>text</th>\n      <th>title_vector</th>\n      <th>content_vector</th>\n      <th>vector_id</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>1</td>\n      <td>https://simple.wikipedia.org/wiki/April</td>\n      <td>April</td>\n      <td>April is the fourth month of the year in the J...</td>\n      <td>[0.001009464613161981, -0.020700545981526375, ...</td>\n      <td>[-0.011253940872848034, -0.013491976074874401,...</td>\n      <td>0</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>2</td>\n      <td>https://simple.wikipedia.org/wiki/August</td>\n      <td>August</td>\n      <td>August (Aug.) is the eighth month of the year ...</td>\n      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>6</td>\n      <td>https://simple.wikipedia.org/wiki/Art</td>\n      <td>Art</td>\n      <td>Art is a creative activity that expresses imag...</td>\n      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>8</td>\n      <td>https://simple.wikipedia.org/wiki/A</td>\n      <td>A</td>\n      <td>A or a is the first letter of the English alph...</td>\n      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n      <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>9</td>\n      <td>https://simple.wikipedia.org/wiki/Air</td>\n      <td>Air</td>\n      <td>Air refers to the Earth's atmosphere."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: bb39b733-f88f-45bf-811f-7a7eca3551e6
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb", "title": "Using Qdrant for Embeddings Search", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"bb39b733-f88f-45bf-811f-7a7eca3551e6\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4c7aafd6-3704-425b-af10-41f8c20b95bb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"368a0b23a141651962d829ce5d9cde0d2bec897f204a469e2bbbcd633b8ef4bd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"948f32d7-385c-475f-a3b0-cd04c8fe5b4f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ca09f368fa35f43f2272297b04b7fbe10a5785381c21a6b6f688449c57b8cab7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c7de26de-afc4-4f8a-87cc-be8b1b920fc3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a2e2794715ca33f7edb3e3243a822ebed4a0d149e6d4411dcee5adfc28af0189\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4730, \"end_char_idx\": 6928, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "doc_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "ref_doc_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "chroma:document": "is the eighth month of the year ...</td>\n      <td>[0.0009286514250561595, 0.000820168002974242, ...</td>\n      <td>[0.0003609954728744924, 0.007262262050062418, ...</td>\n      <td>1</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>6</td>\n      <td>https://simple.wikipedia.org/wiki/Art</td>\n      <td>Art</td>\n      <td>Art is a creative activity that expresses imag...</td>\n      <td>[0.003393713850528002, 0.0061537534929811954, ...</td>\n      <td>[-0.004959689453244209, 0.015772193670272827, ...</td>\n      <td>2</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>8</td>\n      <td>https://simple.wikipedia.org/wiki/A</td>\n      <td>A</td>\n      <td>A or a is the first letter of the English alph...</td>\n      <td>[0.0153952119871974, -0.013759135268628597, 0....</td>\n      <td>[0.024894846603274345, -0.022186409682035446, ...</td>\n      <td>3</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>9</td>\n      <td>https://simple.wikipedia.org/wiki/Air</td>\n      <td>Air</td>\n      <td>Air refers to the Earth's atmosphere. Air is a...</td>\n      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n      <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n\n```python\n# Read vectors from strings back into a list\narticle_df['title_vector'] = article_df.title_vector.apply(literal_eval)\narticle_df['content_vector'] = article_df.content_vector.apply(literal_eval)\n\n# Set vector_id to be a string\narticle_df['vector_id'] = article_df['vector_id'].apply(str)\n```\n\n\n```python\narticle_df.info(show_counts=True)\n```\n\n    <class 'pandas.core.frame.DataFrame'>\n    RangeIndex: 25000 entries, 0 to 24999\n    Data columns (total 7 columns):\n     #   Column          Non-Null Count  Dtype \n    ---  ------          --------------  ----- \n     0   id              25000 non-null  int64 \n     1   url             25000 non-null  object\n     2   title           25000 non-null  object\n     3   text            25000 non-null  object\n     4   title_vector    25000 non-null  object\n     5   content_vector  25000 non-null  object\n     6   vector_id       25000 non-null  object\n    dtypes: int64(1), object(6)\n    memory usage: 1.3+ MB"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: c7de26de-afc4-4f8a-87cc-be8b1b920fc3
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb", "title": "Using Qdrant for Embeddings Search", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c7de26de-afc4-4f8a-87cc-be8b1b920fc3\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4c7aafd6-3704-425b-af10-41f8c20b95bb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"368a0b23a141651962d829ce5d9cde0d2bec897f204a469e2bbbcd633b8ef4bd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bb39b733-f88f-45bf-811f-7a7eca3551e6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"0164a32f1b695d05f05c56cab04cf5d39ad7a0179a5451fec52dba3ea727c399\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a17e3a8f-995c-4dcc-9faa-e6cd1d00dd21\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"24a890fe0c2283ebf7216b072a6120e9869a51de01fb9f1b3b5bd054afbe0457\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5761, \"end_char_idx\": 8586, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "doc_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "ref_doc_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "chroma:document": "Air is a...</td>\n      <td>[0.02224554680287838, -0.02044147066771984, -0...</td>\n      <td>[0.021524671465158463, 0.018522677943110466, -...</td>\n      <td>4</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n\n```python\n# Read vectors from strings back into a list\narticle_df['title_vector'] = article_df.title_vector.apply(literal_eval)\narticle_df['content_vector'] = article_df.content_vector.apply(literal_eval)\n\n# Set vector_id to be a string\narticle_df['vector_id'] = article_df['vector_id'].apply(str)\n```\n\n\n```python\narticle_df.info(show_counts=True)\n```\n\n    <class 'pandas.core.frame.DataFrame'>\n    RangeIndex: 25000 entries, 0 to 24999\n    Data columns (total 7 columns):\n     #   Column          Non-Null Count  Dtype \n    ---  ------          --------------  ----- \n     0   id              25000 non-null  int64 \n     1   url             25000 non-null  object\n     2   title           25000 non-null  object\n     3   text            25000 non-null  object\n     4   title_vector    25000 non-null  object\n     5   content_vector  25000 non-null  object\n     6   vector_id       25000 non-null  object\n    dtypes: int64(1), object(6)\n    memory usage: 1.3+ MB\n\n\n## Qdrant\n\n**[Qdrant](https://qdrant.tech/)**. is a high-performant vector search database written in Rust. It offers both on-premise and cloud version, but for the purposes of that example we're going to use the local deployment mode.\n\nSetting everything up will require:\n- Spinning up a local instance of Qdrant\n- Configuring the collection and storing the data in it\n- Trying out with some queries\n\n### Setup\n\nFor the local deployment, we are going to use Docker, according to the Qdrant documentation: https://qdrant.tech/documentation/quick_start/. Qdrant requires just a single container, but an example of the docker-compose.yaml file is available at `./qdrant/docker-compose.yaml` in this repo.\n\nYou can start Qdrant instance locally by navigating to this directory and running `docker-compose up -d `\n\n> You might need to increase the memory limit for Docker to 8GB or more. Or Qdrant might fail to execute with an error message like `7 Killed`.\n\n\n\n```python\nqdrant = qdrant_client.QdrantClient(host=\"localhost\", port=6333)\n```\n\n\n```python\nqdrant.get_collections()\n```\n\n\n\n\n    CollectionsResponse(collections=[CollectionDescription(name='Articles')])\n\n\n\n### Index data\n\nQdrant stores data in __collections__ where each object is described by at least one vector and may contain an additional metadata called __payload__. Our collection will be called **Articles** and each object will be described by both **title** and **content** vectors.\n\nWe'll be using an official [qdrant-client](https://github.com/qdrant/qdrant_client) package that has all the utility methods already built-in.\n\n\n```python\nfrom qdrant_client.http import models as rest\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: a17e3a8f-995c-4dcc-9faa-e6cd1d00dd21
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb", "title": "Using Qdrant for Embeddings Search", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a17e3a8f-995c-4dcc-9faa-e6cd1d00dd21\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4c7aafd6-3704-425b-af10-41f8c20b95bb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"368a0b23a141651962d829ce5d9cde0d2bec897f204a469e2bbbcd633b8ef4bd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c7de26de-afc4-4f8a-87cc-be8b1b920fc3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"30f8b192993d465ce631467efc3f440dce0d912f663d6797627688d8b85b3005\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"30c24038-df87-408d-ad2e-1e51d83e738b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7972cb12100cfd532bd745f9322057a3922d878419375f87e31ff44abede49b3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6931, \"end_char_idx\": 10143, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "doc_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "ref_doc_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "chroma:document": "## Qdrant\n\n**[Qdrant](https://qdrant.tech/)**. is a high-performant vector search database written in Rust. It offers both on-premise and cloud version, but for the purposes of that example we're going to use the local deployment mode.\n\nSetting everything up will require:\n- Spinning up a local instance of Qdrant\n- Configuring the collection and storing the data in it\n- Trying out with some queries\n\n### Setup\n\nFor the local deployment, we are going to use Docker, according to the Qdrant documentation: https://qdrant.tech/documentation/quick_start/. Qdrant requires just a single container, but an example of the docker-compose.yaml file is available at `./qdrant/docker-compose.yaml` in this repo.\n\nYou can start Qdrant instance locally by navigating to this directory and running `docker-compose up -d `\n\n> You might need to increase the memory limit for Docker to 8GB or more. Or Qdrant might fail to execute with an error message like `7 Killed`.\n\n\n\n```python\nqdrant = qdrant_client.QdrantClient(host=\"localhost\", port=6333)\n```\n\n\n```python\nqdrant.get_collections()\n```\n\n\n\n\n    CollectionsResponse(collections=[CollectionDescription(name='Articles')])\n\n\n\n### Index data\n\nQdrant stores data in __collections__ where each object is described by at least one vector and may contain an additional metadata called __payload__. Our collection will be called **Articles** and each object will be described by both **title** and **content** vectors.\n\nWe'll be using an official [qdrant-client](https://github.com/qdrant/qdrant_client) package that has all the utility methods already built-in.\n\n\n```python\nfrom qdrant_client.http import models as rest\n```\n\n\n```python\n# Get the vector size from the first row to set up the collection\nvector_size = len(article_df['content_vector'][0])\n\n# Set up the collection with the vector configuration. You need to declare the vector size and distance metric for the collection. Distance metric enables vector database to index and search vectors efficiently.\nqdrant.recreate_collection(\n    collection_name='Articles',\n    vectors_config={\n        'title': rest.VectorParams(\n            distance=rest.Distance.COSINE,\n            size=vector_size,\n        ),\n        'content': rest.VectorParams(\n            distance=rest.Distance.COSINE,\n            size=vector_size,\n        ),\n    }\n)\n```\n\n\n\n\n    True\n\n\n\n\n```python\nvector_size = len(article_df['content_vector'][0])\n\nqdrant.recreate_collection(\n    collection_name='Articles',\n    vectors_config={\n        'title': rest.VectorParams(\n            distance=rest.Distance.COSINE,\n            size=vector_size,\n        ),\n        'content': rest.VectorParams(\n            distance=rest.Distance.COSINE,\n            size=vector_size,\n        ),\n    }\n)\n```\n\n\n\n\n    True\n\n\n\nIn addition to the vector configuration defined under `vector`, we can also define the `payload` configuration. Payload is an optional field that allows you to store additional metadata alongside the vectors. In our case, we'll store the `id`, `title`, and `url` of the articles. As we return the title of nearest articles in the search results from payload, we can also provide the user with the URL to the article (which is part of the meta-data)."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 30c24038-df87-408d-ad2e-1e51d83e738b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb", "title": "Using Qdrant for Embeddings Search", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"30c24038-df87-408d-ad2e-1e51d83e738b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4c7aafd6-3704-425b-af10-41f8c20b95bb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"368a0b23a141651962d829ce5d9cde0d2bec897f204a469e2bbbcd633b8ef4bd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a17e3a8f-995c-4dcc-9faa-e6cd1d00dd21\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8456de798d4433816952d94840d855141959b71486924ab0d6e2513c1a9bbb7c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f4644917-118c-46b6-a912-204971467d06\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d6864c3b4d940639568791231196dad3853dd5423ec5b07d40f8241bb6688b23\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8527, \"end_char_idx\": 11525, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "doc_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "ref_doc_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "chroma:document": "```python\nfrom qdrant_client.http import models as rest\n```\n\n\n```python\n# Get the vector size from the first row to set up the collection\nvector_size = len(article_df['content_vector'][0])\n\n# Set up the collection with the vector configuration. You need to declare the vector size and distance metric for the collection. Distance metric enables vector database to index and search vectors efficiently.\nqdrant.recreate_collection(\n    collection_name='Articles',\n    vectors_config={\n        'title': rest.VectorParams(\n            distance=rest.Distance.COSINE,\n            size=vector_size,\n        ),\n        'content': rest.VectorParams(\n            distance=rest.Distance.COSINE,\n            size=vector_size,\n        ),\n    }\n)\n```\n\n\n\n\n    True\n\n\n\n\n```python\nvector_size = len(article_df['content_vector'][0])\n\nqdrant.recreate_collection(\n    collection_name='Articles',\n    vectors_config={\n        'title': rest.VectorParams(\n            distance=rest.Distance.COSINE,\n            size=vector_size,\n        ),\n        'content': rest.VectorParams(\n            distance=rest.Distance.COSINE,\n            size=vector_size,\n        ),\n    }\n)\n```\n\n\n\n\n    True\n\n\n\nIn addition to the vector configuration defined under `vector`, we can also define the `payload` configuration. Payload is an optional field that allows you to store additional metadata alongside the vectors. In our case, we'll store the `id`, `title`, and `url` of the articles. As we return the title of nearest articles in the search results from payload, we can also provide the user with the URL to the article (which is part of the meta-data).\n\n\n```python\nfrom qdrant_client.models import PointStruct # Import the PointStruct to store the vector and payload\nfrom tqdm import tqdm # Library to show the progress bar \n\n# Populate collection with vectors using tqdm to show progress\nfor k, v in tqdm(article_df.iterrows(), desc=\"Upserting articles\", total=len(article_df)):\n    try:\n        qdrant.upsert(\n            collection_name='Articles',\n            points=[\n                PointStruct(\n                    id=k,\n                    vector={'title': v['title_vector'], \n                            'content': v['content_vector']},\n                    payload={\n                        'id': v['id'],\n                        'title': v['title'],\n                        'url': v['url']\n                    }\n                )\n            ]\n        )\n    except Exception as e:\n        print(f\"Failed to upsert row {k}: {v}\")\n        print(f\"Exception: {e}\")\n```\n\n    Upserting articles: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 25000/25000 [01:18<00:00, 318.84it/s]\n\n\n\n```python\n# Check the collection size to make sure all the points have been stored\nqdrant.count(collection_name='Articles')\n```\n\n\n\n\n    CountResult(count=25000)\n\n\n\n### Search Data\n\nOnce the data is put into Qdrant we will start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: f4644917-118c-46b6-a912-204971467d06
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb", "title": "Using Qdrant for Embeddings Search", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"f4644917-118c-46b6-a912-204971467d06\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4c7aafd6-3704-425b-af10-41f8c20b95bb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"368a0b23a141651962d829ce5d9cde0d2bec897f204a469e2bbbcd633b8ef4bd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"30c24038-df87-408d-ad2e-1e51d83e738b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a74e305ad2e672fedf54a3d358fb432036896dd5d553352630e17d7dc5041010\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d2c6a0e4-448f-44db-983b-0d818876803f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d4be03ffeb75ab350fa38dd69b3189672db94bce4868ef8e163ae0cf77eeda76\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9694, \"end_char_idx\": 12068, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "doc_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "ref_doc_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "chroma:document": "In addition to the vector configuration defined under `vector`, we can also define the `payload` configuration. Payload is an optional field that allows you to store additional metadata alongside the vectors. In our case, we'll store the `id`, `title`, and `url` of the articles. As we return the title of nearest articles in the search results from payload, we can also provide the user with the URL to the article (which is part of the meta-data).\n\n\n```python\nfrom qdrant_client.models import PointStruct # Import the PointStruct to store the vector and payload\nfrom tqdm import tqdm # Library to show the progress bar \n\n# Populate collection with vectors using tqdm to show progress\nfor k, v in tqdm(article_df.iterrows(), desc=\"Upserting articles\", total=len(article_df)):\n    try:\n        qdrant.upsert(\n            collection_name='Articles',\n            points=[\n                PointStruct(\n                    id=k,\n                    vector={'title': v['title_vector'], \n                            'content': v['content_vector']},\n                    payload={\n                        'id': v['id'],\n                        'title': v['title'],\n                        'url': v['url']\n                    }\n                )\n            ]\n        )\n    except Exception as e:\n        print(f\"Failed to upsert row {k}: {v}\")\n        print(f\"Exception: {e}\")\n```\n\n    Upserting articles: 100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 25000/25000 [01:18<00:00, 318.84it/s]\n\n\n\n```python\n# Check the collection size to make sure all the points have been stored\nqdrant.count(collection_name='Articles')\n```\n\n\n\n\n    CountResult(count=25000)\n\n\n\n### Search Data\n\nOnce the data is put into Qdrant we will start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search.\n\n\n```python\ndef query_qdrant(query, collection_name, vector_name='title', top_k=20):\n\n    # Creates embedding vector from user query\n    embedded_query = openai.embeddings.create(\n        input=query,\n        model=EMBEDDING_MODEL,\n    ).data[0].embedding # We take the first embedding from the list\n    \n    query_results = qdrant.search(\n        collection_name=collection_name,\n        query_vector=(\n            vector_name, embedded_query\n        ),\n        limit=top_k, \n        query_filter=None\n    )\n    \n    return query_results\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: d2c6a0e4-448f-44db-983b-0d818876803f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb", "title": "Using Qdrant for Embeddings Search", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"d2c6a0e4-448f-44db-983b-0d818876803f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4c7aafd6-3704-425b-af10-41f8c20b95bb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"368a0b23a141651962d829ce5d9cde0d2bec897f204a469e2bbbcd633b8ef4bd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f4644917-118c-46b6-a912-204971467d06\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"91602cf862330da4c2b3c922ad3e1f48746afa009f4931ca946305d7d7f7199c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"eb4a90cf-4993-41e5-b950-9481a0113682\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"628a7bfd3ba8f2f3546e41b86e37212a28dcd765201f974837e0d43e64f303cd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11150, \"end_char_idx\": 14102, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "doc_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "ref_doc_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "chroma:document": "```python\n# Check the collection size to make sure all the points have been stored\nqdrant.count(collection_name='Articles')\n```\n\n\n\n\n    CountResult(count=25000)\n\n\n\n### Search Data\n\nOnce the data is put into Qdrant we will start querying the collection for the closest vectors. We may provide an additional parameter `vector_name` to switch from title to content based search.\n\n\n```python\ndef query_qdrant(query, collection_name, vector_name='title', top_k=20):\n\n    # Creates embedding vector from user query\n    embedded_query = openai.embeddings.create(\n        input=query,\n        model=EMBEDDING_MODEL,\n    ).data[0].embedding # We take the first embedding from the list\n    \n    query_results = qdrant.search(\n        collection_name=collection_name,\n        query_vector=(\n            vector_name, embedded_query\n        ),\n        limit=top_k, \n        query_filter=None\n    )\n    \n    return query_results\n```\n\n\n```python\nquery_results = query_qdrant('modern art in Europe', 'Articles', 'title')\nfor i, article in enumerate(query_results):\n    print(f'{i + 1}. {article.payload[\"title\"]}, URL: {article.payload[\"url\"]} (Score: {round(article.score, 3)})')\n```\n\n    1. Museum of Modern Art, URL: https://simple.wikipedia.org/wiki/Museum%20of%20Modern%20Art (Score: 0.875)\n    2. Western Europe, URL: https://simple.wikipedia.org/wiki/Western%20Europe (Score: 0.867)\n    3. Renaissance art, URL: https://simple.wikipedia.org/wiki/Renaissance%20art (Score: 0.864)\n    4. Pop art, URL: https://simple.wikipedia.org/wiki/Pop%20art (Score: 0.86)\n    5. Northern Europe, URL: https://simple.wikipedia.org/wiki/Northern%20Europe (Score: 0.855)\n    6. Hellenistic art, URL: https://simple.wikipedia.org/wiki/Hellenistic%20art (Score: 0.853)\n    7. Modernist literature, URL: https://simple.wikipedia.org/wiki/Modernist%20literature (Score: 0.847)\n    8. Art film, URL: https://simple.wikipedia.org/wiki/Art%20film (Score: 0.843)\n    9. Central Europe, URL: https://simple.wikipedia.org/wiki/Central%20Europe (Score: 0.843)\n    10. European, URL: https://simple.wikipedia.org/wiki/European (Score: 0.841)\n    11. Art, URL: https://simple.wikipedia.org/wiki/Art (Score: 0.841)\n    12. Byzantine art, URL: https://simple.wikipedia.org/wiki/Byzantine%20art (Score: 0.841)\n    13. Postmodernism, URL: https://simple.wikipedia.org/wiki/Postmodernism (Score: 0.84)\n    14. Eastern Europe, URL: https://simple.wikipedia.org/wiki/Eastern%20Europe (Score: 0.839)\n    15. Cubism, URL: https://simple.wikipedia.org/wiki/Cubism (Score: 0.839)\n    16. Europe, URL: https://simple.wikipedia.org/wiki/Europe (Score: 0.839)\n    17. Impressionism, URL: https://simple.wikipedia.org/wiki/Impressionism (Score: 0.838)\n    18. Bauhaus, URL: https://simple.wikipedia.org/wiki/Bauhaus (Score: 0.838)\n    19. Surrealism, URL: https://simple.wikipedia.org/wiki/Surrealism (Score: 0.837)\n    20. Expressionism, URL: https://simple.wikipedia.org/wiki/Expressionism (Score: 0.837)"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: eb4a90cf-4993-41e5-b950-9481a0113682
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb", "title": "Using Qdrant for Embeddings Search", "tokens": 4316, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"eb4a90cf-4993-41e5-b950-9481a0113682\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4c7aafd6-3704-425b-af10-41f8c20b95bb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"368a0b23a141651962d829ce5d9cde0d2bec897f204a469e2bbbcd633b8ef4bd\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d2c6a0e4-448f-44db-983b-0d818876803f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/qdrant/Using_Qdrant_for_embeddings_search.ipynb\", \"title\": \"Using Qdrant for Embeddings Search\", \"tokens\": 4316, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"168c5fa1af08d5ba11400800fd12265eb3d90fe650b590c4556f3c25c6c62413\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14106, \"end_char_idx\": 16172, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "doc_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "ref_doc_id": "4c7aafd6-3704-425b-af10-41f8c20b95bb", "chroma:document": "```python\n# This time we'll query using content vector\nquery_results = query_qdrant('Famous battles in Scottish history', 'Articles', 'content')\nfor i, article in enumerate(query_results):\n    print(f'{i + 1}. {article.payload[\"title\"]}, URL: {article.payload[\"url\"]} (Score: {round(article.score, 3)})')\n```\n\n    1. Battle of Bannockburn, URL: https://simple.wikipedia.org/wiki/Battle%20of%20Bannockburn (Score: 0.869)\n    2. Wars of Scottish Independence, URL: https://simple.wikipedia.org/wiki/Wars%20of%20Scottish%20Independence (Score: 0.861)\n    3. 1651, URL: https://simple.wikipedia.org/wiki/1651 (Score: 0.853)\n    4. First War of Scottish Independence, URL: https://simple.wikipedia.org/wiki/First%20War%20of%20Scottish%20Independence (Score: 0.85)\n    5. Robert I of Scotland, URL: https://simple.wikipedia.org/wiki/Robert%20I%20of%20Scotland (Score: 0.846)\n    6. 841, URL: https://simple.wikipedia.org/wiki/841 (Score: 0.844)\n    7. 1716, URL: https://simple.wikipedia.org/wiki/1716 (Score: 0.844)\n    8. 1314, URL: https://simple.wikipedia.org/wiki/1314 (Score: 0.837)\n    9. 1263, URL: https://simple.wikipedia.org/wiki/1263 (Score: 0.836)\n    10. William Wallace, URL: https://simple.wikipedia.org/wiki/William%20Wallace (Score: 0.835)\n    11. Stirling, URL: https://simple.wikipedia.org/wiki/Stirling (Score: 0.831)\n    12. 1306, URL: https://simple.wikipedia.org/wiki/1306 (Score: 0.831)\n    13. 1746, URL: https://simple.wikipedia.org/wiki/1746 (Score: 0.83)\n    14. 1040s, URL: https://simple.wikipedia.org/wiki/1040s (Score: 0.828)\n    15. 1106, URL: https://simple.wikipedia.org/wiki/1106 (Score: 0.827)\n    16. 1304, URL: https://simple.wikipedia.org/wiki/1304 (Score: 0.826)\n    17. David II of Scotland, URL: https://simple.wikipedia.org/wiki/David%20II%20of%20Scotland (Score: 0.825)\n    18. Braveheart, URL: https://simple.wikipedia.org/wiki/Braveheart (Score: 0.824)\n    19. 1124, URL: https://simple.wikipedia.org/wiki/1124 (Score: 0.824)\n    20. July 27, URL: https://simple.wikipedia.org/wiki/July%2027 (Score: 0.823)\n\n\n\n```python\n\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 2e20383a-2c49-4cc3-b5f5-4f88949f4b22
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Astra DB", "tokens": 4213, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"2e20383a-2c49-4cc3-b5f5-4f88949f4b22\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"31366239-82b7-497b-9cbd-19245c1a7b03\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"11ecc2b89c2290c6c16b28186eb94888f10182554adae700c5a4235db2a57141\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"719d3539-17c6-473f-b00c-0d4b4c919d1b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ab180f4fa0a8c795bf3f03b7a1b348cc1d2278e23074955f1906bcabcaf2719a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2845, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "ref_doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "chroma:document": "# Philosophy with Vector Embeddings, OpenAI and Astra DB\n\n### AstraPy version\n\nIn this quickstart you will learn how to build a \"philosophy quote finder & generator\" using OpenAI's vector embeddings and DataStax [Astra DB](https://docs.datastax.com/en/astra/home/astra.html) as the vector store for data persistence.\n\nThe basic workflow of this notebook is outlined below. You will evaluate and store the vector embeddings for a number of quotes by famous philosophers, use them to build a powerful search engine and, after that, even a generator of new quotes!\n\nThe notebook exemplifies some of the standard usage patterns of vector search -- while showing how easy is it to get started with [Astra DB](https://docs.datastax.com/en/astra/home/astra.html).\n\nFor a background on using vector search and text embeddings to build a question-answering system, please check out this excellent hands-on notebook: [Question answering using embeddings](https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb).\n\nTable of contents:\n- Setup\n- Create vector collection\n- Connect to OpenAI\n- Load quotes into the Vector Store\n- Use case 1: **quote search engine**\n- Use case 2: **quote generator**\n- Cleanup\n\n### How it works\n\n**Indexing**\n\nEach quote is made into an embedding vector with OpenAI's `Embedding`. These are saved in the Vector Store for later use in searching. Some metadata, including the author's name and a few other pre-computed tags, are stored alongside, to allow for search customization.\n\n![1_vector_indexing](https://user-images.githubusercontent.com/14221764/282422016-1d540607-eed4-4240-9c3d-22ee3a3bc90f.png)\n\n**Search**\n\nTo find a quote similar to the provided search quote, the latter is made into an embedding vector on the fly, and this vector is used to query the store for similar vectors ... i.e. similar quotes that were previously indexed. The search can optionally be constrained by additional metadata (\"find me quotes by Spinoza similar to this one ...\").\n\n![2_vector_search](https://user-images.githubusercontent.com/14221764/282422033-0a1297c4-63bb-4e04-b120-dfd98dc1a689.png)\n\nThe key point here is that \"quotes similar in content\" translates, in vector space, to vectors that are metrically close to each other: thus, vector similarity search effectively implements semantic similarity. _This is the key reason vector embeddings are so powerful._\n\nThe sketch below tries to convey this idea. Each quote, once it's made into a vector, is a point in space. Well, in this case it's on a sphere, since OpenAI's embedding vectors, as most others, are normalized to _unit length_. Oh, and the sphere is actually not three-dimensional, rather 1536-dimensional!\n\nSo, in essence, a similarity search in vector space returns the vectors that are closest to the query vector:\n\n!"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 719d3539-17c6-473f-b00c-0d4b4c919d1b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Astra DB", "tokens": 4213, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"719d3539-17c6-473f-b00c-0d4b4c919d1b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"31366239-82b7-497b-9cbd-19245c1a7b03\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"11ecc2b89c2290c6c16b28186eb94888f10182554adae700c5a4235db2a57141\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2e20383a-2c49-4cc3-b5f5-4f88949f4b22\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"bb310b4940bde621974587766d02ee9a957146f9595860205f2358bc4c46d0c3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5f54a1e8-9130-4857-a97e-33b71b311912\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"29a7c8c0dbc05c6cff49691bebc6f9c06ca66312a8c27582e64dba2b8dbb6336\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1347, \"end_char_idx\": 4009, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "ref_doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "chroma:document": "These are saved in the Vector Store for later use in searching. Some metadata, including the author's name and a few other pre-computed tags, are stored alongside, to allow for search customization.\n\n![1_vector_indexing](https://user-images.githubusercontent.com/14221764/282422016-1d540607-eed4-4240-9c3d-22ee3a3bc90f.png)\n\n**Search**\n\nTo find a quote similar to the provided search quote, the latter is made into an embedding vector on the fly, and this vector is used to query the store for similar vectors ... i.e. similar quotes that were previously indexed. The search can optionally be constrained by additional metadata (\"find me quotes by Spinoza similar to this one ...\").\n\n![2_vector_search](https://user-images.githubusercontent.com/14221764/282422033-0a1297c4-63bb-4e04-b120-dfd98dc1a689.png)\n\nThe key point here is that \"quotes similar in content\" translates, in vector space, to vectors that are metrically close to each other: thus, vector similarity search effectively implements semantic similarity. _This is the key reason vector embeddings are so powerful._\n\nThe sketch below tries to convey this idea. Each quote, once it's made into a vector, is a point in space. Well, in this case it's on a sphere, since OpenAI's embedding vectors, as most others, are normalized to _unit length_. Oh, and the sphere is actually not three-dimensional, rather 1536-dimensional!\n\nSo, in essence, a similarity search in vector space returns the vectors that are closest to the query vector:\n\n![3_vector_space](https://user-images.githubusercontent.com/14221764/262321363-c8c625c1-8be9-450e-8c68-b1ed518f990d.png)\n\n**Generation**\n\nGiven a suggestion (a topic or a tentative quote), the search step is performed, and the first returned results (quotes) are fed into an LLM prompt which asks the generative model to invent a new text along the lines of the passed examples _and_ the initial suggestion.\n\n![4_quote_generation](https://user-images.githubusercontent.com/14221764/282422050-2e209ff5-07d6-41ac-99ac-f442e090b3bb.png)\n\n## Setup\n\nInstall and import the necessary dependencies:\n\n\n```python\n!pip install --quiet \"astrapy>=0.6.0\" \"openai>=1.0.0\" datasets\n```\n\n\n```python\nfrom getpass import getpass\nfrom collections import Counter\n\nfrom astrapy.db import AstraDB\nimport openai\nfrom datasets import load_dataset\n```\n\n### Connection parameters\n\nPlease retrieve your database credentials on your Astra dashboard ([info](https://docs.datastax.com/en/astra/astra-db-vector/)): you will supply them momentarily.\n\nExample values:\n\n- API Endpoint: `https://01234567-89ab-cdef-0123-456789abcdef-us-east1.apps.astra.datastax.com`\n- Token: `AstraCS:6gBhNmsk135...`"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 5f54a1e8-9130-4857-a97e-33b71b311912
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Astra DB", "tokens": 4213, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"5f54a1e8-9130-4857-a97e-33b71b311912\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"31366239-82b7-497b-9cbd-19245c1a7b03\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"11ecc2b89c2290c6c16b28186eb94888f10182554adae700c5a4235db2a57141\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"719d3539-17c6-473f-b00c-0d4b4c919d1b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"1f966e5e01147515f1c416715f7c66bbb76c537476196b73f48376bf55d6aa62\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e0302f83-591f-43bf-afc9-9e218eb7f56b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0ce899b54b844a5aa2e799d742f1933cfe8fc4a1a972e28260739b83f34b39d0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2533, \"end_char_idx\": 5091, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "ref_doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "chroma:document": "Well, in this case it's on a sphere, since OpenAI's embedding vectors, as most others, are normalized to _unit length_. Oh, and the sphere is actually not three-dimensional, rather 1536-dimensional!\n\nSo, in essence, a similarity search in vector space returns the vectors that are closest to the query vector:\n\n![3_vector_space](https://user-images.githubusercontent.com/14221764/262321363-c8c625c1-8be9-450e-8c68-b1ed518f990d.png)\n\n**Generation**\n\nGiven a suggestion (a topic or a tentative quote), the search step is performed, and the first returned results (quotes) are fed into an LLM prompt which asks the generative model to invent a new text along the lines of the passed examples _and_ the initial suggestion.\n\n![4_quote_generation](https://user-images.githubusercontent.com/14221764/282422050-2e209ff5-07d6-41ac-99ac-f442e090b3bb.png)\n\n## Setup\n\nInstall and import the necessary dependencies:\n\n\n```python\n!pip install --quiet \"astrapy>=0.6.0\" \"openai>=1.0.0\" datasets\n```\n\n\n```python\nfrom getpass import getpass\nfrom collections import Counter\n\nfrom astrapy.db import AstraDB\nimport openai\nfrom datasets import load_dataset\n```\n\n### Connection parameters\n\nPlease retrieve your database credentials on your Astra dashboard ([info](https://docs.datastax.com/en/astra/astra-db-vector/)): you will supply them momentarily.\n\nExample values:\n\n- API Endpoint: `https://01234567-89ab-cdef-0123-456789abcdef-us-east1.apps.astra.datastax.com`\n- Token: `AstraCS:6gBhNmsk135...`\n\n\n```python\nASTRA_DB_API_ENDPOINT = input(\"Please enter your API Endpoint:\")\nASTRA_DB_APPLICATION_TOKEN = getpass(\"Please enter your Token\")\n```\n\n    Please enter your API Endpoint: https://4f835778-ec78-42b0-9ae3-29e3cf45b596-us-east1.apps.astra.datastax.com\n    Please enter your Token \u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\n\n\n### Instantiate an Astra DB client\n\n\n```python\nastra_db = AstraDB(\n    api_endpoint=ASTRA_DB_API_ENDPOINT,\n    token=ASTRA_DB_APPLICATION_TOKEN,\n)\n```\n\n## Create vector collection\n\nThe only parameter to specify, other than the collection name, is the dimension of the vectors you'll store. Other parameters, notably the similarity metric to use for searches, are optional.\n\n\n```python\ncoll_name = \"philosophers_astra_db\"\ncollection = astra_db.create_collection(coll_name, dimension=1536)\n```\n\n## Connect to OpenAI\n\n### Set up your secret key\n\n\n```python\nOPENAI_API_KEY = getpass(\"Please enter your OpenAI API Key: \")\n```\n\n    Please enter your OpenAI API Key:  \u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\n\n\n### A test call for embeddings\n\nQuickly check how one can get the embedding vectors for a list of input texts:"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: e0302f83-591f-43bf-afc9-9e218eb7f56b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Astra DB", "tokens": 4213, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e0302f83-591f-43bf-afc9-9e218eb7f56b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"31366239-82b7-497b-9cbd-19245c1a7b03\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"11ecc2b89c2290c6c16b28186eb94888f10182554adae700c5a4235db2a57141\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5f54a1e8-9130-4857-a97e-33b71b311912\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"2659e9bcbf8755c65da3845a34be54f98020795817908cb9fd735978520ebf13\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fa43a680-22ea-4f40-9bee-23dc4158ec10\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"169441b9efdee16ad99ddbc10eee6e264af7c0a4008eaa31ed4c43c54dd009b6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4012, \"end_char_idx\": 6454, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "ref_doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "chroma:document": "```python\nASTRA_DB_API_ENDPOINT = input(\"Please enter your API Endpoint:\")\nASTRA_DB_APPLICATION_TOKEN = getpass(\"Please enter your Token\")\n```\n\n    Please enter your API Endpoint: https://4f835778-ec78-42b0-9ae3-29e3cf45b596-us-east1.apps.astra.datastax.com\n    Please enter your Token \u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\n\n\n### Instantiate an Astra DB client\n\n\n```python\nastra_db = AstraDB(\n    api_endpoint=ASTRA_DB_API_ENDPOINT,\n    token=ASTRA_DB_APPLICATION_TOKEN,\n)\n```\n\n## Create vector collection\n\nThe only parameter to specify, other than the collection name, is the dimension of the vectors you'll store. Other parameters, notably the similarity metric to use for searches, are optional.\n\n\n```python\ncoll_name = \"philosophers_astra_db\"\ncollection = astra_db.create_collection(coll_name, dimension=1536)\n```\n\n## Connect to OpenAI\n\n### Set up your secret key\n\n\n```python\nOPENAI_API_KEY = getpass(\"Please enter your OpenAI API Key: \")\n```\n\n    Please enter your OpenAI API Key:  \u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\n\n\n### A test call for embeddings\n\nQuickly check how one can get the embedding vectors for a list of input texts:\n\n\n```python\nclient = openai.OpenAI(api_key=OPENAI_API_KEY)\nembedding_model_name = \"text-embedding-3-small\"\n\nresult = client.embeddings.create(\n    input=[\n        \"This is a sentence\",\n        \"A second sentence\"\n    ],\n    model=embedding_model_name,\n)\n```\n\n_Note: the above is the syntax for OpenAI v1.0+. If using previous versions, the code to get the embeddings will look different._\n\n\n```python\nprint(f\"len(result.data)              = {len(result.data)}\")\nprint(f\"result.data[1].embedding      = {str(result.data[1].embedding)[:55]}...\")\nprint(f\"len(result.data[1].embedding) = {len(result.data[1].embedding)}\")\n```\n\n    len(result.data)              = 2\n    result.data[1].embedding      = [-0.0108176339417696, 0.0013546717818826437, 0.00362232...\n    len(result.data[1].embedding) = 1536\n\n\n## Load quotes into the Vector Store\n\nGet a dataset with the quotes. _(We adapted and augmented the data from [this Kaggle dataset](https://www.kaggle.com/datasets/mertbozkurt5/quotes-by-philosophers), ready to use in this demo.)_\n\n\n```python\nphilo_dataset = load_dataset(\"datastax/philosopher-quotes\")[\"train\"]\n```\n\nA quick inspection:\n\n\n```python\nprint(\"An example entry:\")\nprint(philo_dataset[16])\n```\n\n    An example entry:\n    {'author': 'aristotle', 'quote': 'Love well, be loved and do something of value.', 'tags': 'love;ethics'}\n\n\nCheck the dataset size:"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: fa43a680-22ea-4f40-9bee-23dc4158ec10
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Astra DB", "tokens": 4213, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"fa43a680-22ea-4f40-9bee-23dc4158ec10\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"31366239-82b7-497b-9cbd-19245c1a7b03\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"11ecc2b89c2290c6c16b28186eb94888f10182554adae700c5a4235db2a57141\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e0302f83-591f-43bf-afc9-9e218eb7f56b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"0f5e38bbfdea66c52971ec9216b1270c384121f5ed90ee47c16fb23dd7e9ab44\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"32e6b973-8d0d-4ea1-a560-8c337a55e750\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"854dbf941b3f333d41e0c2f2e89d75b56943c35422aaee4d6517978e4030a52e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4981, \"end_char_idx\": 7654, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "ref_doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "chroma:document": "### A test call for embeddings\n\nQuickly check how one can get the embedding vectors for a list of input texts:\n\n\n```python\nclient = openai.OpenAI(api_key=OPENAI_API_KEY)\nembedding_model_name = \"text-embedding-3-small\"\n\nresult = client.embeddings.create(\n    input=[\n        \"This is a sentence\",\n        \"A second sentence\"\n    ],\n    model=embedding_model_name,\n)\n```\n\n_Note: the above is the syntax for OpenAI v1.0+. If using previous versions, the code to get the embeddings will look different._\n\n\n```python\nprint(f\"len(result.data)              = {len(result.data)}\")\nprint(f\"result.data[1].embedding      = {str(result.data[1].embedding)[:55]}...\")\nprint(f\"len(result.data[1].embedding) = {len(result.data[1].embedding)}\")\n```\n\n    len(result.data)              = 2\n    result.data[1].embedding      = [-0.0108176339417696, 0.0013546717818826437, 0.00362232...\n    len(result.data[1].embedding) = 1536\n\n\n## Load quotes into the Vector Store\n\nGet a dataset with the quotes. _(We adapted and augmented the data from [this Kaggle dataset](https://www.kaggle.com/datasets/mertbozkurt5/quotes-by-philosophers), ready to use in this demo.)_\n\n\n```python\nphilo_dataset = load_dataset(\"datastax/philosopher-quotes\")[\"train\"]\n```\n\nA quick inspection:\n\n\n```python\nprint(\"An example entry:\")\nprint(philo_dataset[16])\n```\n\n    An example entry:\n    {'author': 'aristotle', 'quote': 'Love well, be loved and do something of value.', 'tags': 'love;ethics'}\n\n\nCheck the dataset size:\n\n\n```python\nauthor_count = Counter(entry[\"author\"] for entry in philo_dataset)\nprint(f\"Total: {len(philo_dataset)} quotes. By author:\")\nfor author, count in author_count.most_common():\n    print(f\"    {author:<20}: {count} quotes\")\n```\n\n    Total: 450 quotes. By author:\n        aristotle           : 50 quotes\n        schopenhauer        : 50 quotes\n        spinoza             : 50 quotes\n        hegel               : 50 quotes\n        freud               : 50 quotes\n        nietzsche           : 50 quotes\n        sartre              : 50 quotes\n        plato               : 50 quotes\n        kant                : 50 quotes\n\n\n### Write to the vector collection\n\nYou will compute the embeddings for the quotes and save them into the Vector Store, along with the text itself and the metadata you'll use later.\n\nTo optimize speed and reduce the calls, you'll perform batched calls to the embedding OpenAI service.\n\nTo store the quote objects, you will use the `insert_many` method of the collection (one call per batch). When preparing the documents for insertion you will choose suitable field names -- keep in mind, however, that the embedding vector must be the fixed special `$vector` field."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 32e6b973-8d0d-4ea1-a560-8c337a55e750
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Astra DB", "tokens": 4213, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"32e6b973-8d0d-4ea1-a560-8c337a55e750\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"31366239-82b7-497b-9cbd-19245c1a7b03\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"11ecc2b89c2290c6c16b28186eb94888f10182554adae700c5a4235db2a57141\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fa43a680-22ea-4f40-9bee-23dc4158ec10\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"08b1bcbe26c4d3a2826fc02755ff59631c92438f31c58251f759ffef009f77d7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"54f701a4-2950-4c49-8293-644ed6e8f7a4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f0e52235ed9aaab6d406c85b38d226f19fdd457265c29ef37083fac930d16e65\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6124, \"end_char_idx\": 9033, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "ref_doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "chroma:document": "```python\nphilo_dataset = load_dataset(\"datastax/philosopher-quotes\")[\"train\"]\n```\n\nA quick inspection:\n\n\n```python\nprint(\"An example entry:\")\nprint(philo_dataset[16])\n```\n\n    An example entry:\n    {'author': 'aristotle', 'quote': 'Love well, be loved and do something of value.', 'tags': 'love;ethics'}\n\n\nCheck the dataset size:\n\n\n```python\nauthor_count = Counter(entry[\"author\"] for entry in philo_dataset)\nprint(f\"Total: {len(philo_dataset)} quotes. By author:\")\nfor author, count in author_count.most_common():\n    print(f\"    {author:<20}: {count} quotes\")\n```\n\n    Total: 450 quotes. By author:\n        aristotle           : 50 quotes\n        schopenhauer        : 50 quotes\n        spinoza             : 50 quotes\n        hegel               : 50 quotes\n        freud               : 50 quotes\n        nietzsche           : 50 quotes\n        sartre              : 50 quotes\n        plato               : 50 quotes\n        kant                : 50 quotes\n\n\n### Write to the vector collection\n\nYou will compute the embeddings for the quotes and save them into the Vector Store, along with the text itself and the metadata you'll use later.\n\nTo optimize speed and reduce the calls, you'll perform batched calls to the embedding OpenAI service.\n\nTo store the quote objects, you will use the `insert_many` method of the collection (one call per batch). When preparing the documents for insertion you will choose suitable field names -- keep in mind, however, that the embedding vector must be the fixed special `$vector` field.\n\n\n```python\nBATCH_SIZE = 20\n\nnum_batches = ((len(philo_dataset) + BATCH_SIZE - 1) // BATCH_SIZE)\n\nquotes_list = philo_dataset[\"quote\"]\nauthors_list = philo_dataset[\"author\"]\ntags_list = philo_dataset[\"tags\"]\n\nprint(\"Starting to store entries: \", end=\"\")\nfor batch_i in range(num_batches):\n    b_start = batch_i * BATCH_SIZE\n    b_end = (batch_i + 1) * BATCH_SIZE\n    # compute the embedding vectors for this batch\n    b_emb_results = client.embeddings.create(\n        input=quotes_list[b_start : b_end],\n        model=embedding_model_name,\n    )\n    # prepare the documents for insertion\n    b_docs = []\n    for entry_idx, emb_result in zip(range(b_start, b_end), b_emb_results.data):\n        if tags_list[entry_idx]:\n            tags = {\n                tag: True\n                for tag in tags_list[entry_idx].split(\";\")\n            }\n        else:\n            tags = {}\n        b_docs.append({\n            \"quote\": quotes_list[entry_idx],\n            \"$vector\": emb_result.embedding,\n            \"author\": authors_list[entry_idx],\n            \"tags\": tags,\n        })\n    # write to the vector collection\n    collection.insert_many(b_docs)\n    print(f\"[{len(b_docs)}]\", end=\"\")\n\nprint(\"\\nFinished storing entries.\")\n```\n\n    Starting to store entries: [20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][10]\n    Finished storing entries."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 54f701a4-2950-4c49-8293-644ed6e8f7a4
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Astra DB", "tokens": 4213, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"54f701a4-2950-4c49-8293-644ed6e8f7a4\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"31366239-82b7-497b-9cbd-19245c1a7b03\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"11ecc2b89c2290c6c16b28186eb94888f10182554adae700c5a4235db2a57141\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"32e6b973-8d0d-4ea1-a560-8c337a55e750\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3b6b1dcaa389277ed13f0385d356299965b78aa29e7bb0889d249fbb11b697e5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"efea22d3-e749-41c6-ae59-2adb13a8a318\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"200cc493de808d61f5b2b3cfb6ba74fdcd35a0334001a15f19fc5109b60bdab5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7657, \"end_char_idx\": 10815, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "ref_doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "chroma:document": "```python\nBATCH_SIZE = 20\n\nnum_batches = ((len(philo_dataset) + BATCH_SIZE - 1) // BATCH_SIZE)\n\nquotes_list = philo_dataset[\"quote\"]\nauthors_list = philo_dataset[\"author\"]\ntags_list = philo_dataset[\"tags\"]\n\nprint(\"Starting to store entries: \", end=\"\")\nfor batch_i in range(num_batches):\n    b_start = batch_i * BATCH_SIZE\n    b_end = (batch_i + 1) * BATCH_SIZE\n    # compute the embedding vectors for this batch\n    b_emb_results = client.embeddings.create(\n        input=quotes_list[b_start : b_end],\n        model=embedding_model_name,\n    )\n    # prepare the documents for insertion\n    b_docs = []\n    for entry_idx, emb_result in zip(range(b_start, b_end), b_emb_results.data):\n        if tags_list[entry_idx]:\n            tags = {\n                tag: True\n                for tag in tags_list[entry_idx].split(\";\")\n            }\n        else:\n            tags = {}\n        b_docs.append({\n            \"quote\": quotes_list[entry_idx],\n            \"$vector\": emb_result.embedding,\n            \"author\": authors_list[entry_idx],\n            \"tags\": tags,\n        })\n    # write to the vector collection\n    collection.insert_many(b_docs)\n    print(f\"[{len(b_docs)}]\", end=\"\")\n\nprint(\"\\nFinished storing entries.\")\n```\n\n    Starting to store entries: [20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][20][10]\n    Finished storing entries.\n\n\n## Use case 1: **quote search engine**\n\nFor the quote-search functionality, you need first to make the input quote into a vector, and then use it to query the store (besides handling the optional metadata into the search call, that is).\n\nEncapsulate the search-engine functionality into a function for ease of re-use. At its core is the `vector_find` method of the collection:\n\n\n```python\ndef find_quote_and_author(query_quote, n, author=None, tags=None):\n    query_vector = client.embeddings.create(\n        input=[query_quote],\n        model=embedding_model_name,\n    ).data[0].embedding\n    filter_clause = {}\n    if author:\n        filter_clause[\"author\"] = author\n    if tags:\n        filter_clause[\"tags\"] = {}\n        for tag in tags:\n            filter_clause[\"tags\"][tag] = True\n    #\n    results = collection.vector_find(\n        query_vector,\n        limit=n,\n        filter=filter_clause,\n        fields=[\"quote\", \"author\"]\n    )\n    return [\n        (result[\"quote\"], result[\"author\"])\n        for result in results\n    ]\n```\n\n### Putting search to test\n\nPassing just a quote:\n\n\n```python\nfind_quote_and_author(\"We struggle all our life for nothing\", 3)\n```\n\n\n\n\n    [('Life to the great majority is only a constant struggle for mere existence, with the certainty of losing it at last.',\n      'schopenhauer'),\n     ('We give up leisure in order that we may have leisure, just as we go to war in order that we may have peace.',\n      'aristotle'),\n     ('Perhaps the gods are kind to us, by making life more disagreeable as we grow older. In the end death seems less intolerable than the manifold burdens we carry',\n      'freud')]\n\n\n\nSearch restricted to an author:\n\n\n```python\nfind_quote_and_author(\"We struggle all our life for nothing\", 2, author=\"nietzsche\")\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: efea22d3-e749-41c6-ae59-2adb13a8a318
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Astra DB", "tokens": 4213, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"efea22d3-e749-41c6-ae59-2adb13a8a318\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"31366239-82b7-497b-9cbd-19245c1a7b03\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"11ecc2b89c2290c6c16b28186eb94888f10182554adae700c5a4235db2a57141\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"54f701a4-2950-4c49-8293-644ed6e8f7a4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"180f4cc35f952f8f8599bb203c13a7acf455f6c11fabc7caa62d5a6fbd28e90b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bd3b20f2-7b3a-4a9c-9e1d-6a8c7727ec8e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e9bac3a77834727f2424f18bfaf5a527f17c8d8e545a159a1b262d3c24ab7e5c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9415, \"end_char_idx\": 12809, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "ref_doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "chroma:document": "```python\ndef find_quote_and_author(query_quote, n, author=None, tags=None):\n    query_vector = client.embeddings.create(\n        input=[query_quote],\n        model=embedding_model_name,\n    ).data[0].embedding\n    filter_clause = {}\n    if author:\n        filter_clause[\"author\"] = author\n    if tags:\n        filter_clause[\"tags\"] = {}\n        for tag in tags:\n            filter_clause[\"tags\"][tag] = True\n    #\n    results = collection.vector_find(\n        query_vector,\n        limit=n,\n        filter=filter_clause,\n        fields=[\"quote\", \"author\"]\n    )\n    return [\n        (result[\"quote\"], result[\"author\"])\n        for result in results\n    ]\n```\n\n### Putting search to test\n\nPassing just a quote:\n\n\n```python\nfind_quote_and_author(\"We struggle all our life for nothing\", 3)\n```\n\n\n\n\n    [('Life to the great majority is only a constant struggle for mere existence, with the certainty of losing it at last.',\n      'schopenhauer'),\n     ('We give up leisure in order that we may have leisure, just as we go to war in order that we may have peace.',\n      'aristotle'),\n     ('Perhaps the gods are kind to us, by making life more disagreeable as we grow older. In the end death seems less intolerable than the manifold burdens we carry',\n      'freud')]\n\n\n\nSearch restricted to an author:\n\n\n```python\nfind_quote_and_author(\"We struggle all our life for nothing\", 2, author=\"nietzsche\")\n```\n\n\n\n\n    [('To live is to suffer, to survive is to find some meaning in the suffering.',\n      'nietzsche'),\n     ('What makes us heroic?--Confronting simultaneously our supreme suffering and our supreme hope.',\n      'nietzsche')]\n\n\n\nSearch constrained to a tag (out of those saved earlier with the quotes):\n\n\n```python\nfind_quote_and_author(\"We struggle all our life for nothing\", 2, tags=[\"politics\"])\n```\n\n\n\n\n    [('He who seeks equality between unequals seeks an absurdity.', 'spinoza'),\n     ('The people are that part of the state that does not know what it wants.',\n      'hegel')]\n\n\n\n### Cutting out irrelevant results\n\nThe vector similarity search generally returns the vectors that are closest to the query, even if that means results that might be somewhat irrelevant if there's nothing better.\n\nTo keep this issue under control, you can get the actual \"similarity\" between the query and each result, and then implement a cutoff on it, effectively discarding results that are beyond that threshold.\nTuning this threshold correctly is not an easy problem: here, we'll just show you the way.\n\nTo get a feeling on how this works, try the following query and play with the choice of quote and threshold to compare the results. Note that the similarity is returned as the special `$similarity` field in each result document - and it will be returned by default, unless you pass `include_similarity = False` to the search method.\n\n_Note (for the mathematically inclined): this value is **a rescaling between zero and one** of the cosine difference between the vectors, i.e. of the scalar product divided by the product of the norms of the two vectors. In other words, this is 0 for opposite-facing vectors and +1 for parallel vectors. For other measures of similarity (cosine is the default), check the `metric` parameter in `AstraDB.create_collection` and the [documentation on allowed values](https://docs.datastax.com/en/astra-serverless/docs/develop/dev-with-json.html#metric-types)._"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: bd3b20f2-7b3a-4a9c-9e1d-6a8c7727ec8e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Astra DB", "tokens": 4213, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"bd3b20f2-7b3a-4a9c-9e1d-6a8c7727ec8e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"31366239-82b7-497b-9cbd-19245c1a7b03\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"11ecc2b89c2290c6c16b28186eb94888f10182554adae700c5a4235db2a57141\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"efea22d3-e749-41c6-ae59-2adb13a8a318\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"432dd9819d4fb006a774e9be398fa31ffaac798be99f2f726411c48119c0f9d3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"27cd0e4a-6bc6-4b93-b5fa-f699666d75db\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f90b576f53a61e402f47adb759bc54571029d3af9550cc33d991be8c8c56ef71\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11050, \"end_char_idx\": 14070, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "ref_doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "chroma:document": "Search constrained to a tag (out of those saved earlier with the quotes):\n\n\n```python\nfind_quote_and_author(\"We struggle all our life for nothing\", 2, tags=[\"politics\"])\n```\n\n\n\n\n    [('He who seeks equality between unequals seeks an absurdity.', 'spinoza'),\n     ('The people are that part of the state that does not know what it wants.',\n      'hegel')]\n\n\n\n### Cutting out irrelevant results\n\nThe vector similarity search generally returns the vectors that are closest to the query, even if that means results that might be somewhat irrelevant if there's nothing better.\n\nTo keep this issue under control, you can get the actual \"similarity\" between the query and each result, and then implement a cutoff on it, effectively discarding results that are beyond that threshold.\nTuning this threshold correctly is not an easy problem: here, we'll just show you the way.\n\nTo get a feeling on how this works, try the following query and play with the choice of quote and threshold to compare the results. Note that the similarity is returned as the special `$similarity` field in each result document - and it will be returned by default, unless you pass `include_similarity = False` to the search method.\n\n_Note (for the mathematically inclined): this value is **a rescaling between zero and one** of the cosine difference between the vectors, i.e. of the scalar product divided by the product of the norms of the two vectors. In other words, this is 0 for opposite-facing vectors and +1 for parallel vectors. For other measures of similarity (cosine is the default), check the `metric` parameter in `AstraDB.create_collection` and the [documentation on allowed values](https://docs.datastax.com/en/astra-serverless/docs/develop/dev-with-json.html#metric-types)._\n\n\n```python\nquote = \"Animals are our equals.\"\n# quote = \"Be good.\"\n# quote = \"This teapot is strange.\"\n\nmetric_threshold = 0.92\n\nquote_vector = client.embeddings.create(\n    input=[quote],\n    model=embedding_model_name,\n).data[0].embedding\n\nresults_full = collection.vector_find(\n    quote_vector,\n    limit=8,\n    fields=[\"quote\"]\n)\nresults = [res for res in results_full if res[\"$similarity\"] >= metric_threshold]\n\nprint(f\"{len(results)} quotes within the threshold:\")\nfor idx, result in enumerate(results):\n    print(f\"    {idx}. [similarity={result['$similarity']:.3f}] \\\"{result['quote'][:70]}...\\\"\")\n```\n\n    3 quotes within the threshold:\n        0. [similarity=0.927] \"The assumption that animals are without rights, and the illusion that ...\"\n        1. [similarity=0.922] \"Animals are in possession of themselves; their soul is in possession o...\"\n        2. [similarity=0.920] \"At his best, man is the noblest of all animals; separated from law and...\"\n\n\n## Use case 2: **quote generator**\n\nFor this task you need another component from OpenAI, namely an LLM to generate the quote for us (based on input obtained by querying the Vector Store).\n\nYou also need a template for the prompt that will be filled for the generate-quote LLM completion task."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 27cd0e4a-6bc6-4b93-b5fa-f699666d75db
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Astra DB", "tokens": 4213, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"27cd0e4a-6bc6-4b93-b5fa-f699666d75db\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"31366239-82b7-497b-9cbd-19245c1a7b03\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"11ecc2b89c2290c6c16b28186eb94888f10182554adae700c5a4235db2a57141\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bd3b20f2-7b3a-4a9c-9e1d-6a8c7727ec8e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"5fbf00135d71e7ef4668250fd60e3cd6d8cea0b35ce4c2f745dac24360f8b8be\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"267f7a12-d3fd-4976-9b2e-2963f51bb1bf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"87d649dcc77f63aabadcd8955cdbafc2e494ca55c06478ba4ce43b4092fd36dc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12812, \"end_char_idx\": 15716, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "ref_doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "chroma:document": "```python\nquote = \"Animals are our equals.\"\n# quote = \"Be good.\"\n# quote = \"This teapot is strange.\"\n\nmetric_threshold = 0.92\n\nquote_vector = client.embeddings.create(\n    input=[quote],\n    model=embedding_model_name,\n).data[0].embedding\n\nresults_full = collection.vector_find(\n    quote_vector,\n    limit=8,\n    fields=[\"quote\"]\n)\nresults = [res for res in results_full if res[\"$similarity\"] >= metric_threshold]\n\nprint(f\"{len(results)} quotes within the threshold:\")\nfor idx, result in enumerate(results):\n    print(f\"    {idx}. [similarity={result['$similarity']:.3f}] \\\"{result['quote'][:70]}...\\\"\")\n```\n\n    3 quotes within the threshold:\n        0. [similarity=0.927] \"The assumption that animals are without rights, and the illusion that ...\"\n        1. [similarity=0.922] \"Animals are in possession of themselves; their soul is in possession o...\"\n        2. [similarity=0.920] \"At his best, man is the noblest of all animals; separated from law and...\"\n\n\n## Use case 2: **quote generator**\n\nFor this task you need another component from OpenAI, namely an LLM to generate the quote for us (based on input obtained by querying the Vector Store).\n\nYou also need a template for the prompt that will be filled for the generate-quote LLM completion task.\n\n\n```python\ncompletion_model_name = \"gpt-3.5-turbo\"\n\ngeneration_prompt_template = \"\"\"\"Generate a single short philosophical quote on the given topic,\nsimilar in spirit and form to the provided actual example quotes.\nDo not exceed 20-30 words in your quote.\n\nREFERENCE TOPIC: \"{topic}\"\n\nACTUAL EXAMPLES:\n{examples}\n\"\"\"\n```\n\nLike for search, this functionality is best wrapped into a handy function (which internally uses search):\n\n\n```python\ndef generate_quote(topic, n=2, author=None, tags=None):\n    quotes = find_quote_and_author(query_quote=topic, n=n, author=author, tags=tags)\n    if quotes:\n        prompt = generation_prompt_template.format(\n            topic=topic,\n            examples=\"\\n\".join(f\"  - {quote[0]}\" for quote in quotes),\n        )\n        # a little logging:\n        print(\"** quotes found:\")\n        for q, a in quotes:\n            print(f\"**    - {q} ({a})\")\n        print(\"** end of logging\")\n        #\n        response = client.chat.completions.create(\n            model=completion_model_name,\n            messages=[{\"role\": \"user\", \"content\": prompt}],\n            temperature=0.7,\n            max_tokens=320,\n        )\n        return response.choices[0].message.content.replace('\"', '').strip()\n    else:\n        print(\"** no quotes found.\")\n        return None\n```\n\n_Note: similar to the case of the embedding computation, the code for the Chat Completion API would be slightly different for OpenAI prior to v1.0._\n\n#### Putting quote generation to test\n\nJust passing a text (a \"quote\", but one can actually just suggest a topic since its vector embedding will still end up at the right place in the vector space):"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 267f7a12-d3fd-4976-9b2e-2963f51bb1bf
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Astra DB", "tokens": 4213, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"267f7a12-d3fd-4976-9b2e-2963f51bb1bf\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"31366239-82b7-497b-9cbd-19245c1a7b03\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"11ecc2b89c2290c6c16b28186eb94888f10182554adae700c5a4235db2a57141\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"27cd0e4a-6bc6-4b93-b5fa-f699666d75db\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a8d7bfb6d4a3baadc8848d8022e1d89507a7f3c5b08a345c9a7d631db51eeb94\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b9401ff3-9dc5-41ea-9ea1-2b5f1fb95082\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3e1b18dd4edc7b304f7549d0b75288c20dde2dab2b067974e6388fb9dfcb23c4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14073, \"end_char_idx\": 17573, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "ref_doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "chroma:document": "```python\ncompletion_model_name = \"gpt-3.5-turbo\"\n\ngeneration_prompt_template = \"\"\"\"Generate a single short philosophical quote on the given topic,\nsimilar in spirit and form to the provided actual example quotes.\nDo not exceed 20-30 words in your quote.\n\nREFERENCE TOPIC: \"{topic}\"\n\nACTUAL EXAMPLES:\n{examples}\n\"\"\"\n```\n\nLike for search, this functionality is best wrapped into a handy function (which internally uses search):\n\n\n```python\ndef generate_quote(topic, n=2, author=None, tags=None):\n    quotes = find_quote_and_author(query_quote=topic, n=n, author=author, tags=tags)\n    if quotes:\n        prompt = generation_prompt_template.format(\n            topic=topic,\n            examples=\"\\n\".join(f\"  - {quote[0]}\" for quote in quotes),\n        )\n        # a little logging:\n        print(\"** quotes found:\")\n        for q, a in quotes:\n            print(f\"**    - {q} ({a})\")\n        print(\"** end of logging\")\n        #\n        response = client.chat.completions.create(\n            model=completion_model_name,\n            messages=[{\"role\": \"user\", \"content\": prompt}],\n            temperature=0.7,\n            max_tokens=320,\n        )\n        return response.choices[0].message.content.replace('\"', '').strip()\n    else:\n        print(\"** no quotes found.\")\n        return None\n```\n\n_Note: similar to the case of the embedding computation, the code for the Chat Completion API would be slightly different for OpenAI prior to v1.0._\n\n#### Putting quote generation to test\n\nJust passing a text (a \"quote\", but one can actually just suggest a topic since its vector embedding will still end up at the right place in the vector space):\n\n\n```python\nq_topic = generate_quote(\"politics and virtue\")\nprint(\"\\nA new generated quote:\")\nprint(q_topic)\n```\n\n    ** quotes found:\n    **    - Happiness is the reward of virtue. (aristotle)\n    **    - Our moral virtues benefit mainly other people; intellectual virtues, on the other hand, benefit primarily ourselves; therefore the former make us universally popular, the latter unpopular. (schopenhauer)\n    ** end of logging\n    \n    A new generated quote:\n    True politics lies in the virtuous pursuit of justice, for it is through virtue that we build a better world for all.\n\n\nUse inspiration from just a single philosopher:\n\n\n```python\nq_topic = generate_quote(\"animals\", author=\"schopenhauer\")\nprint(\"\\nA new generated quote:\")\nprint(q_topic)\n```\n\n    ** quotes found:\n    **    - Because Christian morality leaves animals out of account, they are at once outlawed in philosophical morals; they are mere 'things,' mere means to any ends whatsoever. They can therefore be used for vivisection, hunting, coursing, bullfights, and horse racing, and can be whipped to death as they struggle along with heavy carts of stone. Shame on such a morality that is worthy of pariahs, and that fails to recognize the eternal essence that exists in every living thing, and shines forth with inscrutable significance from all eyes that see the sun! (schopenhauer)\n    **    - The assumption that animals are without rights, and the illusion that our treatment of them has no moral significance, is a positively outrageous example of Western crudity and barbarity. Universal compassion is the only guarantee of morality. (schopenhauer)\n    ** end of logging\n    \n    A new generated quote:\n    Excluding animals from ethical consideration reveals a moral blindness that allows for their exploitation and suffering. True morality embraces universal compassion."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: b9401ff3-9dc5-41ea-9ea1-2b5f1fb95082
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Astra DB", "tokens": 4213, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"b9401ff3-9dc5-41ea-9ea1-2b5f1fb95082\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"31366239-82b7-497b-9cbd-19245c1a7b03\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"11ecc2b89c2290c6c16b28186eb94888f10182554adae700c5a4235db2a57141\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"267f7a12-d3fd-4976-9b2e-2963f51bb1bf\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_AstraPy.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Astra DB\", \"tokens\": 4213, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"2f2b0c1eef644bdac0b5ba8eece7b4ebe6e954b8d7eea624d0e72b30fa4d39e8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15719, \"end_char_idx\": 17814, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "ref_doc_id": "31366239-82b7-497b-9cbd-19245c1a7b03", "chroma:document": "```python\nq_topic = generate_quote(\"politics and virtue\")\nprint(\"\\nA new generated quote:\")\nprint(q_topic)\n```\n\n    ** quotes found:\n    **    - Happiness is the reward of virtue. (aristotle)\n    **    - Our moral virtues benefit mainly other people; intellectual virtues, on the other hand, benefit primarily ourselves; therefore the former make us universally popular, the latter unpopular. (schopenhauer)\n    ** end of logging\n    \n    A new generated quote:\n    True politics lies in the virtuous pursuit of justice, for it is through virtue that we build a better world for all.\n\n\nUse inspiration from just a single philosopher:\n\n\n```python\nq_topic = generate_quote(\"animals\", author=\"schopenhauer\")\nprint(\"\\nA new generated quote:\")\nprint(q_topic)\n```\n\n    ** quotes found:\n    **    - Because Christian morality leaves animals out of account, they are at once outlawed in philosophical morals; they are mere 'things,' mere means to any ends whatsoever. They can therefore be used for vivisection, hunting, coursing, bullfights, and horse racing, and can be whipped to death as they struggle along with heavy carts of stone. Shame on such a morality that is worthy of pariahs, and that fails to recognize the eternal essence that exists in every living thing, and shines forth with inscrutable significance from all eyes that see the sun! (schopenhauer)\n    **    - The assumption that animals are without rights, and the illusion that our treatment of them has no moral significance, is a positively outrageous example of Western crudity and barbarity. Universal compassion is the only guarantee of morality. (schopenhauer)\n    ** end of logging\n    \n    A new generated quote:\n    Excluding animals from ethical consideration reveals a moral blindness that allows for their exploitation and suffering. True morality embraces universal compassion.\n\n\n## Cleanup\n\nIf you want to remove all resources used for this demo, run this cell (_warning: this will irreversibly delete the collection and its data!_):\n\n\n```python\nastra_db.delete_collection(coll_name)\n```\n\n\n\n\n    {'status': {'ok': 1}}"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: da742d8b-bba2-470d-ada6-a02a238f18b6
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB", "tokens": 7788, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"da742d8b-bba2-470d-ada6-a02a238f18b6\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb5efc1b-a20b-44a8-a5a4-548d3f7d961b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c612319605ec9e5f26c9165ca32a7036b2f07e2ac5ee91dc11787d133a1fcb44\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c10bd153-137c-4c30-99c0-d25dcc336e50\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"18f245ededb9bdabd0c5caae93afbc75bcf293d33731a39326b9e92ef3742d16\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3309, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "ref_doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "chroma:document": "# Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\n\n### CQL Version\n\nIn this quickstart you will learn how to build a \"philosophy quote finder & generator\" using OpenAI's vector embeddings and [Apache Cassandra\u00ae](https://cassandra.apache.org), or equivalently DataStax [Astra DB through CQL](https://docs.datastax.com/en/astra-serverless/docs/vector-search/quickstart.html), as the vector store for data persistence.\n\nThe basic workflow of this notebook is outlined below. You will evaluate and store the vector embeddings for a number of quotes by famous philosophers, use them to build a powerful search engine and, after that, even a generator of new quotes!\n\nThe notebook exemplifies some of the standard usage patterns of vector search -- while showing how easy is it to get started with the vector capabilities of [Cassandra](https://cassandra.apache.org/doc/trunk/cassandra/vector-search/overview.html) / [Astra DB through CQL](https://docs.datastax.com/en/astra-serverless/docs/vector-search/quickstart.html).\n\nFor a background on using vector search and text embeddings to build a question-answering system, please check out this excellent hands-on notebook: [Question answering using embeddings](https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb).\n\n#### _Choose-your-framework_\n\nPlease note that this notebook uses the [Cassandra drivers](https://docs.datastax.com/en/developer/python-driver/latest/) and runs CQL (Cassandra Query Language) statements directly, but we cover other choices of technology to accomplish the same task. Check out this folder's [README](https://github.com/openai/openai-cookbook/tree/main/examples/vector_databases/cassandra_astradb) for other options. This notebook can run either as a Colab notebook or as a regular Jupyter notebook.\n\nTable of contents:\n- Setup\n- Get DB connection\n- Connect to OpenAI\n- Load quotes into the Vector Store\n- Use case 1: **quote search engine**\n- Use case 2: **quote generator**\n- (Optional) exploit partitioning in the Vector Store\n\n### How it works\n\n**Indexing**\n\nEach quote is made into an embedding vector with OpenAI's `Embedding`. These are saved in the Vector Store for later use in searching. Some metadata, including the author's name and a few other pre-computed tags, are stored alongside, to allow for search customization.\n\n![1_vector_indexing_cql](https://user-images.githubusercontent.com/14221764/282437237-1e763166-a863-4332-99b8-323ba23d1b87.png)\n\n**Search**\n\nTo find a quote similar to the provided search quote, the latter is made into an embedding vector on the fly, and this vector is used to query the store for similar vectors ... i.e. similar quotes that were previously indexed. The search can optionally be constrained by additional metadata (\"find me quotes by Spinoza similar to this one ...\").\n\n![2_vector_search_cql](https://user-images.githubusercontent.com/14221764/282437291-85335612-a845-444e-bed7-e4cf014a9f17.png)\n\nThe key point here is that \"quotes similar in content\" translates, in vector space, to vectors that are metrically close to each other: thus, vector similarity search effectively implements semantic similarity. _This is the key reason vector embeddings are so powerful._\n\nThe sketch below tries to convey this idea."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: c10bd153-137c-4c30-99c0-d25dcc336e50
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB", "tokens": 7788, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"c10bd153-137c-4c30-99c0-d25dcc336e50\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb5efc1b-a20b-44a8-a5a4-548d3f7d961b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c612319605ec9e5f26c9165ca32a7036b2f07e2ac5ee91dc11787d133a1fcb44\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"da742d8b-bba2-470d-ada6-a02a238f18b6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"f3741748ae6e0e8d448e5b5b9da843d342127003da9af3adb804c6c0311eee04\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"45544280-68cb-4b08-b1a7-6c29dd98acf6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4ba7476368c73702a8b1893c7f1e17ce54d968d96282289ce6bc8d853e770d85\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1762, \"end_char_idx\": 4721, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "ref_doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "chroma:document": "This notebook can run either as a Colab notebook or as a regular Jupyter notebook.\n\nTable of contents:\n- Setup\n- Get DB connection\n- Connect to OpenAI\n- Load quotes into the Vector Store\n- Use case 1: **quote search engine**\n- Use case 2: **quote generator**\n- (Optional) exploit partitioning in the Vector Store\n\n### How it works\n\n**Indexing**\n\nEach quote is made into an embedding vector with OpenAI's `Embedding`. These are saved in the Vector Store for later use in searching. Some metadata, including the author's name and a few other pre-computed tags, are stored alongside, to allow for search customization.\n\n![1_vector_indexing_cql](https://user-images.githubusercontent.com/14221764/282437237-1e763166-a863-4332-99b8-323ba23d1b87.png)\n\n**Search**\n\nTo find a quote similar to the provided search quote, the latter is made into an embedding vector on the fly, and this vector is used to query the store for similar vectors ... i.e. similar quotes that were previously indexed. The search can optionally be constrained by additional metadata (\"find me quotes by Spinoza similar to this one ...\").\n\n![2_vector_search_cql](https://user-images.githubusercontent.com/14221764/282437291-85335612-a845-444e-bed7-e4cf014a9f17.png)\n\nThe key point here is that \"quotes similar in content\" translates, in vector space, to vectors that are metrically close to each other: thus, vector similarity search effectively implements semantic similarity. _This is the key reason vector embeddings are so powerful._\n\nThe sketch below tries to convey this idea. Each quote, once it's made into a vector, is a point in space. Well, in this case it's on a sphere, since OpenAI's embedding vectors, as most others, are normalized to _unit length_. Oh, and the sphere is actually not three-dimensional, rather 1536-dimensional!\n\nSo, in essence, a similarity search in vector space returns the vectors that are closest to the query vector:\n\n![3_vector_space](https://user-images.githubusercontent.com/14221764/262321363-c8c625c1-8be9-450e-8c68-b1ed518f990d.png)\n\n**Generation**\n\nGiven a suggestion (a topic or a tentative quote), the search step is performed, and the first returned results (quotes) are fed into an LLM prompt which asks the generative model to invent a new text along the lines of the passed examples _and_ the initial suggestion.\n\n![4_quote_generation](https://user-images.githubusercontent.com/14221764/282437321-881bd273-3443-4987-9a11-350d3288dd8e.png)\n\n## Setup\n\nInstall and import the necessary dependencies:\n\n\n```python\n!pip install --quiet \"cassandra-driver>=0.28.0\" \"openai>=1.0.0\" datasets\n```\n\n\n```python\nimport os\nfrom uuid import uuid4\nfrom getpass import getpass\nfrom collections import Counter\n\nfrom cassandra.cluster import Cluster\nfrom cassandra.auth import PlainTextAuthProvider\n\nimport openai\nfrom datasets import load_dataset\n```\n\n_Don't mind the next cell too much, we need it to detect Colabs and let you upload the SCB file (see below):_"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 45544280-68cb-4b08-b1a7-6c29dd98acf6
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB", "tokens": 7788, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"45544280-68cb-4b08-b1a7-6c29dd98acf6\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb5efc1b-a20b-44a8-a5a4-548d3f7d961b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c612319605ec9e5f26c9165ca32a7036b2f07e2ac5ee91dc11787d133a1fcb44\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c10bd153-137c-4c30-99c0-d25dcc336e50\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"6539cb1f1117e70225127a46c9530f5673fd0c90d9b80005da7c26a1610b17bd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a2cc1aa1-3ec0-4bee-9c49-455dff5e130b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b6ba24bff7420aa87fdfb8c7f0d1ed3b2dff57ba0744df5be5efe9487d9fdb99\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3205, \"end_char_idx\": 6192, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "ref_doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "chroma:document": "_This is the key reason vector embeddings are so powerful._\n\nThe sketch below tries to convey this idea. Each quote, once it's made into a vector, is a point in space. Well, in this case it's on a sphere, since OpenAI's embedding vectors, as most others, are normalized to _unit length_. Oh, and the sphere is actually not three-dimensional, rather 1536-dimensional!\n\nSo, in essence, a similarity search in vector space returns the vectors that are closest to the query vector:\n\n![3_vector_space](https://user-images.githubusercontent.com/14221764/262321363-c8c625c1-8be9-450e-8c68-b1ed518f990d.png)\n\n**Generation**\n\nGiven a suggestion (a topic or a tentative quote), the search step is performed, and the first returned results (quotes) are fed into an LLM prompt which asks the generative model to invent a new text along the lines of the passed examples _and_ the initial suggestion.\n\n![4_quote_generation](https://user-images.githubusercontent.com/14221764/282437321-881bd273-3443-4987-9a11-350d3288dd8e.png)\n\n## Setup\n\nInstall and import the necessary dependencies:\n\n\n```python\n!pip install --quiet \"cassandra-driver>=0.28.0\" \"openai>=1.0.0\" datasets\n```\n\n\n```python\nimport os\nfrom uuid import uuid4\nfrom getpass import getpass\nfrom collections import Counter\n\nfrom cassandra.cluster import Cluster\nfrom cassandra.auth import PlainTextAuthProvider\n\nimport openai\nfrom datasets import load_dataset\n```\n\n_Don't mind the next cell too much, we need it to detect Colabs and let you upload the SCB file (see below):_\n\n\n```python\ntry:\n    from google.colab import files\n    IS_COLAB = True\nexcept ModuleNotFoundError:\n    IS_COLAB = False\n```\n\n## Get DB connection\n\nA couple of secrets are required to create a `Session` object (a connection to your Astra DB instance).\n\n_(Note: some steps will be slightly different on Google Colab and on local Jupyter, that's why the notebook will detect the runtime type.)_\n\n\n```python\n# Your database's Secure Connect Bundle zip file is needed:\nif IS_COLAB:\n    print('Please upload your Secure Connect Bundle zipfile: ')\n    uploaded = files.upload()\n    if uploaded:\n        astraBundleFileTitle = list(uploaded.keys())[0]\n        ASTRA_DB_SECURE_BUNDLE_PATH = os.path.join(os.getcwd(), astraBundleFileTitle)\n    else:\n        raise ValueError(\n            'Cannot proceed without Secure Connect Bundle. Please re-run the cell.'\n        )\nelse:\n    # you are running a local-jupyter notebook:\n    ASTRA_DB_SECURE_BUNDLE_PATH = input(\"Please provide the full path to your Secure Connect Bundle zipfile: \")\n\nASTRA_DB_APPLICATION_TOKEN = getpass(\"Please provide your Database Token ('AstraCS:...' string): \")\nASTRA_DB_KEYSPACE = input(\"Please provide the Keyspace name for your Database: \")\n```\n\n    Please provide the full path to your Secure Connect Bundle zipfile:  /path/to/secure-connect-DatabaseName.zip\n    Please provide your Database Token ('AstraCS:...' string):  \u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\n    Please provide the Keyspace name for your Database:  my_keyspace"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: a2cc1aa1-3ec0-4bee-9c49-455dff5e130b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB", "tokens": 7788, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a2cc1aa1-3ec0-4bee-9c49-455dff5e130b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb5efc1b-a20b-44a8-a5a4-548d3f7d961b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c612319605ec9e5f26c9165ca32a7036b2f07e2ac5ee91dc11787d133a1fcb44\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"45544280-68cb-4b08-b1a7-6c29dd98acf6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"322ad0e0a9abe36c8eaac551c4453b3a10480c50fa5e69c04d1eb11c02078425\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3904d728-a89a-4cb3-ab8e-f51712f93191\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"846fbd7358330c2354f0aed084f8166087fd00dff881eee1b6423ef6cebcc020\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4724, \"end_char_idx\": 7917, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "ref_doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "chroma:document": "```python\ntry:\n    from google.colab import files\n    IS_COLAB = True\nexcept ModuleNotFoundError:\n    IS_COLAB = False\n```\n\n## Get DB connection\n\nA couple of secrets are required to create a `Session` object (a connection to your Astra DB instance).\n\n_(Note: some steps will be slightly different on Google Colab and on local Jupyter, that's why the notebook will detect the runtime type.)_\n\n\n```python\n# Your database's Secure Connect Bundle zip file is needed:\nif IS_COLAB:\n    print('Please upload your Secure Connect Bundle zipfile: ')\n    uploaded = files.upload()\n    if uploaded:\n        astraBundleFileTitle = list(uploaded.keys())[0]\n        ASTRA_DB_SECURE_BUNDLE_PATH = os.path.join(os.getcwd(), astraBundleFileTitle)\n    else:\n        raise ValueError(\n            'Cannot proceed without Secure Connect Bundle. Please re-run the cell.'\n        )\nelse:\n    # you are running a local-jupyter notebook:\n    ASTRA_DB_SECURE_BUNDLE_PATH = input(\"Please provide the full path to your Secure Connect Bundle zipfile: \")\n\nASTRA_DB_APPLICATION_TOKEN = getpass(\"Please provide your Database Token ('AstraCS:...' string): \")\nASTRA_DB_KEYSPACE = input(\"Please provide the Keyspace name for your Database: \")\n```\n\n    Please provide the full path to your Secure Connect Bundle zipfile:  /path/to/secure-connect-DatabaseName.zip\n    Please provide your Database Token ('AstraCS:...' string):  \u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\n    Please provide the Keyspace name for your Database:  my_keyspace\n\n\n### Creation of the DB connection\n\nThis is how you create a connection to Astra DB:\n\n_(Incidentally, you could also use any Cassandra cluster (as long as it provides Vector capabilities), just by [changing the parameters](https://docs.datastax.com/en/developer/python-driver/latest/getting_started/#connecting-to-cassandra) to the following `Cluster` instantiation.)_\n\n\n```python\n# Don't mind the \"Closing connection\" error after \"downgrading protocol...\" messages you may see,\n# it is really just a warning: the connection will work smoothly.\ncluster = Cluster(\n    cloud={\n        \"secure_connect_bundle\": ASTRA_DB_SECURE_BUNDLE_PATH,\n    },\n    auth_provider=PlainTextAuthProvider(\n        \"token\",\n        ASTRA_DB_APPLICATION_TOKEN,\n    ),\n)\n\nsession = cluster.connect()\nkeyspace = ASTRA_DB_KEYSPACE\n```\n\n### Creation of the Vector table in CQL\n\nYou need a table which support vectors and is equipped with metadata. Call it \"philosophers_cql\".\n\nEach row will store: a quote, its vector embedding, the quote author and a set of \"tags\". You also need a primary key to ensure uniqueness of rows.\n\nThe following is the full CQL command that creates the table (check out [this page](https://docs.datastax.com/en/dse/6.7/cql/cql/cqlQuickReference.html) for more on the CQL syntax of this and the following statements):\n\n\n```python\ncreate_table_statement = f\"\"\"CREATE TABLE IF NOT EXISTS {keyspace}.philosophers_cql (\n    quote_id UUID PRIMARY KEY,\n    body TEXT,\n    embedding_vector VECTOR<FLOAT, 1536>,\n    author TEXT,\n    tags SET<TEXT>\n);\"\"\"\n```\n\nPass this statement to your database Session to execute it:\n\n\n```python\nsession.execute(create_table_statement)\n```\n\n\n\n\n    <cassandra.cluster.ResultSet at 0x7feee37b3460>"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 3904d728-a89a-4cb3-ab8e-f51712f93191
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB", "tokens": 7788, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"3904d728-a89a-4cb3-ab8e-f51712f93191\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb5efc1b-a20b-44a8-a5a4-548d3f7d961b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c612319605ec9e5f26c9165ca32a7036b2f07e2ac5ee91dc11787d133a1fcb44\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a2cc1aa1-3ec0-4bee-9c49-455dff5e130b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"e47db5e16ea8b2a4de492c4092208819fd932baf01b98fceb0335fc3b0bd938d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"1000e3e6-3c9f-483d-8e71-712bab5e9873\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"98bd0cbdc96ccb5b62bf6e326b1a22570532a217d06593394b9141832f311119\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6565, \"end_char_idx\": 9807, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "ref_doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "chroma:document": "```python\n# Don't mind the \"Closing connection\" error after \"downgrading protocol...\" messages you may see,\n# it is really just a warning: the connection will work smoothly.\ncluster = Cluster(\n    cloud={\n        \"secure_connect_bundle\": ASTRA_DB_SECURE_BUNDLE_PATH,\n    },\n    auth_provider=PlainTextAuthProvider(\n        \"token\",\n        ASTRA_DB_APPLICATION_TOKEN,\n    ),\n)\n\nsession = cluster.connect()\nkeyspace = ASTRA_DB_KEYSPACE\n```\n\n### Creation of the Vector table in CQL\n\nYou need a table which support vectors and is equipped with metadata. Call it \"philosophers_cql\".\n\nEach row will store: a quote, its vector embedding, the quote author and a set of \"tags\". You also need a primary key to ensure uniqueness of rows.\n\nThe following is the full CQL command that creates the table (check out [this page](https://docs.datastax.com/en/dse/6.7/cql/cql/cqlQuickReference.html) for more on the CQL syntax of this and the following statements):\n\n\n```python\ncreate_table_statement = f\"\"\"CREATE TABLE IF NOT EXISTS {keyspace}.philosophers_cql (\n    quote_id UUID PRIMARY KEY,\n    body TEXT,\n    embedding_vector VECTOR<FLOAT, 1536>,\n    author TEXT,\n    tags SET<TEXT>\n);\"\"\"\n```\n\nPass this statement to your database Session to execute it:\n\n\n```python\nsession.execute(create_table_statement)\n```\n\n\n\n\n    <cassandra.cluster.ResultSet at 0x7feee37b3460>\n\n\n\n#### Add a vector index for ANN search\n\nIn order to run ANN (approximate-nearest-neighbor) searches on the vectors in the table, you need to create a specific index on the `embedding_vector` column.\n\n_When creating the index, you can [optionally choose](https://docs.datastax.com/en/astra-serverless/docs/vector-search/cql.html#_create_the_vector_schema_and_load_the_data_into_the_database) the \"similarity function\" used to compute vector distances: since for unit-length vectors (such as those from OpenAI) the \"cosine difference\" is the same as the \"dot product\", you'll use the latter which is computationally less expensive._\n\nRun this CQL statement:\n\n\n```python\ncreate_vector_index_statement = f\"\"\"CREATE CUSTOM INDEX IF NOT EXISTS idx_embedding_vector\n    ON {keyspace}.philosophers_cql (embedding_vector)\n    USING 'org.apache.cassandra.index.sai.StorageAttachedIndex'\n    WITH OPTIONS = {{'similarity_function' : 'dot_product'}};\n\"\"\"\n# Note: the double '{{' and '}}' are just the F-string escape sequence for '{' and '}'\n\nsession.execute(create_vector_index_statement)\n```\n\n\n\n\n    <cassandra.cluster.ResultSet at 0x7feeefd3da00>\n\n\n\n#### Add indexes for author and tag filtering\n\nThat is enough to run vector searches on the table ... but you want to be able to optionally specify an author and/or some tags to restrict the quote search. Create two other indexes to support this:\n\n\n```python\ncreate_author_index_statement = f\"\"\"CREATE CUSTOM INDEX IF NOT EXISTS idx_author\n    ON {keyspace}.philosophers_cql (author)\n    USING 'org.apache.cassandra.index.sai.StorageAttachedIndex';\n\"\"\"\nsession.execute(create_author_index_statement)\n\ncreate_tags_index_statement = f\"\"\"CREATE CUSTOM INDEX IF NOT EXISTS idx_tags\n    ON {keyspace}.philosophers_cql (VALUES(tags))\n    USING 'org.apache.cassandra.index.sai.StorageAttachedIndex';\n\"\"\"\nsession.execute(create_tags_index_statement)\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 1000e3e6-3c9f-483d-8e71-712bab5e9873
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB", "tokens": 7788, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"1000e3e6-3c9f-483d-8e71-712bab5e9873\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb5efc1b-a20b-44a8-a5a4-548d3f7d961b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c612319605ec9e5f26c9165ca32a7036b2f07e2ac5ee91dc11787d133a1fcb44\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3904d728-a89a-4cb3-ab8e-f51712f93191\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"9104e5a7515dae27406e8134c22657923cb42a5c92ff24b37c148bb3b2fe3e83\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"73dd296b-512c-4dfd-8a20-85a7aa336e0b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8a90b60290cc805efdc757e3284a312f389a183e886aa73742ee3e95b4fbff95\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8579, \"end_char_idx\": 11519, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "ref_doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "chroma:document": "```python\ncreate_vector_index_statement = f\"\"\"CREATE CUSTOM INDEX IF NOT EXISTS idx_embedding_vector\n    ON {keyspace}.philosophers_cql (embedding_vector)\n    USING 'org.apache.cassandra.index.sai.StorageAttachedIndex'\n    WITH OPTIONS = {{'similarity_function' : 'dot_product'}};\n\"\"\"\n# Note: the double '{{' and '}}' are just the F-string escape sequence for '{' and '}'\n\nsession.execute(create_vector_index_statement)\n```\n\n\n\n\n    <cassandra.cluster.ResultSet at 0x7feeefd3da00>\n\n\n\n#### Add indexes for author and tag filtering\n\nThat is enough to run vector searches on the table ... but you want to be able to optionally specify an author and/or some tags to restrict the quote search. Create two other indexes to support this:\n\n\n```python\ncreate_author_index_statement = f\"\"\"CREATE CUSTOM INDEX IF NOT EXISTS idx_author\n    ON {keyspace}.philosophers_cql (author)\n    USING 'org.apache.cassandra.index.sai.StorageAttachedIndex';\n\"\"\"\nsession.execute(create_author_index_statement)\n\ncreate_tags_index_statement = f\"\"\"CREATE CUSTOM INDEX IF NOT EXISTS idx_tags\n    ON {keyspace}.philosophers_cql (VALUES(tags))\n    USING 'org.apache.cassandra.index.sai.StorageAttachedIndex';\n\"\"\"\nsession.execute(create_tags_index_statement)\n```\n\n\n\n\n    <cassandra.cluster.ResultSet at 0x7fef2c64af70>\n\n\n\n## Connect to OpenAI\n\n### Set up your secret key\n\n\n```python\nOPENAI_API_KEY = getpass(\"Please enter your OpenAI API Key: \")\n```\n\n    Please enter your OpenAI API Key:  \u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\n\n\n### A test call for embeddings\n\nQuickly check how one can get the embedding vectors for a list of input texts:\n\n\n```python\nclient = openai.OpenAI(api_key=OPENAI_API_KEY)\nembedding_model_name = \"text-embedding-3-small\"\n\nresult = client.embeddings.create(\n    input=[\n        \"This is a sentence\",\n        \"A second sentence\"\n    ],\n    model=embedding_model_name,\n)\n```\n\n_Note: the above is the syntax for OpenAI v1.0+. If using previous versions, the code to get the embeddings will look different._\n\n\n```python\nprint(f\"len(result.data)              = {len(result.data)}\")\nprint(f\"result.data[1].embedding      = {str(result.data[1].embedding)[:55]}...\")\nprint(f\"len(result.data[1].embedding) = {len(result.data[1].embedding)}\")\n```\n\n    len(result.data)              = 2\n    result.data[1].embedding      = [-0.0108176339417696, 0.0013546717818826437, 0.00362232...\n    len(result.data[1].embedding) = 1536\n\n\n## Load quotes into the Vector Store\n\nGet a dataset with the quotes. _(We adapted and augmented the data from [this Kaggle dataset](https://www.kaggle.com/datasets/mertbozkurt5/quotes-by-philosophers), ready to use in this demo.)_\n\n\n```python\nphilo_dataset = load_dataset(\"datastax/philosopher-quotes\")[\"train\"]\n```\n\nA quick inspection:\n\n\n```python\nprint(\"An example entry:\")\nprint(philo_dataset[16])\n```\n\n    An example entry:\n    {'author': 'aristotle', 'quote': 'Love well, be loved and do something of value.', 'tags': 'love;ethics'}\n\n\nCheck the dataset size:"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 73dd296b-512c-4dfd-8a20-85a7aa336e0b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB", "tokens": 7788, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"73dd296b-512c-4dfd-8a20-85a7aa336e0b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb5efc1b-a20b-44a8-a5a4-548d3f7d961b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c612319605ec9e5f26c9165ca32a7036b2f07e2ac5ee91dc11787d133a1fcb44\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"1000e3e6-3c9f-483d-8e71-712bab5e9873\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"44554f3c3934b9a96349b85bbd4bffea307a188b8502cab6e422fe42dd60716d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0e1d54d1-29c0-4ce3-9e3e-a93dceb1aa2f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4f2d58adf9afe963b75e4e36f0e53cc4fa77ca26b1881004bd6cde402b7dfce6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10046, \"end_char_idx\": 12814, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "ref_doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "chroma:document": "### A test call for embeddings\n\nQuickly check how one can get the embedding vectors for a list of input texts:\n\n\n```python\nclient = openai.OpenAI(api_key=OPENAI_API_KEY)\nembedding_model_name = \"text-embedding-3-small\"\n\nresult = client.embeddings.create(\n    input=[\n        \"This is a sentence\",\n        \"A second sentence\"\n    ],\n    model=embedding_model_name,\n)\n```\n\n_Note: the above is the syntax for OpenAI v1.0+. If using previous versions, the code to get the embeddings will look different._\n\n\n```python\nprint(f\"len(result.data)              = {len(result.data)}\")\nprint(f\"result.data[1].embedding      = {str(result.data[1].embedding)[:55]}...\")\nprint(f\"len(result.data[1].embedding) = {len(result.data[1].embedding)}\")\n```\n\n    len(result.data)              = 2\n    result.data[1].embedding      = [-0.0108176339417696, 0.0013546717818826437, 0.00362232...\n    len(result.data[1].embedding) = 1536\n\n\n## Load quotes into the Vector Store\n\nGet a dataset with the quotes. _(We adapted and augmented the data from [this Kaggle dataset](https://www.kaggle.com/datasets/mertbozkurt5/quotes-by-philosophers), ready to use in this demo.)_\n\n\n```python\nphilo_dataset = load_dataset(\"datastax/philosopher-quotes\")[\"train\"]\n```\n\nA quick inspection:\n\n\n```python\nprint(\"An example entry:\")\nprint(philo_dataset[16])\n```\n\n    An example entry:\n    {'author': 'aristotle', 'quote': 'Love well, be loved and do something of value.', 'tags': 'love;ethics'}\n\n\nCheck the dataset size:\n\n\n```python\nauthor_count = Counter(entry[\"author\"] for entry in philo_dataset)\nprint(f\"Total: {len(philo_dataset)} quotes. By author:\")\nfor author, count in author_count.most_common():\n    print(f\"    {author:<20}: {count} quotes\")\n```\n\n    Total: 450 quotes. By author:\n        aristotle           : 50 quotes\n        schopenhauer        : 50 quotes\n        spinoza             : 50 quotes\n        hegel               : 50 quotes\n        freud               : 50 quotes\n        nietzsche           : 50 quotes\n        sartre              : 50 quotes\n        plato               : 50 quotes\n        kant                : 50 quotes\n\n\n### Insert quotes into vector store\n\nYou will compute the embeddings for the quotes and save them into the Vector Store, along with the text itself and the metadata planned for later use.\n\nTo optimize speed and reduce the calls, you'll perform batched calls to the embedding OpenAI service.\n\nThe DB write is accomplished with a CQL statement. But since you'll run this particular insertion several times (albeit with different values), it's best to _prepare_ the statement and then just run it over and over.\n\n_(Note: for faster insertion, the Cassandra drivers would let you do concurrent inserts, which we don't do here for a more straightforward demo code.)_"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 0e1d54d1-29c0-4ce3-9e3e-a93dceb1aa2f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB", "tokens": 7788, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0e1d54d1-29c0-4ce3-9e3e-a93dceb1aa2f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb5efc1b-a20b-44a8-a5a4-548d3f7d961b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c612319605ec9e5f26c9165ca32a7036b2f07e2ac5ee91dc11787d133a1fcb44\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"73dd296b-512c-4dfd-8a20-85a7aa336e0b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"9b3362b4817093b41310615d896f72b137f795bf4f3051de7ab06ef2ff67d2c4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"430ddddc-6556-4b3e-8324-3b5dbbf593ca\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0cf4ba4945eeee8734ce4368d0353141f2dcae9f3243b5adb8f5cb617b2b5fd8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11189, \"end_char_idx\": 15199, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "ref_doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "chroma:document": "```python\nphilo_dataset = load_dataset(\"datastax/philosopher-quotes\")[\"train\"]\n```\n\nA quick inspection:\n\n\n```python\nprint(\"An example entry:\")\nprint(philo_dataset[16])\n```\n\n    An example entry:\n    {'author': 'aristotle', 'quote': 'Love well, be loved and do something of value.', 'tags': 'love;ethics'}\n\n\nCheck the dataset size:\n\n\n```python\nauthor_count = Counter(entry[\"author\"] for entry in philo_dataset)\nprint(f\"Total: {len(philo_dataset)} quotes. By author:\")\nfor author, count in author_count.most_common():\n    print(f\"    {author:<20}: {count} quotes\")\n```\n\n    Total: 450 quotes. By author:\n        aristotle           : 50 quotes\n        schopenhauer        : 50 quotes\n        spinoza             : 50 quotes\n        hegel               : 50 quotes\n        freud               : 50 quotes\n        nietzsche           : 50 quotes\n        sartre              : 50 quotes\n        plato               : 50 quotes\n        kant                : 50 quotes\n\n\n### Insert quotes into vector store\n\nYou will compute the embeddings for the quotes and save them into the Vector Store, along with the text itself and the metadata planned for later use.\n\nTo optimize speed and reduce the calls, you'll perform batched calls to the embedding OpenAI service.\n\nThe DB write is accomplished with a CQL statement. But since you'll run this particular insertion several times (albeit with different values), it's best to _prepare_ the statement and then just run it over and over.\n\n_(Note: for faster insertion, the Cassandra drivers would let you do concurrent inserts, which we don't do here for a more straightforward demo code.)_\n\n\n```python\nprepared_insertion = session.prepare(\n    f\"INSERT INTO {keyspace}.philosophers_cql (quote_id, author, body, embedding_vector, tags) VALUES (?, ?, ?, ?, ?);\"\n)\n\nBATCH_SIZE = 20\n\nnum_batches = ((len(philo_dataset) + BATCH_SIZE - 1) // BATCH_SIZE)\n\nquotes_list = philo_dataset[\"quote\"]\nauthors_list = philo_dataset[\"author\"]\ntags_list = philo_dataset[\"tags\"]\n\nprint(\"Starting to store entries:\")\nfor batch_i in range(num_batches):\n    b_start = batch_i * BATCH_SIZE\n    b_end = (batch_i + 1) * BATCH_SIZE\n    # compute the embedding vectors for this batch\n    b_emb_results = client.embeddings.create(\n        input=quotes_list[b_start : b_end],\n        model=embedding_model_name,\n    )\n    # prepare the rows for insertion\n    print(\"B \", end=\"\")\n    for entry_idx, emb_result in zip(range(b_start, b_end), b_emb_results.data):\n        if tags_list[entry_idx]:\n            tags = {\n                tag\n                for tag in tags_list[entry_idx].split(\";\")\n            }\n        else:\n            tags = set()\n        author = authors_list[entry_idx]\n        quote = quotes_list[entry_idx]\n        quote_id = uuid4()  # a new random ID for each quote. In a production app you'll want to have better control...\n        session.execute(\n            prepared_insertion,\n            (quote_id, author, quote, emb_result.embedding, tags),\n        )\n        print(\"*\", end=\"\")\n    print(f\" done ({len(b_emb_results.data)})\")\n\nprint(\"\\nFinished storing entries.\")\n```\n\n    Starting to store entries:\n    B ******************** done (20)\n    B ******************** done (20)\n    B ******************** done (20)\n    B ******************** done (20)\n    B ******************** done (20)\n    B ******************** done (20)\n    B ******************** done (20)\n    B ******************** done (20)\n    B ******************** done (20)\n    B ******************** done (20)\n    B ******************** done (20)\n    B ******************** done (20)\n    B ******************** done (20)\n    B ******************** done (20)\n    B ******************** done (20)\n    B ******************** done (20)\n    B ******************** done (20)\n    B ******************** done (20)\n    B ******************** done (20)\n    B ******************** done (20)\n    B ******************** done (20)\n    B ******************** done (20)\n    B ********** done (10)\n    \n    Finished storing entries."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 430ddddc-6556-4b3e-8324-3b5dbbf593ca
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB", "tokens": 7788, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"430ddddc-6556-4b3e-8324-3b5dbbf593ca\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb5efc1b-a20b-44a8-a5a4-548d3f7d961b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c612319605ec9e5f26c9165ca32a7036b2f07e2ac5ee91dc11787d133a1fcb44\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0e1d54d1-29c0-4ce3-9e3e-a93dceb1aa2f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"ca9db875183d25189a8e44a4948849a8320da8028929437ab618a36f53799614\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"92d62579-2f9d-4a44-9d25-959a951fcf0c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a010ca8a2abc0c035cd080ff5b4b38de855471586df5854e3b0f5c3c9af6e1dd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15202, \"end_char_idx\": 18357, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "ref_doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "chroma:document": "## Use case 1: **quote search engine**\n\nFor the quote-search functionality, you need first to make the input quote into a vector, and then use it to query the store (besides handling the optional metadata into the search call, that is).\n\nEncapsulate the search-engine functionality into a function for ease of re-use:\n\n\n```python\ndef find_quote_and_author(query_quote, n, author=None, tags=None):\n    query_vector = client.embeddings.create(\n        input=[query_quote],\n        model=embedding_model_name,\n    ).data[0].embedding\n    # depending on what conditions are passed, the WHERE clause in the statement may vary.\n    where_clauses = []\n    where_values = []\n    if author:\n        where_clauses += [\"author = %s\"]\n        where_values += [author]\n    if tags:\n        for tag in tags:\n            where_clauses += [\"tags CONTAINS %s\"]\n            where_values += [tag]\n    # The reason for these two lists above is that when running the CQL search statement the values passed\n    # must match the sequence of \"?\" marks in the statement.\n    if where_clauses:\n        search_statement = f\"\"\"SELECT body, author FROM {keyspace}.philosophers_cql\n            WHERE {' AND '.join(where_clauses)}\n            ORDER BY embedding_vector ANN OF %s\n            LIMIT %s;\n        \"\"\"\n    else:\n        search_statement = f\"\"\"SELECT body, author FROM {keyspace}.philosophers_cql\n            ORDER BY embedding_vector ANN OF %s\n            LIMIT %s;\n        \"\"\"\n    # For best performance, one should keep a cache of prepared statements (see the insertion code above)\n    # for the various possible statements used here.\n    # (We'll leave it as an exercise to the reader to avoid making this code too long.\n    # Remember: to prepare a statement you use '?' instead of '%s'.)\n    query_values = tuple(where_values + [query_vector] + [n])\n    result_rows = session.execute(search_statement, query_values)\n    return [\n        (result_row.body, result_row.author)\n        for result_row in result_rows\n    ]\n```\n\n### Putting search to test\n\nPassing just a quote:\n\n\n```python\nfind_quote_and_author(\"We struggle all our life for nothing\", 3)\n```\n\n\n\n\n    [('Life to the great majority is only a constant struggle for mere existence, with the certainty of losing it at last.',\n      'schopenhauer'),\n     ('We give up leisure in order that we may have leisure, just as we go to war in order that we may have peace.',\n      'aristotle'),\n     ('Perhaps the gods are kind to us, by making life more disagreeable as we grow older. In the end death seems less intolerable than the manifold burdens we carry',\n      'freud')]\n\n\n\nSearch restricted to an author:\n\n\n```python\nfind_quote_and_author(\"We struggle all our life for nothing\", 2, author=\"nietzsche\")\n```\n\n\n\n\n    [('To live is to suffer, to survive is to find some meaning in the suffering.',\n      'nietzsche'),\n     ('What makes us heroic?--Confronting simultaneously our supreme suffering and our supreme hope.',\n      'nietzsche')]\n\n\n\nSearch constrained to a tag (out of those saved earlier with the quotes):\n\n\n```python\nfind_quote_and_author(\"We struggle all our life for nothing\", 2, tags=[\"politics\"])\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 92d62579-2f9d-4a44-9d25-959a951fcf0c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB", "tokens": 7788, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"92d62579-2f9d-4a44-9d25-959a951fcf0c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb5efc1b-a20b-44a8-a5a4-548d3f7d961b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c612319605ec9e5f26c9165ca32a7036b2f07e2ac5ee91dc11787d133a1fcb44\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"430ddddc-6556-4b3e-8324-3b5dbbf593ca\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"a570e2a6e6c1858f6fe1bf9b4ed2dc6c2ce33ad8f6e975ae846150f3b6786387\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3c567ef9-d6a6-4f06-afe2-c1a45fe341c0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cd4547429557361b73151049a4912257ae0f1bf1b53bd91711af637f6a52a8bf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17262, \"end_char_idx\": 19920, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "ref_doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "chroma:document": "```python\nfind_quote_and_author(\"We struggle all our life for nothing\", 3)\n```\n\n\n\n\n    [('Life to the great majority is only a constant struggle for mere existence, with the certainty of losing it at last.',\n      'schopenhauer'),\n     ('We give up leisure in order that we may have leisure, just as we go to war in order that we may have peace.',\n      'aristotle'),\n     ('Perhaps the gods are kind to us, by making life more disagreeable as we grow older. In the end death seems less intolerable than the manifold burdens we carry',\n      'freud')]\n\n\n\nSearch restricted to an author:\n\n\n```python\nfind_quote_and_author(\"We struggle all our life for nothing\", 2, author=\"nietzsche\")\n```\n\n\n\n\n    [('To live is to suffer, to survive is to find some meaning in the suffering.',\n      'nietzsche'),\n     ('What makes us heroic?--Confronting simultaneously our supreme suffering and our supreme hope.',\n      'nietzsche')]\n\n\n\nSearch constrained to a tag (out of those saved earlier with the quotes):\n\n\n```python\nfind_quote_and_author(\"We struggle all our life for nothing\", 2, tags=[\"politics\"])\n```\n\n\n\n\n    [('Mankind will never see an end of trouble until lovers of wisdom come to hold political power, or the holders of power become lovers of wisdom',\n      'plato'),\n     ('Everything the State says is a lie, and everything it has it has stolen.',\n      'nietzsche')]\n\n\n\n### Cutting out irrelevant results\n\nThe vector similarity search generally returns the vectors that are closest to the query, even if that means results that might be somewhat irrelevant if there's nothing better.\n\nTo keep this issue under control, you can get the actual \"similarity\" between the query and each result, and then set a cutoff on it, effectively discarding results that are beyond that threshold.\nTuning this threshold correctly is not an easy problem: here, we'll just show you the way.\n\nTo get a feeling on how this works, try the following query and play with the choice of quote and threshold to compare the results:\n\n_Note (for the mathematically inclined): this value is **a rescaling between zero and one** of the cosine difference between the vectors, i.e. of the scalar product divided by the product of the norms of the two vectors. In other words, this is 0 for opposite-facing vecors and +1 for parallel vectors. For other measures of similarity, check the [documentation](https://docs.datastax.com/en/astra-serverless/docs/vector-search/cql.html#_create_the_vector_schema_and_load_the_data_into_the_database) -- and keep in mind that the metric in the `SELECT` query should match the one used when creating the index earlier for meaningful, ordered results._"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 3c567ef9-d6a6-4f06-afe2-c1a45fe341c0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB", "tokens": 7788, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"3c567ef9-d6a6-4f06-afe2-c1a45fe341c0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb5efc1b-a20b-44a8-a5a4-548d3f7d961b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c612319605ec9e5f26c9165ca32a7036b2f07e2ac5ee91dc11787d133a1fcb44\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"92d62579-2f9d-4a44-9d25-959a951fcf0c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3a52e5f840d6ad8c22f822e183160be223244d104dbc0e9b4c6f545fcc21bf11\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"94e96faf-8771-4095-a2b0-4bb5bf03b52c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"35e0a28c622327eb5917eb9ae789bfd488cd3dbba9af0f5ba0158af98001d469\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18184, \"end_char_idx\": 21249, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "ref_doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "chroma:document": "Search constrained to a tag (out of those saved earlier with the quotes):\n\n\n```python\nfind_quote_and_author(\"We struggle all our life for nothing\", 2, tags=[\"politics\"])\n```\n\n\n\n\n    [('Mankind will never see an end of trouble until lovers of wisdom come to hold political power, or the holders of power become lovers of wisdom',\n      'plato'),\n     ('Everything the State says is a lie, and everything it has it has stolen.',\n      'nietzsche')]\n\n\n\n### Cutting out irrelevant results\n\nThe vector similarity search generally returns the vectors that are closest to the query, even if that means results that might be somewhat irrelevant if there's nothing better.\n\nTo keep this issue under control, you can get the actual \"similarity\" between the query and each result, and then set a cutoff on it, effectively discarding results that are beyond that threshold.\nTuning this threshold correctly is not an easy problem: here, we'll just show you the way.\n\nTo get a feeling on how this works, try the following query and play with the choice of quote and threshold to compare the results:\n\n_Note (for the mathematically inclined): this value is **a rescaling between zero and one** of the cosine difference between the vectors, i.e. of the scalar product divided by the product of the norms of the two vectors. In other words, this is 0 for opposite-facing vecors and +1 for parallel vectors. For other measures of similarity, check the [documentation](https://docs.datastax.com/en/astra-serverless/docs/vector-search/cql.html#_create_the_vector_schema_and_load_the_data_into_the_database) -- and keep in mind that the metric in the `SELECT` query should match the one used when creating the index earlier for meaningful, ordered results._\n\n\n```python\nquote = \"Animals are our equals.\"\n# quote = \"Be good.\"\n# quote = \"This teapot is strange.\"\n\nsimilarity_threshold = 0.92\n\nquote_vector = client.embeddings.create(\n    input=[quote],\n    model=embedding_model_name,\n).data[0].embedding\n\n# Once more: remember to prepare your statements in production for greater performance...\n\nsearch_statement = f\"\"\"SELECT body, similarity_dot_product(embedding_vector, %s) as similarity\n    FROM {keyspace}.philosophers_cql\n    ORDER BY embedding_vector ANN OF %s\n    LIMIT %s;\n\"\"\"\nquery_values = (quote_vector, quote_vector, 8)\n\nresult_rows = session.execute(search_statement, query_values)\nresults = [\n    (result_row.body, result_row.similarity)\n    for result_row in result_rows\n    if result_row.similarity >= similarity_threshold\n]\n\nprint(f\"{len(results)} quotes within the threshold:\")\nfor idx, (r_body, r_similarity) in enumerate(results):\n    print(f\"    {idx}. [similarity={r_similarity:.3f}] \\\"{r_body[:70]}...\\\"\")\n```\n\n    3 quotes within the threshold:\n        0. [similarity=0.927] \"The assumption that animals are without rights, and the illusion that ...\"\n        1. [similarity=0.922] \"Animals are in possession of themselves; their soul is in possession o...\"\n        2. [similarity=0.920] \"At his best, man is the noblest of all animals; separated from law and...\""}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 94e96faf-8771-4095-a2b0-4bb5bf03b52c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB", "tokens": 7788, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"94e96faf-8771-4095-a2b0-4bb5bf03b52c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb5efc1b-a20b-44a8-a5a4-548d3f7d961b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c612319605ec9e5f26c9165ca32a7036b2f07e2ac5ee91dc11787d133a1fcb44\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3c567ef9-d6a6-4f06-afe2-c1a45fe341c0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c3e18780cd7099f1b9e667062efb27a34c06f21fdb8c1659b61cae3cc41737fb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7fb08850-5886-4f3d-bbc9-89fc0cb64d8c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8f8e89e495ea0bf57ec41a8d98b454633d0e930ca2ddb83ddb178bd5346129ea\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19923, \"end_char_idx\": 21974, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "ref_doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "chroma:document": "```python\nquote = \"Animals are our equals.\"\n# quote = \"Be good.\"\n# quote = \"This teapot is strange.\"\n\nsimilarity_threshold = 0.92\n\nquote_vector = client.embeddings.create(\n    input=[quote],\n    model=embedding_model_name,\n).data[0].embedding\n\n# Once more: remember to prepare your statements in production for greater performance...\n\nsearch_statement = f\"\"\"SELECT body, similarity_dot_product(embedding_vector, %s) as similarity\n    FROM {keyspace}.philosophers_cql\n    ORDER BY embedding_vector ANN OF %s\n    LIMIT %s;\n\"\"\"\nquery_values = (quote_vector, quote_vector, 8)\n\nresult_rows = session.execute(search_statement, query_values)\nresults = [\n    (result_row.body, result_row.similarity)\n    for result_row in result_rows\n    if result_row.similarity >= similarity_threshold\n]\n\nprint(f\"{len(results)} quotes within the threshold:\")\nfor idx, (r_body, r_similarity) in enumerate(results):\n    print(f\"    {idx}. [similarity={r_similarity:.3f}] \\\"{r_body[:70]}...\\\"\")\n```\n\n    3 quotes within the threshold:\n        0. [similarity=0.927] \"The assumption that animals are without rights, and the illusion that ...\"\n        1. [similarity=0.922] \"Animals are in possession of themselves; their soul is in possession o...\"\n        2. [similarity=0.920] \"At his best, man is the noblest of all animals; separated from law and...\"\n\n\n## Use case 2: **quote generator**\n\nFor this task you need another component from OpenAI, namely an LLM to generate the quote for us (based on input obtained by querying the Vector Store).\n\nYou also need a template for the prompt that will be filled for the generate-quote LLM completion task.\n\n\n```python\ncompletion_model_name = \"gpt-3.5-turbo\"\n\ngeneration_prompt_template = \"\"\"\"Generate a single short philosophical quote on the given topic,\nsimilar in spirit and form to the provided actual example quotes.\nDo not exceed 20-30 words in your quote.\n\nREFERENCE TOPIC: \"{topic}\"\n\nACTUAL EXAMPLES:\n{examples}\n\"\"\"\n```\n\nLike for search, this functionality is best wrapped into a handy function (which internally uses search):"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 7fb08850-5886-4f3d-bbc9-89fc0cb64d8c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB", "tokens": 7788, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"7fb08850-5886-4f3d-bbc9-89fc0cb64d8c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb5efc1b-a20b-44a8-a5a4-548d3f7d961b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c612319605ec9e5f26c9165ca32a7036b2f07e2ac5ee91dc11787d133a1fcb44\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"94e96faf-8771-4095-a2b0-4bb5bf03b52c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"031b28ae1b763aa335348dc1c0db1b5221754ba63f64547ffdfcbfdd4177c921\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"73ffd070-68ec-49a4-9c9b-92fa33af6e5a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"18b6c30867e69f224bf4bb2bb5a7d6aed099f4a0e97c5b7ac60a2a2edb176206\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21252, \"end_char_idx\": 23809, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "ref_doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "chroma:document": "## Use case 2: **quote generator**\n\nFor this task you need another component from OpenAI, namely an LLM to generate the quote for us (based on input obtained by querying the Vector Store).\n\nYou also need a template for the prompt that will be filled for the generate-quote LLM completion task.\n\n\n```python\ncompletion_model_name = \"gpt-3.5-turbo\"\n\ngeneration_prompt_template = \"\"\"\"Generate a single short philosophical quote on the given topic,\nsimilar in spirit and form to the provided actual example quotes.\nDo not exceed 20-30 words in your quote.\n\nREFERENCE TOPIC: \"{topic}\"\n\nACTUAL EXAMPLES:\n{examples}\n\"\"\"\n```\n\nLike for search, this functionality is best wrapped into a handy function (which internally uses search):\n\n\n```python\ndef generate_quote(topic, n=2, author=None, tags=None):\n    quotes = find_quote_and_author(query_quote=topic, n=n, author=author, tags=tags)\n    if quotes:\n        prompt = generation_prompt_template.format(\n            topic=topic,\n            examples=\"\\n\".join(f\"  - {quote[0]}\" for quote in quotes),\n        )\n        # a little logging:\n        print(\"** quotes found:\")\n        for q, a in quotes:\n            print(f\"**    - {q} ({a})\")\n        print(\"** end of logging\")\n        #\n        response = client.chat.completions.create(\n            model=completion_model_name,\n            messages=[{\"role\": \"user\", \"content\": prompt}],\n            temperature=0.7,\n            max_tokens=320,\n        )\n        return response.choices[0].message.content.replace('\"', '').strip()\n    else:\n        print(\"** no quotes found.\")\n        return None\n```\n\n_Note: similar to the case of the embedding computation, the code for the Chat Completion API would be slightly different for OpenAI prior to v1.0._\n\n#### Putting quote generation to test\n\nJust passing a text (a \"quote\", but one can actually just suggest a topic since its vector embedding will still end up at the right place in the vector space):\n\n\n```python\nq_topic = generate_quote(\"politics and virtue\")\nprint(\"\\nA new generated quote:\")\nprint(q_topic)\n```\n\n    ** quotes found:\n    **    - Happiness is the reward of virtue. (aristotle)\n    **    - Our moral virtues benefit mainly other people; intellectual virtues, on the other hand, benefit primarily ourselves; therefore the former make us universally popular, the latter unpopular. (schopenhauer)\n    ** end of logging\n    \n    A new generated quote:\n    True politics is not the pursuit of power, but the cultivation of virtue for the betterment of all.\n\n\nUse inspiration from just a single philosopher:"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 73ffd070-68ec-49a4-9c9b-92fa33af6e5a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB", "tokens": 7788, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"73ffd070-68ec-49a4-9c9b-92fa33af6e5a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb5efc1b-a20b-44a8-a5a4-548d3f7d961b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c612319605ec9e5f26c9165ca32a7036b2f07e2ac5ee91dc11787d133a1fcb44\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7fb08850-5886-4f3d-bbc9-89fc0cb64d8c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c6f98c5b8fd9de5a22d4f58588aa5147984fde473302c48f508326b3d7d1bc55\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ccb0ede0-0efb-486d-90c0-b49ff0cd682e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"559834002524a2623061746df5b16836c5c72f794672d9f40dd6840bbec767fe\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 23194, \"end_char_idx\": 26107, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "ref_doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "chroma:document": "```python\nq_topic = generate_quote(\"politics and virtue\")\nprint(\"\\nA new generated quote:\")\nprint(q_topic)\n```\n\n    ** quotes found:\n    **    - Happiness is the reward of virtue. (aristotle)\n    **    - Our moral virtues benefit mainly other people; intellectual virtues, on the other hand, benefit primarily ourselves; therefore the former make us universally popular, the latter unpopular. (schopenhauer)\n    ** end of logging\n    \n    A new generated quote:\n    True politics is not the pursuit of power, but the cultivation of virtue for the betterment of all.\n\n\nUse inspiration from just a single philosopher:\n\n\n```python\nq_topic = generate_quote(\"animals\", author=\"schopenhauer\")\nprint(\"\\nA new generated quote:\")\nprint(q_topic)\n```\n\n    ** quotes found:\n    **    - Because Christian morality leaves animals out of account, they are at once outlawed in philosophical morals; they are mere 'things,' mere means to any ends whatsoever. They can therefore be used for vivisection, hunting, coursing, bullfights, and horse racing, and can be whipped to death as they struggle along with heavy carts of stone. Shame on such a morality that is worthy of pariahs, and that fails to recognize the eternal essence that exists in every living thing, and shines forth with inscrutable significance from all eyes that see the sun! (schopenhauer)\n    **    - The assumption that animals are without rights, and the illusion that our treatment of them has no moral significance, is a positively outrageous example of Western crudity and barbarity. Universal compassion is the only guarantee of morality. (schopenhauer)\n    ** end of logging\n    \n    A new generated quote:\n    Do not judge the worth of a soul by its outward form, for within every animal lies an eternal essence that deserves our compassion and respect.\n\n\n## (Optional) **Partitioning**\n\nThere's an interesting topic to examine before completing this quickstart. While, generally, tags and quotes can be in any relationship (e.g. a quote having multiple tags), _authors_ are effectively an exact grouping (they define a \"disjoint partitioning\" on the set of quotes): each quote has exactly one author (for us, at least).\n\nNow, suppose you know in advance your application will usually (or always) run queries on a _single author_. Then you can take full advantage of the underlying database structure: if you group quotes in **partitions** (one per author), vector queries on just an author will use less resources and return much faster.\n\nWe'll not dive into the details here, which have to do with the Cassandra storage internals: the important message is that **if your queries are run within a group, consider partitioning accordingly to boost performance**.\n\nYou'll now see this choice in action.\n\nThe partitioning per author calls for a new table schema: create a new table called \"philosophers_cql_partitioned\", along with the necessary indexes:"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: ccb0ede0-0efb-486d-90c0-b49ff0cd682e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB", "tokens": 7788, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"ccb0ede0-0efb-486d-90c0-b49ff0cd682e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb5efc1b-a20b-44a8-a5a4-548d3f7d961b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c612319605ec9e5f26c9165ca32a7036b2f07e2ac5ee91dc11787d133a1fcb44\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"73ffd070-68ec-49a4-9c9b-92fa33af6e5a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"b26798290da598cd146d242f0641b1a8ab89e88cd749304f877844cfb7485462\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cd5ec1ea-8232-40cc-91f2-c930329056a7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5e684e0bc6b577dd11b0f0ac541e6af48426e7bf8551ac19a56a67e833b2b5c3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 25011, \"end_char_idx\": 28012, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "ref_doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "chroma:document": "## (Optional) **Partitioning**\n\nThere's an interesting topic to examine before completing this quickstart. While, generally, tags and quotes can be in any relationship (e.g. a quote having multiple tags), _authors_ are effectively an exact grouping (they define a \"disjoint partitioning\" on the set of quotes): each quote has exactly one author (for us, at least).\n\nNow, suppose you know in advance your application will usually (or always) run queries on a _single author_. Then you can take full advantage of the underlying database structure: if you group quotes in **partitions** (one per author), vector queries on just an author will use less resources and return much faster.\n\nWe'll not dive into the details here, which have to do with the Cassandra storage internals: the important message is that **if your queries are run within a group, consider partitioning accordingly to boost performance**.\n\nYou'll now see this choice in action.\n\nThe partitioning per author calls for a new table schema: create a new table called \"philosophers_cql_partitioned\", along with the necessary indexes:\n\n\n```python\ncreate_table_p_statement = f\"\"\"CREATE TABLE IF NOT EXISTS {keyspace}.philosophers_cql_partitioned (\n    author TEXT,\n    quote_id UUID,\n    body TEXT,\n    embedding_vector VECTOR<FLOAT, 1536>,\n    tags SET<TEXT>,\n    PRIMARY KEY ( (author), quote_id )\n) WITH CLUSTERING ORDER BY (quote_id ASC);\"\"\"\n\nsession.execute(create_table_p_statement)\n\ncreate_vector_index_p_statement = f\"\"\"CREATE CUSTOM INDEX IF NOT EXISTS idx_embedding_vector_p\n    ON {keyspace}.philosophers_cql_partitioned (embedding_vector)\n    USING 'org.apache.cassandra.index.sai.StorageAttachedIndex'\n    WITH OPTIONS = {{'similarity_function' : 'dot_product'}};\n\"\"\"\n\nsession.execute(create_vector_index_p_statement)\n\ncreate_tags_index_p_statement = f\"\"\"CREATE CUSTOM INDEX IF NOT EXISTS idx_tags_p\n    ON {keyspace}.philosophers_cql_partitioned (VALUES(tags))\n    USING 'org.apache.cassandra.index.sai.StorageAttachedIndex';\n\"\"\"\nsession.execute(create_tags_index_p_statement)\n```\n\n\n\n\n    <cassandra.cluster.ResultSet at 0x7fef149d7940>\n\n\n\nNow repeat the compute-embeddings-and-insert step on the new table.\n\nYou could use the very same insertion code as you did earlier, because the differences are hidden \"behind the scenes\": the database will store the inserted rows differently according to the partitioning scheme of this new table.\n\nHowever, by way of demonstration, you will take advantage of a handy facility offered by the Cassandra drivers to easily run several queries (in this case, `INSERT`s) concurrently. This is something that Cassandra / Astra DB through CQL supports very well and can lead to a significant speedup, with very little changes in the client code.\n\n_(Note: one could additionally have cached the embeddings computed previously to save a few API tokens -- here, however, we wanted to keep the code easier to inspect.)_\n\n\n```python\nfrom cassandra.concurrent import execute_concurrent_with_args\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: cd5ec1ea-8232-40cc-91f2-c930329056a7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB", "tokens": 7788, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"cd5ec1ea-8232-40cc-91f2-c930329056a7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb5efc1b-a20b-44a8-a5a4-548d3f7d961b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c612319605ec9e5f26c9165ca32a7036b2f07e2ac5ee91dc11787d133a1fcb44\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ccb0ede0-0efb-486d-90c0-b49ff0cd682e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"e0253ee5d1a3e4df8635c125d936dd31c90a715d8ee6ce12c88113d630c8b3da\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a159b731-4067-48a1-85a2-57c28eb619de\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c8a725cd29aabffd2a427b3396cf76958d07bdc84351a4a5a7f68749950cd885\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 27075, \"end_char_idx\": 30229, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "ref_doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "chroma:document": "<cassandra.cluster.ResultSet at 0x7fef149d7940>\n\n\n\nNow repeat the compute-embeddings-and-insert step on the new table.\n\nYou could use the very same insertion code as you did earlier, because the differences are hidden \"behind the scenes\": the database will store the inserted rows differently according to the partitioning scheme of this new table.\n\nHowever, by way of demonstration, you will take advantage of a handy facility offered by the Cassandra drivers to easily run several queries (in this case, `INSERT`s) concurrently. This is something that Cassandra / Astra DB through CQL supports very well and can lead to a significant speedup, with very little changes in the client code.\n\n_(Note: one could additionally have cached the embeddings computed previously to save a few API tokens -- here, however, we wanted to keep the code easier to inspect.)_\n\n\n```python\nfrom cassandra.concurrent import execute_concurrent_with_args\n```\n\n\n```python\nprepared_insertion = session.prepare(\n    f\"INSERT INTO {keyspace}.philosophers_cql_partitioned (quote_id, author, body, embedding_vector, tags) VALUES (?, ?, ?, ?, ?);\"\n)\n\nBATCH_SIZE = 50\n\nnum_batches = ((len(philo_dataset) + BATCH_SIZE - 1) // BATCH_SIZE)\n\nquotes_list = philo_dataset[\"quote\"]\nauthors_list = philo_dataset[\"author\"]\ntags_list = philo_dataset[\"tags\"]\n\nprint(\"Starting to store entries:\")\nfor batch_i in range(num_batches):\n    print(\"[...\", end=\"\")\n    b_start = batch_i * BATCH_SIZE\n    b_end = (batch_i + 1) * BATCH_SIZE\n    # compute the embedding vectors for this batch\n    b_emb_results = client.embeddings.create(\n        input=quotes_list[b_start : b_end],\n        model=embedding_model_name,\n    )\n    # prepare this batch's entries for insertion\n    tuples_to_insert = []\n    for entry_idx, emb_result in zip(range(b_start, b_end), b_emb_results.data):\n        if tags_list[entry_idx]:\n            tags = {\n                tag\n                for tag in tags_list[entry_idx].split(\";\")\n            }\n        else:\n            tags = set()\n        author = authors_list[entry_idx]\n        quote = quotes_list[entry_idx]\n        quote_id = uuid4()  # a new random ID for each quote. In a production app you'll want to have better control...\n        # append a *tuple* to the list, and in the tuple the values are ordered to match \"?\" in the prepared statement:\n        tuples_to_insert.append((quote_id, author, quote, emb_result.embedding, tags))\n    # insert the batch at once through the driver's concurrent primitive\n    conc_results = execute_concurrent_with_args(\n        session,\n        prepared_insertion,\n        tuples_to_insert,\n    )\n    # check that all insertions succeed (better to always do this):\n    if any([not success for success, _ in conc_results]):\n        print(\"Something failed during the insertions!\")\n    else:\n        print(f\"{len(b_emb_results.data)}] \", end=\"\")\n\nprint(\"\\nFinished storing entries.\")\n```\n\n    Starting to store entries:\n    [...50] [...50] [...50] [...50] [...50] [...50] [...50] [...50] [...50] \n    Finished storing entries.\n\n\nDespite the different table schema, the DB query behind the similarity search is essentially the same:"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: a159b731-4067-48a1-85a2-57c28eb619de
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB", "tokens": 7788, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"a159b731-4067-48a1-85a2-57c28eb619de\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb5efc1b-a20b-44a8-a5a4-548d3f7d961b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c612319605ec9e5f26c9165ca32a7036b2f07e2ac5ee91dc11787d133a1fcb44\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cd5ec1ea-8232-40cc-91f2-c930329056a7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"defa52d24270babdd12166482baee4950b4cb1ac45bec0e608321be16e3acaed\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8c912fbe-da1f-429a-8160-b07b9ca5262c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7250c5ac93bd71b4c3e2a9c5bdc4e077fa886918fa3c3ad1c4cf64718e8bb1a3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 30127, \"end_char_idx\": 32570, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "ref_doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "chroma:document": "Despite the different table schema, the DB query behind the similarity search is essentially the same:\n\n\n```python\ndef find_quote_and_author_p(query_quote, n, author=None, tags=None):\n    query_vector = client.embeddings.create(\n        input=[query_quote],\n        model=embedding_model_name,\n    ).data[0].embedding\n    # Depending on what conditions are passed, the WHERE clause in the statement may vary.\n    # Construct it accordingly:\n    where_clauses = []\n    where_values = []\n    if author:\n        where_clauses += [\"author = %s\"]\n        where_values += [author]\n    if tags:\n        for tag in tags:\n            where_clauses += [\"tags CONTAINS %s\"]\n            where_values += [tag]\n    if where_clauses:\n        search_statement = f\"\"\"SELECT body, author FROM {keyspace}.philosophers_cql_partitioned\n            WHERE {' AND '.join(where_clauses)}\n            ORDER BY embedding_vector ANN OF %s\n            LIMIT %s;\n        \"\"\"\n    else:\n        search_statement = f\"\"\"SELECT body, author FROM {keyspace}.philosophers_cql_partitioned\n            ORDER BY embedding_vector ANN OF %s\n            LIMIT %s;\n        \"\"\"\n    query_values = tuple(where_values + [query_vector] + [n])\n    result_rows = session.execute(search_statement, query_values)\n    return [\n        (result_row.body, result_row.author)\n        for result_row in result_rows\n    ]\n```\n\nThat's it: the new table still supports the \"generic\" similarity searches all right ...\n\n\n```python\nfind_quote_and_author_p(\"We struggle all our life for nothing\", 3)\n```\n\n\n\n\n    [('Life to the great majority is only a constant struggle for mere existence, with the certainty of losing it at last.',\n      'schopenhauer'),\n     ('We give up leisure in order that we may have leisure, just as we go to war in order that we may have peace.',\n      'aristotle'),\n     ('Perhaps the gods are kind to us, by making life more disagreeable as we grow older. In the end death seems less intolerable than the manifold burdens we carry',\n      'freud')]\n\n\n\n... but it's when an author is specified that you would notice a _huge_ performance advantage:\n\n\n```python\nfind_quote_and_author_p(\"We struggle all our life for nothing\", 2, author=\"nietzsche\")\n```\n\n\n\n\n    [('To live is to suffer, to survive is to find some meaning in the suffering.',\n      'nietzsche'),\n     ('What makes us heroic?--Confronting simultaneously our supreme suffering and our supreme hope.',\n      'nietzsche')]"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 8c912fbe-da1f-429a-8160-b07b9ca5262c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB", "tokens": 7788, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"8c912fbe-da1f-429a-8160-b07b9ca5262c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"cb5efc1b-a20b-44a8-a5a4-548d3f7d961b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c612319605ec9e5f26c9165ca32a7036b2f07e2ac5ee91dc11787d133a1fcb44\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a159b731-4067-48a1-85a2-57c28eb619de\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_CQL.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB\", \"tokens\": 7788, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"41095687a58c8947e124ef269d26a8935eb621596e0968c0a60d77fd3c7ec6c5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 31585, \"end_char_idx\": 33995, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "ref_doc_id": "cb5efc1b-a20b-44a8-a5a4-548d3f7d961b", "chroma:document": "```python\nfind_quote_and_author_p(\"We struggle all our life for nothing\", 3)\n```\n\n\n\n\n    [('Life to the great majority is only a constant struggle for mere existence, with the certainty of losing it at last.',\n      'schopenhauer'),\n     ('We give up leisure in order that we may have leisure, just as we go to war in order that we may have peace.',\n      'aristotle'),\n     ('Perhaps the gods are kind to us, by making life more disagreeable as we grow older. In the end death seems less intolerable than the manifold burdens we carry',\n      'freud')]\n\n\n\n... but it's when an author is specified that you would notice a _huge_ performance advantage:\n\n\n```python\nfind_quote_and_author_p(\"We struggle all our life for nothing\", 2, author=\"nietzsche\")\n```\n\n\n\n\n    [('To live is to suffer, to survive is to find some meaning in the suffering.',\n      'nietzsche'),\n     ('What makes us heroic?--Confronting simultaneously our supreme suffering and our supreme hope.',\n      'nietzsche')]\n\n\n\nWell, you _would_ notice a performance gain, if you had a realistic-size dataset. In this demo, with a few tens of entries, there's no noticeable difference -- but you get the idea.\n\n## Conclusion\n\nCongratulations! You have learned how to use OpenAI for vector embeddings and Astra DB / Cassandra for storage in order to build a sophisticated philosophical search engine and quote generator.\n\nThis example used the [Cassandra drivers](https://docs.datastax.com/en/developer/python-driver/latest/) and runs CQL (Cassandra Query Language) statements directly to interface with the Vector Store - but this is not the only choice. Check the [README](https://github.com/openai/openai-cookbook/tree/main/examples/vector_databases/cassandra_astradb) for other options and integration with popular frameworks.\n\nTo find out more on how Astra DB's Vector Search capabilities can be a key ingredient in your ML/GenAI applications, visit [Astra DB](https://docs.datastax.com/en/astra-serverless/docs/vector-search/overview.html)'s web page on the topic.\n\n## Cleanup\n\nIf you want to remove all resources used for this demo, run this cell (_warning: this will delete the tables and the data inserted in them!_):\n\n\n```python\nsession.execute(f\"DROP TABLE IF EXISTS {keyspace}.philosophers_cql;\")\nsession.execute(f\"DROP TABLE IF EXISTS {keyspace}.philosophers_cql_partitioned;\")\n```\n\n\n\n\n    <cassandra.cluster.ResultSet at 0x7fef149096a0>"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 803f091b-d82f-4da7-bcae-8cb8bc5a801a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL", "tokens": 6051, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"803f091b-d82f-4da7-bcae-8cb8bc5a801a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8938aaa0-39b6-4913-afa3-644dfcdc84cb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"723f825cd19bd5d01794b8111cf19db861b8b27b9247f5207316ec1dc1f05a13\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"018bfaf7-fccb-438f-9684-a4dbcaab449e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"63d89d7e1f58572f09ed18c4f10302344b73c1eb419266b310ab6fda313e9fe9\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3274, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "doc_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "ref_doc_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "chroma:document": "# Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\n\n### CassIO version\n\nIn this quickstart you will learn how to build a \"philosophy quote finder & generator\" using OpenAI's vector embeddings and [Apache Cassandra\u00ae](https://cassandra.apache.org), or equivalently DataStax [Astra DB through CQL](https://docs.datastax.com/en/astra-serverless/docs/vector-search/quickstart.html), as the vector store for data persistence.\n\nThe basic workflow of this notebook is outlined below. You will evaluate and store the vector embeddings for a number of quotes by famous philosophers, use them to build a powerful search engine and, after that, even a generator of new quotes!\n\nThe notebook exemplifies some of the standard usage patterns of vector search -- while showing how easy is it to get started with the vector capabilities of [Cassandra](https://cassandra.apache.org/doc/trunk/cassandra/vector-search/overview.html) / [Astra DB through CQL](https://docs.datastax.com/en/astra-serverless/docs/vector-search/quickstart.html).\n\nFor a background on using vector search and text embeddings to build a question-answering system, please check out this excellent hands-on notebook: [Question answering using embeddings](https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb).\n\n#### _Choose-your-framework_\n\nPlease note that this notebook uses the [CassIO library](https://cassio.org), but we cover other choices of technology to accomplish the same task. Check out this folder's [README](https://github.com/openai/openai-cookbook/tree/main/examples/vector_databases/cassandra_astradb) for other options. This notebook can run either as a Colab notebook or as a regular Jupyter notebook.\n\nTable of contents:\n- Setup\n- Get DB connection\n- Connect to OpenAI\n- Load quotes into the Vector Store\n- Use case 1: **quote search engine**\n- Use case 2: **quote generator**\n- (Optional) exploit partitioning in the Vector Store\n\n### How it works\n\n**Indexing**\n\nEach quote is made into an embedding vector with OpenAI's `Embedding`. These are saved in the Vector Store for later use in searching. Some metadata, including the author's name and a few other pre-computed tags, are stored alongside, to allow for search customization.\n\n![1_vector_indexing](https://user-images.githubusercontent.com/14221764/282440878-dc3ed680-7d0e-4b30-9a74-d2d66a7394f7.png)\n\n**Search**\n\nTo find a quote similar to the provided search quote, the latter is made into an embedding vector on the fly, and this vector is used to query the store for similar vectors ... i.e. similar quotes that were previously indexed. The search can optionally be constrained by additional metadata (\"find me quotes by Spinoza similar to this one ...\").\n\n![2_vector_search](https://user-images.githubusercontent.com/14221764/282440908-683e3ee1-0bf1-46b3-8621-86c31fc7f9c9.png)\n\nThe key point here is that \"quotes similar in content\" translates, in vector space, to vectors that are metrically close to each other: thus, vector similarity search effectively implements semantic similarity. _This is the key reason vector embeddings are so powerful._\n\nThe sketch below tries to convey this idea. Each quote, once it's made into a vector, is a point in space."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 018bfaf7-fccb-438f-9684-a4dbcaab449e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL", "tokens": 6051, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"018bfaf7-fccb-438f-9684-a4dbcaab449e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8938aaa0-39b6-4913-afa3-644dfcdc84cb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"723f825cd19bd5d01794b8111cf19db861b8b27b9247f5207316ec1dc1f05a13\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"803f091b-d82f-4da7-bcae-8cb8bc5a801a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"cc0bd0b46c4466a5d8ce7cbcd87d74087449cb2530f8f43348ef9e824f8d2d46\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"561a98df-9ed0-4ab9-bae6-8fe0fabf7ab5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e066639a27df3a783c88b5a99d58b44a03f292a74a25a563e24033ee2302d39d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1672, \"end_char_idx\": 4246, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "doc_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "ref_doc_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "chroma:document": "This notebook can run either as a Colab notebook or as a regular Jupyter notebook.\n\nTable of contents:\n- Setup\n- Get DB connection\n- Connect to OpenAI\n- Load quotes into the Vector Store\n- Use case 1: **quote search engine**\n- Use case 2: **quote generator**\n- (Optional) exploit partitioning in the Vector Store\n\n### How it works\n\n**Indexing**\n\nEach quote is made into an embedding vector with OpenAI's `Embedding`. These are saved in the Vector Store for later use in searching. Some metadata, including the author's name and a few other pre-computed tags, are stored alongside, to allow for search customization.\n\n![1_vector_indexing](https://user-images.githubusercontent.com/14221764/282440878-dc3ed680-7d0e-4b30-9a74-d2d66a7394f7.png)\n\n**Search**\n\nTo find a quote similar to the provided search quote, the latter is made into an embedding vector on the fly, and this vector is used to query the store for similar vectors ... i.e. similar quotes that were previously indexed. The search can optionally be constrained by additional metadata (\"find me quotes by Spinoza similar to this one ...\").\n\n![2_vector_search](https://user-images.githubusercontent.com/14221764/282440908-683e3ee1-0bf1-46b3-8621-86c31fc7f9c9.png)\n\nThe key point here is that \"quotes similar in content\" translates, in vector space, to vectors that are metrically close to each other: thus, vector similarity search effectively implements semantic similarity. _This is the key reason vector embeddings are so powerful._\n\nThe sketch below tries to convey this idea. Each quote, once it's made into a vector, is a point in space. Well, in this case it's on a sphere, since OpenAI's embedding vectors, as most others, are normalized to _unit length_. Oh, and the sphere is actually not three-dimensional, rather 1536-dimensional!\n\nSo, in essence, a similarity search in vector space returns the vectors that are closest to the query vector:\n\n![3_vector_space](https://user-images.githubusercontent.com/14221764/262321363-c8c625c1-8be9-450e-8c68-b1ed518f990d.png)\n\n**Generation**\n\nGiven a suggestion (a topic or a tentative quote), the search step is performed, and the first returned results (quotes) are fed into an LLM prompt which asks the generative model to invent a new text along the lines of the passed examples _and_ the initial suggestion.\n\n![4_quote_generation](https://user-images.githubusercontent.com/14221764/282440927-d56f36eb-d611-4342-8026-7736edc6f5c9.png)\n\n## Setup\n\nFirst install some required packages:\n\n\n```python\n!pip install --quiet \"cassio>=0.1.3\" \"openai>=1.0.0\" datasets\n```"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 561a98df-9ed0-4ab9-bae6-8fe0fabf7ab5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL", "tokens": 6051, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"561a98df-9ed0-4ab9-bae6-8fe0fabf7ab5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8938aaa0-39b6-4913-afa3-644dfcdc84cb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"723f825cd19bd5d01794b8111cf19db861b8b27b9247f5207316ec1dc1f05a13\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"018bfaf7-fccb-438f-9684-a4dbcaab449e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"19b6f2faef47131a6fa8587cd08b2a33b202779078b71c32bf7ee449669a3378\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cb18b98c-de1a-4e46-b48f-c5cffbf94922\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"e05764c860cfdd91fb216182a0e4081a2031f0d1899c736affb67bdc921c9af1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2773, \"end_char_idx\": 5604, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "doc_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "ref_doc_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "chroma:document": "![2_vector_search](https://user-images.githubusercontent.com/14221764/282440908-683e3ee1-0bf1-46b3-8621-86c31fc7f9c9.png)\n\nThe key point here is that \"quotes similar in content\" translates, in vector space, to vectors that are metrically close to each other: thus, vector similarity search effectively implements semantic similarity. _This is the key reason vector embeddings are so powerful._\n\nThe sketch below tries to convey this idea. Each quote, once it's made into a vector, is a point in space. Well, in this case it's on a sphere, since OpenAI's embedding vectors, as most others, are normalized to _unit length_. Oh, and the sphere is actually not three-dimensional, rather 1536-dimensional!\n\nSo, in essence, a similarity search in vector space returns the vectors that are closest to the query vector:\n\n![3_vector_space](https://user-images.githubusercontent.com/14221764/262321363-c8c625c1-8be9-450e-8c68-b1ed518f990d.png)\n\n**Generation**\n\nGiven a suggestion (a topic or a tentative quote), the search step is performed, and the first returned results (quotes) are fed into an LLM prompt which asks the generative model to invent a new text along the lines of the passed examples _and_ the initial suggestion.\n\n![4_quote_generation](https://user-images.githubusercontent.com/14221764/282440927-d56f36eb-d611-4342-8026-7736edc6f5c9.png)\n\n## Setup\n\nFirst install some required packages:\n\n\n```python\n!pip install --quiet \"cassio>=0.1.3\" \"openai>=1.0.0\" datasets\n```\n\n\n```python\nfrom getpass import getpass\nfrom collections import Counter\n\nimport cassio\nfrom cassio.table import MetadataVectorCassandraTable\n\nimport openai\nfrom datasets import load_dataset\n```\n\n## Get DB connection\n\nIn order to connect to your Astra DB through CQL, you need two things:\n- A Token, with role \"Database Administrator\" (it looks like `AstraCS:...`)\n- the database ID (it looks like `3df2a5b6-...`)\n\n    Make sure you have both strings -- which are obtained in the [Astra UI](https://astra.datastax.com) once you sign in. For more information, see here: [database ID](https://awesome-astra.github.io/docs/pages/astra/faq/#where-should-i-find-a-database-identifier) and [Token](https://awesome-astra.github.io/docs/pages/astra/create-token/#c-procedure).\n\nIf you want to _connect to a Cassandra cluster_ (which however must [support](https://cassandra.apache.org/doc/trunk/cassandra/vector-search/overview.html) Vector Search), replace with `cassio.init(session=..., keyspace=...)` with suitable Session and keyspace name for your cluster.\n\n\n```python\nastra_token = getpass(\"Please enter your Astra token ('AstraCS:...')\")\ndatabase_id = input(\"Please enter your database id ('3df2a5b6-...')\")\n```\n\n    Please enter your Astra token ('AstraCS:...') \u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\n    Please enter your database id ('3df2a5b6-...') 01234567-89ab-dcef-0123-456789abcdef"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: cb18b98c-de1a-4e46-b48f-c5cffbf94922
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL", "tokens": 6051, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"cb18b98c-de1a-4e46-b48f-c5cffbf94922\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8938aaa0-39b6-4913-afa3-644dfcdc84cb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"723f825cd19bd5d01794b8111cf19db861b8b27b9247f5207316ec1dc1f05a13\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"561a98df-9ed0-4ab9-bae6-8fe0fabf7ab5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"10db62beb0c3df97d85066800a6b354f27f779763aea3b8ac5436554ae62f8b4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0f15e0ef-46bd-4be2-84a8-37485fe5f163\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"96e9cccddaf4bbbc3eea2886909f129d3b80a095db90b304ebb92206d1f228cc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4171, \"end_char_idx\": 6996, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "doc_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "ref_doc_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "chroma:document": "```python\n!pip install --quiet \"cassio>=0.1.3\" \"openai>=1.0.0\" datasets\n```\n\n\n```python\nfrom getpass import getpass\nfrom collections import Counter\n\nimport cassio\nfrom cassio.table import MetadataVectorCassandraTable\n\nimport openai\nfrom datasets import load_dataset\n```\n\n## Get DB connection\n\nIn order to connect to your Astra DB through CQL, you need two things:\n- A Token, with role \"Database Administrator\" (it looks like `AstraCS:...`)\n- the database ID (it looks like `3df2a5b6-...`)\n\n    Make sure you have both strings -- which are obtained in the [Astra UI](https://astra.datastax.com) once you sign in. For more information, see here: [database ID](https://awesome-astra.github.io/docs/pages/astra/faq/#where-should-i-find-a-database-identifier) and [Token](https://awesome-astra.github.io/docs/pages/astra/create-token/#c-procedure).\n\nIf you want to _connect to a Cassandra cluster_ (which however must [support](https://cassandra.apache.org/doc/trunk/cassandra/vector-search/overview.html) Vector Search), replace with `cassio.init(session=..., keyspace=...)` with suitable Session and keyspace name for your cluster.\n\n\n```python\nastra_token = getpass(\"Please enter your Astra token ('AstraCS:...')\")\ndatabase_id = input(\"Please enter your database id ('3df2a5b6-...')\")\n```\n\n    Please enter your Astra token ('AstraCS:...') \u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\n    Please enter your database id ('3df2a5b6-...') 01234567-89ab-dcef-0123-456789abcdef\n\n\n\n```python\ncassio.init(token=astra_token, database_id=database_id)\n```\n\n### Creation of the DB connection\n\nThis is how you create a connection to Astra DB through CQL:\n\n_(Incidentally, you could also use any Cassandra cluster (as long as it provides Vector capabilities), just by [changing the parameters](https://docs.datastax.com/en/developer/python-driver/latest/getting_started/#connecting-to-cassandra) to the following `Cluster` instantiation.)_\n\n### Creation of the Vector Store through CassIO\n\nYou need a table which support vectors and is equipped with metadata. Call it \"philosophers_cassio\":\n\n\n```python\nv_table = MetadataVectorCassandraTable(table=\"philosophers_cassio\", vector_dimension=1536)\n```\n\n## Connect to OpenAI\n\n### Set up your secret key\n\n\n```python\nOPENAI_API_KEY = getpass(\"Please enter your OpenAI API Key: \")\n```\n\n    Please enter your OpenAI API Key:  \u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\n\n\n### A test call for embeddings\n\nQuickly check how one can get the embedding vectors for a list of input texts:\n\n\n```python\nclient = openai.OpenAI(api_key=OPENAI_API_KEY)\nembedding_model_name = \"text-embedding-3-small\"\n\nresult = client.embeddings.create(\n    input=[\n        \"This is a sentence\",\n        \"A second sentence\"\n    ],\n    model=embedding_model_name,\n)\n```\n\n_Note: the above is the syntax for OpenAI v1.0+. If using previous versions, the code to get the embeddings will look different._"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 0f15e0ef-46bd-4be2-84a8-37485fe5f163
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL", "tokens": 6051, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0f15e0ef-46bd-4be2-84a8-37485fe5f163\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8938aaa0-39b6-4913-afa3-644dfcdc84cb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"723f825cd19bd5d01794b8111cf19db861b8b27b9247f5207316ec1dc1f05a13\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cb18b98c-de1a-4e46-b48f-c5cffbf94922\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c5b85ac5d0b8ac2b90c54bd1b4b38465bc6df8da00c50a98505bc7247fc806c7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e8bddefe-2fd3-4184-8338-9aa074e5fa84\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d21d4aa5e27b2be76776c89aaee9d1fb592d04eba872e3be0a9f6ec03f396635\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5608, \"end_char_idx\": 8538, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "doc_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "ref_doc_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "chroma:document": "```python\ncassio.init(token=astra_token, database_id=database_id)\n```\n\n### Creation of the DB connection\n\nThis is how you create a connection to Astra DB through CQL:\n\n_(Incidentally, you could also use any Cassandra cluster (as long as it provides Vector capabilities), just by [changing the parameters](https://docs.datastax.com/en/developer/python-driver/latest/getting_started/#connecting-to-cassandra) to the following `Cluster` instantiation.)_\n\n### Creation of the Vector Store through CassIO\n\nYou need a table which support vectors and is equipped with metadata. Call it \"philosophers_cassio\":\n\n\n```python\nv_table = MetadataVectorCassandraTable(table=\"philosophers_cassio\", vector_dimension=1536)\n```\n\n## Connect to OpenAI\n\n### Set up your secret key\n\n\n```python\nOPENAI_API_KEY = getpass(\"Please enter your OpenAI API Key: \")\n```\n\n    Please enter your OpenAI API Key:  \u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\u00b7\n\n\n### A test call for embeddings\n\nQuickly check how one can get the embedding vectors for a list of input texts:\n\n\n```python\nclient = openai.OpenAI(api_key=OPENAI_API_KEY)\nembedding_model_name = \"text-embedding-3-small\"\n\nresult = client.embeddings.create(\n    input=[\n        \"This is a sentence\",\n        \"A second sentence\"\n    ],\n    model=embedding_model_name,\n)\n```\n\n_Note: the above is the syntax for OpenAI v1.0+. If using previous versions, the code to get the embeddings will look different._\n\n\n```python\nprint(f\"len(result.data)              = {len(result.data)}\")\nprint(f\"result.data[1].embedding      = {str(result.data[1].embedding)[:55]}...\")\nprint(f\"len(result.data[1].embedding) = {len(result.data[1].embedding)}\")\n```\n\n    len(result.data)              = 2\n    result.data[1].embedding      = [-0.010821706615388393, 0.001387271680869162, 0.0035479...\n    len(result.data[1].embedding) = 1536\n\n\n## Load quotes into the Vector Store\n\n_Note: the above is the syntax for OpenAI v1.0+. If using previous versions, the code to get the embeddings will look different._\n\n\n```python\nphilo_dataset = load_dataset(\"datastax/philosopher-quotes\")[\"train\"]\n```\n\nA quick inspection:\n\n\n```python\nprint(\"An example entry:\")\nprint(philo_dataset[16])\n```\n\n    An example entry:\n    {'author': 'aristotle', 'quote': 'Love well, be loved and do something of value.', 'tags': 'love;ethics'}\n\n\nCheck the dataset size:\n\n\n```python\nauthor_count = Counter(entry[\"author\"] for entry in philo_dataset)\nprint(f\"Total: {len(philo_dataset)} quotes. By author:\")\nfor author, count in author_count.most_common():\n    print(f\"    {author:<20}: {count} quotes\")\n```\n\n    Total: 450 quotes. By author:\n        aristotle           : 50 quotes\n        schopenhauer        : 50 quotes\n        spinoza             : 50 quotes\n        hegel               : 50 quotes\n        freud               : 50 quotes\n        nietzsche           : 50 quotes\n        sartre              : 50 quotes\n        plato               : 50 quotes\n        kant                : 50 quotes"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: e8bddefe-2fd3-4184-8338-9aa074e5fa84
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL", "tokens": 6051, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"e8bddefe-2fd3-4184-8338-9aa074e5fa84\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8938aaa0-39b6-4913-afa3-644dfcdc84cb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"723f825cd19bd5d01794b8111cf19db861b8b27b9247f5207316ec1dc1f05a13\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0f15e0ef-46bd-4be2-84a8-37485fe5f163\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"3d9e895b8c6c03a4359005c73e41ee0031e2e7032633227c29ae0cb02b4dd726\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8cdc399a-029c-43c7-be3c-11b583258ee0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7a5f4b27ee2c6e7a393c1d8cfbf8278d8cfe88f364df39cf1691eab6e117218d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7407, \"end_char_idx\": 9092, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "doc_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "ref_doc_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "chroma:document": "## Load quotes into the Vector Store\n\n_Note: the above is the syntax for OpenAI v1.0+. If using previous versions, the code to get the embeddings will look different._\n\n\n```python\nphilo_dataset = load_dataset(\"datastax/philosopher-quotes\")[\"train\"]\n```\n\nA quick inspection:\n\n\n```python\nprint(\"An example entry:\")\nprint(philo_dataset[16])\n```\n\n    An example entry:\n    {'author': 'aristotle', 'quote': 'Love well, be loved and do something of value.', 'tags': 'love;ethics'}\n\n\nCheck the dataset size:\n\n\n```python\nauthor_count = Counter(entry[\"author\"] for entry in philo_dataset)\nprint(f\"Total: {len(philo_dataset)} quotes. By author:\")\nfor author, count in author_count.most_common():\n    print(f\"    {author:<20}: {count} quotes\")\n```\n\n    Total: 450 quotes. By author:\n        aristotle           : 50 quotes\n        schopenhauer        : 50 quotes\n        spinoza             : 50 quotes\n        hegel               : 50 quotes\n        freud               : 50 quotes\n        nietzsche           : 50 quotes\n        sartre              : 50 quotes\n        plato               : 50 quotes\n        kant                : 50 quotes\n\n\n### Insert quotes into vector store\n\nYou will compute the embeddings for the quotes and save them into the Vector Store, along with the text itself and the metadata planned for later use. Note that the author is added as a metadata field along with the \"tags\" already found with the quote itself.\n\nTo optimize speed and reduce the calls, you'll perform batched calls to the embedding OpenAI service.\n\n_(Note: for faster execution, Cassandra and CassIO would let you do concurrent inserts, which we don't do here for a more straightforward demo code.)_"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 8cdc399a-029c-43c7-be3c-11b583258ee0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL", "tokens": 6051, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"8cdc399a-029c-43c7-be3c-11b583258ee0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8938aaa0-39b6-4913-afa3-644dfcdc84cb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"723f825cd19bd5d01794b8111cf19db861b8b27b9247f5207316ec1dc1f05a13\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e8bddefe-2fd3-4184-8338-9aa074e5fa84\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"c5cbd6cd8354df2186d432871f980e893d087ebaf638c7cc2a5eaaa07e63d439\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3812499b-8ac6-467a-8cdc-12c438ef06b6\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"56239839ff89ff002fd281811addc799de22c6b9d3caf1c6fd33e52904cd7888\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7577, \"end_char_idx\": 11047, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "doc_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "ref_doc_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "chroma:document": "```python\nphilo_dataset = load_dataset(\"datastax/philosopher-quotes\")[\"train\"]\n```\n\nA quick inspection:\n\n\n```python\nprint(\"An example entry:\")\nprint(philo_dataset[16])\n```\n\n    An example entry:\n    {'author': 'aristotle', 'quote': 'Love well, be loved and do something of value.', 'tags': 'love;ethics'}\n\n\nCheck the dataset size:\n\n\n```python\nauthor_count = Counter(entry[\"author\"] for entry in philo_dataset)\nprint(f\"Total: {len(philo_dataset)} quotes. By author:\")\nfor author, count in author_count.most_common():\n    print(f\"    {author:<20}: {count} quotes\")\n```\n\n    Total: 450 quotes. By author:\n        aristotle           : 50 quotes\n        schopenhauer        : 50 quotes\n        spinoza             : 50 quotes\n        hegel               : 50 quotes\n        freud               : 50 quotes\n        nietzsche           : 50 quotes\n        sartre              : 50 quotes\n        plato               : 50 quotes\n        kant                : 50 quotes\n\n\n### Insert quotes into vector store\n\nYou will compute the embeddings for the quotes and save them into the Vector Store, along with the text itself and the metadata planned for later use. Note that the author is added as a metadata field along with the \"tags\" already found with the quote itself.\n\nTo optimize speed and reduce the calls, you'll perform batched calls to the embedding OpenAI service.\n\n_(Note: for faster execution, Cassandra and CassIO would let you do concurrent inserts, which we don't do here for a more straightforward demo code.)_\n\n\n```python\nBATCH_SIZE = 50\n\nnum_batches = ((len(philo_dataset) + BATCH_SIZE - 1) // BATCH_SIZE)\n\nquotes_list = philo_dataset[\"quote\"]\nauthors_list = philo_dataset[\"author\"]\ntags_list = philo_dataset[\"tags\"]\n\nprint(\"Starting to store entries:\")\nfor batch_i in range(num_batches):\n    b_start = batch_i * BATCH_SIZE\n    b_end = (batch_i + 1) * BATCH_SIZE\n    # compute the embedding vectors for this batch\n    b_emb_results = client.embeddings.create(\n        input=quotes_list[b_start : b_end],\n        model=embedding_model_name,\n    )\n    # prepare the rows for insertion\n    print(\"B \", end=\"\")\n    for entry_idx, emb_result in zip(range(b_start, b_end), b_emb_results.data):\n        if tags_list[entry_idx]:\n            tags = {\n                tag\n                for tag in tags_list[entry_idx].split(\";\")\n            }\n        else:\n            tags = set()\n        author = authors_list[entry_idx]\n        quote = quotes_list[entry_idx]\n        v_table.put(\n            row_id=f\"q_{author}_{entry_idx}\",\n            body_blob=quote,\n            vector=emb_result.embedding,\n            metadata={**{tag: True for tag in tags}, **{\"author\": author}},\n        )\n        print(\"*\", end=\"\")\n    print(f\" done ({len(b_emb_results.data)})\")\n\nprint(\"\\nFinished storing entries.\")\n```\n\n    Starting to store entries:\n    B ************************************************** done (50)\n    B ************************************************** done (50)\n    B ************************************************** done (50)\n    B ************************************************** done (50)\n    B ************************************************** done (50)\n    B ************************************************** done (50)\n    B ************************************************** done (50)\n    B ************************************************** done (50)\n    B ************************************************** done (50)\n    \n    Finished storing entries."}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 3812499b-8ac6-467a-8cdc-12c438ef06b6
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL", "tokens": 6051, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"3812499b-8ac6-467a-8cdc-12c438ef06b6\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8938aaa0-39b6-4913-afa3-644dfcdc84cb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"723f825cd19bd5d01794b8111cf19db861b8b27b9247f5207316ec1dc1f05a13\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8cdc399a-029c-43c7-be3c-11b583258ee0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"8d90550b45b02037461caf24de1b246e10d1cddef61039549176249461a05583\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0472123c-3436-4578-a8e3-09a38a1f6c2e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3d1b896f62928984106f1753247dcadafa3a18f0ba2c4cfa1c04b79a79d2a576\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9095, \"end_char_idx\": 12581, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "doc_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "ref_doc_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "chroma:document": "```python\nBATCH_SIZE = 50\n\nnum_batches = ((len(philo_dataset) + BATCH_SIZE - 1) // BATCH_SIZE)\n\nquotes_list = philo_dataset[\"quote\"]\nauthors_list = philo_dataset[\"author\"]\ntags_list = philo_dataset[\"tags\"]\n\nprint(\"Starting to store entries:\")\nfor batch_i in range(num_batches):\n    b_start = batch_i * BATCH_SIZE\n    b_end = (batch_i + 1) * BATCH_SIZE\n    # compute the embedding vectors for this batch\n    b_emb_results = client.embeddings.create(\n        input=quotes_list[b_start : b_end],\n        model=embedding_model_name,\n    )\n    # prepare the rows for insertion\n    print(\"B \", end=\"\")\n    for entry_idx, emb_result in zip(range(b_start, b_end), b_emb_results.data):\n        if tags_list[entry_idx]:\n            tags = {\n                tag\n                for tag in tags_list[entry_idx].split(\";\")\n            }\n        else:\n            tags = set()\n        author = authors_list[entry_idx]\n        quote = quotes_list[entry_idx]\n        v_table.put(\n            row_id=f\"q_{author}_{entry_idx}\",\n            body_blob=quote,\n            vector=emb_result.embedding,\n            metadata={**{tag: True for tag in tags}, **{\"author\": author}},\n        )\n        print(\"*\", end=\"\")\n    print(f\" done ({len(b_emb_results.data)})\")\n\nprint(\"\\nFinished storing entries.\")\n```\n\n    Starting to store entries:\n    B ************************************************** done (50)\n    B ************************************************** done (50)\n    B ************************************************** done (50)\n    B ************************************************** done (50)\n    B ************************************************** done (50)\n    B ************************************************** done (50)\n    B ************************************************** done (50)\n    B ************************************************** done (50)\n    B ************************************************** done (50)\n    \n    Finished storing entries.\n\n\n## Use case 1: **quote search engine**\n\nFor the quote-search functionality, you need first to make the input quote into a vector, and then use it to query the store (besides handling the optional metadata into the search call, that is).\n\nEncapsulate the search-engine functionality into a function for ease of re-use:\n\n\n```python\ndef find_quote_and_author(query_quote, n, author=None, tags=None):\n    query_vector = client.embeddings.create(\n        input=[query_quote],\n        model=embedding_model_name,\n    ).data[0].embedding\n    metadata = {}\n    if author:\n        metadata[\"author\"] = author\n    if tags:\n        for tag in tags:\n            metadata[tag] = True\n    #\n    results = v_table.ann_search(\n        query_vector,\n        n=n,\n        metadata=metadata,\n    )\n    return [\n        (result[\"body_blob\"], result[\"metadata\"][\"author\"])\n        for result in results\n    ]\n```\n\n### Putting search to test\n\nPassing just a quote:\n\n\n```python\nfind_quote_and_author(\"We struggle all our life for nothing\", 3)\n```\n\n\n\n\n    [('Life to the great majority is only a constant struggle for mere existence, with the certainty of losing it at last.',\n      'schopenhauer'),\n     ('We give up leisure in order that we may have leisure, just as we go to war in order that we may have peace.',\n      'aristotle'),\n     ('Perhaps the gods are kind to us, by making life more disagreeable as we grow older. In the end death seems less intolerable than the manifold burdens we carry',\n      'freud')]\n\n\n\nSearch restricted to an author:"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 0472123c-3436-4578-a8e3-09a38a1f6c2e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL", "tokens": 6051, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"0472123c-3436-4578-a8e3-09a38a1f6c2e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8938aaa0-39b6-4913-afa3-644dfcdc84cb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"723f825cd19bd5d01794b8111cf19db861b8b27b9247f5207316ec1dc1f05a13\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3812499b-8ac6-467a-8cdc-12c438ef06b6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"e5f0820022b91c403566c4d1e598952f37aa7625e16a4a70209c2449a6b96330\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"917f5c67-4765-4236-bbd1-e5c2895d11e4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"674d2ce49e35837bb7699f4d5ccdd387f1f9d1cf740937254d492f9ceb101336\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11050, \"end_char_idx\": 13363, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "doc_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "ref_doc_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "chroma:document": "## Use case 1: **quote search engine**\n\nFor the quote-search functionality, you need first to make the input quote into a vector, and then use it to query the store (besides handling the optional metadata into the search call, that is).\n\nEncapsulate the search-engine functionality into a function for ease of re-use:\n\n\n```python\ndef find_quote_and_author(query_quote, n, author=None, tags=None):\n    query_vector = client.embeddings.create(\n        input=[query_quote],\n        model=embedding_model_name,\n    ).data[0].embedding\n    metadata = {}\n    if author:\n        metadata[\"author\"] = author\n    if tags:\n        for tag in tags:\n            metadata[tag] = True\n    #\n    results = v_table.ann_search(\n        query_vector,\n        n=n,\n        metadata=metadata,\n    )\n    return [\n        (result[\"body_blob\"], result[\"metadata\"][\"author\"])\n        for result in results\n    ]\n```\n\n### Putting search to test\n\nPassing just a quote:\n\n\n```python\nfind_quote_and_author(\"We struggle all our life for nothing\", 3)\n```\n\n\n\n\n    [('Life to the great majority is only a constant struggle for mere existence, with the certainty of losing it at last.',\n      'schopenhauer'),\n     ('We give up leisure in order that we may have leisure, just as we go to war in order that we may have peace.',\n      'aristotle'),\n     ('Perhaps the gods are kind to us, by making life more disagreeable as we grow older. In the end death seems less intolerable than the manifold burdens we carry',\n      'freud')]\n\n\n\nSearch restricted to an author:\n\n\n```python\nfind_quote_and_author(\"We struggle all our life for nothing\", 2, author=\"nietzsche\")\n```\n\n\n\n\n    [('To live is to suffer, to survive is to find some meaning in the suffering.',\n      'nietzsche'),\n     ('What makes us heroic?--Confronting simultaneously our supreme suffering and our supreme hope.',\n      'nietzsche')]\n\n\n\nSearch constrained to a tag (out of those saved earlier with the quotes):\n\n\n```python\nfind_quote_and_author(\"We struggle all our life for nothing\", 2, tags=[\"politics\"])\n```\n\n\n\n\n    [('Mankind will never see an end of trouble until lovers of wisdom come to hold political power, or the holders of power become lovers of wisdom',\n      'plato'),\n     ('Everything the State says is a lie, and everything it has it has stolen.',\n      'nietzsche')]"}


created_at: 2024-08-26 20:31:21
topic: persistent://default/default/47f0afae-77ee-4e4a-bcd4-0b050f4de62c
id: 917f5c67-4765-4236-bbd1-e5c2895d11e4
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb", "title": "Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL", "tokens": 6051, "retrieve_doc": true, "source": "openai_cookbooks", "_node_content": "{\"id_\": \"917f5c67-4765-4236-bbd1-e5c2895d11e4\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8938aaa0-39b6-4913-afa3-644dfcdc84cb\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"723f825cd19bd5d01794b8111cf19db861b8b27b9247f5207316ec1dc1f05a13\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0472123c-3436-4578-a8e3-09a38a1f6c2e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/vector_databases/cassandra_astradb/Philosophical_Quotes_cassIO.ipynb\", \"title\": \"Philosophy with Vector Embeddings, OpenAI and Cassandra / Astra DB through CQL\", \"tokens\": 6051, \"retrieve_doc\": true, \"source\": \"openai_cookbooks\"}, \"hash\": \"2e0235d8613999234e13485a56665ee4b7159cb978e5b0aaf8f8f24739770aae\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"814e8160-7045-4570-937c-07fb489ff94c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"46b54953fac4856142fa32a9ed617716bebde4b7627df54619e571f59a013c1d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11995, \"end_char_idx\": 14557, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "doc_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "ref_doc_id": "8938aaa0-39b6-4913-afa3-644dfcdc84cb", "chroma:document": "```python\nfind_quote_and_author(\"We struggle all our life for nothing\", 3)\n```\n\n\n\n\n    [('Life to the great majority is only a constant struggle for mere existence, with the certainty of losing it at last.',\n      'schopenhauer'),\n     ('We give up leisure in order that we may have leisure, just as we go to war in order that we may have peace.',\n      'aristotle'),\n     ('Perhaps the gods are kind to us, by making life more disagreeable as we grow older. In the end death seems less intolerable than the manifold burdens we carry',\n      'freud')]\n\n\n\nSearch restricted to an author:\n\n\n```python\nfind_quote_and_author(\"We struggle all our life for nothing\", 2, author=\"nietzsche\")\n```\n\n\n\n\n    [('To live is to suffer, to survive is to find some meaning in the suffering.',\n      'nietzsche'),\n     ('What makes us heroic?--Confronting simultaneously our supreme suffering and our supreme hope.',\n      'nietzsche')]\n\n\n\nSearch constrained to a tag (out of those saved earlier with the quotes):\n\n\n```python\nfind_quote_and_author(\"We struggle all our life for nothing\", 2, tags=[\"politics\"])\n```\n\n\n\n\n    [('Mankind will never see an end of trouble until lovers of wisdom come to hold political power, or the holders of power become lovers of wisdom',\n      'plato'),\n     ('Everything the State says is a lie, and everything it has it has stolen.',\n      'nietzsche')]\n\n\n\n### Cutting out irrelevant results\n\nThe vector similarity search generally returns the vectors that are closest to the query, even if that means results that might be somewhat irrelevant if there's nothing better.\n\nTo keep this issue under control, you can get the actual \"distance\" between the query and each result, and then set a cutoff on it, effectively discarding results that are beyond that threshold.\nTuning this threshold correctly is not an easy problem: here, we'll just show you the way.\n\nTo get a feeling on how this works, try the following query and play with the choice of quote and threshold to compare the results:\n\n_Note (for the mathematically inclined): this \"distance\" is exactly the cosine similarity between the vectors, i.e. the scalar product divided by the product of the norms of the two vectors. As such, it is a number ranging from -1 to +1, where -1 is for exactly opposite-facing vectors and +1 for identically-oriented vectors. Elsewhere (e.g. in the \"CQL\" counterpart of this demo) you would get a rescaling of this quantity to fit the [0, 1] interval, which means the resulting numerical values and adequate thresholds there are transformed accordingly._"}


