# embeddings_queue

created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: cecf4b08-ad32-4e94-8d2f-3a56f3dde475
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"cecf4b08-ad32-4e94-8d2f-3a56f3dde475\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d41c37b5-11ff-416c-9303-0daab905e61f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"bd6309d85754359747104e4d5a7c9b5f0a223073ce49b3d0e67a24c91449ebb3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3009, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "# Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\n\nThe aim of this notebook is to walk through a comprehensive example of how to fine-tune OpenAI models for Retrieval Augmented Generation (RAG). \n\nWe will also be integrating Qdrant and Few-Shot Learning to boost the model's performance and reduce hallucinations. This could serve as a practical guide for ML practitioners, data scientists, and AI Engineers interested in leveraging the power of OpenAI models for specific use-cases. \ud83e\udd29\n\n## Why should you read this blog?\n\nYou want to learn how to \n- [Fine-tune OpenAI models](https://platform.openai.com/docs/guides/fine-tuning/) for specific use-cases\n- Use [Qdrant](https://qdrant.tech/documentation/) to improve the performance of your RAG model\n- Use fine-tuning to improve the correctness of your RAG model and reduce hallucinations\n\nTo begin, we've selected a dataset where we've a guarantee that the retrieval is perfect. We've selected a subset of the [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/) dataset, which is a collection of questions and answers about Wikipedia articles. We've also included samples where the answer is not present in the context, to demonstrate how RAG handles this case.\n\n## Table of Contents\n1. Setting up the Environment\n\n### Section A: Zero-Shot Learning\n2. Data Preparation: SQuADv2 Dataset\n3. Answering using Base gpt-3.5-turbo-0613 model\n4. Fine-tuning and Answering using Fine-tuned model\n5. **Evaluation**: How well does the model perform?\n\n### Section B: Few-Shot Learning\n\n6. Using Qdrant to Improve RAG Prompt\n7. Fine-Tuning OpenAI Model with Qdrant\n8. Evaluation\n\n9. **Conclusion**\n    - Aggregate Results\n    - Observations\n\n## Terms, Definitions, and References\n\n**Retrieval Augmented Generation (RAG)?**\nThe phrase Retrieval Augmented Generation (RAG) comes from a [recent paper](https://arxiv.org/abs/2005.11401) by Lewis et al. from Facebook AI. The idea is to use a pre-trained language model (LM) to generate text, but to use a separate retrieval system to find relevant documents to condition the LM on. \n\n**What is Qdrant?**\nQdrant is an open-source vector search engine that allows you to search for similar vectors in a large dataset. It is built in Rust and here we'll use the Python client to interact with it. This is the Retrieval part of RAG.\n\n**What is Few-Shot Learning?**\nFew-shot learning is a type of machine learning where the model is \"improved\" via training or fine-tuning on a small amount of data. In this case, we'll use it to fine-tune the RAG model on a small number of examples from the SQuAD dataset. This is the Augmented part of RAG.\n\n**What is Zero-Shot Learning?**\nZero-shot learning is a type of machine learning where the model is \"improved\" via training or fine-tuning without any dataset specific information. \n\n**What is Fine-Tuning?**\nFine-tuning is a type of machine learning where the model is \"improved\" via training or fine-tuning on a small amount of data."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: d41c37b5-11ff-416c-9303-0daab905e61f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"d41c37b5-11ff-416c-9303-0daab905e61f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cecf4b08-ad32-4e94-8d2f-3a56f3dde475\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f6e31535485cab75ed7ede00ee67d5df7591527b88e7c2b3bdca44370b03837e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"19cc82c2-91c5-4fa4-8ed7-086d07dab76d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"19882e9be2e5a9df2b233c2312f3da3ffc7779d8b9286d05594df5ddda2a7578\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1442, \"end_char_idx\": 4059, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "Fine-tuning and Answering using Fine-tuned model\n5. **Evaluation**: How well does the model perform?\n\n### Section B: Few-Shot Learning\n\n6. Using Qdrant to Improve RAG Prompt\n7. Fine-Tuning OpenAI Model with Qdrant\n8. Evaluation\n\n9. **Conclusion**\n    - Aggregate Results\n    - Observations\n\n## Terms, Definitions, and References\n\n**Retrieval Augmented Generation (RAG)?**\nThe phrase Retrieval Augmented Generation (RAG) comes from a [recent paper](https://arxiv.org/abs/2005.11401) by Lewis et al. from Facebook AI. The idea is to use a pre-trained language model (LM) to generate text, but to use a separate retrieval system to find relevant documents to condition the LM on. \n\n**What is Qdrant?**\nQdrant is an open-source vector search engine that allows you to search for similar vectors in a large dataset. It is built in Rust and here we'll use the Python client to interact with it. This is the Retrieval part of RAG.\n\n**What is Few-Shot Learning?**\nFew-shot learning is a type of machine learning where the model is \"improved\" via training or fine-tuning on a small amount of data. In this case, we'll use it to fine-tune the RAG model on a small number of examples from the SQuAD dataset. This is the Augmented part of RAG.\n\n**What is Zero-Shot Learning?**\nZero-shot learning is a type of machine learning where the model is \"improved\" via training or fine-tuning without any dataset specific information. \n\n**What is Fine-Tuning?**\nFine-tuning is a type of machine learning where the model is \"improved\" via training or fine-tuning on a small amount of data. In this case, we'll use it to fine-tune the RAG model on a small number of examples from the SQuAD dataset. The LLM is what makes the Generation part of RAG.\n\n## 1. Setting Up the Environment\n\n### Install and Import Dependencies\n\n\n```python\n!pip install pandas openai tqdm tenacity scikit-learn tiktoken python-dotenv seaborn --upgrade --quiet\n```\n\n\n```python\nimport json\nimport os\nimport time\n\nimport pandas as pd\nfrom openai import OpenAI\nimport tiktoken\nimport seaborn as sns\nfrom tenacity import retry, wait_exponential\nfrom tqdm import tqdm\nfrom collections import defaultdict\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport numpy as np\nfrom sklearn.metrics import confusion_matrix\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\ntqdm.pandas()\n\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n\n```\n\n### Set your keys\nGet your OpenAI keys [here](https://platform.openai.com/account/api-keys) and Qdrant keys after making a free cluster [here](https://cloud.qdrant.io/login)."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 19cc82c2-91c5-4fa4-8ed7-086d07dab76d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"19cc82c2-91c5-4fa4-8ed7-086d07dab76d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d41c37b5-11ff-416c-9303-0daab905e61f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"dc4685e884045b34ce2fb518f38ae16eac4dbdc621264c4531a3dc9aa2e1b9f4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8f71d7b9-b4a4-43bb-a703-5ac579df4ae5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7afdd8b85611d4f227050ddcbfda91240c9004a269ef33b06ba69acbac0892b5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2531, \"end_char_idx\": 4967, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "In this case, we'll use it to fine-tune the RAG model on a small number of examples from the SQuAD dataset. This is the Augmented part of RAG.\n\n**What is Zero-Shot Learning?**\nZero-shot learning is a type of machine learning where the model is \"improved\" via training or fine-tuning without any dataset specific information. \n\n**What is Fine-Tuning?**\nFine-tuning is a type of machine learning where the model is \"improved\" via training or fine-tuning on a small amount of data. In this case, we'll use it to fine-tune the RAG model on a small number of examples from the SQuAD dataset. The LLM is what makes the Generation part of RAG.\n\n## 1. Setting Up the Environment\n\n### Install and Import Dependencies\n\n\n```python\n!pip install pandas openai tqdm tenacity scikit-learn tiktoken python-dotenv seaborn --upgrade --quiet\n```\n\n\n```python\nimport json\nimport os\nimport time\n\nimport pandas as pd\nfrom openai import OpenAI\nimport tiktoken\nimport seaborn as sns\nfrom tenacity import retry, wait_exponential\nfrom tqdm import tqdm\nfrom collections import defaultdict\nimport numpy as np\nimport matplotlib.pyplot as plt\nimport numpy as np\nfrom sklearn.metrics import confusion_matrix\n\nimport warnings\nwarnings.filterwarnings('ignore')\n\ntqdm.pandas()\n\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n\n```\n\n### Set your keys\nGet your OpenAI keys [here](https://platform.openai.com/account/api-keys) and Qdrant keys after making a free cluster [here](https://cloud.qdrant.io/login).\n\n\n```python\nos.environ[\"QDRANT_URL\"] = \"https://xxx.cloud.qdrant.io:6333\"\nos.environ[\"QDRANT_API_KEY\"] = \"xxx\"\n```\n\n## Section A\n\n## 2. Data Preparation: SQuADv2 Data Subsets\n\nFor the purpose of demonstration, we'll make small slices from the train and validation splits of the [SQuADv2](https://rajpurkar.github.io/SQuAD-explorer/) dataset. This dataset has questions and contexts where the answer is not present in the context, to help us evaluate how LLM handles this case.\n\nWe'll read the data from the JSON files and create a dataframe with the following columns: `question`, `context`, `answer`, `is_impossible`.\n\n### Download the Data\n\n\n```python\n# !mkdir -p local_cache\n# !wget https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v2.0.json -O local_cache/train.json\n# !wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v2.0.json -O local_cache/dev.json\n```\n\n### Read JSON to DataFrame"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 8f71d7b9-b4a4-43bb-a703-5ac579df4ae5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"8f71d7b9-b4a4-43bb-a703-5ac579df4ae5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"19cc82c2-91c5-4fa4-8ed7-086d07dab76d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"fbf5b0a4563d22c078f31eb84d75475532a766c71d5a31177df1f2174c8e89b6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ce4b90f3-322a-425b-b382-eea2f9312d99\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a42d924a367cff538724c55c136c21c210e8bfa39cf70b01b6d4e16bc4a83f25\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4062, \"end_char_idx\": 7384, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "```python\nos.environ[\"QDRANT_URL\"] = \"https://xxx.cloud.qdrant.io:6333\"\nos.environ[\"QDRANT_API_KEY\"] = \"xxx\"\n```\n\n## Section A\n\n## 2. Data Preparation: SQuADv2 Data Subsets\n\nFor the purpose of demonstration, we'll make small slices from the train and validation splits of the [SQuADv2](https://rajpurkar.github.io/SQuAD-explorer/) dataset. This dataset has questions and contexts where the answer is not present in the context, to help us evaluate how LLM handles this case.\n\nWe'll read the data from the JSON files and create a dataframe with the following columns: `question`, `context`, `answer`, `is_impossible`.\n\n### Download the Data\n\n\n```python\n# !mkdir -p local_cache\n# !wget https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v2.0.json -O local_cache/train.json\n# !wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v2.0.json -O local_cache/dev.json\n```\n\n### Read JSON to DataFrame\n\n\n```python\ndef json_to_dataframe_with_titles(json_data):\n    qas = []\n    context = []\n    is_impossible = []\n    answers = []\n    titles = []\n\n    for article in json_data['data']:\n        title = article['title']\n        for paragraph in article['paragraphs']:\n            for qa in paragraph['qas']:\n                qas.append(qa['question'].strip())\n                context.append(paragraph['context'])\n                is_impossible.append(qa['is_impossible'])\n                \n                ans_list = []\n                for ans in qa['answers']:\n                    ans_list.append(ans['text'])\n                answers.append(ans_list)\n                titles.append(title)\n\n    df = pd.DataFrame({'title': titles, 'question': qas, 'context': context, 'is_impossible': is_impossible, 'answers': answers})\n    return df\n\ndef get_diverse_sample(df, sample_size=100, random_state=42):\n    \"\"\"\n    Get a diverse sample of the dataframe by sampling from each title\n    \"\"\"\n    sample_df = df.groupby(['title', 'is_impossible']).apply(lambda x: x.sample(min(len(x), max(1, sample_size // 50)), random_state=random_state)).reset_index(drop=True)\n    \n    if len(sample_df) < sample_size:\n        remaining_sample_size = sample_size - len(sample_df)\n        remaining_df = df.drop(sample_df.index).sample(remaining_sample_size, random_state=random_state)\n        sample_df = pd.concat([sample_df, remaining_df]).sample(frac=1, random_state=random_state).reset_index(drop=True)\n\n    return sample_df.sample(min(sample_size, len(sample_df)), random_state=random_state).reset_index(drop=True)\n\ntrain_df = json_to_dataframe_with_titles(json.load(open('local_cache/train.json')))\nval_df = json_to_dataframe_with_titles(json.load(open('local_cache/dev.json')))\n\ndf = get_diverse_sample(val_df, sample_size=100, random_state=42)\n```\n\n## 3. Answering using Base gpt-3.5-turbo-0613 model\n\n### 3.1 Zero Shot Prompt\n\nLet's start by using the base gpt-3.5-turbo-0613 model to answer the questions. This prompt is a simple concatenation of the question and context, with a separator token in between: `\\n\\n`. We've a simple instruction part of the prompt: \n\n> Answer the following Question based on the Context only. Only answer from the Context. If you don't know the answer, say 'I don't know'.\n\nOther prompts are possible, but this is a good starting point. We'll use this prompt to answer the questions in the validation set."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: ce4b90f3-322a-425b-b382-eea2f9312d99
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"ce4b90f3-322a-425b-b382-eea2f9312d99\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8f71d7b9-b4a4-43bb-a703-5ac579df4ae5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"018bae4a499e3d1a55637000df95359f6b9c222ca3a4d5641d9fa37502717c5c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6cff5e70-283b-4275-802b-b248f56aabfa\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5ec93f2cc416bf59589b037262979f8372d781ef569a34f7ad36309f7d723244\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "```python\n# Function to get prompt messages\ndef get_prompt(row):\n    return [\n        {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n        {\n            \"role\": \"user\",\n            \"content\": f\"\"\"Answer the following Question based on the Context only. Only answer from the Context. If you don't know the answer, say 'I don't know'.\n    Question: {row.question}\\n\\n\n    Context: {row.context}\\n\\n\n    Answer:\\n\"\"\",\n        },\n    ]\n```\n\n### 3.2 Answering using Zero Shot Prompt\n\nNext, you'll need some re-usable functions which make an OpenAI API Call and return the answer. You'll use the `ChatCompletion.create` endpoint of the API, which takes a prompt and returns the completed text.\n\n\n```python\n# Function with tenacity for retries\n@retry(wait=wait_exponential(multiplier=1, min=2, max=6))\ndef api_call(messages, model):\n    return client.chat.completions.create(\n        model=model,\n        messages=messages,\n        stop=[\"\\n\\n\"],\n        max_tokens=100,\n        temperature=0.0,\n    )\n\n\n# Main function to answer question\ndef answer_question(row, prompt_func=get_prompt, model=\"gpt-3.5-turbo\"):\n    messages = prompt_func(row)\n    response = api_call(messages, model)\n    return response.choices[0].message.content\n```\n\n\u23f0 **Time to run: ~3 min**, \ud83d\udedc Needs Internet Connection\n\n\n```python\n# Use progress_apply with tqdm for progress bar\ndf[\"generated_answer\"] = df.progress_apply(answer_question, axis=1)\ndf.to_json(\"local_cache/100_val.json\", orient=\"records\", lines=True)\ndf = pd.read_json(\"local_cache/100_val.json\", orient=\"records\", lines=True)\n```\n\n\n```python\ndf\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>title</th>\n      <th>question</th>\n      <th>context</th>\n      <th>is_impossible</th>\n      <th>answers</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Scottish_Parliament</td>\n      <td>What consequence of establishing the Scottish .</td>\n      <td>A procedural consequence of the establishment .</td>\n      <td>False</td>\n      <td>[able to vote on domestic legislation that app.</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Imperialism</td>\n      <td>Imperialism is less often associated with whic.</td>\n      <td>The principles of imperialism are often genera.</td>\n      <td>True</td>\n      <td>[]</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Economic_inequality</td>\n      <td>What issues can't prevent women from working o.</td>\n      <td>When a person\u2019s capabilities are lowered, they."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 6cff5e70-283b-4275-802b-b248f56aabfa
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"6cff5e70-283b-4275-802b-b248f56aabfa\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ce4b90f3-322a-425b-b382-eea2f9312d99\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"669af375e2f21650af57bf68528e2ede388ed2203b8c29869b2037389154dd5f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b502214c-8b64-40e6-b691-c0c047c14aca\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"efbbedffe65f71d470b5caef81673f53f61d570a03f7d2c364ec3ec1a03c3694\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "```python\ndf\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>title</th>\n      <th>question</th>\n      <th>context</th>\n      <th>is_impossible</th>\n      <th>answers</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Scottish_Parliament</td>\n      <td>What consequence of establishing the Scottish .</td>\n      <td>A procedural consequence of the establishment .</td>\n      <td>False</td>\n      <td>[able to vote on domestic legislation that app.</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Imperialism</td>\n      <td>Imperialism is less often associated with whic.</td>\n      <td>The principles of imperialism are often genera.</td>\n      <td>True</td>\n      <td>[]</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Economic_inequality</td>\n      <td>What issues can't prevent women from working o.</td>\n      <td>When a person\u2019s capabilities are lowered, they.</td>\n      <td>True</td>\n      <td>[]</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>Southern_California</td>\n      <td>What county are Los Angeles, Orange, San Diego.</td>\n      <td>Its counties of Los Angeles, Orange, San Diego.</td>\n      <td>True</td>\n      <td>[]</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>French_and_Indian_War</td>\n      <td>When was the deportation of Canadians?</td>\n      <td>Britain gained control of French Canada and Ac.</td>\n      <td>True</td>\n      <td>[]</td>\n    </tr>\n    <tr>\n      <th>.</th>\n      <td>.</td>\n      <td>.</td>\n      <td>.</td>\n      <td>.</td>\n      <td>.</td>\n    </tr>\n    <tr>\n      <th>95</th>\n      <td>Geology</td>\n      <td>In the layered Earth model, what is the inner .</td>\n      <td>Seismologists can use the arrival times of sei.</td>\n      <td>True</td>\n      <td>[]</td>\n    </tr>\n    <tr>\n      <th>96</th>\n      <td>Prime_number</td>\n      <td>What type of value would the Basel function ha.</td>\n      <td>The zeta function is closely related to prime .</td>\n      <td>True</td>\n      <td>[]</td>\n    </tr>\n    <tr>\n      <th>97</th>\n      <td>Fresno,"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: b502214c-8b64-40e6-b691-c0c047c14aca
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"b502214c-8b64-40e6-b691-c0c047c14aca\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"6cff5e70-283b-4275-802b-b248f56aabfa\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"606cadac54ba494b641a67bdbb73c2b1fe13a075dcb53f305b84c9981171ffa6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cfa84bf0-56ba-43dc-8fd0-1136792109ec\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5e4794ada91c8216cc43a36727992fa6a9e01640642155da2528350b4fd6f811\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "they.</td>\n      <td>True</td>\n      <td>[]</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>Southern_California</td>\n      <td>What county are Los Angeles, Orange, San Diego.</td>\n      <td>Its counties of Los Angeles, Orange, San Diego.</td>\n      <td>True</td>\n      <td>[]</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>French_and_Indian_War</td>\n      <td>When was the deportation of Canadians?</td>\n      <td>Britain gained control of French Canada and Ac.</td>\n      <td>True</td>\n      <td>[]</td>\n    </tr>\n    <tr>\n      <th>.</th>\n      <td>.</td>\n      <td>.</td>\n      <td>.</td>\n      <td>.</td>\n      <td>.</td>\n    </tr>\n    <tr>\n      <th>95</th>\n      <td>Geology</td>\n      <td>In the layered Earth model, what is the inner .</td>\n      <td>Seismologists can use the arrival times of sei.</td>\n      <td>True</td>\n      <td>[]</td>\n    </tr>\n    <tr>\n      <th>96</th>\n      <td>Prime_number</td>\n      <td>What type of value would the Basel function ha.</td>\n      <td>The zeta function is closely related to prime .</td>\n      <td>True</td>\n      <td>[]</td>\n    </tr>\n    <tr>\n      <th>97</th>\n      <td>Fresno,_California</td>\n      <td>What does the San Joaquin Valley Railroad cros.</td>\n      <td>Passenger rail service is provided by Amtrak S.</td>\n      <td>True</td>\n      <td>[]</td>\n    </tr>\n    <tr>\n      <th>98</th>\n      <td>Victoria_(Australia)</td>\n      <td>What party rules in Melbourne's inner regions?</td>\n      <td>The centre-left Australian Labor Party (ALP), .</td>\n      <td>False</td>\n      <td>[The Greens, Australian Greens, Greens]</td>\n    </tr>\n    <tr>\n      <th>99</th>\n      <td>Immune_system</td>\n      <td>The speed of the killing response of the human.</td>\n      <td>In humans, this response is activated by compl.</td>\n      <td>False</td>\n      <td>[signal amplification, signal amplification, s.</td>\n    </tr>\n  </tbody>\n</table>\n<p>100 rows \u00d7 5 columns</p>\n</div>"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: cfa84bf0-56ba-43dc-8fd0-1136792109ec
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"cfa84bf0-56ba-43dc-8fd0-1136792109ec\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b502214c-8b64-40e6-b691-c0c047c14aca\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"72315285d36775efcb0ae431bd57b3848d8a21c5a1c17b651a7e5cc4311dacfd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7a91d318-e7c9-4f49-a3e1-afa9221de462\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"57ff9005f07cbfaec771da9f4ba06230b120d978ef00ec573a97af03aa38c9f8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "what is the inner .</td>\n      <td>Seismologists can use the arrival times of sei.</td>\n      <td>True</td>\n      <td>[]</td>\n    </tr>\n    <tr>\n      <th>96</th>\n      <td>Prime_number</td>\n      <td>What type of value would the Basel function ha.</td>\n      <td>The zeta function is closely related to prime .</td>\n      <td>True</td>\n      <td>[]</td>\n    </tr>\n    <tr>\n      <th>97</th>\n      <td>Fresno,_California</td>\n      <td>What does the San Joaquin Valley Railroad cros.</td>\n      <td>Passenger rail service is provided by Amtrak S.</td>\n      <td>True</td>\n      <td>[]</td>\n    </tr>\n    <tr>\n      <th>98</th>\n      <td>Victoria_(Australia)</td>\n      <td>What party rules in Melbourne's inner regions?</td>\n      <td>The centre-left Australian Labor Party (ALP), .</td>\n      <td>False</td>\n      <td>[The Greens, Australian Greens, Greens]</td>\n    </tr>\n    <tr>\n      <th>99</th>\n      <td>Immune_system</td>\n      <td>The speed of the killing response of the human.</td>\n      <td>In humans, this response is activated by compl.</td>\n      <td>False</td>\n      <td>[signal amplification, signal amplification, s.</td>\n    </tr>\n  </tbody>\n</table>\n<p>100 rows \u00d7 5 columns</p>\n</div>\n\n\n\n## 4. Fine-tuning and Answering using Fine-tuned model\n\nFor the complete fine-tuning process, please refer to the [OpenAI Fine-Tuning Docs](https://platform.openai.com/docs/guides/fine-tuning/use-a-fine-tuned-model).\n\n### 4.1 Prepare the Fine-Tuning Data\n\nWe need to prepare the data for fine-tuning. We'll use a few samples from train split of same dataset as before, but we'll add the answer to the context. This will help the model learn to retrieve the answer from the context. \n\nOur instruction prompt is the same as before, and so is the system prompt."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 7a91d318-e7c9-4f49-a3e1-afa9221de462
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"7a91d318-e7c9-4f49-a3e1-afa9221de462\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cfa84bf0-56ba-43dc-8fd0-1136792109ec\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"2f39cd95e780a82b6d34667740e4ea1d579e674b3a09091fcf60b22448105ba4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"88202e0d-4345-4c63-b155-b9d56ef9857c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6f74cef954d61cb79d144ad5fdf416e110044fb0132d3092b591f38daf10636c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "_California</td>\n      <td>What does the San Joaquin Valley Railroad cros.</td>\n      <td>Passenger rail service is provided by Amtrak S.</td>\n      <td>True</td>\n      <td>[]</td>\n    </tr>\n    <tr>\n      <th>98</th>\n      <td>Victoria_(Australia)</td>\n      <td>What party rules in Melbourne's inner regions?</td>\n      <td>The centre-left Australian Labor Party (ALP), .</td>\n      <td>False</td>\n      <td>[The Greens, Australian Greens, Greens]</td>\n    </tr>\n    <tr>\n      <th>99</th>\n      <td>Immune_system</td>\n      <td>The speed of the killing response of the human.</td>\n      <td>In humans, this response is activated by compl.</td>\n      <td>False</td>\n      <td>[signal amplification, signal amplification, s.</td>\n    </tr>\n  </tbody>\n</table>\n<p>100 rows \u00d7 5 columns</p>\n</div>\n\n\n\n## 4. Fine-tuning and Answering using Fine-tuned model\n\nFor the complete fine-tuning process, please refer to the [OpenAI Fine-Tuning Docs](https://platform.openai.com/docs/guides/fine-tuning/use-a-fine-tuned-model).\n\n### 4.1 Prepare the Fine-Tuning Data\n\nWe need to prepare the data for fine-tuning. We'll use a few samples from train split of same dataset as before, but we'll add the answer to the context. This will help the model learn to retrieve the answer from the context. \n\nOur instruction prompt is the same as before, and so is the system prompt. \n\n\n```python\ndef dataframe_to_jsonl(df):\n    def create_jsonl_entry(row):\n        answer = row[\"answers\"][0] if row[\"answers\"] else \"I don't know\"\n        messages = [\n            {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n            {\n                \"role\": \"user\",\n                \"content\": f\"\"\"Answer the following Question based on the Context only. Only answer from the Context. If you don't know the answer, say 'I don't know'.\n            Question: {row.question}\\n\\n\n            Context: {row.context}\\n\\n\n            Answer:\\n\"\"\",\n            },\n            {\"role\": \"assistant\", \"content\": answer},\n        ]\n        return json.dumps({\"messages\": messages})\n\n    jsonl_output = df.apply(create_jsonl_entry, axis=1)\n    return \"\\n\".join(jsonl_output)\n\ntrain_sample = get_diverse_sample(train_df, sample_size=100, random_state=42)\n\nwith open(\"local_cache/100_train.jsonl\", \"w\") as f:\n    f.write(dataframe_to_jsonl(train_sample))\n```\n\n**Tip: \ud83d\udca1 Verify the Fine-Tuning Data**\n\nYou can see this [cookbook](https://github.com/openai/openai-cookbook/blob/main/examples/Chat_finetuning_data_prep.ipynb) for more details on how to prepare the data for fine-tuning.\n\n### 4.2 Fine-Tune OpenAI Model\n\nIf you're new to OpenAI Model Fine-Tuning, please refer to the [How to finetune Chat models](https://github.com/openai/openai-cookbook/blob/448a0595b84ced3bebc9a1568b625e748f9c1d60/examples/How_to_finetune_chat_models.ipynb) notebook. You can also refer to the [OpenAI Fine-Tuning Docs](platform.openai.com/docs/guides/fine-tuning/use-a-fine-tuned-model) for more details."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 88202e0d-4345-4c63-b155-b9d56ef9857c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"88202e0d-4345-4c63-b155-b9d56ef9857c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7a91d318-e7c9-4f49-a3e1-afa9221de462\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9e80e072e25c9f5bfc8f66c49d807a3d26106aa4b424bb952da3fb5f6102a843\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0b8213cd-6505-40a9-8fff-38d1fca7755b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0578ada283197b1b1d56ba723adb382013efac4a40dee49e006c32e9c5db664e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14308, \"end_char_idx\": 17379, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "```python\nclass OpenAIFineTuner:\n    \"\"\"\n    Class to fine tune OpenAI models\n    \"\"\"\n    def __init__(self, training_file_path, model_name, suffix):\n        self.training_file_path = training_file_path\n        self.model_name = model_name\n        self.suffix = suffix\n        self.file_object = None\n        self.fine_tuning_job = None\n        self.model_id = None\n\n    def create_openai_file(self):\n        self.file_object = client.files.create(\n            file=open(self.training_file_path, \"r\"),\n            purpose=\"fine-tune\",\n        )\n\n    def wait_for_file_processing(self, sleep_time=20):\n        while self.file_object.status != 'processed':\n            time.sleep(sleep_time)\n            self.file_object.refresh()\n            print(\"File Status: \", self.file_object.status)\n\n    def create_fine_tuning_job(self):\n        self.fine_tuning_job = client.fine_tuning.jobs.create(\n            training_file=self.file_object[\"id\"],\n            model=self.model_name,\n            suffix=self.suffix,\n        )\n\n    def wait_for_fine_tuning(self, sleep_time=45):\n        while self.fine_tuning_job.status != 'succeeded':\n            time.sleep(sleep_time)\n            self.fine_tuning_job.refresh()\n            print(\"Job Status: \", self.fine_tuning_job.status)\n\n    def retrieve_fine_tuned_model(self):\n        self.model_id = client.fine_tuning.jobs.retrieve(self.fine_tuning_job[\"id\"]).fine_tuned_model\n        return self.model_id\n\n    def fine_tune_model(self):\n        self.create_openai_file()\n        self.wait_for_file_processing()\n        self.create_fine_tuning_job()\n        self.wait_for_fine_tuning()\n        return self.retrieve_fine_tuned_model()\n\nfine_tuner = OpenAIFineTuner(\n        training_file_path=\"local_cache/100_train.jsonl\",\n        model_name=\"gpt-3.5-turbo\",\n        suffix=\"100trn20230907\"\n    )\n```\n\n\u23f0 **Time to run: ~10-20 minutes**, \ud83d\udedc Needs Internet Connection\n\n\n```python\nmodel_id = fine_tuner.fine_tune_model()\nmodel_id\n```\n\n#### 4.2.1 Try out the Fine-Tuned Model\n\nLet's try out the fine-tuned model on the same validation set as before. You'll use the same prompt as before, but you will use the fine-tuned model instead of the base model. Before you do that, you can make a simple call to get a sense of how the fine-tuned model is doing.\n\n\n```python\ncompletion = client.chat.completions.create(\n    model=model_id,\n    messages=[\n        {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n        {\"role\": \"user\", \"content\": \"Hello!\"},\n        {\"role\": \"assistant\", \"content\": \"Hi, how can I help you today?\"},\n        {\n            \"role\": \"user\",\n            \"content\": \"Can you answer the following question based on the given context? If not, say, I don't know:\\n\\nQuestion: What is the capital of France?\\n\\nContext: The capital of Mars is Gaia. Answer:\",\n        },\n    ],\n)\n\nprint(completion.choices[0].message)\n```\n\n### 4.3 Answer Using the Fine-Tuned Model\n\nThis is the same as before, but you'll use the fine-tuned model instead of the base model.\n\n\u23f0 **Time to run: ~5 min**, \ud83d\udedc Needs Internet Connection"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 0b8213cd-6505-40a9-8fff-38d1fca7755b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"0b8213cd-6505-40a9-8fff-38d1fca7755b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"88202e0d-4345-4c63-b155-b9d56ef9857c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"2416b1fcdd438ab38f8948909e1d6bb9baefd4dd7db87d63c5aeb20325c38bf7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e9bc597c-3f35-48c9-a011-8ff0b31a094c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"72fc6e7c8ac5cfa9916d6ea58a4fa655ae496f8e23a6b2f0de42a9ec376ba68a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16211, \"end_char_idx\": 19255, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "```python\nmodel_id = fine_tuner.fine_tune_model()\nmodel_id\n```\n\n#### 4.2.1 Try out the Fine-Tuned Model\n\nLet's try out the fine-tuned model on the same validation set as before. You'll use the same prompt as before, but you will use the fine-tuned model instead of the base model. Before you do that, you can make a simple call to get a sense of how the fine-tuned model is doing.\n\n\n```python\ncompletion = client.chat.completions.create(\n    model=model_id,\n    messages=[\n        {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n        {\"role\": \"user\", \"content\": \"Hello!\"},\n        {\"role\": \"assistant\", \"content\": \"Hi, how can I help you today?\"},\n        {\n            \"role\": \"user\",\n            \"content\": \"Can you answer the following question based on the given context? If not, say, I don't know:\\n\\nQuestion: What is the capital of France?\\n\\nContext: The capital of Mars is Gaia. Answer:\",\n        },\n    ],\n)\n\nprint(completion.choices[0].message)\n```\n\n### 4.3 Answer Using the Fine-Tuned Model\n\nThis is the same as before, but you'll use the fine-tuned model instead of the base model.\n\n\u23f0 **Time to run: ~5 min**, \ud83d\udedc Needs Internet Connection\n\n\n```python\ndf[\"ft_generated_answer\"] = df.progress_apply(answer_question, model=model_id, axis=1)\n```\n\n## 5. Evaluation: How well does the model perform?\n\nTo evaluate the model's performance, compare the predicted answer to the actual answers -- if any of the actual answers are present in the predicted answer, then it's a match. We've also created error categories to help you understand where the model is struggling.\n\nWhen we know that a correct answer exists in the context, we can measure the model's performance, there are 3 possible outcomes:\n\n1. \u2705 **Answered Correctly**: The model responded the correct answer. It may have also included other answers that were not in the context.\n2. \u274e **Skipped**: The model responded with \"I don't know\" (IDK) while the answer was present in the context. It's better than giving the wrong answer. It's better for the model say \"I don't know\" than giving the wrong answer. In our design, we know that a true answer exists and hence we're able to measure it -- this is not always the case. *This is a model error*. We exclude this from the overall error rate. \n3. \u274c **Wrong**: The model responded with an incorrect answer. **This is a model ERROR.**\n\nWhen we know that a correct answer does not exist in the context, we can measure the model's performance, there are 2 possible outcomes:\n\n4. \u274c **Hallucination**: The model responded with an answer, when \"I don't know\" was expected. **This is a model ERROR.** \n5. \u2705 **I don't know**: The model responded with \"I don't know\" (IDK) and the answer was not present in the context. **This is a model WIN.**\n\n\n```python\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nclass Evaluator:\n    def __init__(self, df):\n        self.df = df\n        self.y_pred = pd.Series()  # Initialize as empty Series\n        self.labels_answer_expected = [\"\u2705 Answered Correctly\","}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: e9bc597c-3f35-48c9-a011-8ff0b31a094c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"e9bc597c-3f35-48c9-a011-8ff0b31a094c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0b8213cd-6505-40a9-8fff-38d1fca7755b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"ae83782fd643c642b1bca10d65bdb6516f7a0e1eba8b4268c0d3764d4ff9af9a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"09be2d6e-8e94-42a4-a453-f072d3643f5c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5858c6253cd9e7d175c88a6911cf65be84d65f6fcac5b570b495ea27d5656bc2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18978, \"end_char_idx\": 22144, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "```python\nimport pandas as pd\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n\nclass Evaluator:\n    def __init__(self, df):\n        self.df = df\n        self.y_pred = pd.Series()  # Initialize as empty Series\n        self.labels_answer_expected = [\"\u2705 Answered Correctly\", \"\u274e Skipped\", \"\u274c Wrong Answer\"]\n        self.labels_idk_expected = [\"\u274c Hallucination\", \"\u2705 I don't know\"]\n\n    def _evaluate_answer_expected(self, row, answers_column):\n        generated_answer = row[answers_column].lower()\n        actual_answers = [ans.lower() for ans in row[\"answers\"]]\n        return (\n            \"\u2705 Answered Correctly\" if any(ans in generated_answer for ans in actual_answers)\n            else \"\u274e Skipped\" if generated_answer == \"i don't know\"\n            else \"\u274c Wrong Answer\"\n        )\n\n    def _evaluate_idk_expected(self, row, answers_column):\n        generated_answer = row[answers_column].lower()\n        return (\n            \"\u274c Hallucination\" if generated_answer != \"i don't know\"\n            else \"\u2705 I don't know\"\n        )\n\n    def _evaluate_single_row(self, row, answers_column):\n        is_impossible = row[\"is_impossible\"]\n        return (\n            self._evaluate_answer_expected(row, answers_column) if not is_impossible\n            else self._evaluate_idk_expected(row, answers_column)\n        )\n\n    def evaluate_model(self, answers_column=\"generated_answer\"):\n        self.y_pred = pd.Series(self.df.apply(self._evaluate_single_row, answers_column=answers_column, axis=1))\n        freq_series = self.y_pred.value_counts()\n        \n        # Counting rows for each scenario\n        total_answer_expected = len(self.df[self.df['is_impossible'] == False])\n        total_idk_expected = len(self.df[self.df['is_impossible'] == True])\n        \n        freq_answer_expected = (freq_series / total_answer_expected * 100).round(2).reindex(self.labels_answer_expected, fill_value=0)\n        freq_idk_expected = (freq_series / total_idk_expected * 100).round(2).reindex(self.labels_idk_expected, fill_value=0)\n        return freq_answer_expected.to_dict(), freq_idk_expected.to_dict()\n\n    def print_eval(self):\n        answer_columns=[\"generated_answer\", \"ft_generated_answer\"]\n        baseline_correctness, baseline_idk = self.evaluate_model()\n        ft_correctness, ft_idk = self.evaluate_model(self.df, answer_columns[1])\n        print(\"When the model should answer correctly:\")\n        eval_df = pd.merge(\n            baseline_correctness.rename(\"Baseline\"),\n            ft_correctness.rename(\"Fine-Tuned\"),\n            left_index=True,\n            right_index=True,\n        )\n        print(eval_df)\n        print(\"\\n\\n\\nWhen the model should say 'I don't know':\")\n        eval_df = pd.merge(\n            baseline_idk.rename(\"Baseline\"),\n            ft_idk.rename(\"Fine-Tuned\"),\n            left_index=True,\n            right_index=True,\n        )\n        print(eval_df)\n    \n    def plot_model_comparison(self, answer_columns=[\"generated_answer\", \"ft_generated_answer\"], scenario=\"answer_expected\", nice_names=[\"Baseline\", \"Fine-Tuned\"]):\n        \n        results = []\n        for col in answer_columns:\n            answer_expected, idk_expected = self."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 09be2d6e-8e94-42a4-a453-f072d3643f5c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"09be2d6e-8e94-42a4-a453-f072d3643f5c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e9bc597c-3f35-48c9-a011-8ff0b31a094c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e926690456b06fc91d7a734ae13b2f477477c710e88ae710991d0397183261f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"76b7962c-30d0-463f-987c-92b08615596f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9368a004517e47a9c6db873231720c30d5af3c13df7ead400b6db56a8d56920c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 20428, \"end_char_idx\": 23521, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "answers_column=answers_column, axis=1))\n        freq_series = self.y_pred.value_counts()\n        \n        # Counting rows for each scenario\n        total_answer_expected = len(self.df[self.df['is_impossible'] == False])\n        total_idk_expected = len(self.df[self.df['is_impossible'] == True])\n        \n        freq_answer_expected = (freq_series / total_answer_expected * 100).round(2).reindex(self.labels_answer_expected, fill_value=0)\n        freq_idk_expected = (freq_series / total_idk_expected * 100).round(2).reindex(self.labels_idk_expected, fill_value=0)\n        return freq_answer_expected.to_dict(), freq_idk_expected.to_dict()\n\n    def print_eval(self):\n        answer_columns=[\"generated_answer\", \"ft_generated_answer\"]\n        baseline_correctness, baseline_idk = self.evaluate_model()\n        ft_correctness, ft_idk = self.evaluate_model(self.df, answer_columns[1])\n        print(\"When the model should answer correctly:\")\n        eval_df = pd.merge(\n            baseline_correctness.rename(\"Baseline\"),\n            ft_correctness.rename(\"Fine-Tuned\"),\n            left_index=True,\n            right_index=True,\n        )\n        print(eval_df)\n        print(\"\\n\\n\\nWhen the model should say 'I don't know':\")\n        eval_df = pd.merge(\n            baseline_idk.rename(\"Baseline\"),\n            ft_idk.rename(\"Fine-Tuned\"),\n            left_index=True,\n            right_index=True,\n        )\n        print(eval_df)\n    \n    def plot_model_comparison(self, answer_columns=[\"generated_answer\", \"ft_generated_answer\"], scenario=\"answer_expected\", nice_names=[\"Baseline\", \"Fine-Tuned\"]):\n        \n        results = []\n        for col in answer_columns:\n            answer_expected, idk_expected = self.evaluate_model(col)\n            if scenario == \"answer_expected\":\n                results.append(answer_expected)\n            elif scenario == \"idk_expected\":\n                results.append(idk_expected)\n            else:\n                raise ValueError(\"Invalid scenario\")\n        \n        \n        results_df = pd.DataFrame(results, index=nice_names)\n        if scenario == \"answer_expected\":\n            results_df = results_df.reindex(self.labels_answer_expected, axis=1)\n        elif scenario == \"idk_expected\":\n            results_df = results_df.reindex(self.labels_idk_expected, axis=1)\n        \n        melted_df = results_df.reset_index().melt(id_vars='index', var_name='Status', value_name='Frequency')\n        sns.set_theme(style=\"whitegrid\", palette=\"icefire\")\n        g = sns.catplot(data=melted_df, x='Frequency', y='index', hue='Status', kind='bar', height=5, aspect=2)\n\n        # Annotating each bar\n        for p in g.ax.patches:\n            g.ax.annotate(f\"{p.get_width():.0f}%\", (p.get_width()+5, p.get_y() + p.get_height() / 2),\n                        textcoords=\"offset points\",\n                        xytext=(0, 0),\n                        ha='center', va='center')\n        plt.ylabel(\"Model\")\n        plt.xlabel(\"Percentage\")\n        plt.xlim(0, 100)\n        plt.tight_layout()\n        plt.title(scenario.replace(\"_\", \" \").title())\n        plt.show()"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 76b7962c-30d0-463f-987c-92b08615596f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"76b7962c-30d0-463f-987c-92b08615596f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"09be2d6e-8e94-42a4-a453-f072d3643f5c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"2b5d4744f05c256108afde979cd0c3c97b19bad467656ff316f4fea5ab5f40c1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"925cf04f-793e-40ed-b68c-02a135d8b694\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"663f178ba2e260e7db4643b5e6bb0297e1060b07d7c8b3c1fef73f98059f028a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21836, \"end_char_idx\": 24602, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": ")\n        print(eval_df)\n    \n    def plot_model_comparison(self, answer_columns=[\"generated_answer\", \"ft_generated_answer\"], scenario=\"answer_expected\", nice_names=[\"Baseline\", \"Fine-Tuned\"]):\n        \n        results = []\n        for col in answer_columns:\n            answer_expected, idk_expected = self.evaluate_model(col)\n            if scenario == \"answer_expected\":\n                results.append(answer_expected)\n            elif scenario == \"idk_expected\":\n                results.append(idk_expected)\n            else:\n                raise ValueError(\"Invalid scenario\")\n        \n        \n        results_df = pd.DataFrame(results, index=nice_names)\n        if scenario == \"answer_expected\":\n            results_df = results_df.reindex(self.labels_answer_expected, axis=1)\n        elif scenario == \"idk_expected\":\n            results_df = results_df.reindex(self.labels_idk_expected, axis=1)\n        \n        melted_df = results_df.reset_index().melt(id_vars='index', var_name='Status', value_name='Frequency')\n        sns.set_theme(style=\"whitegrid\", palette=\"icefire\")\n        g = sns.catplot(data=melted_df, x='Frequency', y='index', hue='Status', kind='bar', height=5, aspect=2)\n\n        # Annotating each bar\n        for p in g.ax.patches:\n            g.ax.annotate(f\"{p.get_width():.0f}%\", (p.get_width()+5, p.get_y() + p.get_height() / 2),\n                        textcoords=\"offset points\",\n                        xytext=(0, 0),\n                        ha='center', va='center')\n        plt.ylabel(\"Model\")\n        plt.xlabel(\"Percentage\")\n        plt.xlim(0, 100)\n        plt.tight_layout()\n        plt.title(scenario.replace(\"_\", \" \").title())\n        plt.show()\n\n\n# Compare the results by merging into one dataframe\nevaluator = Evaluator(df)\n# evaluator.evaluate_model(answers_column=\"ft_generated_answer\")\n# evaluator.plot_model_comparison([\"generated_answer\", \"ft_generated_answer\"], scenario=\"answer_expected\", nice_names=[\"Baseline\", \"Fine-Tuned\"])\n```\n\n\n```python\n# Optionally, save the results to a JSON file\ndf.to_json(\"local_cache/100_val_ft.json\", orient=\"records\", lines=True)\ndf = pd.read_json(\"local_cache/100_val_ft.json\", orient=\"records\", lines=True)\n```\n\n\n```python\nevaluator.plot_model_comparison([\"generated_answer\", \"ft_generated_answer\"], scenario=\"answer_expected\", nice_names=[\"Baseline\", \"Fine-Tuned\"])\n```\n\n\n    \n![png](output_31_0.png)\n    \n\n\nNotice that the fine-tuned model skips questions more often -- and makes fewer mistakes. This is because the fine-tuned model is more conservative and skips questions when it's not sure.\n\n\n```python\nevaluator.plot_model_comparison([\"generated_answer\", \"ft_generated_answer\"], scenario=\"idk_expected\", nice_names=[\"Baseline\", \"Fine-Tuned\"])\n```\n\n\n    \n![png](output_33_0.png)"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 925cf04f-793e-40ed-b68c-02a135d8b694
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"925cf04f-793e-40ed-b68c-02a135d8b694\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"76b7962c-30d0-463f-987c-92b08615596f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"8821028e1b34f82e3101bf80eb8c9a81d0c2a0eacafac339c17394567c7087b8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d4710a7f-aab9-476c-a414-6243401037a5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c61f4fbdbc155314b5d0a1841f69a992b85edfca1e170edba8d2c6d1681c49fc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 22999, \"end_char_idx\": 25988, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "kind='bar', height=5, aspect=2)\n\n        # Annotating each bar\n        for p in g.ax.patches:\n            g.ax.annotate(f\"{p.get_width():.0f}%\", (p.get_width()+5, p.get_y() + p.get_height() / 2),\n                        textcoords=\"offset points\",\n                        xytext=(0, 0),\n                        ha='center', va='center')\n        plt.ylabel(\"Model\")\n        plt.xlabel(\"Percentage\")\n        plt.xlim(0, 100)\n        plt.tight_layout()\n        plt.title(scenario.replace(\"_\", \" \").title())\n        plt.show()\n\n\n# Compare the results by merging into one dataframe\nevaluator = Evaluator(df)\n# evaluator.evaluate_model(answers_column=\"ft_generated_answer\")\n# evaluator.plot_model_comparison([\"generated_answer\", \"ft_generated_answer\"], scenario=\"answer_expected\", nice_names=[\"Baseline\", \"Fine-Tuned\"])\n```\n\n\n```python\n# Optionally, save the results to a JSON file\ndf.to_json(\"local_cache/100_val_ft.json\", orient=\"records\", lines=True)\ndf = pd.read_json(\"local_cache/100_val_ft.json\", orient=\"records\", lines=True)\n```\n\n\n```python\nevaluator.plot_model_comparison([\"generated_answer\", \"ft_generated_answer\"], scenario=\"answer_expected\", nice_names=[\"Baseline\", \"Fine-Tuned\"])\n```\n\n\n    \n![png](output_31_0.png)\n    \n\n\nNotice that the fine-tuned model skips questions more often -- and makes fewer mistakes. This is because the fine-tuned model is more conservative and skips questions when it's not sure.\n\n\n```python\nevaluator.plot_model_comparison([\"generated_answer\", \"ft_generated_answer\"], scenario=\"idk_expected\", nice_names=[\"Baseline\", \"Fine-Tuned\"])\n```\n\n\n    \n![png](output_33_0.png)\n    \n\n\nNotice that the fine-tuned model has learnt to say \"I don't know\" a lot better than the prompt. Or, the model has gotten good at skipping questions.\n\n### Observations\n\n1. The fine-tuned model is better at saying \"I don't know\"\n2. Hallucinations drop from 100% to 15% with fine-tuning\n3. Wrong answers drop from 17% to 6% with fine-tuning\n\n**Correct answers also drop from 83% to 60% with fine-tuning** - this is because the fine-tuned model is **more conservative** and says \"I don't know\" more often. This is a good thing because it's better to say \"I don't know\" than to give a wrong answer.\n\nThat said, we want to improve the correctness of the model, even if that increases the hallucinations. We're looking for a model that is both correct and conservative, striking a balance between the two. We'll use Qdrant and Few-Shot Learning to achieve this.\n\n**\ud83d\udcaa You're 2/3rds of the way there! Keep reading!**\n\n# Section B: Few Shot Learning\n\nWe'll select a few examples from the dataset, including cases where the answer is not present in the context. We'll then use these examples to create a prompt that we can use to fine-tune the model. We'll then measure the performance of the fine-tuned model.\n\n**What is next?**\n\n6. Fine-Tuning OpenAI Model with Qdrant\n    6.1 Embed the Fine-Tuning Data\n    6.2 Embedding the Questions\n7. Using Qdrant to Improve RAG Prompt\n8. Evaluation"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: d4710a7f-aab9-476c-a414-6243401037a5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"d4710a7f-aab9-476c-a414-6243401037a5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"925cf04f-793e-40ed-b68c-02a135d8b694\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"6b249ade6cf361b358707b2ae43bd4785bc129b177eb3fa36ae3a8eca1a49bc3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"92be59a1-c15c-4ae4-b66d-d869c943ae6b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5e5e9556d987a7242052fe0565395d8bf34e4cc5b8a3ae8b6b16e66bc0c9fec8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 24579, \"end_char_idx\": 27505, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "![png](output_33_0.png)\n    \n\n\nNotice that the fine-tuned model has learnt to say \"I don't know\" a lot better than the prompt. Or, the model has gotten good at skipping questions.\n\n### Observations\n\n1. The fine-tuned model is better at saying \"I don't know\"\n2. Hallucinations drop from 100% to 15% with fine-tuning\n3. Wrong answers drop from 17% to 6% with fine-tuning\n\n**Correct answers also drop from 83% to 60% with fine-tuning** - this is because the fine-tuned model is **more conservative** and says \"I don't know\" more often. This is a good thing because it's better to say \"I don't know\" than to give a wrong answer.\n\nThat said, we want to improve the correctness of the model, even if that increases the hallucinations. We're looking for a model that is both correct and conservative, striking a balance between the two. We'll use Qdrant and Few-Shot Learning to achieve this.\n\n**\ud83d\udcaa You're 2/3rds of the way there! Keep reading!**\n\n# Section B: Few Shot Learning\n\nWe'll select a few examples from the dataset, including cases where the answer is not present in the context. We'll then use these examples to create a prompt that we can use to fine-tune the model. We'll then measure the performance of the fine-tuned model.\n\n**What is next?**\n\n6. Fine-Tuning OpenAI Model with Qdrant\n    6.1 Embed the Fine-Tuning Data\n    6.2 Embedding the Questions\n7. Using Qdrant to Improve RAG Prompt\n8. Evaluation\n\n\n## 6. Fine-Tuning OpenAI Model with Qdrant\n\nSo far, we've been using the OpenAI model to answer questions without using examples of the answer. The previous step made it work better on in-context examples, while this one helps it generalize to unseen data, and attempt to learn when to say \"I don't know\" and when to give an answer.\n\nThis is where few-shot learning comes in!\n\nFew-shot learning is a type of transfer learning that allows us to answer questions where the answer is not present in the context. We can do this by providing a few examples of the answer we're looking for, and the model will learn to answer questions where the answer is not present in the context.\n\n### 5.1 Embed the Training Data\n\nEmbeddings are a way to represent sentences as an array of floats. We'll use the embeddings to find the most similar questions to the ones we're looking for.\n\n\n```python\nimport os\nfrom qdrant_client import QdrantClient\nfrom qdrant_client.http import models\nfrom qdrant_client.http.models import PointStruct\nfrom qdrant_client.http.models import Distance, VectorParams\n```\n\nNow that we've the Qdrant imports in place, \n\n\n```python\nqdrant_client = QdrantClient(\n    url=os.getenv(\"QDRANT_URL\"), api_key=os.getenv(\"QDRANT_API_KEY\"), timeout=6000, prefer_grpc=True\n)\n\ncollection_name = \"squadv2-cookbook\"\n\n# # Create the collection, run this only once\n# qdrant_client.recreate_collection(\n#     collection_name=collection_name,\n#     vectors_config=VectorParams(size=384, distance=Distance.COSINE),\n# )\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 92be59a1-c15c-4ae4-b66d-d869c943ae6b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"92be59a1-c15c-4ae4-b66d-d869c943ae6b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d4710a7f-aab9-476c-a414-6243401037a5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"ceaa9a8ef17f26cc89bcd480a5fc54b0ffcdd389d4401a271d12ee790aeea7df\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"92f47724-b9a8-489a-a1a6-72379a70a7d9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4ddbe040edf04031d558295a41f48717105f741abfdcfc4d4e680d46f6d4023a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 25991, \"end_char_idx\": 29363, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "## 6. Fine-Tuning OpenAI Model with Qdrant\n\nSo far, we've been using the OpenAI model to answer questions without using examples of the answer. The previous step made it work better on in-context examples, while this one helps it generalize to unseen data, and attempt to learn when to say \"I don't know\" and when to give an answer.\n\nThis is where few-shot learning comes in!\n\nFew-shot learning is a type of transfer learning that allows us to answer questions where the answer is not present in the context. We can do this by providing a few examples of the answer we're looking for, and the model will learn to answer questions where the answer is not present in the context.\n\n### 5.1 Embed the Training Data\n\nEmbeddings are a way to represent sentences as an array of floats. We'll use the embeddings to find the most similar questions to the ones we're looking for.\n\n\n```python\nimport os\nfrom qdrant_client import QdrantClient\nfrom qdrant_client.http import models\nfrom qdrant_client.http.models import PointStruct\nfrom qdrant_client.http.models import Distance, VectorParams\n```\n\nNow that we've the Qdrant imports in place, \n\n\n```python\nqdrant_client = QdrantClient(\n    url=os.getenv(\"QDRANT_URL\"), api_key=os.getenv(\"QDRANT_API_KEY\"), timeout=6000, prefer_grpc=True\n)\n\ncollection_name = \"squadv2-cookbook\"\n\n# # Create the collection, run this only once\n# qdrant_client.recreate_collection(\n#     collection_name=collection_name,\n#     vectors_config=VectorParams(size=384, distance=Distance.COSINE),\n# )\n```\n\n\n```python\nfrom fastembed.embedding import DefaultEmbedding\nfrom typing import List\nimport numpy as np\nimport pandas as pd\nfrom tqdm.notebook import tqdm\n\ntqdm.pandas()\n\nembedding_model = DefaultEmbedding()\n```\n\n### 5.2 Embedding the Questions\n\nNext, you'll embed the entire training set questions. You'll use the question to question similarity to find the most similar questions to the question we're looking for. This is a workflow which is used in RAG to leverage the OpenAI model ability of incontext learning with more examples. This is what we call Few Shot Learning here.\n\n**\u2757\ufe0f\u23f0 Important Note: This step can take up to 3 hours to complete. Please be patient. If you see Out of Memory errors or Kernel Crashes, please reduce the batch size to 32, restart the kernel and run the notebook again. This code needs to be run only ONCE.**\n\n## Function Breakdown for `generate_points_from_dataframe`\n\n1. **Initialization**: `batch_size = 512` and `total_batches` set the stage for how many questions will be processed in one go. This is to prevent memory issues. If your machine can handle more, feel free to increase the batch size. If your kernel crashes, reduce the batch size to 32 and try again.\n2. **Progress Bar**: `tqdm` gives you a nice progress bar so you don't fall asleep.\n3. **Batch Loop**: The for-loop iterates through batches. `start_idx` and `end_idx` define the slice of the DataFrame to process.\n4. **Generate Embeddings**: `batch_embeddings = embedding_model.embed(batch, batch_size=batch_size)` - This is where the magic happens. Your questions get turned into embeddings.\n5. **PointStruct Generation**: Using `.progress_apply`, it turns each row into a `PointStruct` object. This includes an ID, the embedding vector, and other metadata.\n\nReturns the list of `PointStruct` objects, which can be used to create a collection in Qdrant."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 92f47724-b9a8-489a-a1a6-72379a70a7d9
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"92f47724-b9a8-489a-a1a6-72379a70a7d9\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"92be59a1-c15c-4ae4-b66d-d869c943ae6b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"5acdbd5966f26f3f89e2a913971bac9704801e754bde31c8461104e9d1667c93\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"40728e21-f49d-4d1c-a13b-2330d3a39b81\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"34b05026f1128ad6ed6c5af64c81e0da5574c844af38eb224a71b3a29c6b1676\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 29366, \"end_char_idx\": 32460, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "```python\ndef generate_points_from_dataframe(df: pd.DataFrame) -> List[PointStruct]:\n    batch_size = 512\n    questions = df[\"question\"].tolist()\n    total_batches = len(questions) // batch_size + 1\n    \n    pbar = tqdm(total=len(questions), desc=\"Generating embeddings\")\n    \n    # Generate embeddings in batches to improve performance\n    embeddings = []\n    for i in range(total_batches):\n        start_idx = i * batch_size\n        end_idx = min((i + 1) * batch_size, len(questions))\n        batch = questions[start_idx:end_idx]\n        \n        batch_embeddings = embedding_model.embed(batch, batch_size=batch_size)\n        embeddings.extend(batch_embeddings)\n        pbar.update(len(batch))\n        \n    pbar.close()\n    \n    # Convert embeddings to list of lists\n    embeddings_list = [embedding.tolist() for embedding in embeddings]\n    \n    # Create a temporary DataFrame to hold the embeddings and existing DataFrame columns\n    temp_df = df.copy()\n    temp_df[\"embeddings\"] = embeddings_list\n    temp_df[\"id\"] = temp_df.index\n    \n    # Generate PointStruct objects using DataFrame apply method\n    points = temp_df.progress_apply(\n        lambda row: PointStruct(\n            id=row[\"id\"],\n            vector=row[\"embeddings\"],\n            payload={\n                \"question\": row[\"question\"],\n                \"title\": row[\"title\"],\n                \"context\": row[\"context\"],\n                \"is_impossible\": row[\"is_impossible\"],\n                \"answers\": row[\"answers\"],\n            },\n        ),\n        axis=1,\n    ).tolist()\n\n    return points\n\npoints = generate_points_from_dataframe(train_df)\n```\n\n#### Upload the Embeddings to Qdrant\n\nNote that configuring Qdrant is outside the scope of this notebook. Please refer to the [Qdrant](https://qdrant.tech) for more information. We used a timeout of 600 seconds for the upload, and grpc compression to speed up the upload.\n\n\n```python\noperation_info = qdrant_client.upsert(\n    collection_name=collection_name, wait=True, points=points\n)\nprint(operation_info)\n```\n\n## 6. Using Qdrant to Improve RAG Prompt\n\nNow that we've uploaded the embeddings to Qdrant, we can use Qdrant to find the most similar questions to the question we're looking for. We'll use the top 5 most similar questions to create a prompt that we can use to fine-tune the model. We'll then measure the performance of the fine-tuned model on the same validation set, but with few shot prompting!\n\nOur main function `get_few_shot_prompt` serves as the workhorse for generating prompts for few-shot learning. It does this by retrieving similar questions from Qdrant - a vector search engine, using an embeddings model. Here is the high-level workflow:\n\n1. Retrieve similar questions from Qdrant where the **answer is present** in the context\n2. Retrieve similar questions from Qdrant where the **answer is IMPOSSIBLE** i.e. the expected answer is \"I don't know\" to find in the context\n3. Create a prompt using the retrieved questions\n4. Fine-tune the model using the prompt\n5. Evaluate the fine-tuned model on the validation set with the same prompting technique"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 40728e21-f49d-4d1c-a13b-2330d3a39b81
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"40728e21-f49d-4d1c-a13b-2330d3a39b81\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"92f47724-b9a8-489a-a1a6-72379a70a7d9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7dde6d1cdcc1733d0a4ae51bc23aa2d9fa57408b33b2db4928db6428728d8d8c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"32845cd1-39bd-4430-997b-1116712fd209\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5e3491d2322ca3cdb399f4440c534a9658fef59a6dacb14386e4de1caf7bf43f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 31258, \"end_char_idx\": 33752, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "```python\noperation_info = qdrant_client.upsert(\n    collection_name=collection_name, wait=True, points=points\n)\nprint(operation_info)\n```\n\n## 6. Using Qdrant to Improve RAG Prompt\n\nNow that we've uploaded the embeddings to Qdrant, we can use Qdrant to find the most similar questions to the question we're looking for. We'll use the top 5 most similar questions to create a prompt that we can use to fine-tune the model. We'll then measure the performance of the fine-tuned model on the same validation set, but with few shot prompting!\n\nOur main function `get_few_shot_prompt` serves as the workhorse for generating prompts for few-shot learning. It does this by retrieving similar questions from Qdrant - a vector search engine, using an embeddings model. Here is the high-level workflow:\n\n1. Retrieve similar questions from Qdrant where the **answer is present** in the context\n2. Retrieve similar questions from Qdrant where the **answer is IMPOSSIBLE** i.e. the expected answer is \"I don't know\" to find in the context\n3. Create a prompt using the retrieved questions\n4. Fine-tune the model using the prompt\n5. Evaluate the fine-tuned model on the validation set with the same prompting technique\n\n\n```python\ndef get_few_shot_prompt(row):\n\n    query, row_context = row[\"question\"], row[\"context\"]\n\n    embeddings = list(embedding_model.embed([query]))\n    query_embedding = embeddings[0].tolist()\n\n    num_of_qa_to_retrieve = 5\n\n    # Query Qdrant for similar questions that have an answer\n    q1 = qdrant_client.search(\n        collection_name=collection_name,\n        query_vector=query_embedding,\n        with_payload=True,\n        limit=num_of_qa_to_retrieve,\n        query_filter=models.Filter(\n            must=[\n                models.FieldCondition(\n                    key=\"is_impossible\",\n                    match=models.MatchValue(\n                        value=False,\n                    ),\n                ),\n            ],\n        )\n    )\n\n    # Query Qdrant for similar questions that are IMPOSSIBLE to answer\n    q2 = qdrant_client.search(\n        collection_name=collection_name,\n        query_vector=query_embedding,\n        query_filter=models.Filter(\n            must=[\n                models.FieldCondition(\n                    key=\"is_impossible\",\n                    match=models.MatchValue(\n                        value=True,\n                    ),\n                ),\n            ]\n        ),\n        with_payload=True,\n        limit=num_of_qa_to_retrieve,\n    )"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 32845cd1-39bd-4430-997b-1116712fd209
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"32845cd1-39bd-4430-997b-1116712fd209\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"40728e21-f49d-4d1c-a13b-2330d3a39b81\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"da66e45723fdcc45c75c726cd67ce36526ff8898bfe98238a6c786a5164beedb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"968b773a-5d5c-41df-bf8a-5bfb62f53c17\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"29cd50c31aaeff745085d82438d98308905ae83f27af17558d6287056d9c8211\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 32463, \"end_char_idx\": 35609, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "```python\ndef get_few_shot_prompt(row):\n\n    query, row_context = row[\"question\"], row[\"context\"]\n\n    embeddings = list(embedding_model.embed([query]))\n    query_embedding = embeddings[0].tolist()\n\n    num_of_qa_to_retrieve = 5\n\n    # Query Qdrant for similar questions that have an answer\n    q1 = qdrant_client.search(\n        collection_name=collection_name,\n        query_vector=query_embedding,\n        with_payload=True,\n        limit=num_of_qa_to_retrieve,\n        query_filter=models.Filter(\n            must=[\n                models.FieldCondition(\n                    key=\"is_impossible\",\n                    match=models.MatchValue(\n                        value=False,\n                    ),\n                ),\n            ],\n        )\n    )\n\n    # Query Qdrant for similar questions that are IMPOSSIBLE to answer\n    q2 = qdrant_client.search(\n        collection_name=collection_name,\n        query_vector=query_embedding,\n        query_filter=models.Filter(\n            must=[\n                models.FieldCondition(\n                    key=\"is_impossible\",\n                    match=models.MatchValue(\n                        value=True,\n                    ),\n                ),\n            ]\n        ),\n        with_payload=True,\n        limit=num_of_qa_to_retrieve,\n    )\n\n\n    instruction = \"\"\"Answer the following Question based on the Context only. Only answer from the Context. If you don't know the answer, say 'I don't know'.\\n\\n\"\"\"\n    # If there is a next best question, add it to the prompt\n    \n    def q_to_prompt(q):\n        question, context = q.payload[\"question\"], q.payload[\"context\"]\n        answer = q.payload[\"answers\"][0] if len(q.payload[\"answers\"]) > 0 else \"I don't know\"\n        return [\n            {\n                \"role\": \"user\", \n                \"content\": f\"\"\"Question: {question}\\n\\nContext: {context}\\n\\nAnswer:\"\"\"\n            },\n            {\"role\": \"assistant\", \"content\": answer},\n        ]\n\n    rag_prompt = []\n    \n    if len(q1) >= 1:\n        rag_prompt += q_to_prompt(q1[1])\n    if len(q2) >= 1:\n        rag_prompt += q_to_prompt(q2[1])\n    if len(q1) >= 1:\n        rag_prompt += q_to_prompt(q1[2])\n    \n    \n\n    rag_prompt += [\n        {\n            \"role\": \"user\",\n            \"content\": f\"\"\"Question: {query}\\n\\nContext: {row_context}\\n\\nAnswer:\"\"\"\n        },\n    ]\n\n    rag_prompt = [{\"role\": \"system\", \"content\": instruction}] + rag_prompt\n    return rag_prompt\n```\n\n\n```python\n# \u23f0 Time: 2 min\ntrain_sample[\"few_shot_prompt\"] = train_sample.progress_apply(get_few_shot_prompt, axis=1)\n```\n\n## 7. Fine-Tuning OpenAI Model with Qdrant\n\n### 7.1 Upload the Fine-Tuning Data to OpenAI\n\n\n```python\n# Prepare the OpenAI File format i.e. JSONL from train_sample\ndef dataframe_to_jsonl(df):\n    def create_jsonl_entry(row):\n        messages = row[\"few_shot_prompt\"]\n        return json.dumps({\"messages\": messages})\n\n    jsonl_output = df.progress_apply(create_jsonl_entry, axis=1)\n    return \"\\n\".join(jsonl_output)\n\nwith open(\"local_cache/100_train_few_shot.jsonl\", \"w\") as f:\n    f.write(dataframe_to_jsonl(train_sample))\n```\n\n### 7.2 Fine-Tune the Model\n\n\u23f0 **Time to run: ~15-30 minutes**"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 968b773a-5d5c-41df-bf8a-5bfb62f53c17
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"968b773a-5d5c-41df-bf8a-5bfb62f53c17\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"32845cd1-39bd-4430-997b-1116712fd209\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"2590a03bc2b5a0d3b5d89acb5a0f78212151a8b2a7146166ad0b5a982a3248e1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"31a6ad3b-8815-4fbc-b910-4052370bd2f9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9d8addfcbc9314d55d4b1e2305bb2c3963226c07b97cea868b80904a2ca08f3d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 34894, \"end_char_idx\": 35850, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "```python\n# \u23f0 Time: 2 min\ntrain_sample[\"few_shot_prompt\"] = train_sample.progress_apply(get_few_shot_prompt, axis=1)\n```\n\n## 7. Fine-Tuning OpenAI Model with Qdrant\n\n### 7.1 Upload the Fine-Tuning Data to OpenAI\n\n\n```python\n# Prepare the OpenAI File format i.e. JSONL from train_sample\ndef dataframe_to_jsonl(df):\n    def create_jsonl_entry(row):\n        messages = row[\"few_shot_prompt\"]\n        return json.dumps({\"messages\": messages})\n\n    jsonl_output = df.progress_apply(create_jsonl_entry, axis=1)\n    return \"\\n\".join(jsonl_output)\n\nwith open(\"local_cache/100_train_few_shot.jsonl\", \"w\") as f:\n    f.write(dataframe_to_jsonl(train_sample))\n```\n\n### 7.2 Fine-Tune the Model\n\n\u23f0 **Time to run: ~15-30 minutes**\n\n\n```python\nfine_tuner = OpenAIFineTuner(\n        training_file_path=\"local_cache/100_train_few_shot.jsonl\",\n        model_name=\"gpt-3.5-turbo\",\n        suffix=\"trnfewshot20230907\"\n    )\n\nmodel_id = fine_tuner.fine_tune_model()\nmodel_id\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 31a6ad3b-8815-4fbc-b910-4052370bd2f9
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"31a6ad3b-8815-4fbc-b910-4052370bd2f9\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"968b773a-5d5c-41df-bf8a-5bfb62f53c17\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"0e2e888c8c27c161da9e422cc1a8b19740302c6de422bbc9705df398968f35d9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d7b5c5a3-05f0-4a94-8d60-9678e9bbdd21\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9c3ba18dc6583179f20ffcaf5152e129438a1d3b4a7cb1d090f2024814c6cb41\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 34894, \"end_char_idx\": 38379, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "```python\n# \u23f0 Time: 2 min\ntrain_sample[\"few_shot_prompt\"] = train_sample.progress_apply(get_few_shot_prompt, axis=1)\n```\n\n## 7. Fine-Tuning OpenAI Model with Qdrant\n\n### 7.1 Upload the Fine-Tuning Data to OpenAI\n\n\n```python\n# Prepare the OpenAI File format i.e. JSONL from train_sample\ndef dataframe_to_jsonl(df):\n    def create_jsonl_entry(row):\n        messages = row[\"few_shot_prompt\"]\n        return json.dumps({\"messages\": messages})\n\n    jsonl_output = df.progress_apply(create_jsonl_entry, axis=1)\n    return \"\\n\".join(jsonl_output)\n\nwith open(\"local_cache/100_train_few_shot.jsonl\", \"w\") as f:\n    f.write(dataframe_to_jsonl(train_sample))\n```\n\n### 7.2 Fine-Tune the Model\n\n\u23f0 **Time to run: ~15-30 minutes**\n\n\n```python\nfine_tuner = OpenAIFineTuner(\n        training_file_path=\"local_cache/100_train_few_shot.jsonl\",\n        model_name=\"gpt-3.5-turbo\",\n        suffix=\"trnfewshot20230907\"\n    )\n\nmodel_id = fine_tuner.fine_tune_model()\nmodel_id\n```\n\n\n```python\n# Let's try this out\ncompletion = client.chat.completions.create(\n    model=model_id,\n    messages=[\n        {\"role\": \"system\", \"content\": \"You are a helpful assistant.\"},\n        {\n            \"role\": \"user\",\n            \"content\": \"Can you answer the following question based on the given context? If not, say, I don't know:\\n\\nQuestion: What is the capital of France?\\n\\nContext: The capital of Mars is Gaia. Answer:\",\n        },\n        {\n            \"role\": \"assistant\",\n            \"content\": \"I don't know\",\n        },\n        {\n            \"role\": \"user\",\n            \"content\": \"Question: Where did Maharana Pratap die?\\n\\nContext: Rana Pratap's defiance of the mighty Mughal empire, almost alone and unaided by the other Rajput states, constitute a glorious saga of Rajput valour and the spirit of self sacrifice for cherished principles. Rana Pratap's methods of guerrilla warfare was later elaborated further by Malik Ambar, the Deccani general, and by Emperor Shivaji.\\nAnswer:\",\n        },\n        {\n            \"role\": \"assistant\",\n            \"content\": \"I don't know\",\n        },\n        {\n            \"role\": \"user\",\n            \"content\": \"Question: Who did Rana Pratap fight against?\\n\\nContext: In stark contrast to other Rajput rulers who accommodated and formed alliances with the various Muslim dynasties in the subcontinent, by the time Pratap ascended to the throne, Mewar was going through a long standing conflict with the Mughals which started with the defeat of his grandfather Rana Sanga in the Battle of Khanwa in 1527 and continued with the defeat of his father Udai Singh II in Siege of Chittorgarh in 1568. Pratap Singh, gained distinction for his refusal to form any political alliance with the Mughal Empire and his resistance to Muslim domination. The conflicts between Pratap Singh and Akbar led to the Battle of Haldighati. Answer:\",\n        },\n        {\n            \"role\": \"assistant\",\n            \"content\": \"Akbar\",\n        },\n        {\n            \"role\": \"user\",\n            \"content\": \"Question: Which state is Chittorgarh in?\\n\\nContext: Chittorgarh, located in the southern part of the state of Rajasthan, 233 km (144.8 mi) from Ajmer, midway between Delhi and Mumbai on the National Highway 8 (India) in the road network of Golden Quadrilateral. Chittorgarh is situated where National Highways No. 76 & 79 intersect. Answer:\",\n        },\n    ],\n)\nprint(\"Correct Answer: Rajasthan\\nModel Answer:\")\nprint(completion.choices[0].message)\n```\n\n\u23f0 **Time to run: 5-15 min**"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: d7b5c5a3-05f0-4a94-8d60-9678e9bbdd21
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"d7b5c5a3-05f0-4a94-8d60-9678e9bbdd21\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"31a6ad3b-8815-4fbc-b910-4052370bd2f9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"72d251941c0a3797ad250c33307a680f05ccb8a754e2f6c92a91724aac912c26\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4b40084f-8b75-4e9a-9ced-c96938fea3c8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dd4d342b76d7449e06448f34fbf4c0759927b035843950ff2b0ed8fddc98b52b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 38382, \"end_char_idx\": 39938, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "```python\ndf[\"ft_generated_answer_few_shot\"] = df.progress_apply(answer_question, model=model_id, prompt_func=get_few_shot_prompt, axis=1)\ndf.to_json(\"local_cache/100_val_ft_few_shot.json\", orient=\"records\", lines=True)\n```\n\n## 8. Evaluation\n\nBut how well does the model perform? Let's compare the results from the 3 different models we've looked at so far:\n\n\n```python\nevaluator = Evaluator(df)\nevaluator.plot_model_comparison([\"generated_answer\", \"ft_generated_answer\", \"ft_generated_answer_few_shot\"], scenario=\"answer_expected\", nice_names=[\"Baseline\", \"Fine-Tuned\", \"Fine-Tuned with Few-Shot\"])\n```\n\n\n    \n![png](output_56_0.png)\n    \n\n\nThis is quite amazing -- we're able to get the best of both worlds! We're able to get the model to be both correct and conservative: \n\n1. The model is correct 83% of the time -- this is the same as the base model\n2. The model gives the wrong answer only 8% of the time -- down from 17% with the base model\n\nNext, let's look at the hallucinations. We want to reduce the hallucinations, but not at the cost of correctness. We want to strike a balance between the two. We've struck a good balance here:\n\n1. The model hallucinates 53% of the time -- down from 100% with the base model\n2. The model says \"I don't know\" 47% of the time -- up from NEVER with the base model\n\n\n```python\nevaluator.plot_model_comparison([\"generated_answer\", \"ft_generated_answer\", \"ft_generated_answer_few_shot\"], scenario=\"idk_expected\", nice_names=[\"Baseline\", \"Fine-Tuned\", \"Fine-Tuned with Few-Shot\"])\n```\n\n\n    \n![png](output_59_0.png)"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 4b40084f-8b75-4e9a-9ced-c96938fea3c8
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"4b40084f-8b75-4e9a-9ced-c96938fea3c8\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d7b5c5a3-05f0-4a94-8d60-9678e9bbdd21\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"776a773ddcfbf486b9d6279b170bca065dcbd06a56a6f975b0e318417a61d5b1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0b14aa2b-d687-4003-bca7-d612ecc453aa\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"83efd75536847839db9dd72e908ed395701d3829bd31a091880bcfe2b59e776e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 38742, \"end_char_idx\": 41900, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "```python\nevaluator = Evaluator(df)\nevaluator.plot_model_comparison([\"generated_answer\", \"ft_generated_answer\", \"ft_generated_answer_few_shot\"], scenario=\"answer_expected\", nice_names=[\"Baseline\", \"Fine-Tuned\", \"Fine-Tuned with Few-Shot\"])\n```\n\n\n    \n![png](output_56_0.png)\n    \n\n\nThis is quite amazing -- we're able to get the best of both worlds! We're able to get the model to be both correct and conservative: \n\n1. The model is correct 83% of the time -- this is the same as the base model\n2. The model gives the wrong answer only 8% of the time -- down from 17% with the base model\n\nNext, let's look at the hallucinations. We want to reduce the hallucinations, but not at the cost of correctness. We want to strike a balance between the two. We've struck a good balance here:\n\n1. The model hallucinates 53% of the time -- down from 100% with the base model\n2. The model says \"I don't know\" 47% of the time -- up from NEVER with the base model\n\n\n```python\nevaluator.plot_model_comparison([\"generated_answer\", \"ft_generated_answer\", \"ft_generated_answer_few_shot\"], scenario=\"idk_expected\", nice_names=[\"Baseline\", \"Fine-Tuned\", \"Fine-Tuned with Few-Shot\"])\n```\n\n\n    \n![png](output_59_0.png)\n    \n\n\nFew Shot Fine-Tuning with Qdrant is a great way to control and steer the performance of your RAG system. Here, we made the model less conservative compared to zero shot and more confident by using Qdrant to find similar questions. \n\nYou can also use Qdrant to make the model more conservative. We did this by giving examples of questions where the answer is not present in the context.  \nThis is biasing the model to say \"I don't know\" more often. \n\nSimilarly, one can also use Qdrant to make the model more confident by giving examples of questions where the answer is present in the context. This biases the model to give an answer more often. The trade-off is that the model will also hallucinate more often.\n\nYou can make this trade off by adjusting the training data: distribution of questions and examples, as well as the kind and number of examples you retrieve from Qdrant.\n\n## 9. Conclusion\n\nIn this notebook, we've demonstrated how to fine-tune OpenAI models for specific use-cases. We've also demonstrated how to use Qdrant and Few-Shot Learning to improve the performance of the model.\n\n### Aggregate Results\n\nSo far, we've looked at the results for each scenario separately, i.e. each scenario summed to 100. Let's look at the results as an aggregate to get a broader sense of how the model is performing:\n\n| Category | Base | Fine-Tuned | Fine-Tuned with Qdrant |\n| --- | --- | --- | --- |\n| Correct | 44% | 32% | 44% |\n| Skipped | 0% | 18% | 5% |\n| Wrong | 9% | 3% | 4% |\n| Hallucination | 47% | 7% | 25% |\n| I don't know | 0% | 40% | 22% |\n\n### Observations\n\n#### Compared to base model\n1. The few shot fine-tuned with Qdrant model is as good as the base model at answering questions where the answer is present in the context. \n2. The few shot fine-tuned with Qdrant model is better at saying \"I don't know\" when the answer is not present in the context.\n3. The few shot fine-tuned with Qdrant model is better at reducing hallucinations."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 0b14aa2b-d687-4003-bca7-d612ecc453aa
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"0b14aa2b-d687-4003-bca7-d612ecc453aa\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"424e9d13-5f08-4e53-911a-fc541dfb6f26\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e4b2a4abe288387db31d1ee0dea869c780dc26e93cb567c3013575311a249e0\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4b40084f-8b75-4e9a-9ced-c96938fea3c8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb\", \"title\": \"Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning\", \"tokens\": 10450, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3e86bc105ae45c3c71439b649b2f7348a6c1be6d4e02c3eafac350685c345ecd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 41903, \"end_char_idx\": 42696, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "ref_doc_id": "424e9d13-5f08-4e53-911a-fc541dfb6f26", "chroma:document": "#### Compared to fine-tuned model\n1. The few shot fine-tuned with Qdrant model gets more correct answers than the fine-tuned model: **83% of the questions are answered correctly vs 60%** for the fine-tuned model\n2. The few shot fine-tuned with Qdrant model is better at deciding when to say \"I don't know\" when the answer is not present in the context. **34% skip rate for the plain fine-tuning mode, vs 9% for the few shot fine-tuned with Qdrant model**\n\n\nNow, you should be able to:\n\n1. Notice the trade-offs between number of correct answers and hallucinations -- and how training dataset choice influences that!\n2. Fine-tune OpenAI models for specific use-cases and use Qdrant to improve the performance of your RAG model\n3. Get started on how to evaluate the performance of your RAG model"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: b89fb55f-5861-404b-9457-6de45c06cb19
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"b89fb55f-5861-404b-9457-6de45c06cb19\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7dd4c2d3fce9be1cb1b7af1fe064114b802aeedadb7c9529ce98b07483663b2d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"84ef89b3-df0e-4230-8952-822474819578\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"745a03d972c117d89502888561536d0cd90c1067ef62b129e3757c30418a3f24\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3332, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "doc_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "ref_doc_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "chroma:document": "# How to use GPT-4 Vision with Function Calling \n\nThe new GPT-4 Turbo model, available as gpt-4-turbo-2024-04-09 as of April 2024, now enables function calling with vision capabilities, better reasoning and a knowledge cutoff date of Dec 2023. Using images with function calling will unlock multimodal use cases and the ability to use reasoning, allowing you to go beyond OCR and image descriptions.\n\nWe will go through two examples to demonstrate the use of function calling with GPT-4 Turbo with Vision:\n\n1. Simulating a customer service assistant for delivery exception support\n2. Analyzing an organizational chart to extract employee information\n\n### Installation and Setup\n\n\n```python\n!pip install pymupdf --quiet\n!pip install openai --quiet\n!pip install matplotlib --quiet\n# instructor makes it easy to work with function calling\n!pip install instructor --quiet\n```\n\n\n```python\nimport base64\nimport os\nfrom enum import Enum\nfrom io import BytesIO\nfrom typing import Iterable\nfrom typing import List\nfrom typing import Literal, Optional\n\nimport fitz\n# Instructor is powered by Pydantic, which is powered by type hints. Schema validation, prompting is controlled by type annotations\nimport instructor\nimport matplotlib.pyplot as plt\nimport pandas as pd\nfrom IPython.display import display\nfrom PIL import Image\nfrom openai import OpenAI\nfrom pydantic import BaseModel, Field\n```\n\n## 1. Simulating a customer service assistant for delivery exception support\nWe will simulate a customer service assistant for a delivery service that is equipped to analyze images of packages. The assistant will perform the following actions based on the image analysis:\n- If a package appears damaged in the image, automatically process a refund according to policy.\n- If the package looks wet, initiate a replacement.\n- If the package appears normal and not damaged, escalate to an agent.\n\nLet's look at the sample images of packages that the customer service assistant will analyze to determine the appropriate action. We will encode the images as base64 strings for processing by the model.\n\n\n```python\n# Function to encode the image as base64\ndef encode_image(image_path: str):\n    # check if the image exists\n    if not os.path.exists(image_path):\n        raise FileNotFoundError(f\"Image file not found: {image_path}\")\n    with open(image_path, \"rb\") as image_file:\n        return base64.b64encode(image_file.read()).decode('utf-8')\n\n\n# Sample images for testing\nimage_dir = \"images\"\n\n# encode all images within the directory\nimage_files = os.listdir(image_dir)\nimage_data = {}\nfor image_file in image_files:\n    image_path = os.path.join(image_dir, image_file)\n    # encode the image with key as the image file name\n    image_data[image_file.split('.')[0]] = encode_image(image_path)\n    print(f\"Encoded image: {image_file}\")\n\n\ndef display_images(image_data: dict):\n    fig, axs = plt.subplots(1, 3, figsize=(18, 6))\n    for i, (key, value) in enumerate(image_data.items()):\n        img = Image.open(BytesIO(base64.b64decode(value)))\n        ax = axs[i]\n        ax.imshow(img)\n        ax.axis(\"off\")\n        ax.set_title(key)\n    plt.tight_layout()\n    plt.show()\n\n\ndisplay_images(image_data)\n```\n\n    Encoded image: wet_package.jpg\n    Encoded image: damaged_package.jpg\n    Encoded image: normal_package.jpg\n\n\n\n    \n![png](output_6_1.png)"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 84ef89b3-df0e-4230-8952-822474819578
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"84ef89b3-df0e-4230-8952-822474819578\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7dd4c2d3fce9be1cb1b7af1fe064114b802aeedadb7c9529ce98b07483663b2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b89fb55f-5861-404b-9457-6de45c06cb19\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"46683dc51a593a3ebf5a0460b4280687e6792f8a0f5b1befe8cfde51bd3717f2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"78fc32f9-fcd5-46d7-9e38-19d7888fb871\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d3d45c8c1b85f18165f8e055d7c526a63c9ad5a0c26532dfbd2b013df157db59\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2082, \"end_char_idx\": 3894, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "doc_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "ref_doc_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "chroma:document": "```python\n# Function to encode the image as base64\ndef encode_image(image_path: str):\n    # check if the image exists\n    if not os.path.exists(image_path):\n        raise FileNotFoundError(f\"Image file not found: {image_path}\")\n    with open(image_path, \"rb\") as image_file:\n        return base64.b64encode(image_file.read()).decode('utf-8')\n\n\n# Sample images for testing\nimage_dir = \"images\"\n\n# encode all images within the directory\nimage_files = os.listdir(image_dir)\nimage_data = {}\nfor image_file in image_files:\n    image_path = os.path.join(image_dir, image_file)\n    # encode the image with key as the image file name\n    image_data[image_file.split('.')[0]] = encode_image(image_path)\n    print(f\"Encoded image: {image_file}\")\n\n\ndef display_images(image_data: dict):\n    fig, axs = plt.subplots(1, 3, figsize=(18, 6))\n    for i, (key, value) in enumerate(image_data.items()):\n        img = Image.open(BytesIO(base64.b64decode(value)))\n        ax = axs[i]\n        ax.imshow(img)\n        ax.axis(\"off\")\n        ax.set_title(key)\n    plt.tight_layout()\n    plt.show()\n\n\ndisplay_images(image_data)\n```\n\n    Encoded image: wet_package.jpg\n    Encoded image: damaged_package.jpg\n    Encoded image: normal_package.jpg\n\n\n\n    \n![png](output_6_1.png)\n    \n\n\nWe have successfully encoded the sample images as base64 strings and displayed them. The customer service assistant will analyze these images to determine the appropriate action based on the package condition.\n\nLet's now define the functions/tools for order processing, such as escalating an order to an agent, refunding an order, and replacing an order. We will create placeholder functions to simulate the processing of these actions based on the identified tools. We will be using Pydantic models to define the structure of the data for order actions."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 78fc32f9-fcd5-46d7-9e38-19d7888fb871
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"78fc32f9-fcd5-46d7-9e38-19d7888fb871\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7dd4c2d3fce9be1cb1b7af1fe064114b802aeedadb7c9529ce98b07483663b2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"84ef89b3-df0e-4230-8952-822474819578\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"2b89e86baf3bf8d0c57a75e36e0d3137c43b51245aa93f3cc52469c244a27eb6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"78b42426-1e2d-42a8-9821-09fd51ec4b46\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"601b0f363b96725fb8e2aefc4305a30e273ca81d01b026fe6eca2561b435530a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2426, \"end_char_idx\": 6996, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "doc_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "ref_doc_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "chroma:document": "# Sample images for testing\nimage_dir = \"images\"\n\n# encode all images within the directory\nimage_files = os.listdir(image_dir)\nimage_data = {}\nfor image_file in image_files:\n    image_path = os.path.join(image_dir, image_file)\n    # encode the image with key as the image file name\n    image_data[image_file.split('.')[0]] = encode_image(image_path)\n    print(f\"Encoded image: {image_file}\")\n\n\ndef display_images(image_data: dict):\n    fig, axs = plt.subplots(1, 3, figsize=(18, 6))\n    for i, (key, value) in enumerate(image_data.items()):\n        img = Image.open(BytesIO(base64.b64decode(value)))\n        ax = axs[i]\n        ax.imshow(img)\n        ax.axis(\"off\")\n        ax.set_title(key)\n    plt.tight_layout()\n    plt.show()\n\n\ndisplay_images(image_data)\n```\n\n    Encoded image: wet_package.jpg\n    Encoded image: damaged_package.jpg\n    Encoded image: normal_package.jpg\n\n\n\n    \n![png](output_6_1.png)\n    \n\n\nWe have successfully encoded the sample images as base64 strings and displayed them. The customer service assistant will analyze these images to determine the appropriate action based on the package condition.\n\nLet's now define the functions/tools for order processing, such as escalating an order to an agent, refunding an order, and replacing an order. We will create placeholder functions to simulate the processing of these actions based on the identified tools. We will be using Pydantic models to define the structure of the data for order actions.\n\n\n\n```python\nMODEL = \"gpt-4-turbo-2024-04-09\"\n\nclass Order(BaseModel):\n    \"\"\"Represents an order with details such as order ID, customer name, product name, price, status, and delivery date.\"\"\"\n    order_id: str = Field(..., description=\"The unique identifier of the order\")\n    product_name: str = Field(..., description=\"The name of the product\")\n    price: float = Field(..., description=\"The price of the product\")\n    status: str = Field(..., description=\"The status of the order\")\n    delivery_date: str = Field(..., description=\"The delivery date of the order\")\n# Placeholder functions for order processing\n\ndef get_order_details(order_id):\n    # Placeholder function to retrieve order details based on the order ID\n    return Order(\n        order_id=order_id,\n        product_name=\"Product X\",\n        price=100.0,\n        status=\"Delivered\",\n        delivery_date=\"2024-04-10\",\n    )\n\ndef escalate_to_agent(order: Order, message: str):\n    # Placeholder function to escalate the order to a human agent\n    return f\"Order {order.order_id} has been escalated to an agent with message: `{message}`\"\n\ndef refund_order(order: Order):\n    # Placeholder function to process a refund for the order\n    return f\"Order {order.order_id} has been refunded successfully.\"\n\ndef replace_order(order: Order):\n    # Placeholder function to replace the order with a new one\n    return f\"Order {order.order_id} has been replaced with a new order.\"\n\nclass FunctionCallBase(BaseModel):\n    rationale: Optional[str] = Field(..., description=\"The reason for the action.\")\n    image_description: Optional[str] = Field(\n        ..., description=\"The detailed description of the package image.\"\n    )\n    action: Literal[\"escalate_to_agent\", \"replace_order\", \"refund_order\"]\n    message: Optional[str] = Field(\n        ...,\n        description=\"The message to be escalated to the agent if action is escalate_to_agent\",\n    )\n    # Placeholder functions to process the action based on the order ID\n    def __call__(self, order_id):\n        order: Order = get_order_details(order_id=order_id)\n        if self.action == \"escalate_to_agent\":\n            return escalate_to_agent(order, self.message)\n        if self.action == \"replace_order\":\n            return replace_order(order)\n        if self.action == \"refund_order\":\n            return refund_order(order)\n\nclass EscalateToAgent(FunctionCallBase):\n    \"\"\"Escalate to an agent for further assistance.\"\"\"\n    pass\n\nclass OrderActionBase(FunctionCallBase):\n    pass\n\nclass ReplaceOrder(OrderActionBase):\n    \"\"\"Tool call to replace an order.\"\"\"\n    pass\n\nclass RefundOrder(OrderActionBase):\n    \"\"\"Tool call to refund an order.\"\"\"\n    pass\n```\n\n### Simulating user messages and processing the package images\n\nWe will simulate user messages containing the package images and process the images using the GPT-4 Turbo with Vision model. The model will identify the appropriate tool call based on the image analysis and the predefined actions for damaged, wet, or normal packages. We will then process the identified action based on the order ID and display the results."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 78b42426-1e2d-42a8-9821-09fd51ec4b46
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"78b42426-1e2d-42a8-9821-09fd51ec4b46\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7dd4c2d3fce9be1cb1b7af1fe064114b802aeedadb7c9529ce98b07483663b2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"78fc32f9-fcd5-46d7-9e38-19d7888fb871\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1a9db62f977ec9e31b747334ac3e62fda3ef23c4252cbb761ee2afdb0ccfefb4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2ec348c9-9d83-474f-a958-a7a3455c5988\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1d31a0d166c9e36b48e6c7942431e27b88965cc6df32228d4bcf9656f4bdeca2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6999, \"end_char_idx\": 8706, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "doc_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "ref_doc_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "chroma:document": "```python\n# extract the tool call from the response\nORDER_ID = \"12345\"  # Placeholder order ID for testing\nINSTRUCTION_PROMPT = \"You are a customer service assistant for a delivery service, equipped to analyze images of packages. If a package appears damaged in the image, automatically process a refund according to policy. If the package looks wet, initiate a replacement. If the package appears normal and not damaged, escalate to agent. For any other issues or unclear images, escalate to agent. You must always use tools!\"\n\ndef delivery_exception_support_handler(test_image: str):\n    payload = {\n        \"model\": MODEL,\n        \"response_model\": Iterable[RefundOrder | ReplaceOrder | EscalateToAgent],\n        \"tool_choice\": \"auto\",  # automatically select the tool based on the context\n        \"temperature\": 0.0,  # for less diversity in responses\n        \"seed\": 123,  # Set a seed for reproducibility\n    }\n    payload[\"messages\"] = [\n        {\n            \"role\": \"user\",\n            \"content\": INSTRUCTION_PROMPT,\n        },\n        {\n            \"role\": \"user\",\n            \"content\": [\n                {\n                    \"type\": \"image_url\",\n                    \"image_url\": {\n                        \"url\": f\"data:image/jpeg;base64,{image_data[test_image]}\"\n                    }\n                },\n            ],\n        }\n    ]\n    function_calls = instructor.from_openai(\n        OpenAI(), mode=instructor.Mode.PARALLEL_TOOLS\n    ).chat.completions.create(**payload)\n    for tool in function_calls:\n        print(f\"- Tool call: {tool.action} for provided img: {test_image}\")\n        print(f\"- Parameters: {tool}\")\n        print(f\">> Action result: {tool(ORDER_ID)}\")\n        return tool"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 2ec348c9-9d83-474f-a958-a7a3455c5988
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"2ec348c9-9d83-474f-a958-a7a3455c5988\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7dd4c2d3fce9be1cb1b7af1fe064114b802aeedadb7c9529ce98b07483663b2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"78b42426-1e2d-42a8-9821-09fd51ec4b46\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"8ccddf402cb952492f82a4d1eb6acbebe598c769dc02bb7e510ac6cda2e64355\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f56150a3-1e9c-48d3-ae0c-77ba1d524b9f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0161fce6c3cc5d65361fe0b946bc1e4742fcece4127c254e69f4ff93f4535a45\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6999, \"end_char_idx\": 11389, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "doc_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "ref_doc_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "chroma:document": "```python\n# extract the tool call from the response\nORDER_ID = \"12345\"  # Placeholder order ID for testing\nINSTRUCTION_PROMPT = \"You are a customer service assistant for a delivery service, equipped to analyze images of packages. If a package appears damaged in the image, automatically process a refund according to policy. If the package looks wet, initiate a replacement. If the package appears normal and not damaged, escalate to agent. For any other issues or unclear images, escalate to agent. You must always use tools!\"\n\ndef delivery_exception_support_handler(test_image: str):\n    payload = {\n        \"model\": MODEL,\n        \"response_model\": Iterable[RefundOrder | ReplaceOrder | EscalateToAgent],\n        \"tool_choice\": \"auto\",  # automatically select the tool based on the context\n        \"temperature\": 0.0,  # for less diversity in responses\n        \"seed\": 123,  # Set a seed for reproducibility\n    }\n    payload[\"messages\"] = [\n        {\n            \"role\": \"user\",\n            \"content\": INSTRUCTION_PROMPT,\n        },\n        {\n            \"role\": \"user\",\n            \"content\": [\n                {\n                    \"type\": \"image_url\",\n                    \"image_url\": {\n                        \"url\": f\"data:image/jpeg;base64,{image_data[test_image]}\"\n                    }\n                },\n            ],\n        }\n    ]\n    function_calls = instructor.from_openai(\n        OpenAI(), mode=instructor.Mode.PARALLEL_TOOLS\n    ).chat.completions.create(**payload)\n    for tool in function_calls:\n        print(f\"- Tool call: {tool.action} for provided img: {test_image}\")\n        print(f\"- Parameters: {tool}\")\n        print(f\">> Action result: {tool(ORDER_ID)}\")\n        return tool\n\n\nprint(\"Processing delivery exception support for different package images...\")\n\nprint(\"\\n===================== Simulating user message 1 =====================\")\nassert delivery_exception_support_handler(\"damaged_package\").action == \"refund_order\"\n\nprint(\"\\n===================== Simulating user message 2 =====================\")\nassert delivery_exception_support_handler(\"normal_package\").action == \"escalate_to_agent\"\n\nprint(\"\\n===================== Simulating user message 3 =====================\")\nassert delivery_exception_support_handler(\"wet_package\").action == \"replace_order\"\n```\n\n    Processing delivery exception support for different package images...\n    \n    ===================== Simulating user message 1 =====================\n    - Tool call: refund_order for provided img: damaged_package\n    - Parameters: rationale='The package is visibly damaged with significant tears and crushing, indicating potential harm to the contents.' image_description='The package in the image shows extensive damage, including deep creases and tears in the cardboard. The package is also wrapped with extra tape, suggesting prior attempts to secure it after damage.' action='refund_order' message=None\n    >> Action result: Order 12345 has been refunded successfully.\n    \n    ===================== Simulating user message 2 =====================\n    - Tool call: escalate_to_agent for provided img: normal_package\n    - Parameters: rationale='The package appears normal and not damaged, requiring further assistance for any potential issues not visible in the image.' image_description='A cardboard box on a wooden floor, appearing intact and undamaged, with no visible signs of wear, tear, or wetness.' action='escalate_to_agent' message='Please review this package for any issues not visible in the image. The package appears normal and undamaged.'\n    >> Action result: Order 12345 has been escalated to an agent with message: `Please review this package for any issues not visible in the image. The package appears normal and undamaged.`\n    \n    ===================== Simulating user message 3 =====================\n    - Tool call: replace_order for provided img: wet_package\n    - Parameters: rationale='The package appears wet, which may have compromised the contents, especially since it is labeled as fragile.' image_description=\"The package in the image shows significant wetness on the top surface, indicating potential water damage. The box is labeled 'FRAGILE', which suggests that the contents are delicate and may be more susceptible to damage from moisture.\" action='replace_order' message=None\n    >> Action result: Order 12345 has been replaced with a new order."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: f56150a3-1e9c-48d3-ae0c-77ba1d524b9f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"f56150a3-1e9c-48d3-ae0c-77ba1d524b9f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7dd4c2d3fce9be1cb1b7af1fe064114b802aeedadb7c9529ce98b07483663b2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"2ec348c9-9d83-474f-a958-a7a3455c5988\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"930f89234edc88fab0296112fc212cbd461c3c6d365d56fb8445b32507a31ca1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"88ec2a3b-03d4-4a09-a16c-b830a5c69f95\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a74d0c370790e2b8f38e6374975e7130d77095e5d57d70256b63e9fd766b3d06\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11392, \"end_char_idx\": 14076, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "doc_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "ref_doc_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "chroma:document": "## 2.  Analyzing an organizational chart to extract employee information\n\nFor the second example, we will analyze an organizational chart image to extract employee information, such as employee names, roles, managers, and manager roles. We will use GPT-4 Turbo with Vision to process the organizational chart image and extract structured data about the employees in the organization. Indeed, function calling lets us go beyond OCR to actually deduce and translate hierarchical relationships within the chart.\n\nWe will start with a sample organizational chart in PDF format that we want to analyze and convert the first page of the PDF to a JPEG image for analysis.\n\n\n```python\n# Function to convert a single page PDF page to a JPEG image\ndef convert_pdf_page_to_jpg(pdf_path: str, output_path: str, page_number=0):\n    if not os.path.exists(pdf_path):\n        raise FileNotFoundError(f\"PDF file not found: {pdf_path}\")\n    doc = fitz.open(pdf_path)\n    page = doc.load_page(page_number)  # 0 is the first page\n    pix = page.get_pixmap()\n    # Save the pixmap as a JPEG\n    pix.save(output_path)\n\n\ndef display_img_local(image_path: str):\n    img = Image.open(image_path)\n    display(img)\n\n\npdf_path = 'data/org-chart-sample.pdf'\noutput_path = 'org-chart-sample.jpg'\n\nconvert_pdf_page_to_jpg(pdf_path, output_path)\ndisplay_img_local(output_path)\n```\n\n\n    \n![png](output_12_0.png)\n    \n\n\nThe organizational chart image has been successfully extracted from the PDF file and displayed. Let's now define a function to analyze the organizational chart image using the new GPT4 Turbo with Vision. The function will extract information about the employees, their roles, and their managers from the image. We will use function/tool calling to specify the input parameters for the organizational structure, such as the employee name, role, and manager's name and role. We will use Pydantic models to define the structure of the data.\n\n\n\n```python\nbase64_img = encode_image(output_path)\n\nclass RoleEnum(str, Enum):\n    \"\"\"Defines possible roles within an organization.\"\"\"\n    CEO = \"CEO\"\n    CTO = \"CTO\"\n    CFO = \"CFO\"\n    COO = \"COO\"\n    EMPLOYEE = \"Employee\"\n    MANAGER = \"Manager\"\n    INTERN = \"Intern\"\n    OTHER = \"Other\"\n\nclass Employee(BaseModel):\n    \"\"\"Represents an employee, including their name, role, and optional manager information.\"\"\"\n    employee_name: str = Field(..., description=\"The name of the employee\")\n    role: RoleEnum = Field(..., description=\"The role of the employee\")\n    manager_name: Optional[str] = Field(None, description=\"The manager's name, if applicable\")\n    manager_role: Optional[RoleEnum] = Field(None, description=\"The manager's role, if applicable\")"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 88ec2a3b-03d4-4a09-a16c-b830a5c69f95
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"88ec2a3b-03d4-4a09-a16c-b830a5c69f95\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7dd4c2d3fce9be1cb1b7af1fe064114b802aeedadb7c9529ce98b07483663b2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f56150a3-1e9c-48d3-ae0c-77ba1d524b9f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"396247bc022c9db90842af8fdb09d438cd904ab1997e934f0f8b10092eed4152\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"5bda5212-5e9b-4538-bc05-6ba99883dd78\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8afa97cc5acd2cc2c8a615f65a45c5d50bdf2d608544d1fb1f7bbef4dd07e52b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12490, \"end_char_idx\": 15969, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "doc_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "ref_doc_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "chroma:document": "def display_img_local(image_path: str):\n    img = Image.open(image_path)\n    display(img)\n\n\npdf_path = 'data/org-chart-sample.pdf'\noutput_path = 'org-chart-sample.jpg'\n\nconvert_pdf_page_to_jpg(pdf_path, output_path)\ndisplay_img_local(output_path)\n```\n\n\n    \n![png](output_12_0.png)\n    \n\n\nThe organizational chart image has been successfully extracted from the PDF file and displayed. Let's now define a function to analyze the organizational chart image using the new GPT4 Turbo with Vision. The function will extract information about the employees, their roles, and their managers from the image. We will use function/tool calling to specify the input parameters for the organizational structure, such as the employee name, role, and manager's name and role. We will use Pydantic models to define the structure of the data.\n\n\n\n```python\nbase64_img = encode_image(output_path)\n\nclass RoleEnum(str, Enum):\n    \"\"\"Defines possible roles within an organization.\"\"\"\n    CEO = \"CEO\"\n    CTO = \"CTO\"\n    CFO = \"CFO\"\n    COO = \"COO\"\n    EMPLOYEE = \"Employee\"\n    MANAGER = \"Manager\"\n    INTERN = \"Intern\"\n    OTHER = \"Other\"\n\nclass Employee(BaseModel):\n    \"\"\"Represents an employee, including their name, role, and optional manager information.\"\"\"\n    employee_name: str = Field(..., description=\"The name of the employee\")\n    role: RoleEnum = Field(..., description=\"The role of the employee\")\n    manager_name: Optional[str] = Field(None, description=\"The manager's name, if applicable\")\n    manager_role: Optional[RoleEnum] = Field(None, description=\"The manager's role, if applicable\")\n\n\nclass EmployeeList(BaseModel):\n    \"\"\"A list of employees within the organizational structure.\"\"\"\n    employees: List[Employee] = Field(..., description=\"A list of employees\")\n\ndef parse_orgchart(base64_img: str) -> EmployeeList:\n    response = instructor.from_openai(OpenAI()).chat.completions.create(\n        model='gpt-4-turbo',\n        response_model=EmployeeList,\n        messages=[\n            {\n                \"role\": \"user\",\n                \"content\": 'Analyze the given organizational chart and very carefully extract the information.',\n            },\n            {\n                \"role\": \"user\",\n                \"content\": [\n                    {\n                        \"type\": \"image_url\",\n                        \"image_url\": {\n                            \"url\": f\"data:image/jpeg;base64,{base64_img}\"\n                        }\n                    },\n                ],\n            }\n        ],\n    )\n    return response\n```\n\nNow, we will define a function to parse the response from GPT-4 Turbo with Vision and extract the employee data. We will tabulate the extracted data for easy visualization. Please note that the accuracy of the extracted data may vary based on the complexity and clarity of the input image.\n\n\n```python\n# call the functions to analyze the organizational chart and parse the response\nresult = parse_orgchart(base64_img)\n\n# tabulate the extracted data\ndf = pd.DataFrame([{\n    'employee_name': employee.employee_name,\n    'role': employee.role.value,\n    'manager_name': employee.manager_name,\n    'manager_role': employee.manager_role.value if employee.manager_role else None\n} for employee in result.employees])\n\ndisplay(df)\n```\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 5bda5212-5e9b-4538-bc05-6ba99883dd78
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"5bda5212-5e9b-4538-bc05-6ba99883dd78\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7dd4c2d3fce9be1cb1b7af1fe064114b802aeedadb7c9529ce98b07483663b2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"88ec2a3b-03d4-4a09-a16c-b830a5c69f95\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"0f433f8524864ef7043fc868e55d92de1f48aa107e6258dcbe583be7cdb78d6c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b0d4f12a-091d-406c-b90c-11e272504466\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ece6b649218241031b7087b48f3aca6603da0a823f8531daa215ea826646111d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14079, \"end_char_idx\": 16658, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "doc_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "ref_doc_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "chroma:document": "class EmployeeList(BaseModel):\n    \"\"\"A list of employees within the organizational structure.\"\"\"\n    employees: List[Employee] = Field(..., description=\"A list of employees\")\n\ndef parse_orgchart(base64_img: str) -> EmployeeList:\n    response = instructor.from_openai(OpenAI()).chat.completions.create(\n        model='gpt-4-turbo',\n        response_model=EmployeeList,\n        messages=[\n            {\n                \"role\": \"user\",\n                \"content\": 'Analyze the given organizational chart and very carefully extract the information.',\n            },\n            {\n                \"role\": \"user\",\n                \"content\": [\n                    {\n                        \"type\": \"image_url\",\n                        \"image_url\": {\n                            \"url\": f\"data:image/jpeg;base64,{base64_img}\"\n                        }\n                    },\n                ],\n            }\n        ],\n    )\n    return response\n```\n\nNow, we will define a function to parse the response from GPT-4 Turbo with Vision and extract the employee data. We will tabulate the extracted data for easy visualization. Please note that the accuracy of the extracted data may vary based on the complexity and clarity of the input image.\n\n\n```python\n# call the functions to analyze the organizational chart and parse the response\nresult = parse_orgchart(base64_img)\n\n# tabulate the extracted data\ndf = pd.DataFrame([{\n    'employee_name': employee.employee_name,\n    'role': employee.role.value,\n    'manager_name': employee.manager_name,\n    'manager_role': employee.manager_role.value if employee.manager_role else None\n} for employee in result.employees])\n\ndisplay(df)\n```\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>employee_name</th>\n      <th>role</th>\n      <th>manager_name</th>\n      <th>manager_role</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Juliana Silva</td>\n      <td>CEO</td>\n      <td>None</td>\n      <td>None</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Kim Chun Hei</td>\n      <td>CFO</td>\n      <td>Juliana Silva</td>\n      <td>CEO</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Chad Gibbons</td>\n      <td>CTO</td>\n      <td>Juliana Silva</td>\n      <td>CEO</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>Chiaki Sato</td>"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: b0d4f12a-091d-406c-b90c-11e272504466
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"b0d4f12a-091d-406c-b90c-11e272504466\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7dd4c2d3fce9be1cb1b7af1fe064114b802aeedadb7c9529ce98b07483663b2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"5bda5212-5e9b-4538-bc05-6ba99883dd78\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f759473237d4bb776ab8153a9a5b34b051624e7f7451ac91525de9785e8985f4\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"461d50e5-9d16-4a74-91cc-8c3c710d26e0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1a83601e0883f8d7bd158bd92af26ca072184d6a5c88e490c4591e25bed6e27f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15914, \"end_char_idx\": 17294, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "doc_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "ref_doc_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "chroma:document": "}\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>employee_name</th>\n      <th>role</th>\n      <th>manager_name</th>\n      <th>manager_role</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>Juliana Silva</td>\n      <td>CEO</td>\n      <td>None</td>\n      <td>None</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>Kim Chun Hei</td>\n      <td>CFO</td>\n      <td>Juliana Silva</td>\n      <td>CEO</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>Chad Gibbons</td>\n      <td>CTO</td>\n      <td>Juliana Silva</td>\n      <td>CEO</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>Chiaki Sato</td>\n      <td>COO</td>\n      <td>Juliana Silva</td>\n      <td>CEO</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>Cahaya Dewi</td>\n      <td>Manager</td>\n      <td>Kim Chun Hei</td>\n      <td>CFO</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>Shawn Garcia</td>\n      <td>Manager</td>\n      <td>Chad Gibbons</td>\n      <td>CTO</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>Aaron Loeb</td>\n      <td>Manager</td>\n      <td>Chiaki Sato</td>\n      <td>COO</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>Drew Feig</td>\n      <td>Employee</td>\n      <td>Cahaya Dewi</td>\n      <td>Manager</td>\n    </tr>\n    <tr>\n      <th>8</th>"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 461d50e5-9d16-4a74-91cc-8c3c710d26e0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"461d50e5-9d16-4a74-91cc-8c3c710d26e0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7dd4c2d3fce9be1cb1b7af1fe064114b802aeedadb7c9529ce98b07483663b2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b0d4f12a-091d-406c-b90c-11e272504466\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"58e6e46a7d1c31b57c8413b087293a490306675d4d05fe4bd0c9b9e2fffdac9b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b022f4c1-2ea0-43e7-b8bb-5857d19e87d2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"969460755f127eb82048844676c1719f5c0e279955c7e0963406270c7455c8d1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16638, \"end_char_idx\": 17968, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "doc_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "ref_doc_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "chroma:document": "<td>Chiaki Sato</td>\n      <td>COO</td>\n      <td>Juliana Silva</td>\n      <td>CEO</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>Cahaya Dewi</td>\n      <td>Manager</td>\n      <td>Kim Chun Hei</td>\n      <td>CFO</td>\n    </tr>\n    <tr>\n      <th>5</th>\n      <td>Shawn Garcia</td>\n      <td>Manager</td>\n      <td>Chad Gibbons</td>\n      <td>CTO</td>\n    </tr>\n    <tr>\n      <th>6</th>\n      <td>Aaron Loeb</td>\n      <td>Manager</td>\n      <td>Chiaki Sato</td>\n      <td>COO</td>\n    </tr>\n    <tr>\n      <th>7</th>\n      <td>Drew Feig</td>\n      <td>Employee</td>\n      <td>Cahaya Dewi</td>\n      <td>Manager</td>\n    </tr>\n    <tr>\n      <th>8</th>\n      <td>Richard Sanchez</td>\n      <td>Employee</td>\n      <td>Cahaya Dewi</td>\n      <td>Manager</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>Sacha Dubois</td>\n      <td>Intern</td>\n      <td>Cahaya Dewi</td>\n      <td>Manager</td>\n    </tr>\n    <tr>\n      <th>10</th>\n      <td>Olivia Wilson</td>\n      <td>Employee</td>\n      <td>Shawn Garcia</td>\n      <td>Manager</td>\n    </tr>\n    <tr>\n      <th>11</th>\n      <td>Matt Zhang</td>\n      <td>Intern</td>\n      <td>Shawn Garcia</td>\n      <td>Manager</td>\n    </tr>\n    <tr>\n      <th>12</th>\n      <td>Avery Davis</td>\n      <td>Employee</td>\n      <td>Aaron Loeb</td>\n      <td>Manager</td>\n    </tr>\n    <tr>"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: b022f4c1-2ea0-43e7-b8bb-5857d19e87d2
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"b022f4c1-2ea0-43e7-b8bb-5857d19e87d2\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7dd4c2d3fce9be1cb1b7af1fe064114b802aeedadb7c9529ce98b07483663b2d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"461d50e5-9d16-4a74-91cc-8c3c710d26e0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb\", \"title\": \"How to use GPT-4 Vision with Function Calling\", \"tokens\": 4334, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9df9b951dcf2a73f5903e659cd04e0e72a5acaa6bea7b0abc4b5b6bd7bef6695\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17263, \"end_char_idx\": 18712, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "doc_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "ref_doc_id": "97dbb12b-d1b3-4992-a9fd-e6a3d4eddd99", "chroma:document": "</tr>\n    <tr>\n      <th>8</th>\n      <td>Richard Sanchez</td>\n      <td>Employee</td>\n      <td>Cahaya Dewi</td>\n      <td>Manager</td>\n    </tr>\n    <tr>\n      <th>9</th>\n      <td>Sacha Dubois</td>\n      <td>Intern</td>\n      <td>Cahaya Dewi</td>\n      <td>Manager</td>\n    </tr>\n    <tr>\n      <th>10</th>\n      <td>Olivia Wilson</td>\n      <td>Employee</td>\n      <td>Shawn Garcia</td>\n      <td>Manager</td>\n    </tr>\n    <tr>\n      <th>11</th>\n      <td>Matt Zhang</td>\n      <td>Intern</td>\n      <td>Shawn Garcia</td>\n      <td>Manager</td>\n    </tr>\n    <tr>\n      <th>12</th>\n      <td>Avery Davis</td>\n      <td>Employee</td>\n      <td>Aaron Loeb</td>\n      <td>Manager</td>\n    </tr>\n    <tr>\n      <th>13</th>\n      <td>Harper Russo</td>\n      <td>Employee</td>\n      <td>Aaron Loeb</td>\n      <td>Manager</td>\n    </tr>\n    <tr>\n      <th>14</th>\n      <td>Taylor Alonso</td>\n      <td>Intern</td>\n      <td>Aaron Loeb</td>\n      <td>Manager</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\nThe extracted data from the organizational chart has been successfully parsed and displayed in a DataFrame. This approach allows us to leverage GPT-4 Turbo with Vision capabilities to extract structured information from images, such as organizational charts and diagrams, and process the data for further analysis. By using function calling, we can extend the functionality of multimodal models to perform specific tasks or call external functions."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: c017f2fc-f4ec-4004-bb0b-fea39e231210
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_wandb.ipynb", "title": "Load the embeddings", "tokens": 506, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"c017f2fc-f4ec-4004-bb0b-fea39e231210\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_wandb.ipynb\", \"title\": \"Load the embeddings\", \"tokens\": 506, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"70027c3a-3e35-44f1-ac8f-f84cb111546c\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_wandb.ipynb\", \"title\": \"Load the embeddings\", \"tokens\": 506, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bdf4796b74a9494cda7741f4c640cbe2a485046ad219ebbf3a917050d222672f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2071, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "70027c3a-3e35-44f1-ac8f-f84cb111546c", "doc_id": "70027c3a-3e35-44f1-ac8f-f84cb111546c", "ref_doc_id": "70027c3a-3e35-44f1-ac8f-f84cb111546c", "chroma:document": "## Visualizing embeddings in W&B\n\nWe will upload the data to [Weights & Biases](http://wandb.ai) and use an [Embedding Projector](https://docs.wandb.ai/ref/app/features/panels/weave/embedding-projector) to visualize the embeddings using common dimension reduction algorithms like PCA, UMAP, and t-SNE. The dataset is created in the [Get_embeddings_from_dataset Notebook](Get_embeddings_from_dataset.ipynb).\n\n## What is Weights & Biases?\n\n[Weights & Biases](http://wandb.ai) is a machine learning platform used by OpenAI and other ML teams to build better models faster. They use it to quickly track experiments, evaluate model performance, reproduce models, visualize results, and share findings with colleagues.\n\n### 1. Log the data to W&B\n\nWe create a [W&B Table](https://docs.wandb.ai/guides/data-vis/log-tables) with the original data and the embeddings. Each review is a new row and the 1536 embedding floats are given their own column named `emb_{i}`.\n\n\n```python\nimport pandas as pd\nfrom sklearn.manifold import TSNE\nimport numpy as np\nfrom ast import literal_eval\n\n# Load the embeddings\ndatafile_path = \"data/fine_food_reviews_with_embeddings_1k.csv\"\ndf = pd.read_csv(datafile_path)\n\n# Convert to a list of lists of floats\nmatrix = np.array(df.embedding.apply(literal_eval).to_list())\n```\n\n\n```python\nimport wandb\n\noriginal_cols = df.columns[1:-1].tolist()\nembedding_cols = ['emb_'+str(idx) for idx in range(len(matrix[0]))]\ntable_cols = original_cols + embedding_cols\n\nwith wandb.init(project='openai_embeddings'):\n    table = wandb.Table(columns=table_cols)\n    for i, row in enumerate(df.to_dict(orient=\"records\")):\n        original_data = [row[col_name] for col_name in original_cols]\n        embedding_data = matrix[i].tolist()\n        table.add_data(*(original_data + embedding_data))\n    wandb.log({'openai_embedding_table': table})\n```\n\n### 2. Render as 2D Projection\n\nAfter navigating to the W&B run link, we click the \u2699\ufe0f icon in the top right of the Table and change \"Render As:\" to \"Combined 2D Projection\". \n\nExample: http://wandb.me/openai_embeddings"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: d6449033-40db-41a3-8527-37161a581eea
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb", "title": "How to automate tasks with functions (S3 bucket example)", "tokens": 2600, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"d6449033-40db-41a3-8527-37161a581eea\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb\", \"title\": \"How to automate tasks with functions (S3 bucket example)\", \"tokens\": 2600, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7f16d99f-71cb-4503-bbea-3212c302103b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb\", \"title\": \"How to automate tasks with functions (S3 bucket example)\", \"tokens\": 2600, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"5444006992071eb77abe8eb08005bf61dcc286246fb039d83f6d12c1faff4d89\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fa4c971e-a393-4dd9-a843-5d6f36dfbf21\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5c238dd0605c3618695e3c739f33801a19df5a659132e2b7841934c4807a5fdf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1733, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7f16d99f-71cb-4503-bbea-3212c302103b", "doc_id": "7f16d99f-71cb-4503-bbea-3212c302103b", "ref_doc_id": "7f16d99f-71cb-4503-bbea-3212c302103b", "chroma:document": "# How to automate tasks with functions (S3 bucket example)\n\nThis code demonstrates how to interact with ChatGPT functions to perform tasks related to Amazon S3 buckets. The notebook covers S3 bucket key functionalities such as running simple listing commands, searching for a specific file in all buckets, uploading a file to a bucket, and downloading a file from a bucket. The OpenAI Chat API understands the user instructions, generates the natural language responses, and extracts appropriate function calls based on the user's input.\n\n**Requirements**:\nTo run the notebook generate AWS access key with S3 bucket writing permission and store them in a local environment file alongside the Openai key. The \"`.env`\" file format:\n```\nAWS_ACCESS_KEY_ID=<your-key>\nAWS_SECRET_ACCESS_KEY=<your-key>\nOPENAI_API_KEY=<your-key>\n```\n\n\n```python\n! pip install openai\n! pip install boto3\n! pip install tenacity\n! pip install python-dotenv\n```\n\n\n```python\nfrom openai import OpenAI\nimport json\nimport boto3\nimport os\nimport datetime\nfrom urllib.request import urlretrieve\n\n# load environment variables\nfrom dotenv import load_dotenv\nload_dotenv() \n```\n\n\n\n\n    True\n\n\n\n## Initials\n\n\n```python\nOpenAI.api_key = os.environ.get(\"OPENAI_API_KEY\")\nGPT_MODEL = \"gpt-3.5-turbo\"\n```\n\n\n```python\n# Optional - if you had issues loading the environment file, you can set the AWS values using the below code\n# os.environ['AWS_ACCESS_KEY_ID'] = ''\n# os.environ['AWS_SECRET_ACCESS_KEY'] = ''\n\n# Create S3 client\ns3_client = boto3.client('s3')\n\n# Create openai client\nclient = OpenAI()\n```\n\n## Utilities\n\nTo connect user questions or commands to the appropriate function, we need to provide ChatGPT with the necessary function details and expected parameters."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: fa4c971e-a393-4dd9-a843-5d6f36dfbf21
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb", "title": "How to automate tasks with functions (S3 bucket example)", "tokens": 2600, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"fa4c971e-a393-4dd9-a843-5d6f36dfbf21\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb\", \"title\": \"How to automate tasks with functions (S3 bucket example)\", \"tokens\": 2600, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7f16d99f-71cb-4503-bbea-3212c302103b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb\", \"title\": \"How to automate tasks with functions (S3 bucket example)\", \"tokens\": 2600, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"5444006992071eb77abe8eb08005bf61dcc286246fb039d83f6d12c1faff4d89\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d6449033-40db-41a3-8527-37161a581eea\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb\", \"title\": \"How to automate tasks with functions (S3 bucket example)\", \"tokens\": 2600, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"c9ff3c944c278ab97cd630a9987573e00df4cd2a0ae673e51e787f64a2bc1485\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c24afdc7-c065-402f-94cc-9fcd57b7ed01\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"601087bcb95d65bb471ab51e817d00b8f2b784a8275e195e9023b61bb302d294\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 5303, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7f16d99f-71cb-4503-bbea-3212c302103b", "doc_id": "7f16d99f-71cb-4503-bbea-3212c302103b", "ref_doc_id": "7f16d99f-71cb-4503-bbea-3212c302103b", "chroma:document": "# How to automate tasks with functions (S3 bucket example)\n\nThis code demonstrates how to interact with ChatGPT functions to perform tasks related to Amazon S3 buckets. The notebook covers S3 bucket key functionalities such as running simple listing commands, searching for a specific file in all buckets, uploading a file to a bucket, and downloading a file from a bucket. The OpenAI Chat API understands the user instructions, generates the natural language responses, and extracts appropriate function calls based on the user's input.\n\n**Requirements**:\nTo run the notebook generate AWS access key with S3 bucket writing permission and store them in a local environment file alongside the Openai key. The \"`.env`\" file format:\n```\nAWS_ACCESS_KEY_ID=<your-key>\nAWS_SECRET_ACCESS_KEY=<your-key>\nOPENAI_API_KEY=<your-key>\n```\n\n\n```python\n! pip install openai\n! pip install boto3\n! pip install tenacity\n! pip install python-dotenv\n```\n\n\n```python\nfrom openai import OpenAI\nimport json\nimport boto3\nimport os\nimport datetime\nfrom urllib.request import urlretrieve\n\n# load environment variables\nfrom dotenv import load_dotenv\nload_dotenv() \n```\n\n\n\n\n    True\n\n\n\n## Initials\n\n\n```python\nOpenAI.api_key = os.environ.get(\"OPENAI_API_KEY\")\nGPT_MODEL = \"gpt-3.5-turbo\"\n```\n\n\n```python\n# Optional - if you had issues loading the environment file, you can set the AWS values using the below code\n# os.environ['AWS_ACCESS_KEY_ID'] = ''\n# os.environ['AWS_SECRET_ACCESS_KEY'] = ''\n\n# Create S3 client\ns3_client = boto3.client('s3')\n\n# Create openai client\nclient = OpenAI()\n```\n\n## Utilities\n\nTo connect user questions or commands to the appropriate function, we need to provide ChatGPT with the necessary function details and expected parameters.\n\n\n```python\n# Functions dict to pass S3 operations details for the GPT model\nfunctions = [\n    {   \n        \"type\": \"function\",\n        \"function\":{\n            \"name\": \"list_buckets\",\n            \"description\": \"List all available S3 buckets\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {}\n            }\n        }\n    },\n    {\n        \"type\": \"function\",\n        \"function\":{\n            \"name\": \"list_objects\",\n            \"description\": \"List the objects or files inside a given S3 bucket\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"bucket\": {\"type\": \"string\", \"description\": \"The name of the S3 bucket\"},\n                    \"prefix\": {\"type\": \"string\", \"description\": \"The folder path in the S3 bucket\"},\n                },\n                \"required\": [\"bucket\"],\n            },\n        }\n    },\n    {   \n        \"type\": \"function\",\n        \"function\":{\n            \"name\": \"download_file\",\n            \"description\": \"Download a specific file from an S3 bucket to a local distribution folder.\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"bucket\": {\"type\": \"string\", \"description\": \"The name of the S3 bucket\"},\n                    \"key\": {\"type\": \"string\", \"description\": \"The path to the file inside the bucket\"},\n                    \"directory\": {\"type\": \"string\", \"description\": \"The local destination directory to download the file, should be specificed by the user.\"},\n                },\n                \"required\": [\"bucket\", \"key\", \"directory\"],\n            }\n        }\n    },\n    {\n        \"type\": \"function\",\n        \"function\":{\n            \"name\": \"upload_file\",\n            \"description\": \"Upload a file to an S3 bucket\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"source\": {\"type\": \"string\", \"description\": \"The local source path or remote URL\"},\n                    \"bucket\": {\"type\": \"string\", \"description\": \"The name of the S3 bucket\"},\n                    \"key\": {\"type\": \"string\", \"description\": \"The path to the file inside the bucket\"},\n                    \"is_remote_url\": {\"type\": \"boolean\", \"description\": \"Is the provided source a URL (True) or local path (False)\"},\n                },\n                \"required\": [\"source\", \"bucket\", \"key\", \"is_remote_url\"],\n            }\n        }\n    },\n    {\n        \"type\": \"function\",\n        \"function\":{\n            \"name\": \"search_s3_objects\",\n            \"description\": \"Search for a specific file name inside an S3 bucket\",\n            \"parameters\": {\n                \"type\": \"object\",\n                \"properties\": {\n                    \"search_name\": {\"type\": \"string\", \"description\": \"The name of the file you want to search for\"},\n                    \"bucket\": {\"type\": \"string\", \"description\": \"The name of the S3 bucket\"},\n                    \"prefix\": {\"type\": \"string\", \"description\": \"The folder path in the S3 bucket\"},\n                    \"exact_match\": {\"type\": \"boolean\", \"description\": \"Set exact_match to True if the search should match the exact file name. Set exact_match to False to compare part of the file name string (the file contains)\"}\n                },\n                \"required\": [\"search_name\"],\n            },\n        }\n    }\n]\n```\n\nCreate helper functions to interact with the S3 service, such as listing buckets, listing objects, downloading and uploading files, and searching for specific files."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: c24afdc7-c065-402f-94cc-9fcd57b7ed01
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb", "title": "How to automate tasks with functions (S3 bucket example)", "tokens": 2600, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"c24afdc7-c065-402f-94cc-9fcd57b7ed01\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb\", \"title\": \"How to automate tasks with functions (S3 bucket example)\", \"tokens\": 2600, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7f16d99f-71cb-4503-bbea-3212c302103b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb\", \"title\": \"How to automate tasks with functions (S3 bucket example)\", \"tokens\": 2600, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"5444006992071eb77abe8eb08005bf61dcc286246fb039d83f6d12c1faff4d89\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fa4c971e-a393-4dd9-a843-5d6f36dfbf21\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb\", \"title\": \"How to automate tasks with functions (S3 bucket example)\", \"tokens\": 2600, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"55624379e2d30b347b77bd22bb9f1d79052fc87957def770b418261af238f629\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bd449e7d-405c-4b4b-b9c9-3fa58e5d7c3f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dc11f89f7730b179f1533250e1d70834a4a7ce12f54f9762df55c9ff31349b0d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5306, \"end_char_idx\": 8423, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7f16d99f-71cb-4503-bbea-3212c302103b", "doc_id": "7f16d99f-71cb-4503-bbea-3212c302103b", "ref_doc_id": "7f16d99f-71cb-4503-bbea-3212c302103b", "chroma:document": "```python\ndef datetime_converter(obj):\n    if isinstance(obj, datetime.datetime):\n        return obj.isoformat()\n    raise TypeError(f\"Object of type {obj.__class__.__name__} is not JSON serializable\")\n```\n\n\n```python\ndef list_buckets():\n    response = s3_client.list_buckets()\n    return json.dumps(response['Buckets'], default=datetime_converter)\n\ndef list_objects(bucket, prefix=''):\n    response = s3_client.list_objects_v2(Bucket=bucket, Prefix=prefix)\n    return json.dumps(response.get('Contents', []), default=datetime_converter)\n\ndef download_file(bucket, key, directory):\n    \n    filename = os.path.basename(key)\n    \n    # Resolve destination to the correct file path\n    destination = os.path.join(directory, filename)\n    \n    s3_client.download_file(bucket, key, destination)\n    return json.dumps({\"status\": \"success\", \"bucket\": bucket, \"key\": key, \"destination\": destination})\n\ndef upload_file(source, bucket, key, is_remote_url=False):\n    if is_remote_url:\n        file_name = os.path.basename(source)\n        urlretrieve(source, file_name)\n        source = file_name\n       \n    s3_client.upload_file(source, bucket, key)\n    return json.dumps({\"status\": \"success\", \"source\": source, \"bucket\": bucket, \"key\": key})\n\ndef search_s3_objects(search_name, bucket=None, prefix='', exact_match=True):\n    search_name = search_name.lower()\n    \n    if bucket is None:\n        buckets_response = json.loads(list_buckets())\n        buckets = [bucket_info[\"Name\"] for bucket_info in buckets_response]\n    else:\n        buckets = [bucket]\n\n    results = []\n\n    for bucket_name in buckets:\n        objects_response = json.loads(list_objects(bucket_name, prefix))\n        if exact_match:\n            bucket_results = [obj for obj in objects_response if search_name == obj['Key'].lower()]\n        else:\n            bucket_results = [obj for obj in objects_response if search_name in obj['Key'].lower()]\n\n        if bucket_results:\n            results.extend([{\"Bucket\": bucket_name, \"Object\": obj} for obj in bucket_results])\n\n    return json.dumps(results)\n```\n\nThe below dictionary connects the name with the function to use it for execution based on ChatGPT responses.\n\n\n```python\navailable_functions = {\n    \"list_buckets\": list_buckets,\n    \"list_objects\": list_objects,\n    \"download_file\": download_file,\n    \"upload_file\": upload_file,\n    \"search_s3_objects\": search_s3_objects\n}\n```\n\n## ChatGPT\n\n\n```python\ndef chat_completion_request(messages, functions=None, function_call='auto', \n                            model_name=GPT_MODEL):\n    \n    if functions is not None:\n        return client.chat.completions.create(\n            model=model_name,\n            messages=messages,\n            tools=functions,\n            tool_choice=function_call)\n    else:\n        return client.chat.completions.create(\n            model=model_name,\n            messages=messages)\n```\n\n### Conversation flow\n\nCreate a main function for the chatbot, which takes user input, sends it to the OpenAI Chat API, receives a response, executes any function calls generated by the API, and returns a final response to the user."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: bd449e7d-405c-4b4b-b9c9-3fa58e5d7c3f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb", "title": "How to automate tasks with functions (S3 bucket example)", "tokens": 2600, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"bd449e7d-405c-4b4b-b9c9-3fa58e5d7c3f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb\", \"title\": \"How to automate tasks with functions (S3 bucket example)\", \"tokens\": 2600, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7f16d99f-71cb-4503-bbea-3212c302103b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb\", \"title\": \"How to automate tasks with functions (S3 bucket example)\", \"tokens\": 2600, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"5444006992071eb77abe8eb08005bf61dcc286246fb039d83f6d12c1faff4d89\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c24afdc7-c065-402f-94cc-9fcd57b7ed01\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb\", \"title\": \"How to automate tasks with functions (S3 bucket example)\", \"tokens\": 2600, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f122a31285890ca3dc97967243bbed059dd2b6df294b50fb65faf237731388e7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"579464f9-0ad3-4e14-817c-dfa1e354df8c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a0e08ba5183d28fe846d2380010a8c01ccd9e5d2fa2e3c7295f3f7ff485a7090\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7486, \"end_char_idx\": 10861, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7f16d99f-71cb-4503-bbea-3212c302103b", "doc_id": "7f16d99f-71cb-4503-bbea-3212c302103b", "ref_doc_id": "7f16d99f-71cb-4503-bbea-3212c302103b", "chroma:document": "```python\navailable_functions = {\n    \"list_buckets\": list_buckets,\n    \"list_objects\": list_objects,\n    \"download_file\": download_file,\n    \"upload_file\": upload_file,\n    \"search_s3_objects\": search_s3_objects\n}\n```\n\n## ChatGPT\n\n\n```python\ndef chat_completion_request(messages, functions=None, function_call='auto', \n                            model_name=GPT_MODEL):\n    \n    if functions is not None:\n        return client.chat.completions.create(\n            model=model_name,\n            messages=messages,\n            tools=functions,\n            tool_choice=function_call)\n    else:\n        return client.chat.completions.create(\n            model=model_name,\n            messages=messages)\n```\n\n### Conversation flow\n\nCreate a main function for the chatbot, which takes user input, sends it to the OpenAI Chat API, receives a response, executes any function calls generated by the API, and returns a final response to the user.\n\n\n```python\ndef run_conversation(user_input, topic=\"S3 bucket functions.\", is_log=False):\n\n    system_message=f\"Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous. If the user ask question not related to {topic} response your scope is {topic} only.\"\n    \n    messages = [{\"role\": \"system\", \"content\": system_message},\n                {\"role\": \"user\", \"content\": user_input}]\n    \n    # Call the model to get a response\n    response = chat_completion_request(messages, functions=functions)\n    response_message = response.choices[0].message\n    \n    if is_log:\n        print(response.choices)\n    \n    # check if GPT wanted to call a function\n    if response_message.tool_calls:\n        function_name = response_message.tool_calls[0].function.name\n        function_args = json.loads(response_message.tool_calls[0].function.arguments)\n        \n        # Call the function\n        function_response = available_functions[function_name](**function_args)\n        \n        # Add the response to the conversation\n        messages.append(response_message)\n        messages.append({\n            \"role\": \"tool\",\n            \"content\": function_response,\n            \"tool_call_id\": response_message.tool_calls[0].id,\n        })\n        \n        # Call the model again to summarize the results\n        second_response = chat_completion_request(messages)\n        final_message = second_response.choices[0].message.content\n    else:\n        final_message = response_message.content\n\n    return final_message\n```\n\n### S3 bucket bot testing\nIn the following examples, make sure to replace the placeholders such as `<file_name>`, `<bucket_name>`, and `<directory_path>` with your specific values before execution.\n\n#### Listing and searching\n\nLet's start by listing all the available buckets.\n\n\n```python\nprint(run_conversation('list my S3 buckets'))\n```\n\nYou can ask the assistant to search for a specific file name either in all the buckets or in a specific one.\n\n\n```python\nsearch_file = '<file_name>'\nprint(run_conversation(f'search for a file {search_file} in all buckets'))\n```\n\n\n```python\nsearch_word = '<file_name_part>'\nbucket_name = '<bucket_name>'\nprint(run_conversation(f'search for a file contains {search_word} in {bucket_name}'))\n```\n\nThe model is expected to clarify the ask from the user in case of ambiguity in the parameters values as described in the system message."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 579464f9-0ad3-4e14-817c-dfa1e354df8c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb", "title": "How to automate tasks with functions (S3 bucket example)", "tokens": 2600, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"579464f9-0ad3-4e14-817c-dfa1e354df8c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb\", \"title\": \"How to automate tasks with functions (S3 bucket example)\", \"tokens\": 2600, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"7f16d99f-71cb-4503-bbea-3212c302103b\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb\", \"title\": \"How to automate tasks with functions (S3 bucket example)\", \"tokens\": 2600, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"5444006992071eb77abe8eb08005bf61dcc286246fb039d83f6d12c1faff4d89\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bd449e7d-405c-4b4b-b9c9-3fa58e5d7c3f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb\", \"title\": \"How to automate tasks with functions (S3 bucket example)\", \"tokens\": 2600, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"afa045070176b10c4f0fae0a986653a9feac7eed7ac4b0ffe7982d8449e89eb7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10270, \"end_char_idx\": 12140, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "7f16d99f-71cb-4503-bbea-3212c302103b", "doc_id": "7f16d99f-71cb-4503-bbea-3212c302103b", "ref_doc_id": "7f16d99f-71cb-4503-bbea-3212c302103b", "chroma:document": "```python\nprint(run_conversation('list my S3 buckets'))\n```\n\nYou can ask the assistant to search for a specific file name either in all the buckets or in a specific one.\n\n\n```python\nsearch_file = '<file_name>'\nprint(run_conversation(f'search for a file {search_file} in all buckets'))\n```\n\n\n```python\nsearch_word = '<file_name_part>'\nbucket_name = '<bucket_name>'\nprint(run_conversation(f'search for a file contains {search_word} in {bucket_name}'))\n```\n\nThe model is expected to clarify the ask from the user in case of ambiguity in the parameters values as described in the system message.\n\n\n```python\nprint(run_conversation('search for a file'))\n```\n\n    Sure, to help me find what you're looking for, could you please provide the name of the file you want to search for and the name of the S3 bucket? Also, should the search match the file name exactly, or should it also consider partial matches?\n\n\n#### Validate edge cases\n\nWe also instructed the model to reject irrelevant tasks. Let's test it out and see how it works in action.\n\n\n```python\n# the model should not answer details not related to the scope\nprint(run_conversation('what is the weather today'))\n```\n\n    Apologies for the misunderstanding, but I am only able to assist with S3 bucket functions. Can you please ask a question related to S3 bucket functions?\n\n\nThe provided functions are not limited to just retrieving information. They can also assist the user in uploading or downloading files.\n\n#### Download a file\n\n\n```python\nsearch_file = '<file_name>'\nbucket_name = '<bucket_name>'\nlocal_directory = '<directory_path>'\nprint(run_conversation(f'download {search_file} from {bucket_name} bucket to {local_directory} directory'))\n```\n\n#### Upload a file\n\n\n```python\nlocal_file = '<file_name>'\nbucket_name = '<bucket_name>'\nprint(run_conversation(f'upload {local_file} to {bucket_name} bucket'))\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 613470d5-6189-4cda-8c2c-fc14a61bc50a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"613470d5-6189-4cda-8c2c-fc14a61bc50a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb\", \"title\": \"## Visualizing the embeddings in Kangas\", \"tokens\": 2958, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"87d60355-b825-4e17-b4db-7b1439dea298\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb\", \"title\": \"## Visualizing the embeddings in Kangas\", \"tokens\": 2958, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"02ba365e59cdf44bcbdd8ebcc96ab60833ea2355a08e03daa6ec12cd86063a58\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cfe99ca7-fbb1-4c64-94aa-806f31fa0b59\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"99155a013f90dc7ac8d4103ce716eee1247240b046cb70f460ba84886c0e25ff\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2463, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "87d60355-b825-4e17-b4db-7b1439dea298", "doc_id": "87d60355-b825-4e17-b4db-7b1439dea298", "ref_doc_id": "87d60355-b825-4e17-b4db-7b1439dea298", "chroma:document": "## Visualizing the embeddings in Kangas\n\nIn this Jupyter Notebook, we construct a Kangas DataGrid containing the data and projections of the embeddings into 2 dimensions.\n\n## What is Kangas?\n\n[Kangas](https://github.com/comet-ml/kangas/) as an open source, mixed-media, dataframe-like tool for data scientists. It was developed by [Comet](https://comet.com/), a company designed to help reduce the friction of moving models into production. \n\n### 1. Setup\n\nTo get started, we pip install kangas, and import it.\n\n\n```python\n%pip install kangas --quiet\n```\n\n\n```python\nimport kangas as kg\n```\n\n### 2. Constructing a Kangas DataGrid\n\nWe create a Kangas Datagrid with the original data and the embeddings. The data is composed of a rows of reviews, and the embeddings are composed of 1536 floating-point values. In this example, we get the data directly from github, in case you aren't running this notebook inside OpenAI's repo.\n\nWe use Kangas to read the CSV file into a DataGrid for further processing.\n\n\n```python\ndata = kg.read_csv(\"https://raw.githubusercontent.com/openai/openai-cookbook/main/examples/data/fine_food_reviews_with_embeddings_1k.csv\")\n```\n\n    Loading CSV file 'fine_food_reviews_with_embeddings_1k.csv'...\n\n\n    1001it [00:00, 2412.90it/s]\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 1000/1000 [00:00<00:00, 2899.16it/s]\n\n\nWe can review the fields of the CSV file:\n\n\n```python\ndata.info()\n```\n\n    DataGrid (in memory)\n        Name   : fine_food_reviews_with_embeddings_1k\n        Rows   : 1,000\n        Columns: 9\n    #   Column                Non-Null Count DataGrid Type       \n    --- -------------------- --------------- --------------------\n    1   Column 1                       1,000 INTEGER             \n    2   ProductId                      1,000 TEXT                \n    3   UserId                         1,000 TEXT                \n    4   Score                          1,000 INTEGER             \n    5   Summary                        1,000 TEXT                \n    6   Text                           1,000 TEXT                \n    7   combined                       1,000 TEXT                \n    8   n_tokens                       1,000 INTEGER             \n    9   embedding                      1,000 TEXT                \n\n\nAnd get a glimpse of the first and last rows:\n\n\n```python\ndata\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: cfe99ca7-fbb1-4c64-94aa-806f31fa0b59
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"cfe99ca7-fbb1-4c64-94aa-806f31fa0b59\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb\", \"title\": \"## Visualizing the embeddings in Kangas\", \"tokens\": 2958, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"87d60355-b825-4e17-b4db-7b1439dea298\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb\", \"title\": \"## Visualizing the embeddings in Kangas\", \"tokens\": 2958, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"02ba365e59cdf44bcbdd8ebcc96ab60833ea2355a08e03daa6ec12cd86063a58\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"613470d5-6189-4cda-8c2c-fc14a61bc50a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb\", \"title\": \"## Visualizing the embeddings in Kangas\", \"tokens\": 2958, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"950d8208ccfb75124b813df05cfcdfd51629d83f10ca888921d44806e3936a93\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7e367e11-cb58-4afe-a4c2-f109dab6273a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"6c56da3694b97000668891868af06ad4ef2cd5b6fc8871ebe728ccfe9bf05576\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1004, \"end_char_idx\": 3595, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "87d60355-b825-4e17-b4db-7b1439dea298", "doc_id": "87d60355-b825-4e17-b4db-7b1439dea298", "ref_doc_id": "87d60355-b825-4e17-b4db-7b1439dea298", "chroma:document": "```python\ndata = kg.read_csv(\"https://raw.githubusercontent.com/openai/openai-cookbook/main/examples/data/fine_food_reviews_with_embeddings_1k.csv\")\n```\n\n    Loading CSV file 'fine_food_reviews_with_embeddings_1k.csv'...\n\n\n    1001it [00:00, 2412.90it/s]\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 1000/1000 [00:00<00:00, 2899.16it/s]\n\n\nWe can review the fields of the CSV file:\n\n\n```python\ndata.info()\n```\n\n    DataGrid (in memory)\n        Name   : fine_food_reviews_with_embeddings_1k\n        Rows   : 1,000\n        Columns: 9\n    #   Column                Non-Null Count DataGrid Type       \n    --- -------------------- --------------- --------------------\n    1   Column 1                       1,000 INTEGER             \n    2   ProductId                      1,000 TEXT                \n    3   UserId                         1,000 TEXT                \n    4   Score                          1,000 INTEGER             \n    5   Summary                        1,000 TEXT                \n    6   Text                           1,000 TEXT                \n    7   combined                       1,000 TEXT                \n    8   n_tokens                       1,000 INTEGER             \n    9   embedding                      1,000 TEXT                \n\n\nAnd get a glimpse of the first and last rows:\n\n\n```python\ndata\n```\n\n\n\n\n<table><th colspan='1' >          row-id </th> <th colspan='1' >        Column 1 </th> <th colspan='1' >       ProductId </th> <th colspan='1' >          UserId </th> <th colspan='1' >           Score </th> <th colspan='1' >         Summary </th> <th colspan='1' >            Text </th> <th colspan='1' >        combined </th> <th colspan='1' >        n_tokens </th> <th colspan='1' >       embedding </th> <tr>\n<td colspan='1' >               1 </td> <td colspan='1' >               0 </td> <td colspan='1' >      B003XPF9BO </td> <td colspan='1' >  A3R7JR3FMEBXQB </td> <td colspan='1' >               5 </td> <td colspan='1' > where does one  </td> <td colspan='1' > Wanted to save  </td> <td colspan='1' > Title: where do </td> <td colspan='1' >              52 </td> <td colspan='1' > [0.007018072064 </td> <tr>\n<td colspan='1' >               2 </td> <td colspan='1' >             297 </td> <td colspan='1' >      B003VXHGPK </td> <td colspan='1' >  A21VWSCGW7UUAR </td> <td colspan='1' >               4 </td> <td colspan='1' > Good, but not W </td> <td colspan='1' > Honestly, I hav </td> <td colspan='1' > Title: Good,"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 7e367e11-cb58-4afe-a4c2-f109dab6273a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"7e367e11-cb58-4afe-a4c2-f109dab6273a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb\", \"title\": \"## Visualizing the embeddings in Kangas\", \"tokens\": 2958, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"87d60355-b825-4e17-b4db-7b1439dea298\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb\", \"title\": \"## Visualizing the embeddings in Kangas\", \"tokens\": 2958, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"02ba365e59cdf44bcbdd8ebcc96ab60833ea2355a08e03daa6ec12cd86063a58\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cfe99ca7-fbb1-4c64-94aa-806f31fa0b59\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb\", \"title\": \"## Visualizing the embeddings in Kangas\", \"tokens\": 2958, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"58c163868072ad654d67baa4b1ffc26c60ff2463174edc6401a4975db4858724\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d66c9e00-aab5-4a4c-b0c5-1f51aafd0ee2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"60bdff6a84fa0ce52bc6466d535d76a9881b8113a5d9f83f62075ba131967a28\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2468, \"end_char_idx\": 4477, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "87d60355-b825-4e17-b4db-7b1439dea298", "doc_id": "87d60355-b825-4e17-b4db-7b1439dea298", "ref_doc_id": "87d60355-b825-4e17-b4db-7b1439dea298", "chroma:document": "<table><th colspan='1' >          row-id </th> <th colspan='1' >        Column 1 </th> <th colspan='1' >       ProductId </th> <th colspan='1' >          UserId </th> <th colspan='1' >           Score </th> <th colspan='1' >         Summary </th> <th colspan='1' >            Text </th> <th colspan='1' >        combined </th> <th colspan='1' >        n_tokens </th> <th colspan='1' >       embedding </th> <tr>\n<td colspan='1' >               1 </td> <td colspan='1' >               0 </td> <td colspan='1' >      B003XPF9BO </td> <td colspan='1' >  A3R7JR3FMEBXQB </td> <td colspan='1' >               5 </td> <td colspan='1' > where does one  </td> <td colspan='1' > Wanted to save  </td> <td colspan='1' > Title: where do </td> <td colspan='1' >              52 </td> <td colspan='1' > [0.007018072064 </td> <tr>\n<td colspan='1' >               2 </td> <td colspan='1' >             297 </td> <td colspan='1' >      B003VXHGPK </td> <td colspan='1' >  A21VWSCGW7UUAR </td> <td colspan='1' >               4 </td> <td colspan='1' > Good, but not W </td> <td colspan='1' > Honestly, I hav </td> <td colspan='1' > Title: Good, bu </td> <td colspan='1' >             178 </td> <td colspan='1' > [-0.00314055196 </td> <tr>\n<td colspan='1' >               3 </td> <td colspan='1' >             296 </td> <td colspan='1' >      B008JKTTUA </td> <td colspan='1' >  A34XBAIFT02B60 </td> <td colspan='1' >               1 </td> <td colspan='1' > Should advertis </td> <td colspan='1' > First, these sh </td> <td colspan='1' > Title: Should a </td> <td colspan='1' >              78 </td> <td colspan='1' > [-0.01757248118 </td> <tr>\n<td colspan='1' >               4 </td> <td colspan='1' >             295 </td> <td colspan='1' >      B000LKTTTW </td> <td colspan='1' >  A14MQ40CCU8B13 </td> <td colspan='1' >               5 </td> <td colspan='1' > Best tomato sou </td> <td colspan='1' > I have a hard t </td> <td colspan='1' > Title: Best tom </td> <td colspan='1' >             111 </td> <td colspan='1' > [-0."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: d66c9e00-aab5-4a4c-b0c5-1f51aafd0ee2
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"d66c9e00-aab5-4a4c-b0c5-1f51aafd0ee2\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb\", \"title\": \"## Visualizing the embeddings in Kangas\", \"tokens\": 2958, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"87d60355-b825-4e17-b4db-7b1439dea298\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb\", \"title\": \"## Visualizing the embeddings in Kangas\", \"tokens\": 2958, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"02ba365e59cdf44bcbdd8ebcc96ab60833ea2355a08e03daa6ec12cd86063a58\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7e367e11-cb58-4afe-a4c2-f109dab6273a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb\", \"title\": \"## Visualizing the embeddings in Kangas\", \"tokens\": 2958, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"cfbbf3ea5c2000f2fe51d02022ff1949035ddb616ea7425d95206c060bd19281\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e2da7b3c-fbec-4573-8215-044b5605b71e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"969c92fd9b6fae0ac217144b2ec99033930de793a03be0b417be601d85f7c386\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "87d60355-b825-4e17-b4db-7b1439dea298", "doc_id": "87d60355-b825-4e17-b4db-7b1439dea298", "ref_doc_id": "87d60355-b825-4e17-b4db-7b1439dea298", "chroma:document": "but not W </td> <td colspan='1' > Honestly, I hav </td> <td colspan='1' > Title: Good, bu </td> <td colspan='1' >             178 </td> <td colspan='1' > [-0.00314055196 </td> <tr>\n<td colspan='1' >               3 </td> <td colspan='1' >             296 </td> <td colspan='1' >      B008JKTTUA </td> <td colspan='1' >  A34XBAIFT02B60 </td> <td colspan='1' >               1 </td> <td colspan='1' > Should advertis </td> <td colspan='1' > First, these sh </td> <td colspan='1' > Title: Should a </td> <td colspan='1' >              78 </td> <td colspan='1' > [-0.01757248118 </td> <tr>\n<td colspan='1' >               4 </td> <td colspan='1' >             295 </td> <td colspan='1' >      B000LKTTTW </td> <td colspan='1' >  A14MQ40CCU8B13 </td> <td colspan='1' >               5 </td> <td colspan='1' > Best tomato sou </td> <td colspan='1' > I have a hard t </td> <td colspan='1' > Title: Best tom </td> <td colspan='1' >             111 </td> <td colspan='1' > [-0.00139322795 </td> <tr>\n<td colspan='1' >               5 </td> <td colspan='1' >             294 </td> <td colspan='1' >      B001D09KAM </td> <td colspan='1' >  A34XBAIFT02B60 </td> <td colspan='1' >               1 </td> <td colspan='1' > Should advertis </td> <td colspan='1' > First, these sh </td> <td colspan='1' > Title: Should a </td> <td colspan='1' >              78 </td> <td colspan='1' > [-0.01757248118 </td> <tr>\n<tr><td colspan='10' style='text-align: left;'>.</td></tr><td colspan='1' >             996 </td> <td colspan='1' >             623 </td> <td colspan='1' >      B0000CFXYA </td> <td colspan='1' >  A3GS4GWPIBV0NT </td> <td colspan='1' >               1 </td> <td colspan='1' > Strange inflamm </td> <td colspan='1' > Truthfully wasn </td> <td colspan='1' > Title: Strange  </td> <td colspan='1' >             110 </td> <td colspan='1' > [0."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: e2da7b3c-fbec-4573-8215-044b5605b71e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"e2da7b3c-fbec-4573-8215-044b5605b71e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb\", \"title\": \"## Visualizing the embeddings in Kangas\", \"tokens\": 2958, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"87d60355-b825-4e17-b4db-7b1439dea298\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb\", \"title\": \"## Visualizing the embeddings in Kangas\", \"tokens\": 2958, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"02ba365e59cdf44bcbdd8ebcc96ab60833ea2355a08e03daa6ec12cd86063a58\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d66c9e00-aab5-4a4c-b0c5-1f51aafd0ee2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb\", \"title\": \"## Visualizing the embeddings in Kangas\", \"tokens\": 2958, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7648b55ba193ccb318070eb666fc8a3e14d46c35dc7ef9dff54fe184e05b60a2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"778a2af5-383f-47a4-a4bb-ff56959ba181\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2c3908d5c2f433ac619ad62af16886513f591776e8aa55e2df8f69a05ee059ba\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": null, \"end_char_idx\": null, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "87d60355-b825-4e17-b4db-7b1439dea298", "doc_id": "87d60355-b825-4e17-b4db-7b1439dea298", "ref_doc_id": "87d60355-b825-4e17-b4db-7b1439dea298", "chroma:document": "00139322795 </td> <tr>\n<td colspan='1' >               5 </td> <td colspan='1' >             294 </td> <td colspan='1' >      B001D09KAM </td> <td colspan='1' >  A34XBAIFT02B60 </td> <td colspan='1' >               1 </td> <td colspan='1' > Should advertis </td> <td colspan='1' > First, these sh </td> <td colspan='1' > Title: Should a </td> <td colspan='1' >              78 </td> <td colspan='1' > [-0.01757248118 </td> <tr>\n<tr><td colspan='10' style='text-align: left;'>.</td></tr><td colspan='1' >             996 </td> <td colspan='1' >             623 </td> <td colspan='1' >      B0000CFXYA </td> <td colspan='1' >  A3GS4GWPIBV0NT </td> <td colspan='1' >               1 </td> <td colspan='1' > Strange inflamm </td> <td colspan='1' > Truthfully wasn </td> <td colspan='1' > Title: Strange  </td> <td colspan='1' >             110 </td> <td colspan='1' > [0.000110913533 </td> <tr>\n<td colspan='1' >             997 </td> <td colspan='1' >             624 </td> <td colspan='1' >      B0001BH5YM </td> <td colspan='1' >   A1BZ3HMAKK0NC </td> <td colspan='1' >               5 </td> <td colspan='1' > My favorite and </td> <td colspan='1' > You've just got </td> <td colspan='1' > Title: My favor </td> <td colspan='1' >              80 </td> <td colspan='1' > [-0.02086931467 </td> <tr>\n<td colspan='1' >             998 </td> <td colspan='1' >             625 </td> <td colspan='1' >      B0009ET7TC </td> <td colspan='1' >  A2FSDQY5AI6TNX </td> <td colspan='1' >               5 </td> <td colspan='1' > My furbabies LO </td> <td colspan='1' > Shake the conta </td> <td colspan='1' > Title: My furba </td> <td colspan='1' >              47 </td> <td colspan='1' > [-0.00974910240 </td> <tr>\n<td colspan='1' >             999 </td> <td colspan='1' >             619 </td> <td colspan='1' >      B007PA32L2 </td> <td colspan='1' >  A15FF2P7RPKH6G </td> <td colspan='1' >               5 </td> <td colspan='1' > got this for th </td> <td colspan='1' > all i have hear </td> <td colspan='1' > Title: got this </td> <td colspan='1' >              50 </td> <td colspan='1' > [-0."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 778a2af5-383f-47a4-a4bb-ff56959ba181
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"778a2af5-383f-47a4-a4bb-ff56959ba181\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb\", \"title\": \"## Visualizing the embeddings in Kangas\", \"tokens\": 2958, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"87d60355-b825-4e17-b4db-7b1439dea298\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb\", \"title\": \"## Visualizing the embeddings in Kangas\", \"tokens\": 2958, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"02ba365e59cdf44bcbdd8ebcc96ab60833ea2355a08e03daa6ec12cd86063a58\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e2da7b3c-fbec-4573-8215-044b5605b71e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb\", \"title\": \"## Visualizing the embeddings in Kangas\", \"tokens\": 2958, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"6b7d0266c73a8795c0a97e266cea694de2a7978b30d160babe5be256bbaac98c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2f3b5086-e71d-482a-b50b-dbc1f685984f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b4315d16f37dcefbee10397aa1bfc1df2ed127ec7827285bbfc01c6dbcb42153\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5752, \"end_char_idx\": 7916, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "87d60355-b825-4e17-b4db-7b1439dea298", "doc_id": "87d60355-b825-4e17-b4db-7b1439dea298", "ref_doc_id": "87d60355-b825-4e17-b4db-7b1439dea298", "chroma:document": "02086931467 </td> <tr>\n<td colspan='1' >             998 </td> <td colspan='1' >             625 </td> <td colspan='1' >      B0009ET7TC </td> <td colspan='1' >  A2FSDQY5AI6TNX </td> <td colspan='1' >               5 </td> <td colspan='1' > My furbabies LO </td> <td colspan='1' > Shake the conta </td> <td colspan='1' > Title: My furba </td> <td colspan='1' >              47 </td> <td colspan='1' > [-0.00974910240 </td> <tr>\n<td colspan='1' >             999 </td> <td colspan='1' >             619 </td> <td colspan='1' >      B007PA32L2 </td> <td colspan='1' >  A15FF2P7RPKH6G </td> <td colspan='1' >               5 </td> <td colspan='1' > got this for th </td> <td colspan='1' > all i have hear </td> <td colspan='1' > Title: got this </td> <td colspan='1' >              50 </td> <td colspan='1' > [-0.00521062919 </td> <tr>\n<td colspan='1' >            1000 </td> <td colspan='1' >             999 </td> <td colspan='1' >      B001EQ5GEO </td> <td colspan='1' >  A3VYU0VO6DYV6I </td> <td colspan='1' >               5 </td> <td colspan='1' > I love Maui Cof </td> <td colspan='1' > My first experi </td> <td colspan='1' > Title: I love M </td> <td colspan='1' >             118 </td> <td colspan='1' > [-0.00605782261 </td> <tr>\n<tr>\n<td colspan='10' style=\"text-align: left;\"> [1000 rows x 9 columns] </td> <tr>\n<tr><td colspan='10' style='text-align: left;'></td></tr><tr><td colspan='10' style='text-align: left;'>*  Use DataGrid.save() to save to disk</td></tr><tr><td colspan='10' style='text-align: left;'>** Use DataGrid.show() to start user interface</td></tr></table>\n\n\n\nNow, we create a new DataGrid, converting the numbers into an Embedding:\n\n\n```python\nimport ast # to convert string of a list of numbers into a list of numbers\n\ndg = kg.DataGrid(\n    name=\"openai_embeddings\",\n    columns=data.get_columns(),\n    converters={\"Score\": str},\n)\nfor row in data:\n    embedding = ast.literal_eval(row[8])\n    row[8] = kg.Embedding(\n        embedding, \n        name=str(row[3]), \n        text=\"%s - %.10s\" % (row[3], row[4]),\n        projection=\"umap\",\n    )\n    dg.append(row)\n```\n\nThe new DataGrid now has an Embedding column with proper datatype."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 2f3b5086-e71d-482a-b50b-dbc1f685984f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"2f3b5086-e71d-482a-b50b-dbc1f685984f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb\", \"title\": \"## Visualizing the embeddings in Kangas\", \"tokens\": 2958, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"87d60355-b825-4e17-b4db-7b1439dea298\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb\", \"title\": \"## Visualizing the embeddings in Kangas\", \"tokens\": 2958, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"02ba365e59cdf44bcbdd8ebcc96ab60833ea2355a08e03daa6ec12cd86063a58\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"778a2af5-383f-47a4-a4bb-ff56959ba181\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb\", \"title\": \"## Visualizing the embeddings in Kangas\", \"tokens\": 2958, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"4b36c004048ec176a9dfa44a1f0450ed1dffd02bca988869f9b2007c12ea6b1e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6967, \"end_char_idx\": 9839, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "87d60355-b825-4e17-b4db-7b1439dea298", "doc_id": "87d60355-b825-4e17-b4db-7b1439dea298", "ref_doc_id": "87d60355-b825-4e17-b4db-7b1439dea298", "chroma:document": "00605782261 </td> <tr>\n<tr>\n<td colspan='10' style=\"text-align: left;\"> [1000 rows x 9 columns] </td> <tr>\n<tr><td colspan='10' style='text-align: left;'></td></tr><tr><td colspan='10' style='text-align: left;'>*  Use DataGrid.save() to save to disk</td></tr><tr><td colspan='10' style='text-align: left;'>** Use DataGrid.show() to start user interface</td></tr></table>\n\n\n\nNow, we create a new DataGrid, converting the numbers into an Embedding:\n\n\n```python\nimport ast # to convert string of a list of numbers into a list of numbers\n\ndg = kg.DataGrid(\n    name=\"openai_embeddings\",\n    columns=data.get_columns(),\n    converters={\"Score\": str},\n)\nfor row in data:\n    embedding = ast.literal_eval(row[8])\n    row[8] = kg.Embedding(\n        embedding, \n        name=str(row[3]), \n        text=\"%s - %.10s\" % (row[3], row[4]),\n        projection=\"umap\",\n    )\n    dg.append(row)\n```\n\nThe new DataGrid now has an Embedding column with proper datatype.\n\n\n```python\ndg.info()\n```\n\n    DataGrid (in memory)\n        Name   : openai_embeddings\n        Rows   : 1,000\n        Columns: 9\n    #   Column                Non-Null Count DataGrid Type       \n    --- -------------------- --------------- --------------------\n    1   Column 1                       1,000 INTEGER             \n    2   ProductId                      1,000 TEXT                \n    3   UserId                         1,000 TEXT                \n    4   Score                          1,000 TEXT                \n    5   Summary                        1,000 TEXT                \n    6   Text                           1,000 TEXT                \n    7   combined                       1,000 TEXT                \n    8   n_tokens                       1,000 INTEGER             \n    9   embedding                      1,000 EMBEDDING-ASSET     \n\n\nWe simply save the datagrid, and we're done.\n\n\n```python\ndg.save()\n```\n\n### 3. Render 2D Projections\n\nTo render the data directly in the notebook, simply show it. Note that each row contains an embedding projection. \n\nScroll to far right to see embeddings projection per row.\n\nThe color of the point in projection space represents the Score.\n\n\n```python\ndg.show()\n```\n\n\n\n<iframe\n    width=\"100%\"\n    height=\"750px\"\n    src=\"http://127.0.1.1:4000/?datagrid=openai_embeddings.datagrid&timestamp=1685559502.7515423\"\n    frameborder=\"0\"\n    allowfullscreen\n\n></iframe>\n\n\n\nGroup by \"Score\" to see rows of each group.\n\n\n```python\ndg.show(group=\"Score\", sort=\"Score\", rows=5, select=\"Score,embedding\")\n```\n\n\n\n<iframe\n    width=\"100%\"\n    height=\"750px\"\n    src=\"http://127.0.1.1:4000/?datagrid=openai_embeddings.datagrid&timestamp=1685559502.7515423&group=Score&sort=Score&rows=5&select=Score%2Cembedding\"\n    frameborder=\"0\"\n    allowfullscreen\n\n></iframe>\n\n\n\nAn example of this datagrid is hosted here: https://kangas.comet.com/?datagrid=/data/openai_embeddings.datagrid"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 39a1194a-b765-4d6e-91ea-d54bd7addde4
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_with_Atlas.ipynb", "title": "Load the embeddings", "tokens": 776, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"39a1194a-b765-4d6e-91ea-d54bd7addde4\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_with_Atlas.ipynb\", \"title\": \"Load the embeddings\", \"tokens\": 776, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"05982c2b-8221-4bab-b60e-e90c78a4d92d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_with_Atlas.ipynb\", \"title\": \"Load the embeddings\", \"tokens\": 776, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b6504bb6bfc4d983590b719c62f36a7daea71097cf60099a94b56ac73884627e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e777becd-e03d-4c89-b88a-fde118b3e57c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"58041455514a08016ed1c06ea7a8cc66a5f924abf18ed23fec08050c5b4f5ae4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 1294, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "05982c2b-8221-4bab-b60e-e90c78a4d92d", "doc_id": "05982c2b-8221-4bab-b60e-e90c78a4d92d", "ref_doc_id": "05982c2b-8221-4bab-b60e-e90c78a4d92d", "chroma:document": "## Visualizing Open AI Embeddings in Atlas\n\nIn this example, we will upload food review embeddings to [Atlas](https://atlas.nomic.ai) to visualize the embeddings.\n\n## What is Atlas?\n\n[Atlas](https://atlas.nomic.ai) is a machine learning tool used to visualize massive datasets of embeddings in your web browser. Upload millions of embeddings to Atlas and interact with them in your web browser or jupyter notebook.\n\n### 1. Login to Atlas.\n\n\n\n```python\n!pip install nomic\n```\n\n    \n\n\n```python\nimport pandas as pd\nimport numpy as np\nfrom ast import literal_eval\n\n# Load the embeddings\ndatafile_path = \"data/fine_food_reviews_with_embeddings_1k.csv\"\ndf = pd.read_csv(datafile_path)\n\n# Convert to a list of lists of floats\nembeddings = np.array(df.embedding.apply(literal_eval).to_list())\ndf = df.drop('embedding', axis=1)\ndf = df.rename(columns={'Unnamed: 0': 'id'})\n\n```\n\n\n```python\nimport nomic\nfrom nomic import atlas\nnomic.login('7xDPkYXSYDc1_ErdTPIcoAR9RNd8YDlkS3nVNXcVoIMZ6') #demo account\n\ndata = df.to_dict('records')\nproject = atlas.map_embeddings(embeddings=embeddings, data=data,\n                               id_field='id',\n                               colorable_fields=['Score'])\nmap = project.maps[0]\n```\n\n    \n\n### 2. Interact with your embeddings in Jupyter\n\n\n```python\nmap\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: e777becd-e03d-4c89-b88a-fde118b3e57c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_with_Atlas.ipynb", "title": "Load the embeddings", "tokens": 776, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"e777becd-e03d-4c89-b88a-fde118b3e57c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_with_Atlas.ipynb\", \"title\": \"Load the embeddings\", \"tokens\": 776, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"05982c2b-8221-4bab-b60e-e90c78a4d92d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_with_Atlas.ipynb\", \"title\": \"Load the embeddings\", \"tokens\": 776, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b6504bb6bfc4d983590b719c62f36a7daea71097cf60099a94b56ac73884627e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"39a1194a-b765-4d6e-91ea-d54bd7addde4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_with_Atlas.ipynb\", \"title\": \"Load the embeddings\", \"tokens\": 776, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"771408deab2dd9a7ce8c0f9feaa70e1ea68c74a3af990fbd61b150ac1c349c3f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3fdb9b3b-b5e3-443b-9154-517e44e7670f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5b0d7e8666a5b807e3bbea3400b6775cad72d42af45bacc0f92b52f9d959d8db\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2557, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "05982c2b-8221-4bab-b60e-e90c78a4d92d", "doc_id": "05982c2b-8221-4bab-b60e-e90c78a4d92d", "ref_doc_id": "05982c2b-8221-4bab-b60e-e90c78a4d92d", "chroma:document": "## Visualizing Open AI Embeddings in Atlas\n\nIn this example, we will upload food review embeddings to [Atlas](https://atlas.nomic.ai) to visualize the embeddings.\n\n## What is Atlas?\n\n[Atlas](https://atlas.nomic.ai) is a machine learning tool used to visualize massive datasets of embeddings in your web browser. Upload millions of embeddings to Atlas and interact with them in your web browser or jupyter notebook.\n\n### 1. Login to Atlas.\n\n\n\n```python\n!pip install nomic\n```\n\n    \n\n\n```python\nimport pandas as pd\nimport numpy as np\nfrom ast import literal_eval\n\n# Load the embeddings\ndatafile_path = \"data/fine_food_reviews_with_embeddings_1k.csv\"\ndf = pd.read_csv(datafile_path)\n\n# Convert to a list of lists of floats\nembeddings = np.array(df.embedding.apply(literal_eval).to_list())\ndf = df.drop('embedding', axis=1)\ndf = df.rename(columns={'Unnamed: 0': 'id'})\n\n```\n\n\n```python\nimport nomic\nfrom nomic import atlas\nnomic.login('7xDPkYXSYDc1_ErdTPIcoAR9RNd8YDlkS3nVNXcVoIMZ6') #demo account\n\ndata = df.to_dict('records')\nproject = atlas.map_embeddings(embeddings=embeddings, data=data,\n                               id_field='id',\n                               colorable_fields=['Score'])\nmap = project.maps[0]\n```\n\n    \n\n### 2. Interact with your embeddings in Jupyter\n\n\n```python\nmap\n```\n\n\n\n\n\n    <h3>Project: meek-laborer</h3>\n    <script>\n    destroy = function() {\n        document.getElementById(\"iframe463f4614-7689-47e4-b55b-1da0cc679559\").remove()\n    }\n</script>\n\n<h4>Projection ID: 463f4614-7689-47e4-b55b-1da0cc679559</h4>\n<div class=\"actions\">\n    <div id=\"hide\" class=\"action\" onclick=\"destroy()\">Hide embedded project</div>\n    <div class=\"action\" id=\"out\">\n        <a href=\"https://atlas.nomic.ai/map/fddc0e07-97c5-477c-827c-96bca44519aa/463f4614-7689-47e4-b55b-1da0cc679559\" target=\"_blank\">Explore on atlas.nomic.ai</a>\n    </div>\n</div>\n\n<iframe class=\"iframe\" id=\"iframe463f4614-7689-47e4-b55b-1da0cc679559\" allow=\"clipboard-read; clipboard-write\" src=\"https://atlas.nomic.ai/map/fddc0e07-97c5-477c-827c-96bca44519aa/463f4614-7689-47e4-b55b-1da0cc679559\">\n</iframe>\n\n<style>\n    .iframe {\n        /* vh can be **very** large in vscode ipynb. */\n        height: min(75vh, 66vw);\n        width: 100%;\n    }\n</style>\n\n<style>\n    .actions {\n      display: block;\n    }\n    .action {\n      min-height: 18px;\n      margin: 5px;\n      transition: all 500ms ease-in-out;\n    }\n    .action:hover {\n      cursor: pointer;\n    }\n    #hide:hover::after {\n        content: \" X\";\n    }\n    #out:hover::after {\n        content: \"\";\n    }\n</style>"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 3fdb9b3b-b5e3-443b-9154-517e44e7670f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_with_Atlas.ipynb", "title": "Load the embeddings", "tokens": 776, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"3fdb9b3b-b5e3-443b-9154-517e44e7670f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_with_Atlas.ipynb\", \"title\": \"Load the embeddings\", \"tokens\": 776, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"05982c2b-8221-4bab-b60e-e90c78a4d92d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_with_Atlas.ipynb\", \"title\": \"Load the embeddings\", \"tokens\": 776, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b6504bb6bfc4d983590b719c62f36a7daea71097cf60099a94b56ac73884627e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e777becd-e03d-4c89-b88a-fde118b3e57c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_with_Atlas.ipynb\", \"title\": \"Load the embeddings\", \"tokens\": 776, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"68c0bec86368656c9f2a0a7920b14b558d0a7e10558e0f377a40954a8aeb6dcc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2564, \"end_char_idx\": 2578, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "05982c2b-8221-4bab-b60e-e90c78a4d92d", "doc_id": "05982c2b-8221-4bab-b60e-e90c78a4d92d", "ref_doc_id": "05982c2b-8221-4bab-b60e-e90c78a4d92d", "chroma:document": "```python\n\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 8c8ee268-8fc2-4864-b6cc-079e12f07beb
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"8c8ee268-8fc2-4864-b6cc-079e12f07beb\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8e474bfb-f1be-43d0-86c8-77459454efc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"d6403bb56b167bf497fd43baf379263da481414a794505d22a61f78ce2c3db2e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cb681367-2fd6-4fa1-9a80-b8bae8bb726b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fb1e8c8c2c5bec55ae22dc6ecb3ccb784c51d2c593b1464d455ae73ef3fff809\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3123, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8e474bfb-f1be-43d0-86c8-77459454efc8", "doc_id": "8e474bfb-f1be-43d0-86c8-77459454efc8", "ref_doc_id": "8e474bfb-f1be-43d0-86c8-77459454efc8", "chroma:document": "# Financial Document Analysis with LlamaIndex\n\nIn this example notebook, we showcase how to perform financial analysis over [**10-K**](https://en.wikipedia.org/wiki/Form_10-K) documents with the [**LlamaIndex**](https://gpt-index.readthedocs.io/en/latest/) framework with just a few lines of code.\n\n## Notebook Outline\n* [Introduction](#Introduction)\n* [Setup](#Setup)\n* [Data Loading & Indexing](#Data-Loading-and-Indexing)\n* [Simple QA](#Simple-QA)\n* [Advanced QA - Compare and Contrast](#Advanced-QA---Compare-and-Contrast)\n\n\n## Introduction\n\n### LLamaIndex\n[LlamaIndex](https://gpt-index.readthedocs.io/en/latest/) is a data framework for LLM applications.\nYou can get started with just a few lines of code and build a retrieval-augmented generation (RAG) system in minutes.\nFor more advanced users, LlamaIndex offers a rich toolkit for ingesting and indexing your data, modules for retrieval and re-ranking, and composable components for building custom query engines.\n\nSee [full documentation](https://gpt-index.readthedocs.io/en/latest/) for more details.\n\n### Financial Analysis over 10-K documents\nA key part of a financial analyst's job is to extract information and synthesize insight from long financial documents.\nA great example is the 10-K form - an annual report required by the U.S. Securities and Exchange Commission (SEC), that gives a comprehensive summary of a company's financial performance.\nThese documents typically run hundred of pages in length, and contain domain-specific terminology that makes it challenging for a layperson to digest quickly.\n\n\nWe showcase how LlamaIndex can support a financial analyst in quickly extracting information and synthesize insights **across multiple documents** with very little coding. \n\n## Setup\n\nTo begin, we need to install the llama-index library\n\n\n```python\n!pip install llama-index pypdf\n```\n\nNow, we import all modules used in this tutorial\n\n\n```python\nfrom langchain import OpenAI\n\nfrom llama_index import SimpleDirectoryReader, ServiceContext, VectorStoreIndex\nfrom llama_index import set_global_service_context\nfrom llama_index.response.pprint_utils import pprint_response\nfrom llama_index.tools import QueryEngineTool, ToolMetadata\nfrom llama_index.query_engine import SubQuestionQueryEngine\n```\n\nBefore we start, we can configure the LLM provider and model that will power our RAG system.  \nHere, we pick `gpt-3.5-turbo-instruct` from OpenAI.  \n\n\n```python\nllm = OpenAI(temperature=0, model_name=\"gpt-3.5-turbo-instruct\", max_tokens=-1)\n```\n\nWe construct a `ServiceContext` and set it as the global default, so all subsequent operations that depends on LLM calls will use the model we configured here.\n\n\n```python\nservice_context = ServiceContext.from_defaults(llm=llm)\nset_global_service_context(service_context=service_context)\n```\n\n## Data Loading and Indexing\n\nNow, we load and parse 2 PDFs (one for Uber 10-K in 2021 and another for Lyft 10-k in 2021).    \nUnder the hood, the PDFs are converted to plain text `Document` objects, separate by page.  \n\n> Note: this operation might take a while to run, since each document is more than 100 pages."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: cb681367-2fd6-4fa1-9a80-b8bae8bb726b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"cb681367-2fd6-4fa1-9a80-b8bae8bb726b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8e474bfb-f1be-43d0-86c8-77459454efc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"d6403bb56b167bf497fd43baf379263da481414a794505d22a61f78ce2c3db2e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8c8ee268-8fc2-4864-b6cc-079e12f07beb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"d68884ef68ae6040fe4f01a40d9b66d12e8241d612c39e24261086a0e1592de9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"05167551-a873-40ba-9aea-1d6ae91eb602\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"32f63c194005533c3a439a19e2ed545c88309c0df45aa52f20d5c8f903591004\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1576, \"end_char_idx\": 4689, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8e474bfb-f1be-43d0-86c8-77459454efc8", "doc_id": "8e474bfb-f1be-43d0-86c8-77459454efc8", "ref_doc_id": "8e474bfb-f1be-43d0-86c8-77459454efc8", "chroma:document": "We showcase how LlamaIndex can support a financial analyst in quickly extracting information and synthesize insights **across multiple documents** with very little coding. \n\n## Setup\n\nTo begin, we need to install the llama-index library\n\n\n```python\n!pip install llama-index pypdf\n```\n\nNow, we import all modules used in this tutorial\n\n\n```python\nfrom langchain import OpenAI\n\nfrom llama_index import SimpleDirectoryReader, ServiceContext, VectorStoreIndex\nfrom llama_index import set_global_service_context\nfrom llama_index.response.pprint_utils import pprint_response\nfrom llama_index.tools import QueryEngineTool, ToolMetadata\nfrom llama_index.query_engine import SubQuestionQueryEngine\n```\n\nBefore we start, we can configure the LLM provider and model that will power our RAG system.  \nHere, we pick `gpt-3.5-turbo-instruct` from OpenAI.  \n\n\n```python\nllm = OpenAI(temperature=0, model_name=\"gpt-3.5-turbo-instruct\", max_tokens=-1)\n```\n\nWe construct a `ServiceContext` and set it as the global default, so all subsequent operations that depends on LLM calls will use the model we configured here.\n\n\n```python\nservice_context = ServiceContext.from_defaults(llm=llm)\nset_global_service_context(service_context=service_context)\n```\n\n## Data Loading and Indexing\n\nNow, we load and parse 2 PDFs (one for Uber 10-K in 2021 and another for Lyft 10-k in 2021).    \nUnder the hood, the PDFs are converted to plain text `Document` objects, separate by page.  \n\n> Note: this operation might take a while to run, since each document is more than 100 pages.\n\n\n```python\nlyft_docs = SimpleDirectoryReader(input_files=[\"../data/10k/lyft_2021.pdf\"]).load_data()\nuber_docs = SimpleDirectoryReader(input_files=[\"../data/10k/uber_2021.pdf\"]).load_data()\n```\n\n\n```python\nprint(f'Loaded lyft 10-K with {len(lyft_docs)} pages')\nprint(f'Loaded Uber 10-K with {len(uber_docs)} pages')\n```\n\n    Loaded lyft 10-K with 238 pages\n    Loaded Uber 10-K with 307 pages\n\n\nNow, we can build an (in-memory) `VectorStoreIndex` over the documents that we've loaded.  \n\n> Note: this operation might take a while to run, since it calls OpenAI API for computing vector embedding over document chunks.\n\n\n```python\nlyft_index = VectorStoreIndex.from_documents(lyft_docs)\nuber_index = VectorStoreIndex.from_documents(uber_docs)\n```\n\n## Simple QA\n\nNow we are ready to run some queries against our indices!  \nTo do so, we first configure a `QueryEngine`, which just captures a set of configurations for how we want to query the underlying index.\n\nFor a `VectorStoreIndex`, the most common configuration to adjust is `similarity_top_k` which controls how many document chunks (which we call `Node` objects) are retrieved to use as context for answering our question.\n\n\n```python\nlyft_engine = lyft_index.as_query_engine(similarity_top_k=3)\n```\n\n\n```python\nuber_engine = uber_index.as_query_engine(similarity_top_k=3)\n```\n\nLet's see some queries in action!\n\n\n```python\nresponse = await lyft_engine.aquery('What is the revenue of Lyft in 2021? Answer in millions with page reference')\n```\n\n\n```python\nprint(response)\n```\n\n    \n    $3,208.3 million (page 63)"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 05167551-a873-40ba-9aea-1d6ae91eb602
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"05167551-a873-40ba-9aea-1d6ae91eb602\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8e474bfb-f1be-43d0-86c8-77459454efc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"d6403bb56b167bf497fd43baf379263da481414a794505d22a61f78ce2c3db2e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cb681367-2fd6-4fa1-9a80-b8bae8bb726b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"8360443b0625d043bc72b7a49c304ec36fafc63eff59f1dcede3023ca1d52a96\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ec4fa583-00e4-459d-af63-19e35aa309de\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d5d2afdb820fa2a38202a5e850a5e4ba352e37c2ec00f12ece24f47bf845decb\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3126, \"end_char_idx\": 5855, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8e474bfb-f1be-43d0-86c8-77459454efc8", "doc_id": "8e474bfb-f1be-43d0-86c8-77459454efc8", "ref_doc_id": "8e474bfb-f1be-43d0-86c8-77459454efc8", "chroma:document": "```python\nlyft_docs = SimpleDirectoryReader(input_files=[\"../data/10k/lyft_2021.pdf\"]).load_data()\nuber_docs = SimpleDirectoryReader(input_files=[\"../data/10k/uber_2021.pdf\"]).load_data()\n```\n\n\n```python\nprint(f'Loaded lyft 10-K with {len(lyft_docs)} pages')\nprint(f'Loaded Uber 10-K with {len(uber_docs)} pages')\n```\n\n    Loaded lyft 10-K with 238 pages\n    Loaded Uber 10-K with 307 pages\n\n\nNow, we can build an (in-memory) `VectorStoreIndex` over the documents that we've loaded.  \n\n> Note: this operation might take a while to run, since it calls OpenAI API for computing vector embedding over document chunks.\n\n\n```python\nlyft_index = VectorStoreIndex.from_documents(lyft_docs)\nuber_index = VectorStoreIndex.from_documents(uber_docs)\n```\n\n## Simple QA\n\nNow we are ready to run some queries against our indices!  \nTo do so, we first configure a `QueryEngine`, which just captures a set of configurations for how we want to query the underlying index.\n\nFor a `VectorStoreIndex`, the most common configuration to adjust is `similarity_top_k` which controls how many document chunks (which we call `Node` objects) are retrieved to use as context for answering our question.\n\n\n```python\nlyft_engine = lyft_index.as_query_engine(similarity_top_k=3)\n```\n\n\n```python\nuber_engine = uber_index.as_query_engine(similarity_top_k=3)\n```\n\nLet's see some queries in action!\n\n\n```python\nresponse = await lyft_engine.aquery('What is the revenue of Lyft in 2021? Answer in millions with page reference')\n```\n\n\n```python\nprint(response)\n```\n\n    \n    $3,208.3 million (page 63)\n\n\n\n```python\nresponse = await uber_engine.aquery('What is the revenue of Uber in 2021? Answer in millions, with page reference')\n```\n\n\n```python\nprint(response)\n```\n\n    \n    $17,455 (page 53)\n\n\n## Advanced QA - Compare and Contrast\n\nFor more complex financial analysis, one often needs to reference multiple documents.  \n\nAs a example, let's take a look at how to do compare-and-contrast queries over both Lyft and Uber financials.  \nFor this, we build a `SubQuestionQueryEngine`, which breaks down a complex compare-and-contrast query, into simpler sub-questions to execute on respective sub query engine backed by individual indices.\n\n\n```python\nquery_engine_tools = [\n    QueryEngineTool(\n        query_engine=lyft_engine, \n        metadata=ToolMetadata(name='lyft_10k', description='Provides information about Lyft financials for year 2021')\n    ),\n    QueryEngineTool(\n        query_engine=uber_engine, \n        metadata=ToolMetadata(name='uber_10k', description='Provides information about Uber financials for year 2021')\n    ),\n]\n\ns_engine = SubQuestionQueryEngine.from_defaults(query_engine_tools=query_engine_tools)\n```\n\nLet's see these queries in action!"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: ec4fa583-00e4-459d-af63-19e35aa309de
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"ec4fa583-00e4-459d-af63-19e35aa309de\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8e474bfb-f1be-43d0-86c8-77459454efc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"d6403bb56b167bf497fd43baf379263da481414a794505d22a61f78ce2c3db2e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"05167551-a873-40ba-9aea-1d6ae91eb602\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"cc823631851147cf89120679e80e5c9a59453d562cde4736c63e8ee371126886\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"adf64c23-6bc5-463d-a3d3-ebb8f2ef6dfa\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"51a1b374e2f803bfacc0f73c739c33b50edadbd7cf22bfea926744b34de7080f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4303, \"end_char_idx\": 7987, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8e474bfb-f1be-43d0-86c8-77459454efc8", "doc_id": "8e474bfb-f1be-43d0-86c8-77459454efc8", "ref_doc_id": "8e474bfb-f1be-43d0-86c8-77459454efc8", "chroma:document": "```python\nlyft_engine = lyft_index.as_query_engine(similarity_top_k=3)\n```\n\n\n```python\nuber_engine = uber_index.as_query_engine(similarity_top_k=3)\n```\n\nLet's see some queries in action!\n\n\n```python\nresponse = await lyft_engine.aquery('What is the revenue of Lyft in 2021? Answer in millions with page reference')\n```\n\n\n```python\nprint(response)\n```\n\n    \n    $3,208.3 million (page 63)\n\n\n\n```python\nresponse = await uber_engine.aquery('What is the revenue of Uber in 2021? Answer in millions, with page reference')\n```\n\n\n```python\nprint(response)\n```\n\n    \n    $17,455 (page 53)\n\n\n## Advanced QA - Compare and Contrast\n\nFor more complex financial analysis, one often needs to reference multiple documents.  \n\nAs a example, let's take a look at how to do compare-and-contrast queries over both Lyft and Uber financials.  \nFor this, we build a `SubQuestionQueryEngine`, which breaks down a complex compare-and-contrast query, into simpler sub-questions to execute on respective sub query engine backed by individual indices.\n\n\n```python\nquery_engine_tools = [\n    QueryEngineTool(\n        query_engine=lyft_engine, \n        metadata=ToolMetadata(name='lyft_10k', description='Provides information about Lyft financials for year 2021')\n    ),\n    QueryEngineTool(\n        query_engine=uber_engine, \n        metadata=ToolMetadata(name='uber_10k', description='Provides information about Uber financials for year 2021')\n    ),\n]\n\ns_engine = SubQuestionQueryEngine.from_defaults(query_engine_tools=query_engine_tools)\n```\n\nLet's see these queries in action!\n\n\n```python\nresponse = await s_engine.aquery('Compare and contrast the customer segments and geographies that grew the fastest')\n```\n\n    Generated 4 sub questions.\n    \u001b[36;1m\u001b[1;3m[uber_10k] Q: What customer segments grew the fastest for Uber\n    \u001b[0m\u001b[36;1m\u001b[1;3m[uber_10k] A: in 2021?\n    \n    The customer segments that grew the fastest for Uber in 2021 were its Mobility Drivers, Couriers, Riders, and Eaters. These segments experienced growth due to the continued stay-at-home order demand related to COVID-19, as well as Uber's introduction of its Uber One, Uber Pass, Eats Pass, and Rides Pass membership programs. Additionally, Uber's marketplace-centric advertising helped to connect merchants and brands with its platform network, further driving growth.\n    \u001b[0m\u001b[33;1m\u001b[1;3m[uber_10k] Q: What geographies grew the fastest for Uber\n    \u001b[0m\u001b[33;1m\u001b[1;3m[uber_10k] A: \n    Based on the context information, it appears that Uber experienced the most growth in large metropolitan areas, such as Chicago, Miami, New York City, Sao Paulo, and London. Additionally, Uber experienced growth in suburban and rural areas, as well as in countries such as Argentina, Germany, Italy, Japan, South Korea, and Spain.\n    \u001b[0m\u001b[38;5;200m\u001b[1;3m[lyft_10k] Q: What customer segments grew the fastest for Lyft\n    \u001b[0m\u001b[38;5;200m\u001b[1;3m[lyft_10k] A: \n    The customer segments that grew the fastest for Lyft were ridesharing, light vehicles, and public transit. Ridesharing grew as Lyft was able to predict demand and proactively incentivize drivers to be available for rides in the right place at the right time. Light vehicles grew as users were looking for options that were more active, usually lower-priced, and often more efficient for short trips during heavy traffic. Public transit grew as Lyft integrated third-party public transit data into the Lyft App to offer users a robust view of transportation options around them.\n    \u001b[0m\u001b[32;1m\u001b[1;3m[lyft_10k] Q: What geographies grew the fastest for Lyft\n    \u001b[0m\u001b[32;1m\u001b[1;3m[lyft_10k] A: \n    It is not possible to answer this question with the given context information.\n    \u001b[0m"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: adf64c23-6bc5-463d-a3d3-ebb8f2ef6dfa
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"adf64c23-6bc5-463d-a3d3-ebb8f2ef6dfa\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"8e474bfb-f1be-43d0-86c8-77459454efc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"d6403bb56b167bf497fd43baf379263da481414a794505d22a61f78ce2c3db2e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ec4fa583-00e4-459d-af63-19e35aa309de\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb\", \"title\": \"Financial Document Analysis with LlamaIndex\", \"tokens\": 2638, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"41bc135d2875302036f78eb4139f964d16b7f5ab6e41d69031a16ab790d728ec\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7990, \"end_char_idx\": 10896, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "8e474bfb-f1be-43d0-86c8-77459454efc8", "doc_id": "8e474bfb-f1be-43d0-86c8-77459454efc8", "ref_doc_id": "8e474bfb-f1be-43d0-86c8-77459454efc8", "chroma:document": "```python\nprint(response)\n```\n\n    \n    The customer segments that grew the fastest for Uber in 2021 were its Mobility Drivers, Couriers, Riders, and Eaters. These segments experienced growth due to the continued stay-at-home order demand related to COVID-19, as well as Uber's introduction of its Uber One, Uber Pass, Eats Pass, and Rides Pass membership programs. Additionally, Uber's marketplace-centric advertising helped to connect merchants and brands with its platform network, further driving growth. Uber experienced the most growth in large metropolitan areas, such as Chicago, Miami, New York City, Sao Paulo, and London. Additionally, Uber experienced growth in suburban and rural areas, as well as in countries such as Argentina, Germany, Italy, Japan, South Korea, and Spain.\n    \n    The customer segments that grew the fastest for Lyft were ridesharing, light vehicles, and public transit. Ridesharing grew as Lyft was able to predict demand and proactively incentivize drivers to be available for rides in the right place at the right time. Light vehicles grew as users were looking for options that were more active, usually lower-priced, and often more efficient for short trips during heavy traffic. Public transit grew as Lyft integrated third-party public transit data into the Lyft App to offer users a robust view of transportation options around them. It is not possible to answer the question of which geographies grew the fastest for Lyft with the given context information.\n    \n    In summary, Uber and Lyft both experienced growth in customer segments related to mobility, couriers, riders, and eaters. Uber experienced the most growth in large metropolitan areas, as well as in suburban and rural areas, and in countries such as Argentina, Germany, Italy, Japan, South Korea, and Spain. Lyft experienced the most growth in ridesharing, light vehicles, and public transit. It is not possible to answer the question of which geographies grew the fastest for Lyft with the given context information.\n\n\n\n```python\nresponse = await s_engine.aquery('Compare revenue growth of Uber and Lyft from 2020 to 2021')\n```\n\n    Generated 2 sub questions.\n    \u001b[36;1m\u001b[1;3m[uber_10k] Q: What is the revenue growth of Uber from 2020 to 2021\n    \u001b[0m\u001b[36;1m\u001b[1;3m[uber_10k] A: \n    The revenue growth of Uber from 2020 to 2021 was 57%, or 54% on a constant currency basis.\n    \u001b[0m\u001b[33;1m\u001b[1;3m[lyft_10k] Q: What is the revenue growth of Lyft from 2020 to 2021\n    \u001b[0m\u001b[33;1m\u001b[1;3m[lyft_10k] A: \n    The revenue growth of Lyft from 2020 to 2021 is 36%, increasing from $2,364,681 thousand to $3,208,323 thousand.\n    \u001b[0m\n\n\n```python\nprint(response)\n```\n\n    \n    The revenue growth of Uber from 2020 to 2021 was 57%, or 54% on a constant currency basis, while the revenue growth of Lyft from 2020 to 2021 was 36%. This means that Uber had a higher revenue growth than Lyft from 2020 to 2021."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 3282ac50-3297-46ca-ae22-06adb209c8e6
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"3282ac50-3297-46ca-ae22-06adb209c8e6\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e46118d2-97e0-49bb-b8c5-a99fc16d6c31\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f881672b44bb4e87edc889fb21c6e4678f3ff9286e6b60c0bb53eb4fcf68324d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f9d67534-4b3e-4901-85ba-c1cbec91005b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"363a84282b8220a59fa73aa021e726f76ba41f77e326997513e7208d09cc6883\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3031, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e46118d2-97e0-49bb-b8c5-a99fc16d6c31", "doc_id": "e46118d2-97e0-49bb-b8c5-a99fc16d6c31", "ref_doc_id": "e46118d2-97e0-49bb-b8c5-a99fc16d6c31", "chroma:document": "# OpenAI API Monitoring with W&B Weave\n\n<img src=\"http://wandb.me/logo-im-png\" width=\"400\" alt=\"Weights & Biases\" />\n\n<!--- @wandbcode{weave_openai_client_qs} -->\n\n<a target=\"_blank\" href=\"https://colab.research.google.com/github/wandb/weave/blob/master/examples/prompts/llm_monitoring/openai_client_quickstart.ipynb\">\n  <img src=\"https://colab.research.google.com/assets/colab-badge.svg\" alt=\"Open In Colab\"/>\n</a>\n\n**Note:** you will need an [OpenAI API key](https://platform.openai.com/account/api-keys) to run this colab.\n\nUse the W&B OpenAI integration to monitor OpenAI API calls and understand how your projects and teams are leveraging LLMs.\nIn this example, we'll generate templated Weave Boards: LLM usage monitoring dashboards which you can explore and customize from the UI.\n\n* automatically track LLM usage and aggregate useful metrics like  cost, latency and throughput across your projects/teams\n* dynamically query and derive insights from the logs of all your OpenAI API calls\n* iterate visually to slice, aggregate, and explore your data; customize panels to focus on interesting patterns; share progress more easily with your team through an interactive dashboard\n\n<img src=\"https://raw.githubusercontent.com/wandb/weave/master/docs/assets/full_board_view.png\">\n\n[Play with a live version of this Weave Board \u2192](http://wandb.me/llm-monitoring-board)\n\n#### New to Weights & Biases? [-> Sign up for an account here <-](https://wandb.ai/site)\n\n# Step 0: Setup\n\nInstall dependencies, login to W&B so you can save and share your work, and authenticate with OpenAI.\n\n\n```python\n# if not already installed\n!pip install -qqq weave openai tiktoken wandb\n```\n\n\n```python\nimport wandb\nwandb.login()\n```\n\n\n```python\nimport weave\nimport os\nWANDB_BASE_URL = \"https://api.wandb.ai\"\nos.environ[\"WANDB_BASE_URL\"] = WANDB_BASE_URL\n```\n\n\n```python\n# authenticate with OpenAI\nfrom getpass import getpass\n\nif os.getenv(\"OPENAI_API_KEY\") is None:\n  os.environ[\"OPENAI_API_KEY\"] = getpass(\"Paste your OpenAI key from: https://platform.openai.com/account/api-keys\\n\")\nassert os.getenv(\"OPENAI_API_KEY\", \"\").startswith(\"sk-\"), \"This doesn't look like a valid OpenAI API key\"\nprint(\"OpenAI API key configured\")\n```\n\n# Step 1: Configure data streaming and storage in W&B\n\nSet WB_ENTITY to your wandb username or team name. Log in to W&B and navigate to Home Page at [wandb.ai/home](https://wandb.ai/home) to see valid options under your \"Profile\" and \"Teams\" in the left sidebar.\n\n\n```python\nWB_ENTITY = \"\" # set to your wandb username or team name\nWB_PROJECT = \"weave\" # top-level directory for this work\nSTREAM_NAME = \"openai_logs\" # record table which stores the logs of OpenAI API calls as they stream in\n```\n\n# Step 2: Call init_monitor()\n\nTo start monitoring OpenAI API usage, call `init_monitor(<stream>)`, where `<stream>` has the form `<wandb_team_or_user>/<wandb_project>/<stream_name>`. The stream records and stores all the OpenAI API calls.\n\nRunning this cell will print out a link to view the current project in the Weave UI."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: f9d67534-4b3e-4901-85ba-c1cbec91005b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"f9d67534-4b3e-4901-85ba-c1cbec91005b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e46118d2-97e0-49bb-b8c5-a99fc16d6c31\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f881672b44bb4e87edc889fb21c6e4678f3ff9286e6b60c0bb53eb4fcf68324d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3282ac50-3297-46ca-ae22-06adb209c8e6\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"4e88c6fd0e876a8195fbed3a34439d9cf10944cb207c2eb6876393540fc603b8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"076c8584-9d13-4f2e-a7f4-b707aba1bfe8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"396f67fe8962b03eee56e69bfd8b79cfbafc12da60f5d3b5f9c70ac405c03544\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1581, \"end_char_idx\": 4562, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e46118d2-97e0-49bb-b8c5-a99fc16d6c31", "doc_id": "e46118d2-97e0-49bb-b8c5-a99fc16d6c31", "ref_doc_id": "e46118d2-97e0-49bb-b8c5-a99fc16d6c31", "chroma:document": "```python\n# if not already installed\n!pip install -qqq weave openai tiktoken wandb\n```\n\n\n```python\nimport wandb\nwandb.login()\n```\n\n\n```python\nimport weave\nimport os\nWANDB_BASE_URL = \"https://api.wandb.ai\"\nos.environ[\"WANDB_BASE_URL\"] = WANDB_BASE_URL\n```\n\n\n```python\n# authenticate with OpenAI\nfrom getpass import getpass\n\nif os.getenv(\"OPENAI_API_KEY\") is None:\n  os.environ[\"OPENAI_API_KEY\"] = getpass(\"Paste your OpenAI key from: https://platform.openai.com/account/api-keys\\n\")\nassert os.getenv(\"OPENAI_API_KEY\", \"\").startswith(\"sk-\"), \"This doesn't look like a valid OpenAI API key\"\nprint(\"OpenAI API key configured\")\n```\n\n# Step 1: Configure data streaming and storage in W&B\n\nSet WB_ENTITY to your wandb username or team name. Log in to W&B and navigate to Home Page at [wandb.ai/home](https://wandb.ai/home) to see valid options under your \"Profile\" and \"Teams\" in the left sidebar.\n\n\n```python\nWB_ENTITY = \"\" # set to your wandb username or team name\nWB_PROJECT = \"weave\" # top-level directory for this work\nSTREAM_NAME = \"openai_logs\" # record table which stores the logs of OpenAI API calls as they stream in\n```\n\n# Step 2: Call init_monitor()\n\nTo start monitoring OpenAI API usage, call `init_monitor(<stream>)`, where `<stream>` has the form `<wandb_team_or_user>/<wandb_project>/<stream_name>`. The stream records and stores all the OpenAI API calls.\n\nRunning this cell will print out a link to view the current project in the Weave UI.\n\n\n```python\nfrom weave.monitoring import openai, init_monitor\nm = init_monitor(f\"{WB_ENTITY}/{WB_PROJECT}/{STREAM_NAME}\")\n\n# specifying a single model for simplicity\nOPENAI_MODEL = 'gpt-3.5-turbo'\n\n# prefill with some sample logs\nr = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[{\"role\": \"user\", \"content\": \"hello world!\"}])\nr = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[{\"role\": \"user\", \"content\": \"what is 2+2?\"}])\n```\n\n# Step 3: Preview monitoring dashboard\n\nClick on the link above to preview the data stream, then click \"OpenAI Monitor Board\" in the right sidebar to create a Weave Board for this data stream.\n\n<img src=\"https://raw.githubusercontent.com/wandb/weave/master/docs/assets/short_board_attempt.gif\" width=75%>\n\n# Step 4: Explore & understand your LLM usage\n\nTo save your work, rename the board by clicking on the autogenerated name at the top of the page. To share your board, click \\\"Publish\\\" in the top right.\n\n<img src=\"https://raw.githubusercontent.com/wandb/weave/master/docs/assets/publish_board_short.gif\" width=75%>\n\nTo visualize your work in real-time as you iterate, you can:\n* keep the Board open in a separate tab and refresh to view the latest data\n* rename the Board for easier reference at any point and \\\"Publish\\\" that version to share a link with others\n* find previously saved Boards by navigating to the relevant W&B entity and W&B project name from weave.wandb.ai\n* or open a new instance of a Board template to start fresh with all the data accumulated so far"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 076c8584-9d13-4f2e-a7f4-b707aba1bfe8
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"076c8584-9d13-4f2e-a7f4-b707aba1bfe8\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e46118d2-97e0-49bb-b8c5-a99fc16d6c31\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f881672b44bb4e87edc889fb21c6e4678f3ff9286e6b60c0bb53eb4fcf68324d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f9d67534-4b3e-4901-85ba-c1cbec91005b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1d6aefee754aeee01c8cbcb3ff1a4b089707f75698090eff7889e397aaf919a8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dd1cdfc1-0287-470a-a935-2247266377e3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"12347373e3f59afb296ebc2c75e1ba11ad2c3e89b670fd2ef0e245d794e5230c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3034, \"end_char_idx\": 6586, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e46118d2-97e0-49bb-b8c5-a99fc16d6c31", "doc_id": "e46118d2-97e0-49bb-b8c5-a99fc16d6c31", "ref_doc_id": "e46118d2-97e0-49bb-b8c5-a99fc16d6c31", "chroma:document": "```python\nfrom weave.monitoring import openai, init_monitor\nm = init_monitor(f\"{WB_ENTITY}/{WB_PROJECT}/{STREAM_NAME}\")\n\n# specifying a single model for simplicity\nOPENAI_MODEL = 'gpt-3.5-turbo'\n\n# prefill with some sample logs\nr = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[{\"role\": \"user\", \"content\": \"hello world!\"}])\nr = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[{\"role\": \"user\", \"content\": \"what is 2+2?\"}])\n```\n\n# Step 3: Preview monitoring dashboard\n\nClick on the link above to preview the data stream, then click \"OpenAI Monitor Board\" in the right sidebar to create a Weave Board for this data stream.\n\n<img src=\"https://raw.githubusercontent.com/wandb/weave/master/docs/assets/short_board_attempt.gif\" width=75%>\n\n# Step 4: Explore & understand your LLM usage\n\nTo save your work, rename the board by clicking on the autogenerated name at the top of the page. To share your board, click \\\"Publish\\\" in the top right.\n\n<img src=\"https://raw.githubusercontent.com/wandb/weave/master/docs/assets/publish_board_short.gif\" width=75%>\n\nTo visualize your work in real-time as you iterate, you can:\n* keep the Board open in a separate tab and refresh to view the latest data\n* rename the Board for easier reference at any point and \\\"Publish\\\" that version to share a link with others\n* find previously saved Boards by navigating to the relevant W&B entity and W&B project name from weave.wandb.ai\n* or open a new instance of a Board template to start fresh with all the data accumulated so far\n\n\nNext we'll illustrate a few ways you could track OpenAI API calls. There are many more possibilities depending on your use case, and we can't wait to see what you create from these starter templates.\n\n# Examples\n\n## Example 0: Log a prompt and its completion\n\nMonitor a ChatCompletion request and print the corresponding response, extracting only the text of the completion.\n\n\n```python\nresponse = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[\n        {\"role\": \"user\", \"content\": f\"What is the meaning of life, the universe, and everything?\"},\n    ])\nprint(response['choices'][0]['message']['content'])\n```\n\n## Example 1: Track relevant parameters as attributes\n\nFactor out parameters of interest and track them as attributes on the logged record.\nHere we track the \"system prompt\" separately from the \"prompt template\" and the \"equation\" parameter. This time we'll print the full structured response from the ChatCompletion call.\n\n\n```python\nsystem_prompt = \"you always write in bullet points\"\nprompt_template = 'solve the following equation step by step: {equation}'\nparams = {'equation': '4 * (3 - 1)'}\nopenai.ChatCompletion.create(model=OPENAI_MODEL,\n                             messages=[\n                                    {\"role\": \"system\", \"content\": system_prompt},\n                                    {\"role\": \"user\", \"content\": prompt_template.format(**params)},\n                                ],\n                             # you can add additional attributes to the logged record\n                             # see the monitor_api notebook for more examples\n                             monitor_attributes={\n                                 'system_prompt': system_prompt,\n                                 'prompt_template': prompt_template,\n                                 'params': params\n                             })\n```\n\n## Example 2: Log an ongoing stream of messages\n\nMonitor a stream of messages and log the result as a single record. Note: tokens are not counted in this format."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: dd1cdfc1-0287-470a-a935-2247266377e3
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"dd1cdfc1-0287-470a-a935-2247266377e3\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e46118d2-97e0-49bb-b8c5-a99fc16d6c31\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f881672b44bb4e87edc889fb21c6e4678f3ff9286e6b60c0bb53eb4fcf68324d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"076c8584-9d13-4f2e-a7f4-b707aba1bfe8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bbb1389f389f88ac68a3f42283a9fae916fa1a57ed6c289111fae87a565f6027\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2940026c-817b-4762-a177-b844cb02df60\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"34dfec5596283801792b850a60936b8ebce3d278c76c75954c0abbb19eeefdcf\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4565, \"end_char_idx\": 8061, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e46118d2-97e0-49bb-b8c5-a99fc16d6c31", "doc_id": "e46118d2-97e0-49bb-b8c5-a99fc16d6c31", "ref_doc_id": "e46118d2-97e0-49bb-b8c5-a99fc16d6c31", "chroma:document": "Next we'll illustrate a few ways you could track OpenAI API calls. There are many more possibilities depending on your use case, and we can't wait to see what you create from these starter templates.\n\n# Examples\n\n## Example 0: Log a prompt and its completion\n\nMonitor a ChatCompletion request and print the corresponding response, extracting only the text of the completion.\n\n\n```python\nresponse = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[\n        {\"role\": \"user\", \"content\": f\"What is the meaning of life, the universe, and everything?\"},\n    ])\nprint(response['choices'][0]['message']['content'])\n```\n\n## Example 1: Track relevant parameters as attributes\n\nFactor out parameters of interest and track them as attributes on the logged record.\nHere we track the \"system prompt\" separately from the \"prompt template\" and the \"equation\" parameter. This time we'll print the full structured response from the ChatCompletion call.\n\n\n```python\nsystem_prompt = \"you always write in bullet points\"\nprompt_template = 'solve the following equation step by step: {equation}'\nparams = {'equation': '4 * (3 - 1)'}\nopenai.ChatCompletion.create(model=OPENAI_MODEL,\n                             messages=[\n                                    {\"role\": \"system\", \"content\": system_prompt},\n                                    {\"role\": \"user\", \"content\": prompt_template.format(**params)},\n                                ],\n                             # you can add additional attributes to the logged record\n                             # see the monitor_api notebook for more examples\n                             monitor_attributes={\n                                 'system_prompt': system_prompt,\n                                 'prompt_template': prompt_template,\n                                 'params': params\n                             })\n```\n\n## Example 2: Log an ongoing stream of messages\n\nMonitor a stream of messages and log the result as a single record. Note: tokens are not counted in this format.\n\n\n```python\nfrom weave.monitoring.openai import message_from_stream\nr = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[\n        {\"role\": \"system\", \"content\": \"You are a robot and only speak in robot, like beep bloop bop.\"},\n        {\"role\": \"user\", \"content\": \"Tell me a 50-word story.\"},\n    ], stream=True)\nfor s in message_from_stream(r):\n    print(s, end='')\n```\n\n## Example 3: Structure prompt engineering experiments\n\nHere we compare a few toy options for the system prompt, user question, and intended audience. Try your own experiments and see if any interesting insights emerge as you explore in the Board and group by different parameters.\n\n\n```python\ndef explain_math(system_prompt, prompt_template, params):\n    openai.ChatCompletion.create(model=OPENAI_MODEL,\n                             messages=[\n                                    {\"role\": \"system\", \"content\": system_prompt},\n                                    {\"role\": \"user\", \"content\": prompt_template.format(**params)},\n                                ],\n                             # you can add additional attributes to the logged record\n                             # see the monitor_api notebook for more examples\n                             monitor_attributes={\n                                 'system_prompt': system_prompt,\n                                 'prompt_template': prompt_template,\n                                 'params': params\n                             })\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 2940026c-817b-4762-a177-b844cb02df60
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"2940026c-817b-4762-a177-b844cb02df60\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e46118d2-97e0-49bb-b8c5-a99fc16d6c31\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f881672b44bb4e87edc889fb21c6e4678f3ff9286e6b60c0bb53eb4fcf68324d\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dd1cdfc1-0287-470a-a935-2247266377e3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb\", \"title\": \"OpenAI API Monitoring with W&B Weave\", \"tokens\": 1924, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f980ba25b64ed1fb2efa2755c33114681295307610bf88f782c7fe42b482b86e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6589, \"end_char_idx\": 8659, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e46118d2-97e0-49bb-b8c5-a99fc16d6c31", "doc_id": "e46118d2-97e0-49bb-b8c5-a99fc16d6c31", "ref_doc_id": "e46118d2-97e0-49bb-b8c5-a99fc16d6c31", "chroma:document": "```python\nfrom weave.monitoring.openai import message_from_stream\nr = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[\n        {\"role\": \"system\", \"content\": \"You are a robot and only speak in robot, like beep bloop bop.\"},\n        {\"role\": \"user\", \"content\": \"Tell me a 50-word story.\"},\n    ], stream=True)\nfor s in message_from_stream(r):\n    print(s, end='')\n```\n\n## Example 3: Structure prompt engineering experiments\n\nHere we compare a few toy options for the system prompt, user question, and intended audience. Try your own experiments and see if any interesting insights emerge as you explore in the Board and group by different parameters.\n\n\n```python\ndef explain_math(system_prompt, prompt_template, params):\n    openai.ChatCompletion.create(model=OPENAI_MODEL,\n                             messages=[\n                                    {\"role\": \"system\", \"content\": system_prompt},\n                                    {\"role\": \"user\", \"content\": prompt_template.format(**params)},\n                                ],\n                             # you can add additional attributes to the logged record\n                             # see the monitor_api notebook for more examples\n                             monitor_attributes={\n                                 'system_prompt': system_prompt,\n                                 'prompt_template': prompt_template,\n                                 'params': params\n                             })\n```\n\n\n```python\n# feel free to substitute your own prompts :)\nsystem_prompts = [\"you're extremely flowery and poetic\", \"you're very direct and precise\", \"balance brevity with insight\"]\nprompt_template = 'explain the solution of the following to a {audience}: {equation}'\nequations = ['x^2 + 4x + 9 = 0', '15 * (2 - 6) / 4']\naudience = [\"new student\", \"math genius\"]\n\nfor system_prompt in system_prompts:\n    for equation in equations:\n        for person in audience:\n            params = {\"equation\" : equation, \"audience\" : person}\n            explain_math(system_prompt, prompt_template, params)\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 095cacc4-12ef-44b9-b6f1-bf96db66c595
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"095cacc4-12ef-44b9-b6f1-bf96db66c595\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f9967afa3359b2e7c09990ee29bb86a6dd4321cda0cdc5fbe41510643c5c80a5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a22c2f50-a2f7-4f49-88a9-37da69173b6c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2d781cee85ceb8a18b7dbc93b9f0c9e753a9a6314c0b5f46cc41569d0c33ea40\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2737, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d", "doc_id": "2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d", "ref_doc_id": "2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d", "chroma:document": "# How to use the DALL\u00b7E API\n\nThis notebook shows how to use OpenAI's DALL\u00b7E image API endpoints.\n\nThere are three API endpoints:\n- **Generations:** generates an image or images based on an input caption\n- **Edits:** edits or extends an existing image\n- **Variations:** generates variations of an input image\n\n## Setup\n\n- Import the packages you'll need\n- Import your OpenAI API key: You can do this by running \\``export OPENAI_API_KEY=\"your API key\"`\\` in your terminal.\n- Set a directory to save images to\n\n\n```python\n# imports\nfrom openai import OpenAI  # OpenAI Python library to make API calls\nimport requests  # used to download images\nimport os  # used to access filepaths\nfrom PIL import Image  # used to print and edit images\n\n# initialize OpenAI client\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n\n```\n\n\n```python\n# set a directory to save DALL\u00b7E images to\nimage_dir_name = \"images\"\nimage_dir = os.path.join(os.curdir, image_dir_name)\n\n# create the directory if it doesn't yet exist\nif not os.path.isdir(image_dir):\n    os.mkdir(image_dir)\n\n# print the directory to save to\nprint(f\"{image_dir=}\")\n\n```\n\n    image_dir='./images'\n\n\n## Generations\n\nThe generation API endpoint creates an image based on a text prompt. [API Reference](https://platform.openai.com/docs/api-reference/images/create)\n\n**Required inputs:**\n- `prompt` (str): A text description of the desired image(s). The maximum length is 1000 characters for dall-e-2 and 4000 characters for dall-e-3.\n\n**Optional inputs:**\n- `model` (str): The model to use for image generation. Defaults to dall-e-2\n- `n` (int): The number of images to generate. Must be between 1 and 10. Defaults to 1.\n- `quality` (str): The quality of the image that will be generated. hd creates images with finer details and greater consistency across the image. This param is only supported for dall-e-3.\n- `response_format` (str): The format in which the generated images are returned. Must be one of \"url\" or \"b64_json\". Defaults to \"url\".\n- `size` (str): The size of the generated images. Must be one of 256x256, 512x512, or 1024x1024 for dall-e-2. Must be one of 1024x1024, 1792x1024, or 1024x1792 for dall-e-3 models. Defaults to \"1024x1024\".\n- `style`(str | null): The style of the generated images. Must be one of vivid or natural. Vivid causes the model to lean towards generating hyper-real and dramatic images. Natural causes the model to produce more natural, less hyper-real looking images. This param is only supported for dall-e-3.\n- `user` (str): A unique identifier representing your end-user, which will help OpenAI to monitor and detect abuse. [Learn more.](https://beta.openai.com/docs/usage-policies/end-user-ids)"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: a22c2f50-a2f7-4f49-88a9-37da69173b6c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"a22c2f50-a2f7-4f49-88a9-37da69173b6c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f9967afa3359b2e7c09990ee29bb86a6dd4321cda0cdc5fbe41510643c5c80a5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"095cacc4-12ef-44b9-b6f1-bf96db66c595\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9240d979e920f6f6572bcd12a42b2242fc9efeec6274a41c98b0798fc347e4ea\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f6a8fa65-e0c4-4276-91fd-c73f75a98f8f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cae7f3a0656f5991322954cdf529746689f54e953920540a1bdb209958abb39d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2740, \"end_char_idx\": 4169, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d", "doc_id": "2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d", "ref_doc_id": "2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d", "chroma:document": "```python\n# create an image\n\n# set the prompt\nprompt = \"A cyberpunk monkey hacker dreaming of a beautiful bunch of bananas, digital art\"\n\n# call the OpenAI API\ngeneration_response = client.images.generate(\n    model = \"dall-e-3\",\n    prompt=prompt,\n    n=1,\n    size=\"1024x1024\",\n    response_format=\"url\",\n)\n\n# print response\nprint(generation_response)\n```\n\n    ImagesResponse(created=1701994117, data=[Image(b64_json=None, revised_prompt=None, url='https://oaidalleapiprodscus.blob.core.windows.net/private/org-9HXYFy8ux4r6aboFyec2OLRf/user-8OA8IvMYkfdAcUZXgzAXHS7d/img-ced13hkOk3lXkccQgW1fAQjm.png?st=2023-12-07T23%3A08%3A37Z&se=2023-12-08T01%3A08%3A37Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2023-12-07T16%3A41%3A48Z&ske=2023-12-08T16%3A41%3A48Z&sks=b&skv=2021-08-06&sig=tcD0iyU0ABOvWAKsY89gp5hLVIYkoSXQnrcmH%2Brkric%3D')])\n\n\n\n```python\n# save the image\ngenerated_image_name = \"generated_image.png\"  # any name you like; the filetype should be .png\ngenerated_image_filepath = os.path.join(image_dir, generated_image_name)\ngenerated_image_url = generation_response.data[0].url  # extract image URL from response\ngenerated_image = requests.get(generated_image_url).content  # download the image\n\nwith open(generated_image_filepath, \"wb\") as image_file:\n    image_file.write(generated_image)  # write the image to the file\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: f6a8fa65-e0c4-4276-91fd-c73f75a98f8f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"f6a8fa65-e0c4-4276-91fd-c73f75a98f8f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f9967afa3359b2e7c09990ee29bb86a6dd4321cda0cdc5fbe41510643c5c80a5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a22c2f50-a2f7-4f49-88a9-37da69173b6c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"640ee8bd6c3c9d30555b70f6ac7cea57fb430f29c53bdfc8a02578b849e5a09b\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dcd62c9c-7d87-41e8-be04-5f22f096ae21\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ea5c5938ad4a3ef80b7ee11a01846346a4617862da6dd403ad94bc1025cc1ca3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3673, \"end_char_idx\": 5363, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d", "doc_id": "2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d", "ref_doc_id": "2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d", "chroma:document": "```python\n# save the image\ngenerated_image_name = \"generated_image.png\"  # any name you like; the filetype should be .png\ngenerated_image_filepath = os.path.join(image_dir, generated_image_name)\ngenerated_image_url = generation_response.data[0].url  # extract image URL from response\ngenerated_image = requests.get(generated_image_url).content  # download the image\n\nwith open(generated_image_filepath, \"wb\") as image_file:\n    image_file.write(generated_image)  # write the image to the file\n```\n\n\n```python\n# print the image\nprint(generated_image_filepath)\ndisplay(Image.open(generated_image_filepath))\n\n```\n\n## Variations\n\nThe variations endpoint generates new images (variations) similar to an input image. [API Reference](https://platform.openai.com/docs/api-reference/images/createVariation)\n\nHere we'll generate variations of the image generated above.\n\n**Required inputs:**\n- `image` (str): The image to use as the basis for the variation(s). Must be a valid PNG file, less than 4MB, and square.\n\n**Optional inputs:**\n- `model` (str): The model to use for image variations. Only dall-e-2 is supported at this time.\n- `n` (int): The number of images to generate. Must be between 1 and 10. Defaults to 1.\n- `size` (str): The size of the generated images. Must be one of \"256x256\", \"512x512\", or \"1024x1024\". Smaller images are faster. Defaults to \"1024x1024\".\n- `response_format` (str): The format in which the generated images are returned. Must be one of \"url\" or \"b64_json\". Defaults to \"url\".\n- `user` (str): A unique identifier representing your end-user, which will help OpenAI to monitor and detect abuse. [Learn more.](https://beta.openai.com/docs/usage-policies/end-user-ids)"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: dcd62c9c-7d87-41e8-be04-5f22f096ae21
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"dcd62c9c-7d87-41e8-be04-5f22f096ae21\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f9967afa3359b2e7c09990ee29bb86a6dd4321cda0cdc5fbe41510643c5c80a5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f6a8fa65-e0c4-4276-91fd-c73f75a98f8f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bb8b1183ffce40e3e022f1b88c0a16c4e55bcab386f23e1bde298dc8a2200adf\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d969fdd6-4d6d-4c5b-900c-0ba815d0309f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"becccdac305bcb78ceefb8359364540a75d0fd484b5c57e223ebb1d766a85edc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4172, \"end_char_idx\": 6795, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d", "doc_id": "2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d", "ref_doc_id": "2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d", "chroma:document": "```python\n# print the image\nprint(generated_image_filepath)\ndisplay(Image.open(generated_image_filepath))\n\n```\n\n## Variations\n\nThe variations endpoint generates new images (variations) similar to an input image. [API Reference](https://platform.openai.com/docs/api-reference/images/createVariation)\n\nHere we'll generate variations of the image generated above.\n\n**Required inputs:**\n- `image` (str): The image to use as the basis for the variation(s). Must be a valid PNG file, less than 4MB, and square.\n\n**Optional inputs:**\n- `model` (str): The model to use for image variations. Only dall-e-2 is supported at this time.\n- `n` (int): The number of images to generate. Must be between 1 and 10. Defaults to 1.\n- `size` (str): The size of the generated images. Must be one of \"256x256\", \"512x512\", or \"1024x1024\". Smaller images are faster. Defaults to \"1024x1024\".\n- `response_format` (str): The format in which the generated images are returned. Must be one of \"url\" or \"b64_json\". Defaults to \"url\".\n- `user` (str): A unique identifier representing your end-user, which will help OpenAI to monitor and detect abuse. [Learn more.](https://beta.openai.com/docs/usage-policies/end-user-ids)\n\n\n\n```python\n# create variations\n\n# call the OpenAI API, using `create_variation` rather than `create`\nvariation_response = client.images.create_variation(\n    image=generated_image,  # generated_image is the image generated above\n    n=2,\n    size=\"1024x1024\",\n    response_format=\"url\",\n)\n\n# print response\nprint(variation_response)\n```\n\n    ImagesResponse(created=1701994139, data=[Image(b64_json=None, revised_prompt=None, url='https://oaidalleapiprodscus.blob.core.windows.net/private/org-9HXYFy8ux4r6aboFyec2OLRf/user-8OA8IvMYkfdAcUZXgzAXHS7d/img-noNRGgwaaotRGIe6Y2GVeSpr.png?st=2023-12-07T23%3A08%3A59Z&se=2023-12-08T01%3A08%3A59Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2023-12-07T16%3A39%3A11Z&ske=2023-12-08T16%3A39%3A11Z&sks=b&skv=2021-08-06&sig=ER5RUglhtIk9LWJXw1DsolorT4bnEmFostfnUjY21ns%3D'), Image(b64_json=None, revised_prompt=None, url='https://oaidalleapiprodscus.blob.core.windows.net/private/org-9HXYFy8ux4r6aboFyec2OLRf/user-8OA8IvMYkfdAcUZXgzAXHS7d/img-oz952tL11FFhf9iXXJVIRUZX.png?st=2023-12-07T23%3A08%3A59Z&se=2023-12-08T01%3A08%3A59Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2023-12-07T16%3A39%3A11Z&ske=2023-12-08T16%3A39%3A11Z&sks=b&skv=2021-08-06&sig=99rJOQwDKsfIeerlMXMHholhAhrHfYaQRFJBF8FKv74%3D')])"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: d969fdd6-4d6d-4c5b-900c-0ba815d0309f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"d969fdd6-4d6d-4c5b-900c-0ba815d0309f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f9967afa3359b2e7c09990ee29bb86a6dd4321cda0cdc5fbe41510643c5c80a5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dcd62c9c-7d87-41e8-be04-5f22f096ae21\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"780e65226e3518715ce65edc4a0bb066d0c1ead6ce3c4ee2d774cb510bd56804\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"95095758-1558-4693-9606-455781f5140e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d146b79fe58b50bc0cf29dbad89ef05d81adc82a9dee84541e043cf912021a08\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6799, \"end_char_idx\": 9501, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d", "doc_id": "2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d", "ref_doc_id": "2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d", "chroma:document": "```python\n# save the images\nvariation_urls = [datum.url for datum in variation_response.data]  # extract URLs\nvariation_images = [requests.get(url).content for url in variation_urls]  # download images\nvariation_image_names = [f\"variation_image_{i}.png\" for i in range(len(variation_images))]  # create names\nvariation_image_filepaths = [os.path.join(image_dir, name) for name in variation_image_names]  # create filepaths\nfor image, filepath in zip(variation_images, variation_image_filepaths):  # loop through the variations\n    with open(filepath, \"wb\") as image_file:  # open the file\n        image_file.write(image)  # write the image to the file\n```\n\n\n```python\n# print the original image\nprint(generated_image_filepath)\ndisplay(Image.open(generated_image_filepath))\n\n# print the new variations\nfor variation_image_filepaths in variation_image_filepaths:\n    print(variation_image_filepaths)\n    display(Image.open(variation_image_filepaths))\n\n```\n\n## Edits\n\nThe edit endpoint uses DALL\u00b7E to generate a specified portion of an existing image. Three inputs are needed: the image to edit, a mask specifying the portion to be regenerated, and a prompt describing the desired image. [API Reference](https://platform.openai.com/docs/api-reference/images/createEdit)\n\n**Required inputs:** \n- `image` (str): The image to edit. Must be a valid PNG file, less than 4MB, and square. If mask is not provided, image must have transparency, which will be used as the mask.\n- `prompt` (str): A text description of the desired image(s). The maximum length is 1000 characters.\n\n**Optional inputs:**\n- `mask` (file): An additional image whose fully transparent areas (e.g. where alpha is zero) indicate where image should be edited. Must be a valid PNG file, less than 4MB, and have the same dimensions as image.\n- `model` (str): The model to use for edit image. Only dall-e-2 is supported at this time.\n- `n` (int): The number of images to generate. Must be between 1 and 10. Defaults to 1.\n- `size` (str): The size of the generated images. Must be one of \"256x256\", \"512x512\", or \"1024x1024\". Smaller images are faster. Defaults to \"1024x1024\".\n- `response_format` (str): The format in which the generated images are returned. Must be one of \"url\" or \"b64_json\". Defaults to \"url\".\n- `user` (str): A unique identifier representing your end-user, which will help OpenAI to monitor and detect abuse. [Learn more.](https://beta.openai.com/docs/usage-policies/end-user-ids)\n\n\n### Set Edit Area\n\nAn edit requires a \"mask\" to specify which portion of the image to regenerate. Any pixel with an alpha of 0 (transparent) will be regenerated. The code below creates a 1024x1024 mask where the bottom half is transparent."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 95095758-1558-4693-9606-455781f5140e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"95095758-1558-4693-9606-455781f5140e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f9967afa3359b2e7c09990ee29bb86a6dd4321cda0cdc5fbe41510643c5c80a5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d969fdd6-4d6d-4c5b-900c-0ba815d0309f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9206dd616fe0753dac07572fa680f1cb0aa6e7ae94d4cff3dc93ce3b7d2fdd11\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3740c1bc-dd55-41e6-8497-5b5fa1035d26\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a9f90653b2f6b2afeecea1e3baea97bdebc00fefe65a847bf3f4757133d59617\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9262, \"end_char_idx\": 11576, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d", "doc_id": "2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d", "ref_doc_id": "2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d", "chroma:document": "### Set Edit Area\n\nAn edit requires a \"mask\" to specify which portion of the image to regenerate. Any pixel with an alpha of 0 (transparent) will be regenerated. The code below creates a 1024x1024 mask where the bottom half is transparent.\n\n\n```python\n# create a mask\nwidth = 1024\nheight = 1024\nmask = Image.new(\"RGBA\", (width, height), (0, 0, 0, 1))  # create an opaque image mask\n\n# set the bottom half to be transparent\nfor x in range(width):\n    for y in range(height // 2, height):  # only loop over the bottom half of the mask\n        # set alpha (A) to zero to turn pixel transparent\n        alpha = 0\n        mask.putpixel((x, y), (0, 0, 0, alpha))\n\n# save the mask\nmask_name = \"bottom_half_mask.png\"\nmask_filepath = os.path.join(image_dir, mask_name)\nmask.save(mask_filepath)\n```\n\n### Perform Edit\n\nNow we supply our image, caption and mask to the API to get 5 examples of edits to our image\n\n\n```python\n# edit an image\n\n# call the OpenAI API\nedit_response = client.images.edit(\n    image=open(generated_image_filepath, \"rb\"),  # from the generation section\n    mask=open(mask_filepath, \"rb\"),  # from right above\n    prompt=prompt,  # from the generation section\n    n=1,\n    size=\"1024x1024\",\n    response_format=\"url\",\n)\n\n# print response\nprint(edit_response)\n```\n\n    ImagesResponse(created=1701994167, data=[Image(b64_json=None, revised_prompt=None, url='https://oaidalleapiprodscus.blob.core.windows.net/private/org-9HXYFy8ux4r6aboFyec2OLRf/user-8OA8IvMYkfdAcUZXgzAXHS7d/img-9UOVGC7wB8MS2Q7Rwgj0fFBq.png?st=2023-12-07T23%3A09%3A27Z&se=2023-12-08T01%3A09%3A27Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2023-12-07T16%3A40%3A37Z&ske=2023-12-08T16%3A40%3A37Z&sks=b&skv=2021-08-06&sig=MsRMZ1rN434bVdWr%2B9kIoqu9CIrvZypZBfkQPTOhCl4%3D')])\n\n\n\n```python\n# save the image\nedited_image_name = \"edited_image.png\"  # any name you like; the filetype should be .png\nedited_image_filepath = os.path.join(image_dir, edited_image_name)\nedited_image_url = edit_response.data[0].url  # extract image URL from response\nedited_image = requests.get(edited_image_url).content  # download the image\n\nwith open(edited_image_filepath, \"wb\") as image_file:\n    image_file.write(edited_image)  # write the image to the file\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 3740c1bc-dd55-41e6-8497-5b5fa1035d26
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"3740c1bc-dd55-41e6-8497-5b5fa1035d26\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f9967afa3359b2e7c09990ee29bb86a6dd4321cda0cdc5fbe41510643c5c80a5\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"95095758-1558-4693-9606-455781f5140e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb\", \"title\": \"How to use the DALL\\u00b7E API\", \"tokens\": 3601, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"36ca71a4622c499d3b73ac415c9ec050a145affe3f6c096e2e4e1816641637f7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11113, \"end_char_idx\": 11791, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d", "doc_id": "2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d", "ref_doc_id": "2448e1c8-cd1d-46fd-b116-fa5d7cf03b5d", "chroma:document": "```python\n# save the image\nedited_image_name = \"edited_image.png\"  # any name you like; the filetype should be .png\nedited_image_filepath = os.path.join(image_dir, edited_image_name)\nedited_image_url = edit_response.data[0].url  # extract image URL from response\nedited_image = requests.get(edited_image_url).content  # download the image\n\nwith open(edited_image_filepath, \"wb\") as image_file:\n    image_file.write(edited_image)  # write the image to the file\n```\n\n\n```python\n# print the original image\nprint(generated_image_filepath)\ndisplay(Image.open(generated_image_filepath))\n\n# print edited image\nprint(edited_image_filepath)\ndisplay(Image.open(edited_image_filepath))\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: d43909ab-79e5-40ef-9f69-faa2577a85c1
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"d43909ab-79e5-40ef-9f69-faa2577a85c1\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"eb2c0984-a447-4335-a118-ff90ababec57\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"dfe7789273170d20e0160f316923946bae8dd5425eb9a08d8c8c7abb8244c26e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"25c10399-7251-4383-9bdc-b849d70a24db\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3f872709cd0ad71669adf9a32846c3719ad168a0d9fe6b7f1d34ff216c84207e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2755, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "eb2c0984-a447-4335-a118-ff90ababec57", "doc_id": "eb2c0984-a447-4335-a118-ff90ababec57", "ref_doc_id": "eb2c0984-a447-4335-a118-ff90ababec57", "chroma:document": "# How to create dynamic masks with DALL\u00b7E and Segment Anything\n\nSegment Anything is a model from Meta that can be used to select portions of images. Combined with DALL\u00b7E's ability to inpaint specified portions of images, you can use Segment Anything to easily select any part of an image you'd like to alter.\n\nIn this notebook, we'll use these tools to become fashion designers and dynamically replace our digital models' outfits with tailored, original creations. The notebook follows this flow:\n- **Setup:** Initialise your libraries and any location directories.\n- **Generate original image:** Make an original image that we'll create dynamic masks from.\n- **Generate mask:** Use Segment Anything to create a dynamic mask.\n- **Create new image:** Generate a new image with the masked area inpainted with a fresh prompt.\n\n## Setup\n\nTo get started we'll need to follow the [instructions](https://github.com/facebookresearch/segment-anything) for using the Segment Anything (SAM) model open-sourced by Meta. As of May 2023, the key steps are:\n- Install [Pytorch](https://pytorch.org/get-started/locally/) (version 1.7+).\n- Install the library using ```pip install git+https://github.com/facebookresearch/segment-anything.git```.\n- Install dependencies using ```pip install opencv-python pycocotools matplotlib onnxruntime onnx```.\n- Download a [model checkpoint](https://github.com/facebookresearch/segment-anything#model-checkpoints) to use (default size is 2.4 GB).\n\n\n\n```python\n!pip install torch torchvision torchaudio\n!pip install git+https://github.com/facebookresearch/segment-anything.git\n!pip install opencv-python pycocotools matplotlib onnxruntime onnx\n!pip install requests\n!pip install openai\n!pip install numpy\n```\n\n\n```python\n!wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth\n```\n\n\n```python\nimport cv2\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nfrom matplotlib import rcParams\nimport numpy as np\nfrom openai import OpenAI\nimport os\nfrom PIL import Image\nimport requests\nfrom segment_anything import sam_model_registry, SamAutomaticMaskGenerator, SamPredictor\nimport torch\n\n# Set directories for generation images and edit images\nbase_image_dir = os.path.join(\"images\", \"01_generations\")\nmask_dir = os.path.join(\"images\", \"02_masks\")\nedit_image_dir = os.path.join(\"images\", \"03_edits\")\n\n# Point to your downloaded SAM model\nsam_model_filepath = \"./sam_vit_h_4b8939.pth\"\n\n# Initiate SAM model\nsam = sam_model_registry[\"default\"](checkpoint=sam_model_filepath)\n\n# Initiate openAI client\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n\n```\n\n## Generate original image\n\nFirst we'll create an original image which we'll generate masks from."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 25c10399-7251-4383-9bdc-b849d70a24db
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"25c10399-7251-4383-9bdc-b849d70a24db\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"eb2c0984-a447-4335-a118-ff90ababec57\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"dfe7789273170d20e0160f316923946bae8dd5425eb9a08d8c8c7abb8244c26e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d43909ab-79e5-40ef-9f69-faa2577a85c1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"24ba57ce848bd192322ab4a06ebafa7bff271c0cb97d01a2b4c0058c9e7b34bf\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bf742aa5-3404-4be6-89d2-e977129ffc0d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"150cbdb4185cb59868a7c988f47a1ba89266553763a874c355bb162ad93ba927\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1471, \"end_char_idx\": 4084, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "eb2c0984-a447-4335-a118-ff90ababec57", "doc_id": "eb2c0984-a447-4335-a118-ff90ababec57", "ref_doc_id": "eb2c0984-a447-4335-a118-ff90ababec57", "chroma:document": "```python\n!pip install torch torchvision torchaudio\n!pip install git+https://github.com/facebookresearch/segment-anything.git\n!pip install opencv-python pycocotools matplotlib onnxruntime onnx\n!pip install requests\n!pip install openai\n!pip install numpy\n```\n\n\n```python\n!wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth\n```\n\n\n```python\nimport cv2\nimport matplotlib.pyplot as plt\nimport matplotlib.image as mpimg\nfrom matplotlib import rcParams\nimport numpy as np\nfrom openai import OpenAI\nimport os\nfrom PIL import Image\nimport requests\nfrom segment_anything import sam_model_registry, SamAutomaticMaskGenerator, SamPredictor\nimport torch\n\n# Set directories for generation images and edit images\nbase_image_dir = os.path.join(\"images\", \"01_generations\")\nmask_dir = os.path.join(\"images\", \"02_masks\")\nedit_image_dir = os.path.join(\"images\", \"03_edits\")\n\n# Point to your downloaded SAM model\nsam_model_filepath = \"./sam_vit_h_4b8939.pth\"\n\n# Initiate SAM model\nsam = sam_model_registry[\"default\"](checkpoint=sam_model_filepath)\n\n# Initiate openAI client\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n\n```\n\n## Generate original image\n\nFirst we'll create an original image which we'll generate masks from.\n\n\n```python\ndef process_dalle_images(response, filename, image_dir):\n    # save the images\n    urls = [datum.url for datum in response.data]  # extract URLs\n    images = [requests.get(url).content for url in urls]  # download images\n    image_names = [f\"{filename}_{i + 1}.png\" for i in range(len(images))]  # create names\n    filepaths = [os.path.join(image_dir, name) for name in image_names]  # create filepaths\n    for image, filepath in zip(images, filepaths):  # loop through the variations\n        with open(filepath, \"wb\") as image_file:  # open the file\n            image_file.write(image)  # write the image to the file\n\n    return filepaths\n```\n\n\n```python\ndalle_prompt = '''\nFull length, zoomed out photo of our premium Lederhosen-inspired jumpsuit.\nShowcase the intricate hand-stitched details and high-quality leather, while highlighting the perfect blend of Austrian heritage and modern fashion.\nThis piece appeals to a sophisticated, trendsetting audience who appreciates cultural fusion and innovative design.\n'''\n```\n\n\n```python\n# Generate your images\ngeneration_response = client.images.generate(\n    model = \"dall-e-3\",\n    prompt=dalle_prompt,\n    n=3,\n    size=\"1024x1024\",\n    response_format=\"url\",\n)\n```\n\n\n```python\nfilepaths = process_dalle_images(generation_response, \"generation\", base_image_dir)\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: bf742aa5-3404-4be6-89d2-e977129ffc0d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"bf742aa5-3404-4be6-89d2-e977129ffc0d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"eb2c0984-a447-4335-a118-ff90ababec57\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"dfe7789273170d20e0160f316923946bae8dd5425eb9a08d8c8c7abb8244c26e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"25c10399-7251-4383-9bdc-b849d70a24db\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"46179f97841740cf675edf7339f250832488133b6af80b01943687f09252bc9e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"013fff2c-fbab-44fb-a9a2-a1f0484cf353\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f7f0568eb9192a7ca1476f462fa08aef7b0dfafa43f719ea9763c74f285d8b68\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2758, \"end_char_idx\": 5595, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "eb2c0984-a447-4335-a118-ff90ababec57", "doc_id": "eb2c0984-a447-4335-a118-ff90ababec57", "ref_doc_id": "eb2c0984-a447-4335-a118-ff90ababec57", "chroma:document": "```python\ndef process_dalle_images(response, filename, image_dir):\n    # save the images\n    urls = [datum.url for datum in response.data]  # extract URLs\n    images = [requests.get(url).content for url in urls]  # download images\n    image_names = [f\"{filename}_{i + 1}.png\" for i in range(len(images))]  # create names\n    filepaths = [os.path.join(image_dir, name) for name in image_names]  # create filepaths\n    for image, filepath in zip(images, filepaths):  # loop through the variations\n        with open(filepath, \"wb\") as image_file:  # open the file\n            image_file.write(image)  # write the image to the file\n\n    return filepaths\n```\n\n\n```python\ndalle_prompt = '''\nFull length, zoomed out photo of our premium Lederhosen-inspired jumpsuit.\nShowcase the intricate hand-stitched details and high-quality leather, while highlighting the perfect blend of Austrian heritage and modern fashion.\nThis piece appeals to a sophisticated, trendsetting audience who appreciates cultural fusion and innovative design.\n'''\n```\n\n\n```python\n# Generate your images\ngeneration_response = client.images.generate(\n    model = \"dall-e-3\",\n    prompt=dalle_prompt,\n    n=3,\n    size=\"1024x1024\",\n    response_format=\"url\",\n)\n```\n\n\n```python\nfilepaths = process_dalle_images(generation_response, \"generation\", base_image_dir)\n```\n\n\n```python\n# print the new generations\nfor filepath in filepaths:\n    print(filepath)\n    display(Image.open(filepath))\n```\n\n## Generate Mask\n\nNext we'll load up one of our images and generate masks.\n\nFor this demonstration we're picking a UX where we \"click\" on a point on the image to generate masks from. However, there are [example notebooks](https://github.com/facebookresearch/segment-anything/blob/main/notebooks/automatic_mask_generator_example.ipynb) provided by Meta which show how to generate every possible mask for an image, draw a box, and some other useful approaches.\n\n\n```python\n# Pick one of your generated images\nchosen_image = \"images/01_generations/generation_2.png\"\n```\n\n\n```python\n# Function to display mask using matplotlib\ndef show_mask(mask, ax):\n    color = np.array([30 / 255, 144 / 255, 255 / 255, 0.6])\n    h, w = mask.shape[-2:]\n    mask_image = mask.reshape(h, w, 1) * color.reshape(1, 1, -1)\n    ax.imshow(mask_image)\n\n\n# Function to display where we've \"clicked\"\ndef show_points(coords, labels, ax, marker_size=375):\n    pos_points = coords[labels == 1]\n    neg_points = coords[labels == 0]\n    ax.scatter(\n        pos_points[:, 0],\n        pos_points[:, 1],\n        color=\"green\",\n        marker=\"*\",\n        s=marker_size,\n        edgecolor=\"white\",\n        linewidth=1.25,\n    )\n    ax.scatter(\n        neg_points[:, 0],\n        neg_points[:, 1],\n        color=\"red\",\n        marker=\"*\",\n        s=marker_size,\n        edgecolor=\"white\",\n        linewidth=1.25,\n    )\n\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 013fff2c-fbab-44fb-a9a2-a1f0484cf353
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"013fff2c-fbab-44fb-a9a2-a1f0484cf353\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"eb2c0984-a447-4335-a118-ff90ababec57\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"dfe7789273170d20e0160f316923946bae8dd5425eb9a08d8c8c7abb8244c26e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bf742aa5-3404-4be6-89d2-e977129ffc0d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"88280d0c88de9cbc0575c245395f3a99e74e34250728d5bc5fae8580d77791a1\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"71b60b46-7fc7-40a1-a380-ba6ba0a7466f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2c3c5989065f48bf1f980e73e021ea0346f149315b2fcb667411b0f08938bfbd\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4087, \"end_char_idx\": 7011, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "eb2c0984-a447-4335-a118-ff90ababec57", "doc_id": "eb2c0984-a447-4335-a118-ff90ababec57", "ref_doc_id": "eb2c0984-a447-4335-a118-ff90ababec57", "chroma:document": "```python\n# print the new generations\nfor filepath in filepaths:\n    print(filepath)\n    display(Image.open(filepath))\n```\n\n## Generate Mask\n\nNext we'll load up one of our images and generate masks.\n\nFor this demonstration we're picking a UX where we \"click\" on a point on the image to generate masks from. However, there are [example notebooks](https://github.com/facebookresearch/segment-anything/blob/main/notebooks/automatic_mask_generator_example.ipynb) provided by Meta which show how to generate every possible mask for an image, draw a box, and some other useful approaches.\n\n\n```python\n# Pick one of your generated images\nchosen_image = \"images/01_generations/generation_2.png\"\n```\n\n\n```python\n# Function to display mask using matplotlib\ndef show_mask(mask, ax):\n    color = np.array([30 / 255, 144 / 255, 255 / 255, 0.6])\n    h, w = mask.shape[-2:]\n    mask_image = mask.reshape(h, w, 1) * color.reshape(1, 1, -1)\n    ax.imshow(mask_image)\n\n\n# Function to display where we've \"clicked\"\ndef show_points(coords, labels, ax, marker_size=375):\n    pos_points = coords[labels == 1]\n    neg_points = coords[labels == 0]\n    ax.scatter(\n        pos_points[:, 0],\n        pos_points[:, 1],\n        color=\"green\",\n        marker=\"*\",\n        s=marker_size,\n        edgecolor=\"white\",\n        linewidth=1.25,\n    )\n    ax.scatter(\n        neg_points[:, 0],\n        neg_points[:, 1],\n        color=\"red\",\n        marker=\"*\",\n        s=marker_size,\n        edgecolor=\"white\",\n        linewidth=1.25,\n    )\n\n```\n\n\n```python\n# Load chosen image using opencv\nimage = cv2.imread(chosen_image)\nimage = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n\n# Display our chosen image\nplt.figure(figsize=(10, 10))\nplt.imshow(image)\nplt.axis(\"on\")\nplt.show()\n```\n\n\n    \n![png](output_14_0.png)\n    \n\n\n\n```python\n# Set the pixel coordinates for our \"click\" to assign masks\ninput_point = np.array([[525, 325]])\ninput_label = np.array([1])\n\n# Display the point we've clicked on\nplt.figure(figsize=(10, 10))\nplt.imshow(image)\nshow_points(input_point, input_label, plt.gca())\nplt.axis(\"on\")\nplt.show()\n```\n\n\n    \n![png](output_15_0.png)\n    \n\n\n\n```python\n# Initiate predictor with Segment Anything model\npredictor = SamPredictor(sam)\npredictor.set_image(image)\n\n# Use the predictor to gather masks for the point we clicked\nmasks, scores, logits = predictor.predict(\n    point_coords=input_point,\n    point_labels=input_label,\n    multimask_output=True,\n)\n\n# Check the shape - should be three masks of the same dimensions as our image\nmasks.shape\n```\n\n\n\n\n    (3, 1024, 1024)\n\n\n\n\n```python\n# Display the possible masks we can select along with their confidence\nfor i, (mask, score) in enumerate(zip(masks, scores)):\n    plt.figure(figsize=(10, 10))\n    plt.imshow(image)\n    show_mask(mask, plt.gca())\n    show_points(input_point, input_label, plt.gca())\n    plt.title(f\"Mask {i+1}, Score: {score:.3f}\", fontsize=18)\n    plt.axis(\"off\")\n    plt.show()\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 71b60b46-7fc7-40a1-a380-ba6ba0a7466f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"71b60b46-7fc7-40a1-a380-ba6ba0a7466f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"eb2c0984-a447-4335-a118-ff90ababec57\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"dfe7789273170d20e0160f316923946bae8dd5425eb9a08d8c8c7abb8244c26e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"013fff2c-fbab-44fb-a9a2-a1f0484cf353\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"4144bc860a736b4ae742fa94387f5ec342d97afd3555436364c08d9d7fdbacc7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8c471639-4642-4617-beed-7d236018271e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"cdae518d1ba8cdab98b04dbdfd1b6bd19ac151af4ca5491eb795369b603c069a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5598, \"end_char_idx\": 8235, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "eb2c0984-a447-4335-a118-ff90ababec57", "doc_id": "eb2c0984-a447-4335-a118-ff90ababec57", "ref_doc_id": "eb2c0984-a447-4335-a118-ff90ababec57", "chroma:document": "```python\n# Load chosen image using opencv\nimage = cv2.imread(chosen_image)\nimage = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)\n\n# Display our chosen image\nplt.figure(figsize=(10, 10))\nplt.imshow(image)\nplt.axis(\"on\")\nplt.show()\n```\n\n\n    \n![png](output_14_0.png)\n    \n\n\n\n```python\n# Set the pixel coordinates for our \"click\" to assign masks\ninput_point = np.array([[525, 325]])\ninput_label = np.array([1])\n\n# Display the point we've clicked on\nplt.figure(figsize=(10, 10))\nplt.imshow(image)\nshow_points(input_point, input_label, plt.gca())\nplt.axis(\"on\")\nplt.show()\n```\n\n\n    \n![png](output_15_0.png)\n    \n\n\n\n```python\n# Initiate predictor with Segment Anything model\npredictor = SamPredictor(sam)\npredictor.set_image(image)\n\n# Use the predictor to gather masks for the point we clicked\nmasks, scores, logits = predictor.predict(\n    point_coords=input_point,\n    point_labels=input_label,\n    multimask_output=True,\n)\n\n# Check the shape - should be three masks of the same dimensions as our image\nmasks.shape\n```\n\n\n\n\n    (3, 1024, 1024)\n\n\n\n\n```python\n# Display the possible masks we can select along with their confidence\nfor i, (mask, score) in enumerate(zip(masks, scores)):\n    plt.figure(figsize=(10, 10))\n    plt.imshow(image)\n    show_mask(mask, plt.gca())\n    show_points(input_point, input_label, plt.gca())\n    plt.title(f\"Mask {i+1}, Score: {score:.3f}\", fontsize=18)\n    plt.axis(\"off\")\n    plt.show()\n```\n\n\n    \n![png](output_17_0.png)\n    \n\n\n\n    \n![png](output_17_1.png)\n    \n\n\n\n    \n![png](output_17_2.png)\n    \n\n\n\n```python\n# Choose which mask you'd like to use\nchosen_mask = masks[1]\n\n# We'll now reverse the mask so that it is clear and everything else is white\nchosen_mask = chosen_mask.astype(\"uint8\")\nchosen_mask[chosen_mask != 0] = 255\nchosen_mask[chosen_mask == 0] = 1\nchosen_mask[chosen_mask == 255] = 0\nchosen_mask[chosen_mask == 1] = 255\n\n```\n\n\n```python\n# create a base blank mask\nwidth = 1024\nheight = 1024\nmask = Image.new(\"RGBA\", (width, height), (0, 0, 0, 1))  # create an opaque image mask\n\n# Convert mask back to pixels to add our mask replacing the third dimension\npix = np.array(mask)\npix[:, :, 3] = chosen_mask\n\n# Convert pixels back to an RGBA image and display\nnew_mask = Image.fromarray(pix, \"RGBA\")\nnew_mask\n```\n\n\n\n\n    \n![png](output_19_0.png)\n    \n\n\n\n\n```python\n# We'll save this mask for re-use for our edit\nnew_mask.save(os.path.join(mask_dir, \"new_mask.png\"))\n```\n\n## Create new image\n\nNow we'll combine our original image with the mask and the Edit endpoint for DALLE to inpaint the transparent area according to a new prompt. (as 0f January 2024 dall-e-2 is the only model that supports edits)"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 8c471639-4642-4617-beed-7d236018271e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"8c471639-4642-4617-beed-7d236018271e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"eb2c0984-a447-4335-a118-ff90ababec57\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"dfe7789273170d20e0160f316923946bae8dd5425eb9a08d8c8c7abb8244c26e\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"71b60b46-7fc7-40a1-a380-ba6ba0a7466f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb\", \"title\": \"How to create dynamic masks with DALL\\u00b7E and Segment Anything\", \"tokens\": 2392, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a40098a8d495837a5bd8cfd25d5a368cf33565fcf5f481543eaaaacc2c43ba72\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7019, \"end_char_idx\": 9362, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "eb2c0984-a447-4335-a118-ff90ababec57", "doc_id": "eb2c0984-a447-4335-a118-ff90ababec57", "ref_doc_id": "eb2c0984-a447-4335-a118-ff90ababec57", "chroma:document": "![png](output_17_0.png)\n    \n\n\n\n    \n![png](output_17_1.png)\n    \n\n\n\n    \n![png](output_17_2.png)\n    \n\n\n\n```python\n# Choose which mask you'd like to use\nchosen_mask = masks[1]\n\n# We'll now reverse the mask so that it is clear and everything else is white\nchosen_mask = chosen_mask.astype(\"uint8\")\nchosen_mask[chosen_mask != 0] = 255\nchosen_mask[chosen_mask == 0] = 1\nchosen_mask[chosen_mask == 255] = 0\nchosen_mask[chosen_mask == 1] = 255\n\n```\n\n\n```python\n# create a base blank mask\nwidth = 1024\nheight = 1024\nmask = Image.new(\"RGBA\", (width, height), (0, 0, 0, 1))  # create an opaque image mask\n\n# Convert mask back to pixels to add our mask replacing the third dimension\npix = np.array(mask)\npix[:, :, 3] = chosen_mask\n\n# Convert pixels back to an RGBA image and display\nnew_mask = Image.fromarray(pix, \"RGBA\")\nnew_mask\n```\n\n\n\n\n    \n![png](output_19_0.png)\n    \n\n\n\n\n```python\n# We'll save this mask for re-use for our edit\nnew_mask.save(os.path.join(mask_dir, \"new_mask.png\"))\n```\n\n## Create new image\n\nNow we'll combine our original image with the mask and the Edit endpoint for DALLE to inpaint the transparent area according to a new prompt. (as 0f January 2024 dall-e-2 is the only model that supports edits)\n\n\n\n```python\n# edit an image\nedit_response = client.images.edit(\n    image=open(chosen_image, \"rb\"),  # from the generation section\n    mask=open(os.path.join(mask_dir, \"new_mask.png\"), \"rb\"),  # from right above\n    prompt=\"Brilliant leather Lederhosen with a formal look, detailed, intricate, photorealistic\",  # provide a prompt to fill the space\n    n=3,\n    size=\"1024x1024\",\n    response_format=\"url\",\n)\n\nedit_filepaths = process_dalle_images(edit_response, \"edits\", edit_image_dir)\n\n```\n\n\n```python\n# Display your beautiful creations!\n%matplotlib inline\n\n# figure size in inches optional\nrcParams[\"figure.figsize\"] = 11 ,8\n\n# read images\nimg_A = mpimg.imread(edit_filepaths[0])\nimg_B = mpimg.imread(edit_filepaths[1])\nimg_C = mpimg.imread(edit_filepaths[2])\n\n# display images\nfig, ax = plt.subplots(1,3)\n[a.axis(\"off\") for a in ax]\nax[0].imshow(img_A)\nax[1].imshow(img_B)\nax[2].imshow(img_C)\n```\n\n\n\n\n    <matplotlib.image.AxesImage at 0x791b1f4c58a0>\n\n\n\n\n    \n![png](output_23_1.png)\n    \n\n\nBeautiful!\n\nNow you too can easily create dynamic masks to extend your images - enjoy the APIs, and please share what you build!"}


