# embedding_metadata

key: chroma:document
string_value:
```python
answer_question(olympics_search_fileid, "davinci-instruct-beta-v3", 
            "Where did women's 4 x 100 metres relay event take place during the 2020 Summer Olympics?")
```




    ' Japan National Stadium'



After we fine-tune the model for Q&A we'll be able to use it instead of [`davinci-instruct-beta-v3`](https://beta.openai.com/docs/engines/instruct-series-beta), to obtain better answers when the question can't be answered based on the context. We see a downside of [`davinci-instruct-beta-v3`](https://beta.openai.com/docs/engines/instruct-series-beta), which always attempts to answer the question, regardless of the relevant context being present or not. (Note the second question is asking about a future event, set in 2024.)


```python
answer_question(olympics_search_fileid, "davinci-instruct-beta-v3", 
            "Where did women's 4 x 100 metres relay event take place during the 2048 Summer Olympics?", max_len=1000)
```




    ' Japan National Stadium'



We can see that davinci has a tendency to answer the question, even if the question can't be answered given the context provided. Note the question asked regarding 2048 Summer Olympics, which didn't happen yet, and the retrieved content has only returned results for 2020.

## 2.7 (Optional) Investigation into how likely the search endpoint is to return the relevant context


```python
def check_context(title, heading, question, max_len=1800, search_model='ada', max_rerank=10):
    """
    Evaluate the performance of the search model in retrieving the correct context

    Parameters
    ----------
    title: str
        The title of the Wikipedia page
    heading: str
        The heading of the Wikipedia section
    qusetion: str
        The question
    max_len: int
        The maximum length of the context
    search_model: str
        The search model to use - `ada` is most cost effective
    max_rerank: int
        The maximum number of reranking documents to use the search model on

    Returns
    -------
    rank: int
        The rank of the correct context
    token_length: int
        The number of tokens needed to obtain the correct context
    """
    
    try:
        # TODO: openai.Engine(search_model) is deprecated
        results = openai.Engine(search_model).search(
            search_model=search_model, 
            query=question, 
            max_rerank=max_rerank,
            file=olympics_search_fileid,
            return_metadata=True
        )
        index=-1
        returns = []
        cur_len = 0
        for result in results['data']:
            cur_len += int(result['metadata']) + 4 # we add 4 tokens for the separator `\n\n###\n\n`
            if cur_len > max_len:
                break
            returns.append(result['text'])
            res = result['text'].split('\n')
            if res[0] == title and res[1] == heading:
                index = len(returns) - 1
                break
        return index, cur_len
    except Exception as e:
        #print (e)
        return []
print(check_context("Athletics at the 2020 Summer Olympics – Women's 4 × 100 metres relay", "Summary", "Where did women's 4 x 100 metres relay event take place during the 2020 Summer Olympics?", max_len=10000))
```

    (0, 58)


key: _node_content
string_value:
{"id_": "85190d63-ff96-4fba-976b-c1ee493f1348", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb", "title": "2. Creating a synthetic Q&A dataset", "tokens": 5416, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "da500b3c-9093-4acb-af8a-9bdf5a59d788", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb", "title": "2. Creating a synthetic Q&A dataset", "tokens": 5416, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1dace799e477d710c8a5f5e07dd9eb059b8ce0b711356d9bc729f127541dac5b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3d6b2832-b1d6-4a3a-bdf5-cdff5f08ae8c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb", "title": "2. Creating a synthetic Q&A dataset", "tokens": 5416, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "8bd676687634852a393618a379893324df0f97d472cc602aa9022ca3cbe3f643", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2c0cce0e-fb26-4f3e-98a4-550d80dd05a3", "node_type": "1", "metadata": {}, "hash": "068a6b7ab450d88ef40c83ff16b16f5c60058205e74bc92522d5612de1a2dfa8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17346, "end_char_idx": 20158, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We utilize the generated questions based on context to estimate how often we can retrieve the original context. These questions are noisy, so this is not a perfect estimate.

Our questions and answers are prefixed with numbered bullet points, however due to the way they were generated, they are missing the first number, hence we add "1." to the list of questions (and answers).

We calculate the rank of the section retrieved using ada search, and the number of tokens in the context needed to retrieve the relevant section in full.


```python
ada_results = df.apply(lambda x: [
                    check_context( x.title, 
                                   x.heading, 
                                   q[3:],     # remove the number prefix
                                   max_len=1000000, # set a large number to get the full context 
                                   search_model='ada', 
                                   max_rerank=200,
                                 ) 
                    for q in (x.questions).split('\n') # split the questions
                    if len(q) >10 # remove the empty questions
                ], axis=1)
ada_results.head()
```




    0    [(132, 27104), (-1, 22939), (8, 2151), (2, 121...
    1    [(4, 1737), (0, 130), (8, 744), (96, 17208), (...
    2          [(0, 373), (0, 373), (-1, 40610), (1, 570)]
    3            [(0, 302), (0, 302), (5, 968), (8, 1425)]
    4                      [(0, 167), (0, 167), (2, 1442)]
    Name: ada, dtype: object




```python
out = pd.concat([ada_results], axis=1)
out.columns = ['ada']
out.to_csv('olympics-data/search_engine_results.csv')
```


```python
def expand_lists(out):
    """
    Expand a pandas series containing lists into a series, where each list element becomes a value on its own

    Input is a row per paragraph, which has multiple questions
    Output is a row per question
    """
    cols = [pd.DataFrame(out[name].tolist()).stack().reset_index(level=1, drop=True).rename(name) for name in out.columns] 
    return pd.concat(cols, axis=1)

out_expanded = expand_lists(out)
out_expanded['rank'] = out_expanded.ada.apply(lambda x: x[0] if x != [] else -2)
out_expanded['tokens'] = out_expanded.ada.apply(lambda x: x[1] if x != [] else -2)

```


```python
within_2k = (out_expanded.tokens < 2000).mean()
print(f"{within_2k*100:.1f}% of relevant paragraphs are retrieved within the first 2k tokens")
```

    74.3% of relevant paragraphs are retrieved within the first 2k tokens


The relevant context can be obtained 74% of the time on this dataset


```python
outside_200 = (out_expanded['rank'] == -1).mean()
print(f"{outside_200*100:.1f}% of relevant paragraphs are not retrieved within the first 200 results")
```

    7.4% of relevant paragraphs are not retrieved within the first 200 results


key: _node_content
string_value:
{"id_": "2c0cce0e-fb26-4f3e-98a4-550d80dd05a3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb", "title": "2. Creating a synthetic Q&A dataset", "tokens": 5416, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "da500b3c-9093-4acb-af8a-9bdf5a59d788", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb", "title": "2. Creating a synthetic Q&A dataset", "tokens": 5416, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1dace799e477d710c8a5f5e07dd9eb059b8ce0b711356d9bc729f127541dac5b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "85190d63-ff96-4fba-976b-c1ee493f1348", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb", "title": "2. Creating a synthetic Q&A dataset", "tokens": 5416, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b2999c2b87a8b26ccbd9eb0d69a8327acc499eb67f160ce82bd6ba16f315601b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "850aedcc-ac8e-4815-828c-259c2d9e05fc", "node_type": "1", "metadata": {}, "hash": "062d44d0693558cf08d89eb8e656f1b0c99a05740d3b694619f9f64e1b71fe6e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18856, "end_char_idx": 21286, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
out = pd.concat([ada_results], axis=1)
out.columns = ['ada']
out.to_csv('olympics-data/search_engine_results.csv')
```


```python
def expand_lists(out):
    """
    Expand a pandas series containing lists into a series, where each list element becomes a value on its own

    Input is a row per paragraph, which has multiple questions
    Output is a row per question
    """
    cols = [pd.DataFrame(out[name].tolist()).stack().reset_index(level=1, drop=True).rename(name) for name in out.columns] 
    return pd.concat(cols, axis=1)

out_expanded = expand_lists(out)
out_expanded['rank'] = out_expanded.ada.apply(lambda x: x[0] if x != [] else -2)
out_expanded['tokens'] = out_expanded.ada.apply(lambda x: x[1] if x != [] else -2)

```


```python
within_2k = (out_expanded.tokens < 2000).mean()
print(f"{within_2k*100:.1f}% of relevant paragraphs are retrieved within the first 2k tokens")
```

    74.3% of relevant paragraphs are retrieved within the first 2k tokens


The relevant context can be obtained 74% of the time on this dataset


```python
outside_200 = (out_expanded['rank'] == -1).mean()
print(f"{outside_200*100:.1f}% of relevant paragraphs are not retrieved within the first 200 results")
```

    7.4% of relevant paragraphs are not retrieved within the first 200 results


7.4% of the time, this is due to the keyword search part of the search algorithm not retrieving the relevant context within the first 200 results.
18.3% of the time this is due to the semantic search not placing the relevant context within the first 2000 tokens.


```python
import matplotlib.pyplot as plt

# plot a histogram, and add axis descriptions and title
out_expanded[(out_expanded['rank'] >=0)&(out_expanded['rank'] <30)]['rank'].hist(bins=29)
plt.xlabel('rank')
plt.ylabel('count')
plt.title('Histogram of ranks of retrieved paragraphs')
plt.show()
```


    
![png](output_31_0.png)
    



```python
out_expanded[(out_expanded.tokens>=0)&(out_expanded.tokens < 2000)]['tokens'].hist(bins=29)
plt.xlabel('tokens')
plt.ylabel('count')
plt.title('Histogram of the number of minimum tokens needed')
plt.show()
```


    
![png](output_32_0.png)
    


We can observe that the context is most likely to be returned as one of the first results, and most likely to be returned within the first 200-500 tokens.


```python
# normalized value_counts
out_expanded['rank'].value_counts(normalize=True).sort_index()[:13]
```


key: _node_content
string_value:
{"id_": "850aedcc-ac8e-4815-828c-259c2d9e05fc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb", "title": "2. Creating a synthetic Q&A dataset", "tokens": 5416, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "da500b3c-9093-4acb-af8a-9bdf5a59d788", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb", "title": "2. Creating a synthetic Q&A dataset", "tokens": 5416, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1dace799e477d710c8a5f5e07dd9eb059b8ce0b711356d9bc729f127541dac5b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2c0cce0e-fb26-4f3e-98a4-550d80dd05a3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/olympics-2-create-qa.ipynb", "title": "2. Creating a synthetic Q&A dataset", "tokens": 5416, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a00993fda6ae1f8d6c8c7836490516b49574f6ec100efc9b34bfb90cb2e562b2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19841, "end_char_idx": 21708, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The relevant context can be obtained 74% of the time on this dataset


```python
outside_200 = (out_expanded['rank'] == -1).mean()
print(f"{outside_200*100:.1f}% of relevant paragraphs are not retrieved within the first 200 results")
```

    7.4% of relevant paragraphs are not retrieved within the first 200 results


7.4% of the time, this is due to the keyword search part of the search algorithm not retrieving the relevant context within the first 200 results.
18.3% of the time this is due to the semantic search not placing the relevant context within the first 2000 tokens.


```python
import matplotlib.pyplot as plt

# plot a histogram, and add axis descriptions and title
out_expanded[(out_expanded['rank'] >=0)&(out_expanded['rank'] <30)]['rank'].hist(bins=29)
plt.xlabel('rank')
plt.ylabel('count')
plt.title('Histogram of ranks of retrieved paragraphs')
plt.show()
```


    
![png](output_31_0.png)
    



```python
out_expanded[(out_expanded.tokens>=0)&(out_expanded.tokens < 2000)]['tokens'].hist(bins=29)
plt.xlabel('tokens')
plt.ylabel('count')
plt.title('Histogram of the number of minimum tokens needed')
plt.show()
```


    
![png](output_32_0.png)
    


We can observe that the context is most likely to be returned as one of the first results, and most likely to be returned within the first 200-500 tokens.


```python
# normalized value_counts
out_expanded['rank'].value_counts(normalize=True).sort_index()[:13]
```




    -2     0.000063
    -1     0.074428
     0     0.453420
     1     0.089515
     2     0.047146
     3     0.032437
     4     0.024139
     5     0.019676
     6     0.015967
     7     0.013452
     8     0.011189
     9     0.009869
     10    0.009178
    Name: rank, dtype: float64



probabilities of the relevant context being returned at each rank. (-2 means a processing error, -1 means the rank is >200)


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb


key: title
string_value:
Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning


key: _node_content
string_value:
{"id_": "da298b28-f39e-41b7-97b4-213b9a918b83", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0aa5bf93-3c4c-428b-9e66-66d151e7975a", "node_type": "1", "metadata": {}, "hash": "bd6309d85754359747104e4d5a7c9b5f0a223073ce49b3d0e67a24c91449ebb3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3009, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning

The aim of this notebook is to walk through a comprehensive example of how to fine-tune OpenAI models for Retrieval Augmented Generation (RAG). 

We will also be integrating Qdrant and Few-Shot Learning to boost the model's performance and reduce hallucinations. This could serve as a practical guide for ML practitioners, data scientists, and AI Engineers interested in leveraging the power of OpenAI models for specific use-cases. 🤩

## Why should you read this blog?

You want to learn how to 
- [Fine-tune OpenAI models](https://platform.openai.com/docs/guides/fine-tuning/) for specific use-cases
- Use [Qdrant](https://qdrant.tech/documentation/) to improve the performance of your RAG model
- Use fine-tuning to improve the correctness of your RAG model and reduce hallucinations

To begin, we've selected a dataset where we've a guarantee that the retrieval is perfect. We've selected a subset of the [SQuAD](https://rajpurkar.github.io/SQuAD-explorer/) dataset, which is a collection of questions and answers about Wikipedia articles. We've also included samples where the answer is not present in the context, to demonstrate how RAG handles this case.

## Table of Contents
1. Setting up the Environment

### Section A: Zero-Shot Learning
2. Data Preparation: SQuADv2 Dataset
3. Answering using Base gpt-3.5-turbo-0613 model
4. Fine-tuning and Answering using Fine-tuned model
5. **Evaluation**: How well does the model perform?

### Section B: Few-Shot Learning

6. Using Qdrant to Improve RAG Prompt
7. Fine-Tuning OpenAI Model with Qdrant
8. Evaluation

9. **Conclusion**
    - Aggregate Results
    - Observations

## Terms, Definitions, and References

**Retrieval Augmented Generation (RAG)?**
The phrase Retrieval Augmented Generation (RAG) comes from a [recent paper](https://arxiv.org/abs/2005.11401) by Lewis et al. from Facebook AI. The idea is to use a pre-trained language model (LM) to generate text, but to use a separate retrieval system to find relevant documents to condition the LM on. 

**What is Qdrant?**
Qdrant is an open-source vector search engine that allows you to search for similar vectors in a large dataset. It is built in Rust and here we'll use the Python client to interact with it. This is the Retrieval part of RAG.

**What is Few-Shot Learning?**
Few-shot learning is a type of machine learning where the model is "improved" via training or fine-tuning on a small amount of data. In this case, we'll use it to fine-tune the RAG model on a small number of examples from the SQuAD dataset. This is the Augmented part of RAG.

**What is Zero-Shot Learning?**
Zero-shot learning is a type of machine learning where the model is "improved" via training or fine-tuning without any dataset specific information. 

**What is Fine-Tuning?**
Fine-tuning is a type of machine learning where the model is "improved" via training or fine-tuning on a small amount of data.


key: _node_content
string_value:
{"id_": "0aa5bf93-3c4c-428b-9e66-66d151e7975a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "da298b28-f39e-41b7-97b4-213b9a918b83", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "61118c1d33336410355df7f7601489f39754bb03adc12f3fb442d2a6a79312b9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "75819f84-aa1f-414b-a3fb-f51340a02864", "node_type": "1", "metadata": {}, "hash": "19882e9be2e5a9df2b233c2312f3da3ffc7779d8b9286d05594df5ddda2a7578", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1442, "end_char_idx": 4059, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Fine-tuning and Answering using Fine-tuned model
5. **Evaluation**: How well does the model perform?

### Section B: Few-Shot Learning

6. Using Qdrant to Improve RAG Prompt
7. Fine-Tuning OpenAI Model with Qdrant
8. Evaluation

9. **Conclusion**
    - Aggregate Results
    - Observations

## Terms, Definitions, and References

**Retrieval Augmented Generation (RAG)?**
The phrase Retrieval Augmented Generation (RAG) comes from a [recent paper](https://arxiv.org/abs/2005.11401) by Lewis et al. from Facebook AI. The idea is to use a pre-trained language model (LM) to generate text, but to use a separate retrieval system to find relevant documents to condition the LM on. 

**What is Qdrant?**
Qdrant is an open-source vector search engine that allows you to search for similar vectors in a large dataset. It is built in Rust and here we'll use the Python client to interact with it. This is the Retrieval part of RAG.

**What is Few-Shot Learning?**
Few-shot learning is a type of machine learning where the model is "improved" via training or fine-tuning on a small amount of data. In this case, we'll use it to fine-tune the RAG model on a small number of examples from the SQuAD dataset. This is the Augmented part of RAG.

**What is Zero-Shot Learning?**
Zero-shot learning is a type of machine learning where the model is "improved" via training or fine-tuning without any dataset specific information. 

**What is Fine-Tuning?**
Fine-tuning is a type of machine learning where the model is "improved" via training or fine-tuning on a small amount of data. In this case, we'll use it to fine-tune the RAG model on a small number of examples from the SQuAD dataset. The LLM is what makes the Generation part of RAG.

## 1. Setting Up the Environment

### Install and Import Dependencies


```python
!pip install pandas openai tqdm tenacity scikit-learn tiktoken python-dotenv seaborn --upgrade --quiet
```


```python
import json
import os
import time

import pandas as pd
from openai import OpenAI
import tiktoken
import seaborn as sns
from tenacity import retry, wait_exponential
from tqdm import tqdm
from collections import defaultdict
import numpy as np
import matplotlib.pyplot as plt
import numpy as np
from sklearn.metrics import confusion_matrix

import warnings
warnings.filterwarnings('ignore')

tqdm.pandas()

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))

```

### Set your keys
Get your OpenAI keys [here](https://platform.openai.com/account/api-keys) and Qdrant keys after making a free cluster [here](https://cloud.qdrant.io/login).


key: _node_content
string_value:
{"id_": "75819f84-aa1f-414b-a3fb-f51340a02864", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0aa5bf93-3c4c-428b-9e66-66d151e7975a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "42613a99e181198e4c06b400f4efb7b57eea9a7720aea409e94e3878bf080343", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d0bba0bf-613a-4d4a-8e3e-695ca7a1775f", "node_type": "1", "metadata": {}, "hash": "7afdd8b85611d4f227050ddcbfda91240c9004a269ef33b06ba69acbac0892b5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2531, "end_char_idx": 4967, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
In this case, we'll use it to fine-tune the RAG model on a small number of examples from the SQuAD dataset. This is the Augmented part of RAG.

**What is Zero-Shot Learning?**
Zero-shot learning is a type of machine learning where the model is "improved" via training or fine-tuning without any dataset specific information. 

**What is Fine-Tuning?**
Fine-tuning is a type of machine learning where the model is "improved" via training or fine-tuning on a small amount of data. In this case, we'll use it to fine-tune the RAG model on a small number of examples from the SQuAD dataset. The LLM is what makes the Generation part of RAG.

## 1. Setting Up the Environment

### Install and Import Dependencies


```python
!pip install pandas openai tqdm tenacity scikit-learn tiktoken python-dotenv seaborn --upgrade --quiet
```


```python
import json
import os
import time

import pandas as pd
from openai import OpenAI
import tiktoken
import seaborn as sns
from tenacity import retry, wait_exponential
from tqdm import tqdm
from collections import defaultdict
import numpy as np
import matplotlib.pyplot as plt
import numpy as np
from sklearn.metrics import confusion_matrix

import warnings
warnings.filterwarnings('ignore')

tqdm.pandas()

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))

```

### Set your keys
Get your OpenAI keys [here](https://platform.openai.com/account/api-keys) and Qdrant keys after making a free cluster [here](https://cloud.qdrant.io/login).


```python
os.environ["QDRANT_URL"] = "https://xxx.cloud.qdrant.io:6333"
os.environ["QDRANT_API_KEY"] = "xxx"
```

## Section A

## 2. Data Preparation: SQuADv2 Data Subsets

For the purpose of demonstration, we'll make small slices from the train and validation splits of the [SQuADv2](https://rajpurkar.github.io/SQuAD-explorer/) dataset. This dataset has questions and contexts where the answer is not present in the context, to help us evaluate how LLM handles this case.

We'll read the data from the JSON files and create a dataframe with the following columns: `question`, `context`, `answer`, `is_impossible`.

### Download the Data


```python
# !mkdir -p local_cache
# !wget https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v2.0.json -O local_cache/train.json
# !wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v2.0.json -O local_cache/dev.json
```

### Read JSON to DataFrame


key: _node_content
string_value:
{"id_": "d0bba0bf-613a-4d4a-8e3e-695ca7a1775f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "75819f84-aa1f-414b-a3fb-f51340a02864", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "de7283f094a9bf414bd7690cdd73c04ea16d54534dbf794ef19e1fff772d508d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a19f8c2c-444d-4faa-89ee-34f72158fcbe", "node_type": "1", "metadata": {}, "hash": "a42d924a367cff538724c55c136c21c210e8bfa39cf70b01b6d4e16bc4a83f25", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4062, "end_char_idx": 7384, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
os.environ["QDRANT_URL"] = "https://xxx.cloud.qdrant.io:6333"
os.environ["QDRANT_API_KEY"] = "xxx"
```

## Section A

## 2. Data Preparation: SQuADv2 Data Subsets

For the purpose of demonstration, we'll make small slices from the train and validation splits of the [SQuADv2](https://rajpurkar.github.io/SQuAD-explorer/) dataset. This dataset has questions and contexts where the answer is not present in the context, to help us evaluate how LLM handles this case.

We'll read the data from the JSON files and create a dataframe with the following columns: `question`, `context`, `answer`, `is_impossible`.

### Download the Data


```python
# !mkdir -p local_cache
# !wget https://rajpurkar.github.io/SQuAD-explorer/dataset/train-v2.0.json -O local_cache/train.json
# !wget https://rajpurkar.github.io/SQuAD-explorer/dataset/dev-v2.0.json -O local_cache/dev.json
```

### Read JSON to DataFrame


```python
def json_to_dataframe_with_titles(json_data):
    qas = []
    context = []
    is_impossible = []
    answers = []
    titles = []

    for article in json_data['data']:
        title = article['title']
        for paragraph in article['paragraphs']:
            for qa in paragraph['qas']:
                qas.append(qa['question'].strip())
                context.append(paragraph['context'])
                is_impossible.append(qa['is_impossible'])
                
                ans_list = []
                for ans in qa['answers']:
                    ans_list.append(ans['text'])
                answers.append(ans_list)
                titles.append(title)

    df = pd.DataFrame({'title': titles, 'question': qas, 'context': context, 'is_impossible': is_impossible, 'answers': answers})
    return df

def get_diverse_sample(df, sample_size=100, random_state=42):
    """
    Get a diverse sample of the dataframe by sampling from each title
    """
    sample_df = df.groupby(['title', 'is_impossible']).apply(lambda x: x.sample(min(len(x), max(1, sample_size // 50)), random_state=random_state)).reset_index(drop=True)
    
    if len(sample_df) < sample_size:
        remaining_sample_size = sample_size - len(sample_df)
        remaining_df = df.drop(sample_df.index).sample(remaining_sample_size, random_state=random_state)
        sample_df = pd.concat([sample_df, remaining_df]).sample(frac=1, random_state=random_state).reset_index(drop=True)

    return sample_df.sample(min(sample_size, len(sample_df)), random_state=random_state).reset_index(drop=True)

train_df = json_to_dataframe_with_titles(json.load(open('local_cache/train.json')))
val_df = json_to_dataframe_with_titles(json.load(open('local_cache/dev.json')))

df = get_diverse_sample(val_df, sample_size=100, random_state=42)
```

## 3. Answering using Base gpt-3.5-turbo-0613 model

### 3.1 Zero Shot Prompt

Let's start by using the base gpt-3.5-turbo-0613 model to answer the questions. This prompt is a simple concatenation of the question and context, with a separator token in between: `\n\n`. We've a simple instruction part of the prompt: 

> Answer the following Question based on the Context only. Only answer from the Context. If you don't know the answer, say 'I don't know'.

Other prompts are possible, but this is a good starting point. We'll use this prompt to answer the questions in the validation set.


key: _node_content
string_value:
{"id_": "a19f8c2c-444d-4faa-89ee-34f72158fcbe", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d0bba0bf-613a-4d4a-8e3e-695ca7a1775f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "765fa4aa1ab28f05589ee33dac6f19c268514fcb948909da44f2c9fea4e13bea", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8a736ffd-4cba-48f9-bc30-9394a08dace8", "node_type": "1", "metadata": {}, "hash": "5ec93f2cc416bf59589b037262979f8372d781ef569a34f7ad36309f7d723244", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Function to get prompt messages
def get_prompt(row):
    return [
        {"role": "system", "content": "You are a helpful assistant."},
        {
            "role": "user",
            "content": f"""Answer the following Question based on the Context only. Only answer from the Context. If you don't know the answer, say 'I don't know'.
    Question: {row.question}\n\n
    Context: {row.context}\n\n
    Answer:\n""",
        },
    ]
```

### 3.2 Answering using Zero Shot Prompt

Next, you'll need some re-usable functions which make an OpenAI API Call and return the answer. You'll use the `ChatCompletion.create` endpoint of the API, which takes a prompt and returns the completed text.


```python
# Function with tenacity for retries
@retry(wait=wait_exponential(multiplier=1, min=2, max=6))
def api_call(messages, model):
    return client.chat.completions.create(
        model=model,
        messages=messages,
        stop=["\n\n"],
        max_tokens=100,
        temperature=0.0,
    )


# Main function to answer question
def answer_question(row, prompt_func=get_prompt, model="gpt-3.5-turbo"):
    messages = prompt_func(row)
    response = api_call(messages, model)
    return response.choices[0].message.content
```

⏰ **Time to run: ~3 min**, 🛜 Needs Internet Connection


```python
# Use progress_apply with tqdm for progress bar
df["generated_answer"] = df.progress_apply(answer_question, axis=1)
df.to_json("local_cache/100_val.json", orient="records", lines=True)
df = pd.read_json("local_cache/100_val.json", orient="records", lines=True)
```


```python
df
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>title</th>
      <th>question</th>
      <th>context</th>
      <th>is_impossible</th>
      <th>answers</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Scottish_Parliament</td>
      <td>What consequence of establishing the Scottish .</td>
      <td>A procedural consequence of the establishment .</td>
      <td>False</td>
      <td>[able to vote on domestic legislation that app.</td>
    </tr>
    <tr>
      <th>1</th>
      <td>Imperialism</td>
      <td>Imperialism is less often associated with whic.</td>
      <td>The principles of imperialism are often genera.</td>
      <td>True</td>
      <td>[]</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Economic_inequality</td>
      <td>What issues can't prevent women from working o.</td>
      <td>When a person’s capabilities are lowered, they.


key: _node_content
string_value:
{"id_": "8a736ffd-4cba-48f9-bc30-9394a08dace8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a19f8c2c-444d-4faa-89ee-34f72158fcbe", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "cc61c7e97b40e65e9fd44e397cdb220ba4f1566393097d81084d00257d0fc808", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2ca55eb5-7756-46a9-867e-609b179e2a95", "node_type": "1", "metadata": {}, "hash": "efbbedffe65f71d470b5caef81673f53f61d570a03f7d2c364ec3ec1a03c3694", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
df
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>title</th>
      <th>question</th>
      <th>context</th>
      <th>is_impossible</th>
      <th>answers</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Scottish_Parliament</td>
      <td>What consequence of establishing the Scottish .</td>
      <td>A procedural consequence of the establishment .</td>
      <td>False</td>
      <td>[able to vote on domestic legislation that app.</td>
    </tr>
    <tr>
      <th>1</th>
      <td>Imperialism</td>
      <td>Imperialism is less often associated with whic.</td>
      <td>The principles of imperialism are often genera.</td>
      <td>True</td>
      <td>[]</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Economic_inequality</td>
      <td>What issues can't prevent women from working o.</td>
      <td>When a person’s capabilities are lowered, they.</td>
      <td>True</td>
      <td>[]</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Southern_California</td>
      <td>What county are Los Angeles, Orange, San Diego.</td>
      <td>Its counties of Los Angeles, Orange, San Diego.</td>
      <td>True</td>
      <td>[]</td>
    </tr>
    <tr>
      <th>4</th>
      <td>French_and_Indian_War</td>
      <td>When was the deportation of Canadians?</td>
      <td>Britain gained control of French Canada and Ac.</td>
      <td>True</td>
      <td>[]</td>
    </tr>
    <tr>
      <th>.</th>
      <td>.</td>
      <td>.</td>
      <td>.</td>
      <td>.</td>
      <td>.</td>
    </tr>
    <tr>
      <th>95</th>
      <td>Geology</td>
      <td>In the layered Earth model, what is the inner .</td>
      <td>Seismologists can use the arrival times of sei.</td>
      <td>True</td>
      <td>[]</td>
    </tr>
    <tr>
      <th>96</th>
      <td>Prime_number</td>
      <td>What type of value would the Basel function ha.</td>
      <td>The zeta function is closely related to prime .</td>
      <td>True</td>
      <td>[]</td>
    </tr>
    <tr>
      <th>97</th>
      <td>Fresno,


key: _node_content
string_value:
{"id_": "2ca55eb5-7756-46a9-867e-609b179e2a95", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8a736ffd-4cba-48f9-bc30-9394a08dace8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "7831e60d017808890151e9f3db8332b39850fe0fc7498744d2ac6832cbfbf2d5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d34d29a2-8a9c-497f-a7fd-e25ed5768a2c", "node_type": "1", "metadata": {}, "hash": "5e4794ada91c8216cc43a36727992fa6a9e01640642155da2528350b4fd6f811", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
they.</td>
      <td>True</td>
      <td>[]</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Southern_California</td>
      <td>What county are Los Angeles, Orange, San Diego.</td>
      <td>Its counties of Los Angeles, Orange, San Diego.</td>
      <td>True</td>
      <td>[]</td>
    </tr>
    <tr>
      <th>4</th>
      <td>French_and_Indian_War</td>
      <td>When was the deportation of Canadians?</td>
      <td>Britain gained control of French Canada and Ac.</td>
      <td>True</td>
      <td>[]</td>
    </tr>
    <tr>
      <th>.</th>
      <td>.</td>
      <td>.</td>
      <td>.</td>
      <td>.</td>
      <td>.</td>
    </tr>
    <tr>
      <th>95</th>
      <td>Geology</td>
      <td>In the layered Earth model, what is the inner .</td>
      <td>Seismologists can use the arrival times of sei.</td>
      <td>True</td>
      <td>[]</td>
    </tr>
    <tr>
      <th>96</th>
      <td>Prime_number</td>
      <td>What type of value would the Basel function ha.</td>
      <td>The zeta function is closely related to prime .</td>
      <td>True</td>
      <td>[]</td>
    </tr>
    <tr>
      <th>97</th>
      <td>Fresno,_California</td>
      <td>What does the San Joaquin Valley Railroad cros.</td>
      <td>Passenger rail service is provided by Amtrak S.</td>
      <td>True</td>
      <td>[]</td>
    </tr>
    <tr>
      <th>98</th>
      <td>Victoria_(Australia)</td>
      <td>What party rules in Melbourne's inner regions?</td>
      <td>The centre-left Australian Labor Party (ALP), .</td>
      <td>False</td>
      <td>[The Greens, Australian Greens, Greens]</td>
    </tr>
    <tr>
      <th>99</th>
      <td>Immune_system</td>
      <td>The speed of the killing response of the human.</td>
      <td>In humans, this response is activated by compl.</td>
      <td>False</td>
      <td>[signal amplification, signal amplification, s.</td>
    </tr>
  </tbody>
</table>
<p>100 rows × 5 columns</p>
</div>


key: _node_content
string_value:
{"id_": "d34d29a2-8a9c-497f-a7fd-e25ed5768a2c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2ca55eb5-7756-46a9-867e-609b179e2a95", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f27232efe20d552226e4ac24ad7241b125f8ae209afec0ddb844d2fef34c2cea", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2df108ed-d301-4eea-b41f-b33bd95a7b19", "node_type": "1", "metadata": {}, "hash": "57ff9005f07cbfaec771da9f4ba06230b120d978ef00ec573a97af03aa38c9f8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
what is the inner .</td>
      <td>Seismologists can use the arrival times of sei.</td>
      <td>True</td>
      <td>[]</td>
    </tr>
    <tr>
      <th>96</th>
      <td>Prime_number</td>
      <td>What type of value would the Basel function ha.</td>
      <td>The zeta function is closely related to prime .</td>
      <td>True</td>
      <td>[]</td>
    </tr>
    <tr>
      <th>97</th>
      <td>Fresno,_California</td>
      <td>What does the San Joaquin Valley Railroad cros.</td>
      <td>Passenger rail service is provided by Amtrak S.</td>
      <td>True</td>
      <td>[]</td>
    </tr>
    <tr>
      <th>98</th>
      <td>Victoria_(Australia)</td>
      <td>What party rules in Melbourne's inner regions?</td>
      <td>The centre-left Australian Labor Party (ALP), .</td>
      <td>False</td>
      <td>[The Greens, Australian Greens, Greens]</td>
    </tr>
    <tr>
      <th>99</th>
      <td>Immune_system</td>
      <td>The speed of the killing response of the human.</td>
      <td>In humans, this response is activated by compl.</td>
      <td>False</td>
      <td>[signal amplification, signal amplification, s.</td>
    </tr>
  </tbody>
</table>
<p>100 rows × 5 columns</p>
</div>



## 4. Fine-tuning and Answering using Fine-tuned model

For the complete fine-tuning process, please refer to the [OpenAI Fine-Tuning Docs](https://platform.openai.com/docs/guides/fine-tuning/use-a-fine-tuned-model).

### 4.1 Prepare the Fine-Tuning Data

We need to prepare the data for fine-tuning. We'll use a few samples from train split of same dataset as before, but we'll add the answer to the context. This will help the model learn to retrieve the answer from the context. 

Our instruction prompt is the same as before, and so is the system prompt.


key: _node_content
string_value:
{"id_": "2df108ed-d301-4eea-b41f-b33bd95a7b19", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d34d29a2-8a9c-497f-a7fd-e25ed5768a2c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0149784cb927f00e9c54f3ef1511595e6dbbb7f74d378c00d532093881ae7a58", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6859f0c4-6246-4ea9-a8bd-9e5bae5c946e", "node_type": "1", "metadata": {}, "hash": "6f74cef954d61cb79d144ad5fdf416e110044fb0132d3092b591f38daf10636c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
_California</td>
      <td>What does the San Joaquin Valley Railroad cros.</td>
      <td>Passenger rail service is provided by Amtrak S.</td>
      <td>True</td>
      <td>[]</td>
    </tr>
    <tr>
      <th>98</th>
      <td>Victoria_(Australia)</td>
      <td>What party rules in Melbourne's inner regions?</td>
      <td>The centre-left Australian Labor Party (ALP), .</td>
      <td>False</td>
      <td>[The Greens, Australian Greens, Greens]</td>
    </tr>
    <tr>
      <th>99</th>
      <td>Immune_system</td>
      <td>The speed of the killing response of the human.</td>
      <td>In humans, this response is activated by compl.</td>
      <td>False</td>
      <td>[signal amplification, signal amplification, s.</td>
    </tr>
  </tbody>
</table>
<p>100 rows × 5 columns</p>
</div>



## 4. Fine-tuning and Answering using Fine-tuned model

For the complete fine-tuning process, please refer to the [OpenAI Fine-Tuning Docs](https://platform.openai.com/docs/guides/fine-tuning/use-a-fine-tuned-model).

### 4.1 Prepare the Fine-Tuning Data

We need to prepare the data for fine-tuning. We'll use a few samples from train split of same dataset as before, but we'll add the answer to the context. This will help the model learn to retrieve the answer from the context. 

Our instruction prompt is the same as before, and so is the system prompt. 


```python
def dataframe_to_jsonl(df):
    def create_jsonl_entry(row):
        answer = row["answers"][0] if row["answers"] else "I don't know"
        messages = [
            {"role": "system", "content": "You are a helpful assistant."},
            {
                "role": "user",
                "content": f"""Answer the following Question based on the Context only. Only answer from the Context. If you don't know the answer, say 'I don't know'.
            Question: {row.question}\n\n
            Context: {row.context}\n\n
            Answer:\n""",
            },
            {"role": "assistant", "content": answer},
        ]
        return json.dumps({"messages": messages})

    jsonl_output = df.apply(create_jsonl_entry, axis=1)
    return "\n".join(jsonl_output)

train_sample = get_diverse_sample(train_df, sample_size=100, random_state=42)

with open("local_cache/100_train.jsonl", "w") as f:
    f.write(dataframe_to_jsonl(train_sample))
```

**Tip: 💡 Verify the Fine-Tuning Data**

You can see this [cookbook](https://github.com/openai/openai-cookbook/blob/main/examples/Chat_finetuning_data_prep.ipynb) for more details on how to prepare the data for fine-tuning.

### 4.2 Fine-Tune OpenAI Model

If you're new to OpenAI Model Fine-Tuning, please refer to the [How to finetune Chat models](https://github.com/openai/openai-cookbook/blob/448a0595b84ced3bebc9a1568b625e748f9c1d60/examples/How_to_finetune_chat_models.ipynb) notebook. You can also refer to the [OpenAI Fine-Tuning Docs](platform.openai.com/docs/guides/fine-tuning/use-a-fine-tuned-model) for more details.


key: _node_content
string_value:
{"id_": "6859f0c4-6246-4ea9-a8bd-9e5bae5c946e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2df108ed-d301-4eea-b41f-b33bd95a7b19", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "7095abf517940c201bc7be50b51e3d08c9bf4a16ce96e616819aa410215d667b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4049c1f2-2480-4d5a-98da-bfb5d56f5dd3", "node_type": "1", "metadata": {}, "hash": "0578ada283197b1b1d56ba723adb382013efac4a40dee49e006c32e9c5db664e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14308, "end_char_idx": 17379, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
class OpenAIFineTuner:
    """
    Class to fine tune OpenAI models
    """
    def __init__(self, training_file_path, model_name, suffix):
        self.training_file_path = training_file_path
        self.model_name = model_name
        self.suffix = suffix
        self.file_object = None
        self.fine_tuning_job = None
        self.model_id = None

    def create_openai_file(self):
        self.file_object = client.files.create(
            file=open(self.training_file_path, "r"),
            purpose="fine-tune",
        )

    def wait_for_file_processing(self, sleep_time=20):
        while self.file_object.status != 'processed':
            time.sleep(sleep_time)
            self.file_object.refresh()
            print("File Status: ", self.file_object.status)

    def create_fine_tuning_job(self):
        self.fine_tuning_job = client.fine_tuning.jobs.create(
            training_file=self.file_object["id"],
            model=self.model_name,
            suffix=self.suffix,
        )

    def wait_for_fine_tuning(self, sleep_time=45):
        while self.fine_tuning_job.status != 'succeeded':
            time.sleep(sleep_time)
            self.fine_tuning_job.refresh()
            print("Job Status: ", self.fine_tuning_job.status)

    def retrieve_fine_tuned_model(self):
        self.model_id = client.fine_tuning.jobs.retrieve(self.fine_tuning_job["id"]).fine_tuned_model
        return self.model_id

    def fine_tune_model(self):
        self.create_openai_file()
        self.wait_for_file_processing()
        self.create_fine_tuning_job()
        self.wait_for_fine_tuning()
        return self.retrieve_fine_tuned_model()

fine_tuner = OpenAIFineTuner(
        training_file_path="local_cache/100_train.jsonl",
        model_name="gpt-3.5-turbo",
        suffix="100trn20230907"
    )
```

⏰ **Time to run: ~10-20 minutes**, 🛜 Needs Internet Connection


```python
model_id = fine_tuner.fine_tune_model()
model_id
```

#### 4.2.1 Try out the Fine-Tuned Model

Let's try out the fine-tuned model on the same validation set as before. You'll use the same prompt as before, but you will use the fine-tuned model instead of the base model. Before you do that, you can make a simple call to get a sense of how the fine-tuned model is doing.


```python
completion = client.chat.completions.create(
    model=model_id,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hello!"},
        {"role": "assistant", "content": "Hi, how can I help you today?"},
        {
            "role": "user",
            "content": "Can you answer the following question based on the given context? If not, say, I don't know:\n\nQuestion: What is the capital of France?\n\nContext: The capital of Mars is Gaia. Answer:",
        },
    ],
)

print(completion.choices[0].message)
```

### 4.3 Answer Using the Fine-Tuned Model

This is the same as before, but you'll use the fine-tuned model instead of the base model.

⏰ **Time to run: ~5 min**, 🛜 Needs Internet Connection


key: _node_content
string_value:
{"id_": "4049c1f2-2480-4d5a-98da-bfb5d56f5dd3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6859f0c4-6246-4ea9-a8bd-9e5bae5c946e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "a6a2d9f32a06e425ac72ac1b1126ef0d0ff325a2bb069784b19e0acdfda1b15f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ba66d43c-e36b-4c8a-9fee-4a1c8f8381a7", "node_type": "1", "metadata": {}, "hash": "72fc6e7c8ac5cfa9916d6ea58a4fa655ae496f8e23a6b2f0de42a9ec376ba68a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16211, "end_char_idx": 19255, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
model_id = fine_tuner.fine_tune_model()
model_id
```

#### 4.2.1 Try out the Fine-Tuned Model

Let's try out the fine-tuned model on the same validation set as before. You'll use the same prompt as before, but you will use the fine-tuned model instead of the base model. Before you do that, you can make a simple call to get a sense of how the fine-tuned model is doing.


```python
completion = client.chat.completions.create(
    model=model_id,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {"role": "user", "content": "Hello!"},
        {"role": "assistant", "content": "Hi, how can I help you today?"},
        {
            "role": "user",
            "content": "Can you answer the following question based on the given context? If not, say, I don't know:\n\nQuestion: What is the capital of France?\n\nContext: The capital of Mars is Gaia. Answer:",
        },
    ],
)

print(completion.choices[0].message)
```

### 4.3 Answer Using the Fine-Tuned Model

This is the same as before, but you'll use the fine-tuned model instead of the base model.

⏰ **Time to run: ~5 min**, 🛜 Needs Internet Connection


```python
df["ft_generated_answer"] = df.progress_apply(answer_question, model=model_id, axis=1)
```

## 5. Evaluation: How well does the model perform?

To evaluate the model's performance, compare the predicted answer to the actual answers -- if any of the actual answers are present in the predicted answer, then it's a match. We've also created error categories to help you understand where the model is struggling.

When we know that a correct answer exists in the context, we can measure the model's performance, there are 3 possible outcomes:

1. ✅ **Answered Correctly**: The model responded the correct answer. It may have also included other answers that were not in the context.
2. ❎ **Skipped**: The model responded with "I don't know" (IDK) while the answer was present in the context. It's better than giving the wrong answer. It's better for the model say "I don't know" than giving the wrong answer. In our design, we know that a true answer exists and hence we're able to measure it -- this is not always the case. *This is a model error*. We exclude this from the overall error rate. 
3. ❌ **Wrong**: The model responded with an incorrect answer. **This is a model ERROR.**

When we know that a correct answer does not exist in the context, we can measure the model's performance, there are 2 possible outcomes:

4. ❌ **Hallucination**: The model responded with an answer, when "I don't know" was expected. **This is a model ERROR.** 
5. ✅ **I don't know**: The model responded with "I don't know" (IDK) and the answer was not present in the context. **This is a model WIN.**


```python
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

class Evaluator:
    def __init__(self, df):
        self.df = df
        self.y_pred = pd.Series()  # Initialize as empty Series
        self.labels_answer_expected = ["✅ Answered Correctly",


key: _node_content
string_value:
{"id_": "ba66d43c-e36b-4c8a-9fee-4a1c8f8381a7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4049c1f2-2480-4d5a-98da-bfb5d56f5dd3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "a6de42308c7df856bd34ee95453ff5623c8dcf92a6cbf06434114ceba1ff9673", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7451a9f4-d6e2-47a8-96d1-61a6142b9189", "node_type": "1", "metadata": {}, "hash": "5858c6253cd9e7d175c88a6911cf65be84d65f6fcac5b570b495ea27d5656bc2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18978, "end_char_idx": 22144, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
import pandas as pd
import seaborn as sns
import matplotlib.pyplot as plt

class Evaluator:
    def __init__(self, df):
        self.df = df
        self.y_pred = pd.Series()  # Initialize as empty Series
        self.labels_answer_expected = ["✅ Answered Correctly", "❎ Skipped", "❌ Wrong Answer"]
        self.labels_idk_expected = ["❌ Hallucination", "✅ I don't know"]

    def _evaluate_answer_expected(self, row, answers_column):
        generated_answer = row[answers_column].lower()
        actual_answers = [ans.lower() for ans in row["answers"]]
        return (
            "✅ Answered Correctly" if any(ans in generated_answer for ans in actual_answers)
            else "❎ Skipped" if generated_answer == "i don't know"
            else "❌ Wrong Answer"
        )

    def _evaluate_idk_expected(self, row, answers_column):
        generated_answer = row[answers_column].lower()
        return (
            "❌ Hallucination" if generated_answer != "i don't know"
            else "✅ I don't know"
        )

    def _evaluate_single_row(self, row, answers_column):
        is_impossible = row["is_impossible"]
        return (
            self._evaluate_answer_expected(row, answers_column) if not is_impossible
            else self._evaluate_idk_expected(row, answers_column)
        )

    def evaluate_model(self, answers_column="generated_answer"):
        self.y_pred = pd.Series(self.df.apply(self._evaluate_single_row, answers_column=answers_column, axis=1))
        freq_series = self.y_pred.value_counts()
        
        # Counting rows for each scenario
        total_answer_expected = len(self.df[self.df['is_impossible'] == False])
        total_idk_expected = len(self.df[self.df['is_impossible'] == True])
        
        freq_answer_expected = (freq_series / total_answer_expected * 100).round(2).reindex(self.labels_answer_expected, fill_value=0)
        freq_idk_expected = (freq_series / total_idk_expected * 100).round(2).reindex(self.labels_idk_expected, fill_value=0)
        return freq_answer_expected.to_dict(), freq_idk_expected.to_dict()

    def print_eval(self):
        answer_columns=["generated_answer", "ft_generated_answer"]
        baseline_correctness, baseline_idk = self.evaluate_model()
        ft_correctness, ft_idk = self.evaluate_model(self.df, answer_columns[1])
        print("When the model should answer correctly:")
        eval_df = pd.merge(
            baseline_correctness.rename("Baseline"),
            ft_correctness.rename("Fine-Tuned"),
            left_index=True,
            right_index=True,
        )
        print(eval_df)
        print("\n\n\nWhen the model should say 'I don't know':")
        eval_df = pd.merge(
            baseline_idk.rename("Baseline"),
            ft_idk.rename("Fine-Tuned"),
            left_index=True,
            right_index=True,
        )
        print(eval_df)
    
    def plot_model_comparison(self, answer_columns=["generated_answer", "ft_generated_answer"], scenario="answer_expected", nice_names=["Baseline", "Fine-Tuned"]):
        
        results = []
        for col in answer_columns:
            answer_expected, idk_expected = self.


key: _node_content
string_value:
{"id_": "7451a9f4-d6e2-47a8-96d1-61a6142b9189", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ba66d43c-e36b-4c8a-9fee-4a1c8f8381a7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "d3bc307da9e6f30994b9982b1b93341a1e9cdf00dea3c30a1d51ddab2d61485d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cc078373-de7e-43f7-866f-3c5cc8844207", "node_type": "1", "metadata": {}, "hash": "9368a004517e47a9c6db873231720c30d5af3c13df7ead400b6db56a8d56920c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20428, "end_char_idx": 23521, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
answers_column=answers_column, axis=1))
        freq_series = self.y_pred.value_counts()
        
        # Counting rows for each scenario
        total_answer_expected = len(self.df[self.df['is_impossible'] == False])
        total_idk_expected = len(self.df[self.df['is_impossible'] == True])
        
        freq_answer_expected = (freq_series / total_answer_expected * 100).round(2).reindex(self.labels_answer_expected, fill_value=0)
        freq_idk_expected = (freq_series / total_idk_expected * 100).round(2).reindex(self.labels_idk_expected, fill_value=0)
        return freq_answer_expected.to_dict(), freq_idk_expected.to_dict()

    def print_eval(self):
        answer_columns=["generated_answer", "ft_generated_answer"]
        baseline_correctness, baseline_idk = self.evaluate_model()
        ft_correctness, ft_idk = self.evaluate_model(self.df, answer_columns[1])
        print("When the model should answer correctly:")
        eval_df = pd.merge(
            baseline_correctness.rename("Baseline"),
            ft_correctness.rename("Fine-Tuned"),
            left_index=True,
            right_index=True,
        )
        print(eval_df)
        print("\n\n\nWhen the model should say 'I don't know':")
        eval_df = pd.merge(
            baseline_idk.rename("Baseline"),
            ft_idk.rename("Fine-Tuned"),
            left_index=True,
            right_index=True,
        )
        print(eval_df)
    
    def plot_model_comparison(self, answer_columns=["generated_answer", "ft_generated_answer"], scenario="answer_expected", nice_names=["Baseline", "Fine-Tuned"]):
        
        results = []
        for col in answer_columns:
            answer_expected, idk_expected = self.evaluate_model(col)
            if scenario == "answer_expected":
                results.append(answer_expected)
            elif scenario == "idk_expected":
                results.append(idk_expected)
            else:
                raise ValueError("Invalid scenario")
        
        
        results_df = pd.DataFrame(results, index=nice_names)
        if scenario == "answer_expected":
            results_df = results_df.reindex(self.labels_answer_expected, axis=1)
        elif scenario == "idk_expected":
            results_df = results_df.reindex(self.labels_idk_expected, axis=1)
        
        melted_df = results_df.reset_index().melt(id_vars='index', var_name='Status', value_name='Frequency')
        sns.set_theme(style="whitegrid", palette="icefire")
        g = sns.catplot(data=melted_df, x='Frequency', y='index', hue='Status', kind='bar', height=5, aspect=2)

        # Annotating each bar
        for p in g.ax.patches:
            g.ax.annotate(f"{p.get_width():.0f}%", (p.get_width()+5, p.get_y() + p.get_height() / 2),
                        textcoords="offset points",
                        xytext=(0, 0),
                        ha='center', va='center')
        plt.ylabel("Model")
        plt.xlabel("Percentage")
        plt.xlim(0, 100)
        plt.tight_layout()
        plt.title(scenario.replace("_", " ").title())
        plt.show()


key: _node_content
string_value:
{"id_": "cc078373-de7e-43f7-866f-3c5cc8844207", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7451a9f4-d6e2-47a8-96d1-61a6142b9189", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "21d0feea35e5e6c0d36eccea2298ca6ad68d66ccfbe56bef1571f8b03d40955e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0a7b2661-49db-4fb9-a8d6-9246f0cb486a", "node_type": "1", "metadata": {}, "hash": "663f178ba2e260e7db4643b5e6bb0297e1060b07d7c8b3c1fef73f98059f028a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21836, "end_char_idx": 24602, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
)
        print(eval_df)
    
    def plot_model_comparison(self, answer_columns=["generated_answer", "ft_generated_answer"], scenario="answer_expected", nice_names=["Baseline", "Fine-Tuned"]):
        
        results = []
        for col in answer_columns:
            answer_expected, idk_expected = self.evaluate_model(col)
            if scenario == "answer_expected":
                results.append(answer_expected)
            elif scenario == "idk_expected":
                results.append(idk_expected)
            else:
                raise ValueError("Invalid scenario")
        
        
        results_df = pd.DataFrame(results, index=nice_names)
        if scenario == "answer_expected":
            results_df = results_df.reindex(self.labels_answer_expected, axis=1)
        elif scenario == "idk_expected":
            results_df = results_df.reindex(self.labels_idk_expected, axis=1)
        
        melted_df = results_df.reset_index().melt(id_vars='index', var_name='Status', value_name='Frequency')
        sns.set_theme(style="whitegrid", palette="icefire")
        g = sns.catplot(data=melted_df, x='Frequency', y='index', hue='Status', kind='bar', height=5, aspect=2)

        # Annotating each bar
        for p in g.ax.patches:
            g.ax.annotate(f"{p.get_width():.0f}%", (p.get_width()+5, p.get_y() + p.get_height() / 2),
                        textcoords="offset points",
                        xytext=(0, 0),
                        ha='center', va='center')
        plt.ylabel("Model")
        plt.xlabel("Percentage")
        plt.xlim(0, 100)
        plt.tight_layout()
        plt.title(scenario.replace("_", " ").title())
        plt.show()


# Compare the results by merging into one dataframe
evaluator = Evaluator(df)
# evaluator.evaluate_model(answers_column="ft_generated_answer")
# evaluator.plot_model_comparison(["generated_answer", "ft_generated_answer"], scenario="answer_expected", nice_names=["Baseline", "Fine-Tuned"])
```


```python
# Optionally, save the results to a JSON file
df.to_json("local_cache/100_val_ft.json", orient="records", lines=True)
df = pd.read_json("local_cache/100_val_ft.json", orient="records", lines=True)
```


```python
evaluator.plot_model_comparison(["generated_answer", "ft_generated_answer"], scenario="answer_expected", nice_names=["Baseline", "Fine-Tuned"])
```


    
![png](output_31_0.png)
    


Notice that the fine-tuned model skips questions more often -- and makes fewer mistakes. This is because the fine-tuned model is more conservative and skips questions when it's not sure.


```python
evaluator.plot_model_comparison(["generated_answer", "ft_generated_answer"], scenario="idk_expected", nice_names=["Baseline", "Fine-Tuned"])
```


    
![png](output_33_0.png)


key: _node_content
string_value:
{"id_": "0a7b2661-49db-4fb9-a8d6-9246f0cb486a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cc078373-de7e-43f7-866f-3c5cc8844207", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "3dcbb060eda08f187960285837b0473be5b5407eb8d4641456a581ae350fb26d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c7843728-7d83-484d-81e8-20d6a8bdbdcd", "node_type": "1", "metadata": {}, "hash": "c61f4fbdbc155314b5d0a1841f69a992b85edfca1e170edba8d2c6d1681c49fc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 22999, "end_char_idx": 25988, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
kind='bar', height=5, aspect=2)

        # Annotating each bar
        for p in g.ax.patches:
            g.ax.annotate(f"{p.get_width():.0f}%", (p.get_width()+5, p.get_y() + p.get_height() / 2),
                        textcoords="offset points",
                        xytext=(0, 0),
                        ha='center', va='center')
        plt.ylabel("Model")
        plt.xlabel("Percentage")
        plt.xlim(0, 100)
        plt.tight_layout()
        plt.title(scenario.replace("_", " ").title())
        plt.show()


# Compare the results by merging into one dataframe
evaluator = Evaluator(df)
# evaluator.evaluate_model(answers_column="ft_generated_answer")
# evaluator.plot_model_comparison(["generated_answer", "ft_generated_answer"], scenario="answer_expected", nice_names=["Baseline", "Fine-Tuned"])
```


```python
# Optionally, save the results to a JSON file
df.to_json("local_cache/100_val_ft.json", orient="records", lines=True)
df = pd.read_json("local_cache/100_val_ft.json", orient="records", lines=True)
```


```python
evaluator.plot_model_comparison(["generated_answer", "ft_generated_answer"], scenario="answer_expected", nice_names=["Baseline", "Fine-Tuned"])
```


    
![png](output_31_0.png)
    


Notice that the fine-tuned model skips questions more often -- and makes fewer mistakes. This is because the fine-tuned model is more conservative and skips questions when it's not sure.


```python
evaluator.plot_model_comparison(["generated_answer", "ft_generated_answer"], scenario="idk_expected", nice_names=["Baseline", "Fine-Tuned"])
```


    
![png](output_33_0.png)
    


Notice that the fine-tuned model has learnt to say "I don't know" a lot better than the prompt. Or, the model has gotten good at skipping questions.

### Observations

1. The fine-tuned model is better at saying "I don't know"
2. Hallucinations drop from 100% to 15% with fine-tuning
3. Wrong answers drop from 17% to 6% with fine-tuning

**Correct answers also drop from 83% to 60% with fine-tuning** - this is because the fine-tuned model is **more conservative** and says "I don't know" more often. This is a good thing because it's better to say "I don't know" than to give a wrong answer.

That said, we want to improve the correctness of the model, even if that increases the hallucinations. We're looking for a model that is both correct and conservative, striking a balance between the two. We'll use Qdrant and Few-Shot Learning to achieve this.

**💪 You're 2/3rds of the way there! Keep reading!**

# Section B: Few Shot Learning

We'll select a few examples from the dataset, including cases where the answer is not present in the context. We'll then use these examples to create a prompt that we can use to fine-tune the model. We'll then measure the performance of the fine-tuned model.

**What is next?**

6. Fine-Tuning OpenAI Model with Qdrant
    6.1 Embed the Fine-Tuning Data
    6.2 Embedding the Questions
7. Using Qdrant to Improve RAG Prompt
8. Evaluation


key: _node_content
string_value:
{"id_": "c7843728-7d83-484d-81e8-20d6a8bdbdcd", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0a7b2661-49db-4fb9-a8d6-9246f0cb486a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "75bd175cce004d756277e499920c3f9809d35a1c684b8d803d0d00f7f5be3f92", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8fddfd20-8b8c-4012-97e1-72090263c907", "node_type": "1", "metadata": {}, "hash": "5e5e9556d987a7242052fe0565395d8bf34e4cc5b8a3ae8b6b16e66bc0c9fec8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24579, "end_char_idx": 27505, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
![png](output_33_0.png)
    


Notice that the fine-tuned model has learnt to say "I don't know" a lot better than the prompt. Or, the model has gotten good at skipping questions.

### Observations

1. The fine-tuned model is better at saying "I don't know"
2. Hallucinations drop from 100% to 15% with fine-tuning
3. Wrong answers drop from 17% to 6% with fine-tuning

**Correct answers also drop from 83% to 60% with fine-tuning** - this is because the fine-tuned model is **more conservative** and says "I don't know" more often. This is a good thing because it's better to say "I don't know" than to give a wrong answer.

That said, we want to improve the correctness of the model, even if that increases the hallucinations. We're looking for a model that is both correct and conservative, striking a balance between the two. We'll use Qdrant and Few-Shot Learning to achieve this.

**💪 You're 2/3rds of the way there! Keep reading!**

# Section B: Few Shot Learning

We'll select a few examples from the dataset, including cases where the answer is not present in the context. We'll then use these examples to create a prompt that we can use to fine-tune the model. We'll then measure the performance of the fine-tuned model.

**What is next?**

6. Fine-Tuning OpenAI Model with Qdrant
    6.1 Embed the Fine-Tuning Data
    6.2 Embedding the Questions
7. Using Qdrant to Improve RAG Prompt
8. Evaluation


## 6. Fine-Tuning OpenAI Model with Qdrant

So far, we've been using the OpenAI model to answer questions without using examples of the answer. The previous step made it work better on in-context examples, while this one helps it generalize to unseen data, and attempt to learn when to say "I don't know" and when to give an answer.

This is where few-shot learning comes in!

Few-shot learning is a type of transfer learning that allows us to answer questions where the answer is not present in the context. We can do this by providing a few examples of the answer we're looking for, and the model will learn to answer questions where the answer is not present in the context.

### 5.1 Embed the Training Data

Embeddings are a way to represent sentences as an array of floats. We'll use the embeddings to find the most similar questions to the ones we're looking for.


```python
import os
from qdrant_client import QdrantClient
from qdrant_client.http import models
from qdrant_client.http.models import PointStruct
from qdrant_client.http.models import Distance, VectorParams
```

Now that we've the Qdrant imports in place, 


```python
qdrant_client = QdrantClient(
    url=os.getenv("QDRANT_URL"), api_key=os.getenv("QDRANT_API_KEY"), timeout=6000, prefer_grpc=True
)

collection_name = "squadv2-cookbook"

# # Create the collection, run this only once
# qdrant_client.recreate_collection(
#     collection_name=collection_name,
#     vectors_config=VectorParams(size=384, distance=Distance.COSINE),
# )
```


key: _node_content
string_value:
{"id_": "8fddfd20-8b8c-4012-97e1-72090263c907", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c7843728-7d83-484d-81e8-20d6a8bdbdcd", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "29ea85cec66274dcdac30affb7088360b012840b19a63b4f1612f854bf38a18d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a516d65a-ed72-414b-a7d3-8f55cde58bfb", "node_type": "1", "metadata": {}, "hash": "4ddbe040edf04031d558295a41f48717105f741abfdcfc4d4e680d46f6d4023a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25991, "end_char_idx": 29363, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## 6. Fine-Tuning OpenAI Model with Qdrant

So far, we've been using the OpenAI model to answer questions without using examples of the answer. The previous step made it work better on in-context examples, while this one helps it generalize to unseen data, and attempt to learn when to say "I don't know" and when to give an answer.

This is where few-shot learning comes in!

Few-shot learning is a type of transfer learning that allows us to answer questions where the answer is not present in the context. We can do this by providing a few examples of the answer we're looking for, and the model will learn to answer questions where the answer is not present in the context.

### 5.1 Embed the Training Data

Embeddings are a way to represent sentences as an array of floats. We'll use the embeddings to find the most similar questions to the ones we're looking for.


```python
import os
from qdrant_client import QdrantClient
from qdrant_client.http import models
from qdrant_client.http.models import PointStruct
from qdrant_client.http.models import Distance, VectorParams
```

Now that we've the Qdrant imports in place, 


```python
qdrant_client = QdrantClient(
    url=os.getenv("QDRANT_URL"), api_key=os.getenv("QDRANT_API_KEY"), timeout=6000, prefer_grpc=True
)

collection_name = "squadv2-cookbook"

# # Create the collection, run this only once
# qdrant_client.recreate_collection(
#     collection_name=collection_name,
#     vectors_config=VectorParams(size=384, distance=Distance.COSINE),
# )
```


```python
from fastembed.embedding import DefaultEmbedding
from typing import List
import numpy as np
import pandas as pd
from tqdm.notebook import tqdm

tqdm.pandas()

embedding_model = DefaultEmbedding()
```

### 5.2 Embedding the Questions

Next, you'll embed the entire training set questions. You'll use the question to question similarity to find the most similar questions to the question we're looking for. This is a workflow which is used in RAG to leverage the OpenAI model ability of incontext learning with more examples. This is what we call Few Shot Learning here.

**❗️⏰ Important Note: This step can take up to 3 hours to complete. Please be patient. If you see Out of Memory errors or Kernel Crashes, please reduce the batch size to 32, restart the kernel and run the notebook again. This code needs to be run only ONCE.**

## Function Breakdown for `generate_points_from_dataframe`

1. **Initialization**: `batch_size = 512` and `total_batches` set the stage for how many questions will be processed in one go. This is to prevent memory issues. If your machine can handle more, feel free to increase the batch size. If your kernel crashes, reduce the batch size to 32 and try again.
2. **Progress Bar**: `tqdm` gives you a nice progress bar so you don't fall asleep.
3. **Batch Loop**: The for-loop iterates through batches. `start_idx` and `end_idx` define the slice of the DataFrame to process.
4. **Generate Embeddings**: `batch_embeddings = embedding_model.embed(batch, batch_size=batch_size)` - This is where the magic happens. Your questions get turned into embeddings.
5. **PointStruct Generation**: Using `.progress_apply`, it turns each row into a `PointStruct` object. This includes an ID, the embedding vector, and other metadata.

Returns the list of `PointStruct` objects, which can be used to create a collection in Qdrant.


key: _node_content
string_value:
{"id_": "a516d65a-ed72-414b-a7d3-8f55cde58bfb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8fddfd20-8b8c-4012-97e1-72090263c907", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "a565d02a5da5b3244d9f94a22859aa73383459d0b613e7c9a786c171b92218e9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6199c17c-306c-4b3f-8ec7-364441326097", "node_type": "1", "metadata": {}, "hash": "34b05026f1128ad6ed6c5af64c81e0da5574c844af38eb224a71b3a29c6b1676", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29366, "end_char_idx": 32460, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def generate_points_from_dataframe(df: pd.DataFrame) -> List[PointStruct]:
    batch_size = 512
    questions = df["question"].tolist()
    total_batches = len(questions) // batch_size + 1
    
    pbar = tqdm(total=len(questions), desc="Generating embeddings")
    
    # Generate embeddings in batches to improve performance
    embeddings = []
    for i in range(total_batches):
        start_idx = i * batch_size
        end_idx = min((i + 1) * batch_size, len(questions))
        batch = questions[start_idx:end_idx]
        
        batch_embeddings = embedding_model.embed(batch, batch_size=batch_size)
        embeddings.extend(batch_embeddings)
        pbar.update(len(batch))
        
    pbar.close()
    
    # Convert embeddings to list of lists
    embeddings_list = [embedding.tolist() for embedding in embeddings]
    
    # Create a temporary DataFrame to hold the embeddings and existing DataFrame columns
    temp_df = df.copy()
    temp_df["embeddings"] = embeddings_list
    temp_df["id"] = temp_df.index
    
    # Generate PointStruct objects using DataFrame apply method
    points = temp_df.progress_apply(
        lambda row: PointStruct(
            id=row["id"],
            vector=row["embeddings"],
            payload={
                "question": row["question"],
                "title": row["title"],
                "context": row["context"],
                "is_impossible": row["is_impossible"],
                "answers": row["answers"],
            },
        ),
        axis=1,
    ).tolist()

    return points

points = generate_points_from_dataframe(train_df)
```

#### Upload the Embeddings to Qdrant

Note that configuring Qdrant is outside the scope of this notebook. Please refer to the [Qdrant](https://qdrant.tech) for more information. We used a timeout of 600 seconds for the upload, and grpc compression to speed up the upload.


```python
operation_info = qdrant_client.upsert(
    collection_name=collection_name, wait=True, points=points
)
print(operation_info)
```

## 6. Using Qdrant to Improve RAG Prompt

Now that we've uploaded the embeddings to Qdrant, we can use Qdrant to find the most similar questions to the question we're looking for. We'll use the top 5 most similar questions to create a prompt that we can use to fine-tune the model. We'll then measure the performance of the fine-tuned model on the same validation set, but with few shot prompting!

Our main function `get_few_shot_prompt` serves as the workhorse for generating prompts for few-shot learning. It does this by retrieving similar questions from Qdrant - a vector search engine, using an embeddings model. Here is the high-level workflow:

1. Retrieve similar questions from Qdrant where the **answer is present** in the context
2. Retrieve similar questions from Qdrant where the **answer is IMPOSSIBLE** i.e. the expected answer is "I don't know" to find in the context
3. Create a prompt using the retrieved questions
4. Fine-tune the model using the prompt
5. Evaluate the fine-tuned model on the validation set with the same prompting technique


key: _node_content
string_value:
{"id_": "6199c17c-306c-4b3f-8ec7-364441326097", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a516d65a-ed72-414b-a7d3-8f55cde58bfb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "04d1176066f63efda25c8f9ef47c5355cda44a78ff94ebf8cd5cfcce0c16b5bc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "89adfd3b-0db4-4999-bb09-e4693c1ff85c", "node_type": "1", "metadata": {}, "hash": "5e3491d2322ca3cdb399f4440c534a9658fef59a6dacb14386e4de1caf7bf43f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 31258, "end_char_idx": 33752, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
operation_info = qdrant_client.upsert(
    collection_name=collection_name, wait=True, points=points
)
print(operation_info)
```

## 6. Using Qdrant to Improve RAG Prompt

Now that we've uploaded the embeddings to Qdrant, we can use Qdrant to find the most similar questions to the question we're looking for. We'll use the top 5 most similar questions to create a prompt that we can use to fine-tune the model. We'll then measure the performance of the fine-tuned model on the same validation set, but with few shot prompting!

Our main function `get_few_shot_prompt` serves as the workhorse for generating prompts for few-shot learning. It does this by retrieving similar questions from Qdrant - a vector search engine, using an embeddings model. Here is the high-level workflow:

1. Retrieve similar questions from Qdrant where the **answer is present** in the context
2. Retrieve similar questions from Qdrant where the **answer is IMPOSSIBLE** i.e. the expected answer is "I don't know" to find in the context
3. Create a prompt using the retrieved questions
4. Fine-tune the model using the prompt
5. Evaluate the fine-tuned model on the validation set with the same prompting technique


```python
def get_few_shot_prompt(row):

    query, row_context = row["question"], row["context"]

    embeddings = list(embedding_model.embed([query]))
    query_embedding = embeddings[0].tolist()

    num_of_qa_to_retrieve = 5

    # Query Qdrant for similar questions that have an answer
    q1 = qdrant_client.search(
        collection_name=collection_name,
        query_vector=query_embedding,
        with_payload=True,
        limit=num_of_qa_to_retrieve,
        query_filter=models.Filter(
            must=[
                models.FieldCondition(
                    key="is_impossible",
                    match=models.MatchValue(
                        value=False,
                    ),
                ),
            ],
        )
    )

    # Query Qdrant for similar questions that are IMPOSSIBLE to answer
    q2 = qdrant_client.search(
        collection_name=collection_name,
        query_vector=query_embedding,
        query_filter=models.Filter(
            must=[
                models.FieldCondition(
                    key="is_impossible",
                    match=models.MatchValue(
                        value=True,
                    ),
                ),
            ]
        ),
        with_payload=True,
        limit=num_of_qa_to_retrieve,
    )


key: _node_content
string_value:
{"id_": "89adfd3b-0db4-4999-bb09-e4693c1ff85c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6199c17c-306c-4b3f-8ec7-364441326097", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "94d5ef521d124cf05b4c8c0d7dd8dfb563f317f122055c0568bf81b6ba6cd493", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2a85242b-328f-46a6-bf4b-4e68c1c46cb0", "node_type": "1", "metadata": {}, "hash": "29cd50c31aaeff745085d82438d98308905ae83f27af17558d6287056d9c8211", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 32463, "end_char_idx": 35609, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def get_few_shot_prompt(row):

    query, row_context = row["question"], row["context"]

    embeddings = list(embedding_model.embed([query]))
    query_embedding = embeddings[0].tolist()

    num_of_qa_to_retrieve = 5

    # Query Qdrant for similar questions that have an answer
    q1 = qdrant_client.search(
        collection_name=collection_name,
        query_vector=query_embedding,
        with_payload=True,
        limit=num_of_qa_to_retrieve,
        query_filter=models.Filter(
            must=[
                models.FieldCondition(
                    key="is_impossible",
                    match=models.MatchValue(
                        value=False,
                    ),
                ),
            ],
        )
    )

    # Query Qdrant for similar questions that are IMPOSSIBLE to answer
    q2 = qdrant_client.search(
        collection_name=collection_name,
        query_vector=query_embedding,
        query_filter=models.Filter(
            must=[
                models.FieldCondition(
                    key="is_impossible",
                    match=models.MatchValue(
                        value=True,
                    ),
                ),
            ]
        ),
        with_payload=True,
        limit=num_of_qa_to_retrieve,
    )


    instruction = """Answer the following Question based on the Context only. Only answer from the Context. If you don't know the answer, say 'I don't know'.\n\n"""
    # If there is a next best question, add it to the prompt
    
    def q_to_prompt(q):
        question, context = q.payload["question"], q.payload["context"]
        answer = q.payload["answers"][0] if len(q.payload["answers"]) > 0 else "I don't know"
        return [
            {
                "role": "user", 
                "content": f"""Question: {question}\n\nContext: {context}\n\nAnswer:"""
            },
            {"role": "assistant", "content": answer},
        ]

    rag_prompt = []
    
    if len(q1) >= 1:
        rag_prompt += q_to_prompt(q1[1])
    if len(q2) >= 1:
        rag_prompt += q_to_prompt(q2[1])
    if len(q1) >= 1:
        rag_prompt += q_to_prompt(q1[2])
    
    

    rag_prompt += [
        {
            "role": "user",
            "content": f"""Question: {query}\n\nContext: {row_context}\n\nAnswer:"""
        },
    ]

    rag_prompt = [{"role": "system", "content": instruction}] + rag_prompt
    return rag_prompt
```


```python
# ⏰ Time: 2 min
train_sample["few_shot_prompt"] = train_sample.progress_apply(get_few_shot_prompt, axis=1)
```

## 7. Fine-Tuning OpenAI Model with Qdrant

### 7.1 Upload the Fine-Tuning Data to OpenAI


```python
# Prepare the OpenAI File format i.e. JSONL from train_sample
def dataframe_to_jsonl(df):
    def create_jsonl_entry(row):
        messages = row["few_shot_prompt"]
        return json.dumps({"messages": messages})

    jsonl_output = df.progress_apply(create_jsonl_entry, axis=1)
    return "\n".join(jsonl_output)

with open("local_cache/100_train_few_shot.jsonl", "w") as f:
    f.write(dataframe_to_jsonl(train_sample))
```

### 7.2 Fine-Tune the Model

⏰ **Time to run: ~15-30 minutes**


key: _node_content
string_value:
{"id_": "2a85242b-328f-46a6-bf4b-4e68c1c46cb0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "89adfd3b-0db4-4999-bb09-e4693c1ff85c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9865e38d5456132fe96d9141110eb6304b7823ee436b4b3e2d7397a9eb0213c1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e75207b6-b197-4425-9f89-0bee80541ca6", "node_type": "1", "metadata": {}, "hash": "9d8addfcbc9314d55d4b1e2305bb2c3963226c07b97cea868b80904a2ca08f3d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34894, "end_char_idx": 35850, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# ⏰ Time: 2 min
train_sample["few_shot_prompt"] = train_sample.progress_apply(get_few_shot_prompt, axis=1)
```

## 7. Fine-Tuning OpenAI Model with Qdrant

### 7.1 Upload the Fine-Tuning Data to OpenAI


```python
# Prepare the OpenAI File format i.e. JSONL from train_sample
def dataframe_to_jsonl(df):
    def create_jsonl_entry(row):
        messages = row["few_shot_prompt"]
        return json.dumps({"messages": messages})

    jsonl_output = df.progress_apply(create_jsonl_entry, axis=1)
    return "\n".join(jsonl_output)

with open("local_cache/100_train_few_shot.jsonl", "w") as f:
    f.write(dataframe_to_jsonl(train_sample))
```

### 7.2 Fine-Tune the Model

⏰ **Time to run: ~15-30 minutes**


```python
fine_tuner = OpenAIFineTuner(
        training_file_path="local_cache/100_train_few_shot.jsonl",
        model_name="gpt-3.5-turbo",
        suffix="trnfewshot20230907"
    )

model_id = fine_tuner.fine_tune_model()
model_id
```


key: _node_content
string_value:
{"id_": "e75207b6-b197-4425-9f89-0bee80541ca6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2a85242b-328f-46a6-bf4b-4e68c1c46cb0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "570750d8ebd0a544ecea669c6750312b2f26db2f9a12b4d90e5f40e901334027", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "594a68c9-5a9f-4a3b-9a56-1bbee19a805a", "node_type": "1", "metadata": {}, "hash": "9c3ba18dc6583179f20ffcaf5152e129438a1d3b4a7cb1d090f2024814c6cb41", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34894, "end_char_idx": 38379, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# ⏰ Time: 2 min
train_sample["few_shot_prompt"] = train_sample.progress_apply(get_few_shot_prompt, axis=1)
```

## 7. Fine-Tuning OpenAI Model with Qdrant

### 7.1 Upload the Fine-Tuning Data to OpenAI


```python
# Prepare the OpenAI File format i.e. JSONL from train_sample
def dataframe_to_jsonl(df):
    def create_jsonl_entry(row):
        messages = row["few_shot_prompt"]
        return json.dumps({"messages": messages})

    jsonl_output = df.progress_apply(create_jsonl_entry, axis=1)
    return "\n".join(jsonl_output)

with open("local_cache/100_train_few_shot.jsonl", "w") as f:
    f.write(dataframe_to_jsonl(train_sample))
```

### 7.2 Fine-Tune the Model

⏰ **Time to run: ~15-30 minutes**


```python
fine_tuner = OpenAIFineTuner(
        training_file_path="local_cache/100_train_few_shot.jsonl",
        model_name="gpt-3.5-turbo",
        suffix="trnfewshot20230907"
    )

model_id = fine_tuner.fine_tune_model()
model_id
```


```python
# Let's try this out
completion = client.chat.completions.create(
    model=model_id,
    messages=[
        {"role": "system", "content": "You are a helpful assistant."},
        {
            "role": "user",
            "content": "Can you answer the following question based on the given context? If not, say, I don't know:\n\nQuestion: What is the capital of France?\n\nContext: The capital of Mars is Gaia. Answer:",
        },
        {
            "role": "assistant",
            "content": "I don't know",
        },
        {
            "role": "user",
            "content": "Question: Where did Maharana Pratap die?\n\nContext: Rana Pratap's defiance of the mighty Mughal empire, almost alone and unaided by the other Rajput states, constitute a glorious saga of Rajput valour and the spirit of self sacrifice for cherished principles. Rana Pratap's methods of guerrilla warfare was later elaborated further by Malik Ambar, the Deccani general, and by Emperor Shivaji.\nAnswer:",
        },
        {
            "role": "assistant",
            "content": "I don't know",
        },
        {
            "role": "user",
            "content": "Question: Who did Rana Pratap fight against?\n\nContext: In stark contrast to other Rajput rulers who accommodated and formed alliances with the various Muslim dynasties in the subcontinent, by the time Pratap ascended to the throne, Mewar was going through a long standing conflict with the Mughals which started with the defeat of his grandfather Rana Sanga in the Battle of Khanwa in 1527 and continued with the defeat of his father Udai Singh II in Siege of Chittorgarh in 1568. Pratap Singh, gained distinction for his refusal to form any political alliance with the Mughal Empire and his resistance to Muslim domination. The conflicts between Pratap Singh and Akbar led to the Battle of Haldighati. Answer:",
        },
        {
            "role": "assistant",
            "content": "Akbar",
        },
        {
            "role": "user",
            "content": "Question: Which state is Chittorgarh in?\n\nContext: Chittorgarh, located in the southern part of the state of Rajasthan, 233 km (144.8 mi) from Ajmer, midway between Delhi and Mumbai on the National Highway 8 (India) in the road network of Golden Quadrilateral. Chittorgarh is situated where National Highways No. 76 & 79 intersect. Answer:",
        },
    ],
)
print("Correct Answer: Rajasthan\nModel Answer:")
print(completion.choices[0].message)
```

⏰ **Time to run: 5-15 min**


key: _node_content
string_value:
{"id_": "594a68c9-5a9f-4a3b-9a56-1bbee19a805a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e75207b6-b197-4425-9f89-0bee80541ca6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "776de9eee57b20b92a1e30637c48c98a2a4e7be10645d8b07f1b9985ad30be14", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f79a8e24-2041-48f7-a4da-d8cd41ef2a60", "node_type": "1", "metadata": {}, "hash": "dd4d342b76d7449e06448f34fbf4c0759927b035843950ff2b0ed8fddc98b52b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 38382, "end_char_idx": 39938, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
df["ft_generated_answer_few_shot"] = df.progress_apply(answer_question, model=model_id, prompt_func=get_few_shot_prompt, axis=1)
df.to_json("local_cache/100_val_ft_few_shot.json", orient="records", lines=True)
```

## 8. Evaluation

But how well does the model perform? Let's compare the results from the 3 different models we've looked at so far:


```python
evaluator = Evaluator(df)
evaluator.plot_model_comparison(["generated_answer", "ft_generated_answer", "ft_generated_answer_few_shot"], scenario="answer_expected", nice_names=["Baseline", "Fine-Tuned", "Fine-Tuned with Few-Shot"])
```


    
![png](output_56_0.png)
    


This is quite amazing -- we're able to get the best of both worlds! We're able to get the model to be both correct and conservative: 

1. The model is correct 83% of the time -- this is the same as the base model
2. The model gives the wrong answer only 8% of the time -- down from 17% with the base model

Next, let's look at the hallucinations. We want to reduce the hallucinations, but not at the cost of correctness. We want to strike a balance between the two. We've struck a good balance here:

1. The model hallucinates 53% of the time -- down from 100% with the base model
2. The model says "I don't know" 47% of the time -- up from NEVER with the base model


```python
evaluator.plot_model_comparison(["generated_answer", "ft_generated_answer", "ft_generated_answer_few_shot"], scenario="idk_expected", nice_names=["Baseline", "Fine-Tuned", "Fine-Tuned with Few-Shot"])
```


    
![png](output_59_0.png)


key: _node_content
string_value:
{"id_": "f79a8e24-2041-48f7-a4da-d8cd41ef2a60", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "594a68c9-5a9f-4a3b-9a56-1bbee19a805a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "ff51c1f6e1e8431978f5784e801eae54546d282b95e263f1e3c1008de0c4c447", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6a4f252a-a3e7-486c-8bb7-20984dfc6119", "node_type": "1", "metadata": {}, "hash": "83efd75536847839db9dd72e908ed395701d3829bd31a091880bcfe2b59e776e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 38742, "end_char_idx": 41900, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
evaluator = Evaluator(df)
evaluator.plot_model_comparison(["generated_answer", "ft_generated_answer", "ft_generated_answer_few_shot"], scenario="answer_expected", nice_names=["Baseline", "Fine-Tuned", "Fine-Tuned with Few-Shot"])
```


    
![png](output_56_0.png)
    


This is quite amazing -- we're able to get the best of both worlds! We're able to get the model to be both correct and conservative: 

1. The model is correct 83% of the time -- this is the same as the base model
2. The model gives the wrong answer only 8% of the time -- down from 17% with the base model

Next, let's look at the hallucinations. We want to reduce the hallucinations, but not at the cost of correctness. We want to strike a balance between the two. We've struck a good balance here:

1. The model hallucinates 53% of the time -- down from 100% with the base model
2. The model says "I don't know" 47% of the time -- up from NEVER with the base model


```python
evaluator.plot_model_comparison(["generated_answer", "ft_generated_answer", "ft_generated_answer_few_shot"], scenario="idk_expected", nice_names=["Baseline", "Fine-Tuned", "Fine-Tuned with Few-Shot"])
```


    
![png](output_59_0.png)
    


Few Shot Fine-Tuning with Qdrant is a great way to control and steer the performance of your RAG system. Here, we made the model less conservative compared to zero shot and more confident by using Qdrant to find similar questions. 

You can also use Qdrant to make the model more conservative. We did this by giving examples of questions where the answer is not present in the context.  
This is biasing the model to say "I don't know" more often. 

Similarly, one can also use Qdrant to make the model more confident by giving examples of questions where the answer is present in the context. This biases the model to give an answer more often. The trade-off is that the model will also hallucinate more often.

You can make this trade off by adjusting the training data: distribution of questions and examples, as well as the kind and number of examples you retrieve from Qdrant.

## 9. Conclusion

In this notebook, we've demonstrated how to fine-tune OpenAI models for specific use-cases. We've also demonstrated how to use Qdrant and Few-Shot Learning to improve the performance of the model.

### Aggregate Results

So far, we've looked at the results for each scenario separately, i.e. each scenario summed to 100. Let's look at the results as an aggregate to get a broader sense of how the model is performing:

| Category | Base | Fine-Tuned | Fine-Tuned with Qdrant |
| --- | --- | --- | --- |
| Correct | 44% | 32% | 44% |
| Skipped | 0% | 18% | 5% |
| Wrong | 9% | 3% | 4% |
| Hallucination | 47% | 7% | 25% |
| I don't know | 0% | 40% | 22% |

### Observations

#### Compared to base model
1. The few shot fine-tuned with Qdrant model is as good as the base model at answering questions where the answer is present in the context. 
2. The few shot fine-tuned with Qdrant model is better at saying "I don't know" when the answer is not present in the context.
3. The few shot fine-tuned with Qdrant model is better at reducing hallucinations.


key: _node_content
string_value:
{"id_": "6a4f252a-a3e7-486c-8bb7-20984dfc6119", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "7b96c1f3-71ce-4c5d-b86b-0751a8fc9485", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c3d4e60a5301ff9c4065835cfa44463ecad1db6c2912cad00dd22578ffaf226d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f79a8e24-2041-48f7-a4da-d8cd41ef2a60", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/fine-tuned_qa/ft_retrieval_augmented_generation_qdrant.ipynb", "title": "Fine-Tuning OpenAI Models for Retrieval Augmented Generation (RAG) with Qdrant and Few-Shot Learning", "tokens": 10450, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c055d611d640f319436052228cccdc09f280b9f812ae107ebed5c9d1158545d1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 41903, "end_char_idx": 42696, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
#### Compared to fine-tuned model
1. The few shot fine-tuned with Qdrant model gets more correct answers than the fine-tuned model: **83% of the questions are answered correctly vs 60%** for the fine-tuned model
2. The few shot fine-tuned with Qdrant model is better at deciding when to say "I don't know" when the answer is not present in the context. **34% skip rate for the plain fine-tuning mode, vs 9% for the few shot fine-tuned with Qdrant model**


Now, you should be able to:

1. Notice the trade-offs between number of correct answers and hallucinations -- and how training dataset choice influences that!
2. Fine-tune OpenAI models for specific use-cases and use Qdrant to improve the performance of your RAG model
3. Get started on how to evaluate the performance of your RAG model


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb


key: _node_content
string_value:
{"id_": "2bf76086-1eb6-44c7-960e-ab6a6606bba9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "25b20c91-35fc-4666-b23f-0f187e4bb3d5", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "dd12f61a1673bab75775a08698bc39b904361ac81726c523ce42781ad5048b1c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "94be41bc-365c-4138-8607-134a67703342", "node_type": "1", "metadata": {}, "hash": "745a03d972c117d89502888561536d0cd90c1067ef62b129e3757c30418a3f24", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3332, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to use GPT-4 Vision with Function Calling 

The new GPT-4 Turbo model, available as gpt-4-turbo-2024-04-09 as of April 2024, now enables function calling with vision capabilities, better reasoning and a knowledge cutoff date of Dec 2023. Using images with function calling will unlock multimodal use cases and the ability to use reasoning, allowing you to go beyond OCR and image descriptions.

We will go through two examples to demonstrate the use of function calling with GPT-4 Turbo with Vision:

1. Simulating a customer service assistant for delivery exception support
2. Analyzing an organizational chart to extract employee information

### Installation and Setup


```python
!pip install pymupdf --quiet
!pip install openai --quiet
!pip install matplotlib --quiet
# instructor makes it easy to work with function calling
!pip install instructor --quiet
```


```python
import base64
import os
from enum import Enum
from io import BytesIO
from typing import Iterable
from typing import List
from typing import Literal, Optional

import fitz
# Instructor is powered by Pydantic, which is powered by type hints. Schema validation, prompting is controlled by type annotations
import instructor
import matplotlib.pyplot as plt
import pandas as pd
from IPython.display import display
from PIL import Image
from openai import OpenAI
from pydantic import BaseModel, Field
```

## 1. Simulating a customer service assistant for delivery exception support
We will simulate a customer service assistant for a delivery service that is equipped to analyze images of packages. The assistant will perform the following actions based on the image analysis:
- If a package appears damaged in the image, automatically process a refund according to policy.
- If the package looks wet, initiate a replacement.
- If the package appears normal and not damaged, escalate to an agent.

Let's look at the sample images of packages that the customer service assistant will analyze to determine the appropriate action. We will encode the images as base64 strings for processing by the model.


```python
# Function to encode the image as base64
def encode_image(image_path: str):
    # check if the image exists
    if not os.path.exists(image_path):
        raise FileNotFoundError(f"Image file not found: {image_path}")
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode('utf-8')


# Sample images for testing
image_dir = "images"

# encode all images within the directory
image_files = os.listdir(image_dir)
image_data = {}
for image_file in image_files:
    image_path = os.path.join(image_dir, image_file)
    # encode the image with key as the image file name
    image_data[image_file.split('.')[0]] = encode_image(image_path)
    print(f"Encoded image: {image_file}")


def display_images(image_data: dict):
    fig, axs = plt.subplots(1, 3, figsize=(18, 6))
    for i, (key, value) in enumerate(image_data.items()):
        img = Image.open(BytesIO(base64.b64decode(value)))
        ax = axs[i]
        ax.imshow(img)
        ax.axis("off")
        ax.set_title(key)
    plt.tight_layout()
    plt.show()


display_images(image_data)
```

    Encoded image: wet_package.jpg
    Encoded image: damaged_package.jpg
    Encoded image: normal_package.jpg



    
![png](output_6_1.png)


key: _node_content
string_value:
{"id_": "94be41bc-365c-4138-8607-134a67703342", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "25b20c91-35fc-4666-b23f-0f187e4bb3d5", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "dd12f61a1673bab75775a08698bc39b904361ac81726c523ce42781ad5048b1c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2bf76086-1eb6-44c7-960e-ab6a6606bba9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "eb95dec29c0b405d5b4bdbb826ad97a7a7318a3aac39b30ef15d34a7e575a7af", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c5fec1fa-68fc-465c-a633-91631c07857d", "node_type": "1", "metadata": {}, "hash": "d3d45c8c1b85f18165f8e055d7c526a63c9ad5a0c26532dfbd2b013df157db59", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2082, "end_char_idx": 3894, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Function to encode the image as base64
def encode_image(image_path: str):
    # check if the image exists
    if not os.path.exists(image_path):
        raise FileNotFoundError(f"Image file not found: {image_path}")
    with open(image_path, "rb") as image_file:
        return base64.b64encode(image_file.read()).decode('utf-8')


# Sample images for testing
image_dir = "images"

# encode all images within the directory
image_files = os.listdir(image_dir)
image_data = {}
for image_file in image_files:
    image_path = os.path.join(image_dir, image_file)
    # encode the image with key as the image file name
    image_data[image_file.split('.')[0]] = encode_image(image_path)
    print(f"Encoded image: {image_file}")


def display_images(image_data: dict):
    fig, axs = plt.subplots(1, 3, figsize=(18, 6))
    for i, (key, value) in enumerate(image_data.items()):
        img = Image.open(BytesIO(base64.b64decode(value)))
        ax = axs[i]
        ax.imshow(img)
        ax.axis("off")
        ax.set_title(key)
    plt.tight_layout()
    plt.show()


display_images(image_data)
```

    Encoded image: wet_package.jpg
    Encoded image: damaged_package.jpg
    Encoded image: normal_package.jpg



    
![png](output_6_1.png)
    


We have successfully encoded the sample images as base64 strings and displayed them. The customer service assistant will analyze these images to determine the appropriate action based on the package condition.

Let's now define the functions/tools for order processing, such as escalating an order to an agent, refunding an order, and replacing an order. We will create placeholder functions to simulate the processing of these actions based on the identified tools. We will be using Pydantic models to define the structure of the data for order actions.


key: _node_content
string_value:
{"id_": "c5fec1fa-68fc-465c-a633-91631c07857d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "25b20c91-35fc-4666-b23f-0f187e4bb3d5", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "dd12f61a1673bab75775a08698bc39b904361ac81726c523ce42781ad5048b1c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "94be41bc-365c-4138-8607-134a67703342", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d1c6f555131f1198b7d972d95fa39784bc9dd3ca5bcf54d69425c421acabae96", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f3dcf7e4-16a2-457b-b700-49606fd63c46", "node_type": "1", "metadata": {}, "hash": "601b0f363b96725fb8e2aefc4305a30e273ca81d01b026fe6eca2561b435530a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2426, "end_char_idx": 6996, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Sample images for testing
image_dir = "images"

# encode all images within the directory
image_files = os.listdir(image_dir)
image_data = {}
for image_file in image_files:
    image_path = os.path.join(image_dir, image_file)
    # encode the image with key as the image file name
    image_data[image_file.split('.')[0]] = encode_image(image_path)
    print(f"Encoded image: {image_file}")


def display_images(image_data: dict):
    fig, axs = plt.subplots(1, 3, figsize=(18, 6))
    for i, (key, value) in enumerate(image_data.items()):
        img = Image.open(BytesIO(base64.b64decode(value)))
        ax = axs[i]
        ax.imshow(img)
        ax.axis("off")
        ax.set_title(key)
    plt.tight_layout()
    plt.show()


display_images(image_data)
```

    Encoded image: wet_package.jpg
    Encoded image: damaged_package.jpg
    Encoded image: normal_package.jpg



    
![png](output_6_1.png)
    


We have successfully encoded the sample images as base64 strings and displayed them. The customer service assistant will analyze these images to determine the appropriate action based on the package condition.

Let's now define the functions/tools for order processing, such as escalating an order to an agent, refunding an order, and replacing an order. We will create placeholder functions to simulate the processing of these actions based on the identified tools. We will be using Pydantic models to define the structure of the data for order actions.



```python
MODEL = "gpt-4-turbo-2024-04-09"

class Order(BaseModel):
    """Represents an order with details such as order ID, customer name, product name, price, status, and delivery date."""
    order_id: str = Field(..., description="The unique identifier of the order")
    product_name: str = Field(..., description="The name of the product")
    price: float = Field(..., description="The price of the product")
    status: str = Field(..., description="The status of the order")
    delivery_date: str = Field(..., description="The delivery date of the order")
# Placeholder functions for order processing

def get_order_details(order_id):
    # Placeholder function to retrieve order details based on the order ID
    return Order(
        order_id=order_id,
        product_name="Product X",
        price=100.0,
        status="Delivered",
        delivery_date="2024-04-10",
    )

def escalate_to_agent(order: Order, message: str):
    # Placeholder function to escalate the order to a human agent
    return f"Order {order.order_id} has been escalated to an agent with message: `{message}`"

def refund_order(order: Order):
    # Placeholder function to process a refund for the order
    return f"Order {order.order_id} has been refunded successfully."

def replace_order(order: Order):
    # Placeholder function to replace the order with a new one
    return f"Order {order.order_id} has been replaced with a new order."

class FunctionCallBase(BaseModel):
    rationale: Optional[str] = Field(..., description="The reason for the action.")
    image_description: Optional[str] = Field(
        ..., description="The detailed description of the package image."
    )
    action: Literal["escalate_to_agent", "replace_order", "refund_order"]
    message: Optional[str] = Field(
        ...,
        description="The message to be escalated to the agent if action is escalate_to_agent",
    )
    # Placeholder functions to process the action based on the order ID
    def __call__(self, order_id):
        order: Order = get_order_details(order_id=order_id)
        if self.action == "escalate_to_agent":
            return escalate_to_agent(order, self.message)
        if self.action == "replace_order":
            return replace_order(order)
        if self.action == "refund_order":
            return refund_order(order)

class EscalateToAgent(FunctionCallBase):
    """Escalate to an agent for further assistance."""
    pass

class OrderActionBase(FunctionCallBase):
    pass

class ReplaceOrder(OrderActionBase):
    """Tool call to replace an order."""
    pass

class RefundOrder(OrderActionBase):
    """Tool call to refund an order."""
    pass
```

### Simulating user messages and processing the package images

We will simulate user messages containing the package images and process the images using the GPT-4 Turbo with Vision model. The model will identify the appropriate tool call based on the image analysis and the predefined actions for damaged, wet, or normal packages. We will then process the identified action based on the order ID and display the results.


key: _node_content
string_value:
{"id_": "f3dcf7e4-16a2-457b-b700-49606fd63c46", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "25b20c91-35fc-4666-b23f-0f187e4bb3d5", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "dd12f61a1673bab75775a08698bc39b904361ac81726c523ce42781ad5048b1c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c5fec1fa-68fc-465c-a633-91631c07857d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1f1f0aaa93b8275f4c4d32ed3d7246ca75a7f7e5335920b15e9926effdc5dddf", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ae28ae60-828d-4897-80a2-ae5e7f5cc5c9", "node_type": "1", "metadata": {}, "hash": "1d31a0d166c9e36b48e6c7942431e27b88965cc6df32228d4bcf9656f4bdeca2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6999, "end_char_idx": 8706, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# extract the tool call from the response
ORDER_ID = "12345"  # Placeholder order ID for testing
INSTRUCTION_PROMPT = "You are a customer service assistant for a delivery service, equipped to analyze images of packages. If a package appears damaged in the image, automatically process a refund according to policy. If the package looks wet, initiate a replacement. If the package appears normal and not damaged, escalate to agent. For any other issues or unclear images, escalate to agent. You must always use tools!"

def delivery_exception_support_handler(test_image: str):
    payload = {
        "model": MODEL,
        "response_model": Iterable[RefundOrder | ReplaceOrder | EscalateToAgent],
        "tool_choice": "auto",  # automatically select the tool based on the context
        "temperature": 0.0,  # for less diversity in responses
        "seed": 123,  # Set a seed for reproducibility
    }
    payload["messages"] = [
        {
            "role": "user",
            "content": INSTRUCTION_PROMPT,
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{image_data[test_image]}"
                    }
                },
            ],
        }
    ]
    function_calls = instructor.from_openai(
        OpenAI(), mode=instructor.Mode.PARALLEL_TOOLS
    ).chat.completions.create(**payload)
    for tool in function_calls:
        print(f"- Tool call: {tool.action} for provided img: {test_image}")
        print(f"- Parameters: {tool}")
        print(f">> Action result: {tool(ORDER_ID)}")
        return tool


key: _node_content
string_value:
{"id_": "ae28ae60-828d-4897-80a2-ae5e7f5cc5c9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "25b20c91-35fc-4666-b23f-0f187e4bb3d5", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "dd12f61a1673bab75775a08698bc39b904361ac81726c523ce42781ad5048b1c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f3dcf7e4-16a2-457b-b700-49606fd63c46", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "dc526499f2c39f74dc723a92494bf45ffffed42f225fce30fbb98e15a179df8b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a1f38511-2485-4de7-928d-06a3031c15ef", "node_type": "1", "metadata": {}, "hash": "0161fce6c3cc5d65361fe0b946bc1e4742fcece4127c254e69f4ff93f4535a45", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6999, "end_char_idx": 11389, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# extract the tool call from the response
ORDER_ID = "12345"  # Placeholder order ID for testing
INSTRUCTION_PROMPT = "You are a customer service assistant for a delivery service, equipped to analyze images of packages. If a package appears damaged in the image, automatically process a refund according to policy. If the package looks wet, initiate a replacement. If the package appears normal and not damaged, escalate to agent. For any other issues or unclear images, escalate to agent. You must always use tools!"

def delivery_exception_support_handler(test_image: str):
    payload = {
        "model": MODEL,
        "response_model": Iterable[RefundOrder | ReplaceOrder | EscalateToAgent],
        "tool_choice": "auto",  # automatically select the tool based on the context
        "temperature": 0.0,  # for less diversity in responses
        "seed": 123,  # Set a seed for reproducibility
    }
    payload["messages"] = [
        {
            "role": "user",
            "content": INSTRUCTION_PROMPT,
        },
        {
            "role": "user",
            "content": [
                {
                    "type": "image_url",
                    "image_url": {
                        "url": f"data:image/jpeg;base64,{image_data[test_image]}"
                    }
                },
            ],
        }
    ]
    function_calls = instructor.from_openai(
        OpenAI(), mode=instructor.Mode.PARALLEL_TOOLS
    ).chat.completions.create(**payload)
    for tool in function_calls:
        print(f"- Tool call: {tool.action} for provided img: {test_image}")
        print(f"- Parameters: {tool}")
        print(f">> Action result: {tool(ORDER_ID)}")
        return tool


print("Processing delivery exception support for different package images...")

print("\n===================== Simulating user message 1 =====================")
assert delivery_exception_support_handler("damaged_package").action == "refund_order"

print("\n===================== Simulating user message 2 =====================")
assert delivery_exception_support_handler("normal_package").action == "escalate_to_agent"

print("\n===================== Simulating user message 3 =====================")
assert delivery_exception_support_handler("wet_package").action == "replace_order"
```

    Processing delivery exception support for different package images...
    
    ===================== Simulating user message 1 =====================
    - Tool call: refund_order for provided img: damaged_package
    - Parameters: rationale='The package is visibly damaged with significant tears and crushing, indicating potential harm to the contents.' image_description='The package in the image shows extensive damage, including deep creases and tears in the cardboard. The package is also wrapped with extra tape, suggesting prior attempts to secure it after damage.' action='refund_order' message=None
    >> Action result: Order 12345 has been refunded successfully.
    
    ===================== Simulating user message 2 =====================
    - Tool call: escalate_to_agent for provided img: normal_package
    - Parameters: rationale='The package appears normal and not damaged, requiring further assistance for any potential issues not visible in the image.' image_description='A cardboard box on a wooden floor, appearing intact and undamaged, with no visible signs of wear, tear, or wetness.' action='escalate_to_agent' message='Please review this package for any issues not visible in the image. The package appears normal and undamaged.'
    >> Action result: Order 12345 has been escalated to an agent with message: `Please review this package for any issues not visible in the image. The package appears normal and undamaged.`
    
    ===================== Simulating user message 3 =====================
    - Tool call: replace_order for provided img: wet_package
    - Parameters: rationale='The package appears wet, which may have compromised the contents, especially since it is labeled as fragile.' image_description="The package in the image shows significant wetness on the top surface, indicating potential water damage. The box is labeled 'FRAGILE', which suggests that the contents are delicate and may be more susceptible to damage from moisture." action='replace_order' message=None
    >> Action result: Order 12345 has been replaced with a new order.


key: _node_content
string_value:
{"id_": "a1f38511-2485-4de7-928d-06a3031c15ef", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "25b20c91-35fc-4666-b23f-0f187e4bb3d5", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "dd12f61a1673bab75775a08698bc39b904361ac81726c523ce42781ad5048b1c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ae28ae60-828d-4897-80a2-ae5e7f5cc5c9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "aa1077ed3a231e949968872ba72a9e6105ca7767e76fb2e4ef8f9383eff34089", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c4724204-c7a4-4262-9c16-23d76cdd3608", "node_type": "1", "metadata": {}, "hash": "a74d0c370790e2b8f38e6374975e7130d77095e5d57d70256b63e9fd766b3d06", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11392, "end_char_idx": 14076, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## 2.  Analyzing an organizational chart to extract employee information

For the second example, we will analyze an organizational chart image to extract employee information, such as employee names, roles, managers, and manager roles. We will use GPT-4 Turbo with Vision to process the organizational chart image and extract structured data about the employees in the organization. Indeed, function calling lets us go beyond OCR to actually deduce and translate hierarchical relationships within the chart.

We will start with a sample organizational chart in PDF format that we want to analyze and convert the first page of the PDF to a JPEG image for analysis.


```python
# Function to convert a single page PDF page to a JPEG image
def convert_pdf_page_to_jpg(pdf_path: str, output_path: str, page_number=0):
    if not os.path.exists(pdf_path):
        raise FileNotFoundError(f"PDF file not found: {pdf_path}")
    doc = fitz.open(pdf_path)
    page = doc.load_page(page_number)  # 0 is the first page
    pix = page.get_pixmap()
    # Save the pixmap as a JPEG
    pix.save(output_path)


def display_img_local(image_path: str):
    img = Image.open(image_path)
    display(img)


pdf_path = 'data/org-chart-sample.pdf'
output_path = 'org-chart-sample.jpg'

convert_pdf_page_to_jpg(pdf_path, output_path)
display_img_local(output_path)
```


    
![png](output_12_0.png)
    


The organizational chart image has been successfully extracted from the PDF file and displayed. Let's now define a function to analyze the organizational chart image using the new GPT4 Turbo with Vision. The function will extract information about the employees, their roles, and their managers from the image. We will use function/tool calling to specify the input parameters for the organizational structure, such as the employee name, role, and manager's name and role. We will use Pydantic models to define the structure of the data.



```python
base64_img = encode_image(output_path)

class RoleEnum(str, Enum):
    """Defines possible roles within an organization."""
    CEO = "CEO"
    CTO = "CTO"
    CFO = "CFO"
    COO = "COO"
    EMPLOYEE = "Employee"
    MANAGER = "Manager"
    INTERN = "Intern"
    OTHER = "Other"

class Employee(BaseModel):
    """Represents an employee, including their name, role, and optional manager information."""
    employee_name: str = Field(..., description="The name of the employee")
    role: RoleEnum = Field(..., description="The role of the employee")
    manager_name: Optional[str] = Field(None, description="The manager's name, if applicable")
    manager_role: Optional[RoleEnum] = Field(None, description="The manager's role, if applicable")


key: _node_content
string_value:
{"id_": "c4724204-c7a4-4262-9c16-23d76cdd3608", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "25b20c91-35fc-4666-b23f-0f187e4bb3d5", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "dd12f61a1673bab75775a08698bc39b904361ac81726c523ce42781ad5048b1c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a1f38511-2485-4de7-928d-06a3031c15ef", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ce6c5165ed04b2fc741494e193020bde6222e6e5de6ab9752c59f95d3bbe7ecd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8242b2c9-ab33-46d2-b009-081f84c03c64", "node_type": "1", "metadata": {}, "hash": "8afa97cc5acd2cc2c8a615f65a45c5d50bdf2d608544d1fb1f7bbef4dd07e52b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12490, "end_char_idx": 15969, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
def display_img_local(image_path: str):
    img = Image.open(image_path)
    display(img)


pdf_path = 'data/org-chart-sample.pdf'
output_path = 'org-chart-sample.jpg'

convert_pdf_page_to_jpg(pdf_path, output_path)
display_img_local(output_path)
```


    
![png](output_12_0.png)
    


The organizational chart image has been successfully extracted from the PDF file and displayed. Let's now define a function to analyze the organizational chart image using the new GPT4 Turbo with Vision. The function will extract information about the employees, their roles, and their managers from the image. We will use function/tool calling to specify the input parameters for the organizational structure, such as the employee name, role, and manager's name and role. We will use Pydantic models to define the structure of the data.



```python
base64_img = encode_image(output_path)

class RoleEnum(str, Enum):
    """Defines possible roles within an organization."""
    CEO = "CEO"
    CTO = "CTO"
    CFO = "CFO"
    COO = "COO"
    EMPLOYEE = "Employee"
    MANAGER = "Manager"
    INTERN = "Intern"
    OTHER = "Other"

class Employee(BaseModel):
    """Represents an employee, including their name, role, and optional manager information."""
    employee_name: str = Field(..., description="The name of the employee")
    role: RoleEnum = Field(..., description="The role of the employee")
    manager_name: Optional[str] = Field(None, description="The manager's name, if applicable")
    manager_role: Optional[RoleEnum] = Field(None, description="The manager's role, if applicable")


class EmployeeList(BaseModel):
    """A list of employees within the organizational structure."""
    employees: List[Employee] = Field(..., description="A list of employees")

def parse_orgchart(base64_img: str) -> EmployeeList:
    response = instructor.from_openai(OpenAI()).chat.completions.create(
        model='gpt-4-turbo',
        response_model=EmployeeList,
        messages=[
            {
                "role": "user",
                "content": 'Analyze the given organizational chart and very carefully extract the information.',
            },
            {
                "role": "user",
                "content": [
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": f"data:image/jpeg;base64,{base64_img}"
                        }
                    },
                ],
            }
        ],
    )
    return response
```

Now, we will define a function to parse the response from GPT-4 Turbo with Vision and extract the employee data. We will tabulate the extracted data for easy visualization. Please note that the accuracy of the extracted data may vary based on the complexity and clarity of the input image.


```python
# call the functions to analyze the organizational chart and parse the response
result = parse_orgchart(base64_img)

# tabulate the extracted data
df = pd.DataFrame([{
    'employee_name': employee.employee_name,
    'role': employee.role.value,
    'manager_name': employee.manager_name,
    'manager_role': employee.manager_role.value if employee.manager_role else None
} for employee in result.employees])

display(df)
```


<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;


key: _node_content
string_value:
{"id_": "8242b2c9-ab33-46d2-b009-081f84c03c64", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "25b20c91-35fc-4666-b23f-0f187e4bb3d5", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "dd12f61a1673bab75775a08698bc39b904361ac81726c523ce42781ad5048b1c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c4724204-c7a4-4262-9c16-23d76cdd3608", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "6de0ce211bff2774ec83e80ae37710cae4eb9b30088172611d7ec25bcd604c7a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "33b11382-4d5a-41ab-b7c2-89f400539dbc", "node_type": "1", "metadata": {}, "hash": "ece6b649218241031b7087b48f3aca6603da0a823f8531daa215ea826646111d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14079, "end_char_idx": 16658, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
class EmployeeList(BaseModel):
    """A list of employees within the organizational structure."""
    employees: List[Employee] = Field(..., description="A list of employees")

def parse_orgchart(base64_img: str) -> EmployeeList:
    response = instructor.from_openai(OpenAI()).chat.completions.create(
        model='gpt-4-turbo',
        response_model=EmployeeList,
        messages=[
            {
                "role": "user",
                "content": 'Analyze the given organizational chart and very carefully extract the information.',
            },
            {
                "role": "user",
                "content": [
                    {
                        "type": "image_url",
                        "image_url": {
                            "url": f"data:image/jpeg;base64,{base64_img}"
                        }
                    },
                ],
            }
        ],
    )
    return response
```

Now, we will define a function to parse the response from GPT-4 Turbo with Vision and extract the employee data. We will tabulate the extracted data for easy visualization. Please note that the accuracy of the extracted data may vary based on the complexity and clarity of the input image.


```python
# call the functions to analyze the organizational chart and parse the response
result = parse_orgchart(base64_img)

# tabulate the extracted data
df = pd.DataFrame([{
    'employee_name': employee.employee_name,
    'role': employee.role.value,
    'manager_name': employee.manager_name,
    'manager_role': employee.manager_role.value if employee.manager_role else None
} for employee in result.employees])

display(df)
```


<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>employee_name</th>
      <th>role</th>
      <th>manager_name</th>
      <th>manager_role</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Juliana Silva</td>
      <td>CEO</td>
      <td>None</td>
      <td>None</td>
    </tr>
    <tr>
      <th>1</th>
      <td>Kim Chun Hei</td>
      <td>CFO</td>
      <td>Juliana Silva</td>
      <td>CEO</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Chad Gibbons</td>
      <td>CTO</td>
      <td>Juliana Silva</td>
      <td>CEO</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Chiaki Sato</td>


key: _node_content
string_value:
{"id_": "33b11382-4d5a-41ab-b7c2-89f400539dbc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "25b20c91-35fc-4666-b23f-0f187e4bb3d5", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "dd12f61a1673bab75775a08698bc39b904361ac81726c523ce42781ad5048b1c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8242b2c9-ab33-46d2-b009-081f84c03c64", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f51b16322cc6018674fe38a22b7f4c1f76d78510b893d952af302601f57a9aa6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6f1f45fe-9d2a-412d-ab9c-6cd29357032a", "node_type": "1", "metadata": {}, "hash": "1a83601e0883f8d7bd158bd92af26ca072184d6a5c88e490c4591e25bed6e27f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15914, "end_char_idx": 17294, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
}

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>employee_name</th>
      <th>role</th>
      <th>manager_name</th>
      <th>manager_role</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>Juliana Silva</td>
      <td>CEO</td>
      <td>None</td>
      <td>None</td>
    </tr>
    <tr>
      <th>1</th>
      <td>Kim Chun Hei</td>
      <td>CFO</td>
      <td>Juliana Silva</td>
      <td>CEO</td>
    </tr>
    <tr>
      <th>2</th>
      <td>Chad Gibbons</td>
      <td>CTO</td>
      <td>Juliana Silva</td>
      <td>CEO</td>
    </tr>
    <tr>
      <th>3</th>
      <td>Chiaki Sato</td>
      <td>COO</td>
      <td>Juliana Silva</td>
      <td>CEO</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Cahaya Dewi</td>
      <td>Manager</td>
      <td>Kim Chun Hei</td>
      <td>CFO</td>
    </tr>
    <tr>
      <th>5</th>
      <td>Shawn Garcia</td>
      <td>Manager</td>
      <td>Chad Gibbons</td>
      <td>CTO</td>
    </tr>
    <tr>
      <th>6</th>
      <td>Aaron Loeb</td>
      <td>Manager</td>
      <td>Chiaki Sato</td>
      <td>COO</td>
    </tr>
    <tr>
      <th>7</th>
      <td>Drew Feig</td>
      <td>Employee</td>
      <td>Cahaya Dewi</td>
      <td>Manager</td>
    </tr>
    <tr>
      <th>8</th>


key: _node_content
string_value:
{"id_": "6f1f45fe-9d2a-412d-ab9c-6cd29357032a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "25b20c91-35fc-4666-b23f-0f187e4bb3d5", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "dd12f61a1673bab75775a08698bc39b904361ac81726c523ce42781ad5048b1c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "33b11382-4d5a-41ab-b7c2-89f400539dbc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "65e2eca3add52e7be4d52ace1d434031dead81c0df7e4f9ce99de5b32cbfa736", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4a8c0909-029d-4878-8b5e-fd5afba22701", "node_type": "1", "metadata": {}, "hash": "969460755f127eb82048844676c1719f5c0e279955c7e0963406270c7455c8d1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16638, "end_char_idx": 17968, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<td>Chiaki Sato</td>
      <td>COO</td>
      <td>Juliana Silva</td>
      <td>CEO</td>
    </tr>
    <tr>
      <th>4</th>
      <td>Cahaya Dewi</td>
      <td>Manager</td>
      <td>Kim Chun Hei</td>
      <td>CFO</td>
    </tr>
    <tr>
      <th>5</th>
      <td>Shawn Garcia</td>
      <td>Manager</td>
      <td>Chad Gibbons</td>
      <td>CTO</td>
    </tr>
    <tr>
      <th>6</th>
      <td>Aaron Loeb</td>
      <td>Manager</td>
      <td>Chiaki Sato</td>
      <td>COO</td>
    </tr>
    <tr>
      <th>7</th>
      <td>Drew Feig</td>
      <td>Employee</td>
      <td>Cahaya Dewi</td>
      <td>Manager</td>
    </tr>
    <tr>
      <th>8</th>
      <td>Richard Sanchez</td>
      <td>Employee</td>
      <td>Cahaya Dewi</td>
      <td>Manager</td>
    </tr>
    <tr>
      <th>9</th>
      <td>Sacha Dubois</td>
      <td>Intern</td>
      <td>Cahaya Dewi</td>
      <td>Manager</td>
    </tr>
    <tr>
      <th>10</th>
      <td>Olivia Wilson</td>
      <td>Employee</td>
      <td>Shawn Garcia</td>
      <td>Manager</td>
    </tr>
    <tr>
      <th>11</th>
      <td>Matt Zhang</td>
      <td>Intern</td>
      <td>Shawn Garcia</td>
      <td>Manager</td>
    </tr>
    <tr>
      <th>12</th>
      <td>Avery Davis</td>
      <td>Employee</td>
      <td>Aaron Loeb</td>
      <td>Manager</td>
    </tr>
    <tr>


key: _node_content
string_value:
{"id_": "4a8c0909-029d-4878-8b5e-fd5afba22701", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "25b20c91-35fc-4666-b23f-0f187e4bb3d5", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "dd12f61a1673bab75775a08698bc39b904361ac81726c523ce42781ad5048b1c", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6f1f45fe-9d2a-412d-ab9c-6cd29357032a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/multimodal/Using_GPT4_Vision_With_Function_Calling.ipynb", "title": "How to use GPT-4 Vision with Function Calling", "tokens": 4334, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "324b5399da40f34ea71171c394ebde9102acca5cdd7d35786a53385a288b8310", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17263, "end_char_idx": 18712, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
</tr>
    <tr>
      <th>8</th>
      <td>Richard Sanchez</td>
      <td>Employee</td>
      <td>Cahaya Dewi</td>
      <td>Manager</td>
    </tr>
    <tr>
      <th>9</th>
      <td>Sacha Dubois</td>
      <td>Intern</td>
      <td>Cahaya Dewi</td>
      <td>Manager</td>
    </tr>
    <tr>
      <th>10</th>
      <td>Olivia Wilson</td>
      <td>Employee</td>
      <td>Shawn Garcia</td>
      <td>Manager</td>
    </tr>
    <tr>
      <th>11</th>
      <td>Matt Zhang</td>
      <td>Intern</td>
      <td>Shawn Garcia</td>
      <td>Manager</td>
    </tr>
    <tr>
      <th>12</th>
      <td>Avery Davis</td>
      <td>Employee</td>
      <td>Aaron Loeb</td>
      <td>Manager</td>
    </tr>
    <tr>
      <th>13</th>
      <td>Harper Russo</td>
      <td>Employee</td>
      <td>Aaron Loeb</td>
      <td>Manager</td>
    </tr>
    <tr>
      <th>14</th>
      <td>Taylor Alonso</td>
      <td>Intern</td>
      <td>Aaron Loeb</td>
      <td>Manager</td>
    </tr>
  </tbody>
</table>
</div>


The extracted data from the organizational chart has been successfully parsed and displayed in a DataFrame. This approach allows us to leverage GPT-4 Turbo with Vision capabilities to extract structured information from images, such as organizational charts and diagrams, and process the data for further analysis. By using function calling, we can extend the functionality of multimodal models to perform specific tasks or call external functions.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_wandb.ipynb


key: _node_content
string_value:
{"id_": "6943251d-ee90-433e-b0df-f1e5a236687b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_wandb.ipynb", "title": "Load the embeddings", "tokens": 506, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "0c6c670c-1e11-4e6e-96fd-26c5ab3bc478", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_wandb.ipynb", "title": "Load the embeddings", "tokens": 506, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "9755f69133468856653acb6541fd6cd6611153750fa57cfc2357518380cf3dea", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2071, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Visualizing embeddings in W&B

We will upload the data to [Weights & Biases](http://wandb.ai) and use an [Embedding Projector](https://docs.wandb.ai/ref/app/features/panels/weave/embedding-projector) to visualize the embeddings using common dimension reduction algorithms like PCA, UMAP, and t-SNE. The dataset is created in the [Get_embeddings_from_dataset Notebook](Get_embeddings_from_dataset.ipynb).

## What is Weights & Biases?

[Weights & Biases](http://wandb.ai) is a machine learning platform used by OpenAI and other ML teams to build better models faster. They use it to quickly track experiments, evaluate model performance, reproduce models, visualize results, and share findings with colleagues.

### 1. Log the data to W&B

We create a [W&B Table](https://docs.wandb.ai/guides/data-vis/log-tables) with the original data and the embeddings. Each review is a new row and the 1536 embedding floats are given their own column named `emb_{i}`.


```python
import pandas as pd
from sklearn.manifold import TSNE
import numpy as np
from ast import literal_eval

# Load the embeddings
datafile_path = "data/fine_food_reviews_with_embeddings_1k.csv"
df = pd.read_csv(datafile_path)

# Convert to a list of lists of floats
matrix = np.array(df.embedding.apply(literal_eval).to_list())
```


```python
import wandb

original_cols = df.columns[1:-1].tolist()
embedding_cols = ['emb_'+str(idx) for idx in range(len(matrix[0]))]
table_cols = original_cols + embedding_cols

with wandb.init(project='openai_embeddings'):
    table = wandb.Table(columns=table_cols)
    for i, row in enumerate(df.to_dict(orient="records")):
        original_data = [row[col_name] for col_name in original_cols]
        embedding_data = matrix[i].tolist()
        table.add_data(*(original_data + embedding_data))
    wandb.log({'openai_embedding_table': table})
```

### 2. Render as 2D Projection

After navigating to the W&B run link, we click the ⚙️ icon in the top right of the Table and change "Render As:" to "Combined 2D Projection". 

Example: http://wandb.me/openai_embeddings


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb


key: _node_content
string_value:
{"id_": "ae63c4c9-107d-4aff-a7be-3914924d6dc2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb", "title": "How to automate tasks with functions (S3 bucket example)", "tokens": 2600, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d0b7b93-c6cd-4faa-a630-bd979d0132d8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb", "title": "How to automate tasks with functions (S3 bucket example)", "tokens": 2600, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ba8efa2f971d91bc6653c1258f739403579f86d87cc9b6063d7446084ab60f90", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "176e088b-10d7-45d2-81d6-78a3369ba459", "node_type": "1", "metadata": {}, "hash": "5c238dd0605c3618695e3c739f33801a19df5a659132e2b7841934c4807a5fdf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1733, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to automate tasks with functions (S3 bucket example)

This code demonstrates how to interact with ChatGPT functions to perform tasks related to Amazon S3 buckets. The notebook covers S3 bucket key functionalities such as running simple listing commands, searching for a specific file in all buckets, uploading a file to a bucket, and downloading a file from a bucket. The OpenAI Chat API understands the user instructions, generates the natural language responses, and extracts appropriate function calls based on the user's input.

**Requirements**:
To run the notebook generate AWS access key with S3 bucket writing permission and store them in a local environment file alongside the Openai key. The "`.env`" file format:
```
AWS_ACCESS_KEY_ID=<your-key>
AWS_SECRET_ACCESS_KEY=<your-key>
OPENAI_API_KEY=<your-key>
```


```python
! pip install openai
! pip install boto3
! pip install tenacity
! pip install python-dotenv
```


```python
from openai import OpenAI
import json
import boto3
import os
import datetime
from urllib.request import urlretrieve

# load environment variables
from dotenv import load_dotenv
load_dotenv() 
```




    True



## Initials


```python
OpenAI.api_key = os.environ.get("OPENAI_API_KEY")
GPT_MODEL = "gpt-3.5-turbo"
```


```python
# Optional - if you had issues loading the environment file, you can set the AWS values using the below code
# os.environ['AWS_ACCESS_KEY_ID'] = ''
# os.environ['AWS_SECRET_ACCESS_KEY'] = ''

# Create S3 client
s3_client = boto3.client('s3')

# Create openai client
client = OpenAI()
```

## Utilities

To connect user questions or commands to the appropriate function, we need to provide ChatGPT with the necessary function details and expected parameters.


key: _node_content
string_value:
{"id_": "176e088b-10d7-45d2-81d6-78a3369ba459", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb", "title": "How to automate tasks with functions (S3 bucket example)", "tokens": 2600, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d0b7b93-c6cd-4faa-a630-bd979d0132d8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb", "title": "How to automate tasks with functions (S3 bucket example)", "tokens": 2600, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ba8efa2f971d91bc6653c1258f739403579f86d87cc9b6063d7446084ab60f90", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ae63c4c9-107d-4aff-a7be-3914924d6dc2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb", "title": "How to automate tasks with functions (S3 bucket example)", "tokens": 2600, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "93ff23939b8b3257c38a002c1c93b5aac64304109c5698dcdbe90890fc0eaa84", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a1a54710-b5a0-46a7-acf4-eda17ef33b2c", "node_type": "1", "metadata": {}, "hash": "601087bcb95d65bb471ab51e817d00b8f2b784a8275e195e9023b61bb302d294", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 5303, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to automate tasks with functions (S3 bucket example)

This code demonstrates how to interact with ChatGPT functions to perform tasks related to Amazon S3 buckets. The notebook covers S3 bucket key functionalities such as running simple listing commands, searching for a specific file in all buckets, uploading a file to a bucket, and downloading a file from a bucket. The OpenAI Chat API understands the user instructions, generates the natural language responses, and extracts appropriate function calls based on the user's input.

**Requirements**:
To run the notebook generate AWS access key with S3 bucket writing permission and store them in a local environment file alongside the Openai key. The "`.env`" file format:
```
AWS_ACCESS_KEY_ID=<your-key>
AWS_SECRET_ACCESS_KEY=<your-key>
OPENAI_API_KEY=<your-key>
```


```python
! pip install openai
! pip install boto3
! pip install tenacity
! pip install python-dotenv
```


```python
from openai import OpenAI
import json
import boto3
import os
import datetime
from urllib.request import urlretrieve

# load environment variables
from dotenv import load_dotenv
load_dotenv() 
```




    True



## Initials


```python
OpenAI.api_key = os.environ.get("OPENAI_API_KEY")
GPT_MODEL = "gpt-3.5-turbo"
```


```python
# Optional - if you had issues loading the environment file, you can set the AWS values using the below code
# os.environ['AWS_ACCESS_KEY_ID'] = ''
# os.environ['AWS_SECRET_ACCESS_KEY'] = ''

# Create S3 client
s3_client = boto3.client('s3')

# Create openai client
client = OpenAI()
```

## Utilities

To connect user questions or commands to the appropriate function, we need to provide ChatGPT with the necessary function details and expected parameters.


```python
# Functions dict to pass S3 operations details for the GPT model
functions = [
    {   
        "type": "function",
        "function":{
            "name": "list_buckets",
            "description": "List all available S3 buckets",
            "parameters": {
                "type": "object",
                "properties": {}
            }
        }
    },
    {
        "type": "function",
        "function":{
            "name": "list_objects",
            "description": "List the objects or files inside a given S3 bucket",
            "parameters": {
                "type": "object",
                "properties": {
                    "bucket": {"type": "string", "description": "The name of the S3 bucket"},
                    "prefix": {"type": "string", "description": "The folder path in the S3 bucket"},
                },
                "required": ["bucket"],
            },
        }
    },
    {   
        "type": "function",
        "function":{
            "name": "download_file",
            "description": "Download a specific file from an S3 bucket to a local distribution folder.",
            "parameters": {
                "type": "object",
                "properties": {
                    "bucket": {"type": "string", "description": "The name of the S3 bucket"},
                    "key": {"type": "string", "description": "The path to the file inside the bucket"},
                    "directory": {"type": "string", "description": "The local destination directory to download the file, should be specificed by the user."},
                },
                "required": ["bucket", "key", "directory"],
            }
        }
    },
    {
        "type": "function",
        "function":{
            "name": "upload_file",
            "description": "Upload a file to an S3 bucket",
            "parameters": {
                "type": "object",
                "properties": {
                    "source": {"type": "string", "description": "The local source path or remote URL"},
                    "bucket": {"type": "string", "description": "The name of the S3 bucket"},
                    "key": {"type": "string", "description": "The path to the file inside the bucket"},
                    "is_remote_url": {"type": "boolean", "description": "Is the provided source a URL (True) or local path (False)"},
                },
                "required": ["source", "bucket", "key", "is_remote_url"],
            }
        }
    },
    {
        "type": "function",
        "function":{
            "name": "search_s3_objects",
            "description": "Search for a specific file name inside an S3 bucket",
            "parameters": {
                "type": "object",
                "properties": {
                    "search_name": {"type": "string", "description": "The name of the file you want to search for"},
                    "bucket": {"type": "string", "description": "The name of the S3 bucket"},
                    "prefix": {"type": "string", "description": "The folder path in the S3 bucket"},
                    "exact_match": {"type": "boolean", "description": "Set exact_match to True if the search should match the exact file name. Set exact_match to False to compare part of the file name string (the file contains)"}
                },
                "required": ["search_name"],
            },
        }
    }
]
```

Create helper functions to interact with the S3 service, such as listing buckets, listing objects, downloading and uploading files, and searching for specific files.


key: _node_content
string_value:
{"id_": "a1a54710-b5a0-46a7-acf4-eda17ef33b2c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb", "title": "How to automate tasks with functions (S3 bucket example)", "tokens": 2600, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d0b7b93-c6cd-4faa-a630-bd979d0132d8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb", "title": "How to automate tasks with functions (S3 bucket example)", "tokens": 2600, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ba8efa2f971d91bc6653c1258f739403579f86d87cc9b6063d7446084ab60f90", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "176e088b-10d7-45d2-81d6-78a3369ba459", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb", "title": "How to automate tasks with functions (S3 bucket example)", "tokens": 2600, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ae09f84e5de46a76abc534f5b1485f579006d62c1067bd884cff374dd7e64986", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d15c91e5-0fdb-4573-9f0b-a23a721f838e", "node_type": "1", "metadata": {}, "hash": "dc11f89f7730b179f1533250e1d70834a4a7ce12f54f9762df55c9ff31349b0d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5306, "end_char_idx": 8423, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def datetime_converter(obj):
    if isinstance(obj, datetime.datetime):
        return obj.isoformat()
    raise TypeError(f"Object of type {obj.__class__.__name__} is not JSON serializable")
```


```python
def list_buckets():
    response = s3_client.list_buckets()
    return json.dumps(response['Buckets'], default=datetime_converter)

def list_objects(bucket, prefix=''):
    response = s3_client.list_objects_v2(Bucket=bucket, Prefix=prefix)
    return json.dumps(response.get('Contents', []), default=datetime_converter)

def download_file(bucket, key, directory):
    
    filename = os.path.basename(key)
    
    # Resolve destination to the correct file path
    destination = os.path.join(directory, filename)
    
    s3_client.download_file(bucket, key, destination)
    return json.dumps({"status": "success", "bucket": bucket, "key": key, "destination": destination})

def upload_file(source, bucket, key, is_remote_url=False):
    if is_remote_url:
        file_name = os.path.basename(source)
        urlretrieve(source, file_name)
        source = file_name
       
    s3_client.upload_file(source, bucket, key)
    return json.dumps({"status": "success", "source": source, "bucket": bucket, "key": key})

def search_s3_objects(search_name, bucket=None, prefix='', exact_match=True):
    search_name = search_name.lower()
    
    if bucket is None:
        buckets_response = json.loads(list_buckets())
        buckets = [bucket_info["Name"] for bucket_info in buckets_response]
    else:
        buckets = [bucket]

    results = []

    for bucket_name in buckets:
        objects_response = json.loads(list_objects(bucket_name, prefix))
        if exact_match:
            bucket_results = [obj for obj in objects_response if search_name == obj['Key'].lower()]
        else:
            bucket_results = [obj for obj in objects_response if search_name in obj['Key'].lower()]

        if bucket_results:
            results.extend([{"Bucket": bucket_name, "Object": obj} for obj in bucket_results])

    return json.dumps(results)
```

The below dictionary connects the name with the function to use it for execution based on ChatGPT responses.


```python
available_functions = {
    "list_buckets": list_buckets,
    "list_objects": list_objects,
    "download_file": download_file,
    "upload_file": upload_file,
    "search_s3_objects": search_s3_objects
}
```

## ChatGPT


```python
def chat_completion_request(messages, functions=None, function_call='auto', 
                            model_name=GPT_MODEL):
    
    if functions is not None:
        return client.chat.completions.create(
            model=model_name,
            messages=messages,
            tools=functions,
            tool_choice=function_call)
    else:
        return client.chat.completions.create(
            model=model_name,
            messages=messages)
```

### Conversation flow

Create a main function for the chatbot, which takes user input, sends it to the OpenAI Chat API, receives a response, executes any function calls generated by the API, and returns a final response to the user.


key: _node_content
string_value:
{"id_": "d15c91e5-0fdb-4573-9f0b-a23a721f838e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb", "title": "How to automate tasks with functions (S3 bucket example)", "tokens": 2600, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d0b7b93-c6cd-4faa-a630-bd979d0132d8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb", "title": "How to automate tasks with functions (S3 bucket example)", "tokens": 2600, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ba8efa2f971d91bc6653c1258f739403579f86d87cc9b6063d7446084ab60f90", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a1a54710-b5a0-46a7-acf4-eda17ef33b2c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb", "title": "How to automate tasks with functions (S3 bucket example)", "tokens": 2600, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "10d0d36e63e0a1bab8138097d8ef723cc7cb0bd1f4eac2d66a7d61a385b1d910", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "229bd745-6aa8-441d-89e6-58781908e628", "node_type": "1", "metadata": {}, "hash": "a0e08ba5183d28fe846d2380010a8c01ccd9e5d2fa2e3c7295f3f7ff485a7090", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7486, "end_char_idx": 10861, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
available_functions = {
    "list_buckets": list_buckets,
    "list_objects": list_objects,
    "download_file": download_file,
    "upload_file": upload_file,
    "search_s3_objects": search_s3_objects
}
```

## ChatGPT


```python
def chat_completion_request(messages, functions=None, function_call='auto', 
                            model_name=GPT_MODEL):
    
    if functions is not None:
        return client.chat.completions.create(
            model=model_name,
            messages=messages,
            tools=functions,
            tool_choice=function_call)
    else:
        return client.chat.completions.create(
            model=model_name,
            messages=messages)
```

### Conversation flow

Create a main function for the chatbot, which takes user input, sends it to the OpenAI Chat API, receives a response, executes any function calls generated by the API, and returns a final response to the user.


```python
def run_conversation(user_input, topic="S3 bucket functions.", is_log=False):

    system_message=f"Don't make assumptions about what values to plug into functions. Ask for clarification if a user request is ambiguous. If the user ask question not related to {topic} response your scope is {topic} only."
    
    messages = [{"role": "system", "content": system_message},
                {"role": "user", "content": user_input}]
    
    # Call the model to get a response
    response = chat_completion_request(messages, functions=functions)
    response_message = response.choices[0].message
    
    if is_log:
        print(response.choices)
    
    # check if GPT wanted to call a function
    if response_message.tool_calls:
        function_name = response_message.tool_calls[0].function.name
        function_args = json.loads(response_message.tool_calls[0].function.arguments)
        
        # Call the function
        function_response = available_functions[function_name](**function_args)
        
        # Add the response to the conversation
        messages.append(response_message)
        messages.append({
            "role": "tool",
            "content": function_response,
            "tool_call_id": response_message.tool_calls[0].id,
        })
        
        # Call the model again to summarize the results
        second_response = chat_completion_request(messages)
        final_message = second_response.choices[0].message.content
    else:
        final_message = response_message.content

    return final_message
```

### S3 bucket bot testing
In the following examples, make sure to replace the placeholders such as `<file_name>`, `<bucket_name>`, and `<directory_path>` with your specific values before execution.

#### Listing and searching

Let's start by listing all the available buckets.


```python
print(run_conversation('list my S3 buckets'))
```

You can ask the assistant to search for a specific file name either in all the buckets or in a specific one.


```python
search_file = '<file_name>'
print(run_conversation(f'search for a file {search_file} in all buckets'))
```


```python
search_word = '<file_name_part>'
bucket_name = '<bucket_name>'
print(run_conversation(f'search for a file contains {search_word} in {bucket_name}'))
```

The model is expected to clarify the ask from the user in case of ambiguity in the parameters values as described in the system message.


key: _node_content
string_value:
{"id_": "229bd745-6aa8-441d-89e6-58781908e628", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb", "title": "How to automate tasks with functions (S3 bucket example)", "tokens": 2600, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "3d0b7b93-c6cd-4faa-a630-bd979d0132d8", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb", "title": "How to automate tasks with functions (S3 bucket example)", "tokens": 2600, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ba8efa2f971d91bc6653c1258f739403579f86d87cc9b6063d7446084ab60f90", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d15c91e5-0fdb-4573-9f0b-a23a721f838e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/How_to_automate_S3_storage_with_functions.ipynb", "title": "How to automate tasks with functions (S3 bucket example)", "tokens": 2600, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3298cbc60ac567678ff57c9d8182e02c7b68caa30ffb7eeb449bf0e257457fbb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10270, "end_char_idx": 12140, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
print(run_conversation('list my S3 buckets'))
```

You can ask the assistant to search for a specific file name either in all the buckets or in a specific one.


```python
search_file = '<file_name>'
print(run_conversation(f'search for a file {search_file} in all buckets'))
```


```python
search_word = '<file_name_part>'
bucket_name = '<bucket_name>'
print(run_conversation(f'search for a file contains {search_word} in {bucket_name}'))
```

The model is expected to clarify the ask from the user in case of ambiguity in the parameters values as described in the system message.


```python
print(run_conversation('search for a file'))
```

    Sure, to help me find what you're looking for, could you please provide the name of the file you want to search for and the name of the S3 bucket? Also, should the search match the file name exactly, or should it also consider partial matches?


#### Validate edge cases

We also instructed the model to reject irrelevant tasks. Let's test it out and see how it works in action.


```python
# the model should not answer details not related to the scope
print(run_conversation('what is the weather today'))
```

    Apologies for the misunderstanding, but I am only able to assist with S3 bucket functions. Can you please ask a question related to S3 bucket functions?


The provided functions are not limited to just retrieving information. They can also assist the user in uploading or downloading files.

#### Download a file


```python
search_file = '<file_name>'
bucket_name = '<bucket_name>'
local_directory = '<directory_path>'
print(run_conversation(f'download {search_file} from {bucket_name} bucket to {local_directory} directory'))
```

#### Upload a file


```python
local_file = '<file_name>'
bucket_name = '<bucket_name>'
print(run_conversation(f'upload {local_file} to {bucket_name} bucket'))
```


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb


key: _node_content
string_value:
{"id_": "115930ee-4260-407c-9ec4-8741355430b3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "27b15505-1885-48f9-96f5-a3effb50e0a0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "75a89f006f1de311679cb3df02eccc11236f8ebf85f52e9cfcb503a35e467985", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5ee2f7a0-9254-4f59-9468-fbfa23bedb80", "node_type": "1", "metadata": {}, "hash": "99155a013f90dc7ac8d4103ce716eee1247240b046cb70f460ba84886c0e25ff", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2463, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Visualizing the embeddings in Kangas

In this Jupyter Notebook, we construct a Kangas DataGrid containing the data and projections of the embeddings into 2 dimensions.

## What is Kangas?

[Kangas](https://github.com/comet-ml/kangas/) as an open source, mixed-media, dataframe-like tool for data scientists. It was developed by [Comet](https://comet.com/), a company designed to help reduce the friction of moving models into production. 

### 1. Setup

To get started, we pip install kangas, and import it.


```python
%pip install kangas --quiet
```


```python
import kangas as kg
```

### 2. Constructing a Kangas DataGrid

We create a Kangas Datagrid with the original data and the embeddings. The data is composed of a rows of reviews, and the embeddings are composed of 1536 floating-point values. In this example, we get the data directly from github, in case you aren't running this notebook inside OpenAI's repo.

We use Kangas to read the CSV file into a DataGrid for further processing.


```python
data = kg.read_csv("https://raw.githubusercontent.com/openai/openai-cookbook/main/examples/data/fine_food_reviews_with_embeddings_1k.csv")
```

    Loading CSV file 'fine_food_reviews_with_embeddings_1k.csv'...


    1001it [00:00, 2412.90it/s]
    100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 2899.16it/s]


We can review the fields of the CSV file:


```python
data.info()
```

    DataGrid (in memory)
        Name   : fine_food_reviews_with_embeddings_1k
        Rows   : 1,000
        Columns: 9
    #   Column                Non-Null Count DataGrid Type       
    --- -------------------- --------------- --------------------
    1   Column 1                       1,000 INTEGER             
    2   ProductId                      1,000 TEXT                
    3   UserId                         1,000 TEXT                
    4   Score                          1,000 INTEGER             
    5   Summary                        1,000 TEXT                
    6   Text                           1,000 TEXT                
    7   combined                       1,000 TEXT                
    8   n_tokens                       1,000 INTEGER             
    9   embedding                      1,000 TEXT                


And get a glimpse of the first and last rows:


```python
data
```


key: _node_content
string_value:
{"id_": "5ee2f7a0-9254-4f59-9468-fbfa23bedb80", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "27b15505-1885-48f9-96f5-a3effb50e0a0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "75a89f006f1de311679cb3df02eccc11236f8ebf85f52e9cfcb503a35e467985", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "115930ee-4260-407c-9ec4-8741355430b3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "eb01b02c99c76902db7d736d3b1f0a0d37b111a951907b0fb4411edab558466f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "96953d65-4ab9-4ee4-ad89-75db56cc8898", "node_type": "1", "metadata": {}, "hash": "6c56da3694b97000668891868af06ad4ef2cd5b6fc8871ebe728ccfe9bf05576", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1004, "end_char_idx": 3595, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
data = kg.read_csv("https://raw.githubusercontent.com/openai/openai-cookbook/main/examples/data/fine_food_reviews_with_embeddings_1k.csv")
```

    Loading CSV file 'fine_food_reviews_with_embeddings_1k.csv'...


    1001it [00:00, 2412.90it/s]
    100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 1000/1000 [00:00<00:00, 2899.16it/s]


We can review the fields of the CSV file:


```python
data.info()
```

    DataGrid (in memory)
        Name   : fine_food_reviews_with_embeddings_1k
        Rows   : 1,000
        Columns: 9
    #   Column                Non-Null Count DataGrid Type       
    --- -------------------- --------------- --------------------
    1   Column 1                       1,000 INTEGER             
    2   ProductId                      1,000 TEXT                
    3   UserId                         1,000 TEXT                
    4   Score                          1,000 INTEGER             
    5   Summary                        1,000 TEXT                
    6   Text                           1,000 TEXT                
    7   combined                       1,000 TEXT                
    8   n_tokens                       1,000 INTEGER             
    9   embedding                      1,000 TEXT                


And get a glimpse of the first and last rows:


```python
data
```




<table><th colspan='1' >          row-id </th> <th colspan='1' >        Column 1 </th> <th colspan='1' >       ProductId </th> <th colspan='1' >          UserId </th> <th colspan='1' >           Score </th> <th colspan='1' >         Summary </th> <th colspan='1' >            Text </th> <th colspan='1' >        combined </th> <th colspan='1' >        n_tokens </th> <th colspan='1' >       embedding </th> <tr>
<td colspan='1' >               1 </td> <td colspan='1' >               0 </td> <td colspan='1' >      B003XPF9BO </td> <td colspan='1' >  A3R7JR3FMEBXQB </td> <td colspan='1' >               5 </td> <td colspan='1' > where does one  </td> <td colspan='1' > Wanted to save  </td> <td colspan='1' > Title: where do </td> <td colspan='1' >              52 </td> <td colspan='1' > [0.007018072064 </td> <tr>
<td colspan='1' >               2 </td> <td colspan='1' >             297 </td> <td colspan='1' >      B003VXHGPK </td> <td colspan='1' >  A21VWSCGW7UUAR </td> <td colspan='1' >               4 </td> <td colspan='1' > Good, but not W </td> <td colspan='1' > Honestly, I hav </td> <td colspan='1' > Title: Good,


key: _node_content
string_value:
{"id_": "96953d65-4ab9-4ee4-ad89-75db56cc8898", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "27b15505-1885-48f9-96f5-a3effb50e0a0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "75a89f006f1de311679cb3df02eccc11236f8ebf85f52e9cfcb503a35e467985", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5ee2f7a0-9254-4f59-9468-fbfa23bedb80", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "97259f55fe468c8190a4131223366207f2bd4199bf592627cb7d897cfae0f239", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e342b8be-66b8-49c4-b0d2-111fec398626", "node_type": "1", "metadata": {}, "hash": "60bdff6a84fa0ce52bc6466d535d76a9881b8113a5d9f83f62075ba131967a28", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2468, "end_char_idx": 4477, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<table><th colspan='1' >          row-id </th> <th colspan='1' >        Column 1 </th> <th colspan='1' >       ProductId </th> <th colspan='1' >          UserId </th> <th colspan='1' >           Score </th> <th colspan='1' >         Summary </th> <th colspan='1' >            Text </th> <th colspan='1' >        combined </th> <th colspan='1' >        n_tokens </th> <th colspan='1' >       embedding </th> <tr>
<td colspan='1' >               1 </td> <td colspan='1' >               0 </td> <td colspan='1' >      B003XPF9BO </td> <td colspan='1' >  A3R7JR3FMEBXQB </td> <td colspan='1' >               5 </td> <td colspan='1' > where does one  </td> <td colspan='1' > Wanted to save  </td> <td colspan='1' > Title: where do </td> <td colspan='1' >              52 </td> <td colspan='1' > [0.007018072064 </td> <tr>
<td colspan='1' >               2 </td> <td colspan='1' >             297 </td> <td colspan='1' >      B003VXHGPK </td> <td colspan='1' >  A21VWSCGW7UUAR </td> <td colspan='1' >               4 </td> <td colspan='1' > Good, but not W </td> <td colspan='1' > Honestly, I hav </td> <td colspan='1' > Title: Good, bu </td> <td colspan='1' >             178 </td> <td colspan='1' > [-0.00314055196 </td> <tr>
<td colspan='1' >               3 </td> <td colspan='1' >             296 </td> <td colspan='1' >      B008JKTTUA </td> <td colspan='1' >  A34XBAIFT02B60 </td> <td colspan='1' >               1 </td> <td colspan='1' > Should advertis </td> <td colspan='1' > First, these sh </td> <td colspan='1' > Title: Should a </td> <td colspan='1' >              78 </td> <td colspan='1' > [-0.01757248118 </td> <tr>
<td colspan='1' >               4 </td> <td colspan='1' >             295 </td> <td colspan='1' >      B000LKTTTW </td> <td colspan='1' >  A14MQ40CCU8B13 </td> <td colspan='1' >               5 </td> <td colspan='1' > Best tomato sou </td> <td colspan='1' > I have a hard t </td> <td colspan='1' > Title: Best tom </td> <td colspan='1' >             111 </td> <td colspan='1' > [-0.


key: _node_content
string_value:
{"id_": "e342b8be-66b8-49c4-b0d2-111fec398626", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "27b15505-1885-48f9-96f5-a3effb50e0a0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "75a89f006f1de311679cb3df02eccc11236f8ebf85f52e9cfcb503a35e467985", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "96953d65-4ab9-4ee4-ad89-75db56cc8898", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "29b10349a3fcb01e5cf774d02bb2604090808094eae77661698caffdd504cd15", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ddb4e91e-31e9-4eae-a60d-a3e46461caa9", "node_type": "1", "metadata": {}, "hash": "969c92fd9b6fae0ac217144b2ec99033930de793a03be0b417be601d85f7c386", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
but not W </td> <td colspan='1' > Honestly, I hav </td> <td colspan='1' > Title: Good, bu </td> <td colspan='1' >             178 </td> <td colspan='1' > [-0.00314055196 </td> <tr>
<td colspan='1' >               3 </td> <td colspan='1' >             296 </td> <td colspan='1' >      B008JKTTUA </td> <td colspan='1' >  A34XBAIFT02B60 </td> <td colspan='1' >               1 </td> <td colspan='1' > Should advertis </td> <td colspan='1' > First, these sh </td> <td colspan='1' > Title: Should a </td> <td colspan='1' >              78 </td> <td colspan='1' > [-0.01757248118 </td> <tr>
<td colspan='1' >               4 </td> <td colspan='1' >             295 </td> <td colspan='1' >      B000LKTTTW </td> <td colspan='1' >  A14MQ40CCU8B13 </td> <td colspan='1' >               5 </td> <td colspan='1' > Best tomato sou </td> <td colspan='1' > I have a hard t </td> <td colspan='1' > Title: Best tom </td> <td colspan='1' >             111 </td> <td colspan='1' > [-0.00139322795 </td> <tr>
<td colspan='1' >               5 </td> <td colspan='1' >             294 </td> <td colspan='1' >      B001D09KAM </td> <td colspan='1' >  A34XBAIFT02B60 </td> <td colspan='1' >               1 </td> <td colspan='1' > Should advertis </td> <td colspan='1' > First, these sh </td> <td colspan='1' > Title: Should a </td> <td colspan='1' >              78 </td> <td colspan='1' > [-0.01757248118 </td> <tr>
<tr><td colspan='10' style='text-align: left;'>.</td></tr><td colspan='1' >             996 </td> <td colspan='1' >             623 </td> <td colspan='1' >      B0000CFXYA </td> <td colspan='1' >  A3GS4GWPIBV0NT </td> <td colspan='1' >               1 </td> <td colspan='1' > Strange inflamm </td> <td colspan='1' > Truthfully wasn </td> <td colspan='1' > Title: Strange  </td> <td colspan='1' >             110 </td> <td colspan='1' > [0.


key: _node_content
string_value:
{"id_": "ddb4e91e-31e9-4eae-a60d-a3e46461caa9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "27b15505-1885-48f9-96f5-a3effb50e0a0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "75a89f006f1de311679cb3df02eccc11236f8ebf85f52e9cfcb503a35e467985", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e342b8be-66b8-49c4-b0d2-111fec398626", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "04e066931487bf070cf4257b20244c18df149aae2f1a640bb71f893ca54fd28a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0e2d9df4-eace-43c7-8392-97bc4be1ed44", "node_type": "1", "metadata": {}, "hash": "2c3908d5c2f433ac619ad62af16886513f591776e8aa55e2df8f69a05ee059ba", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": null, "end_char_idx": null, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
00139322795 </td> <tr>
<td colspan='1' >               5 </td> <td colspan='1' >             294 </td> <td colspan='1' >      B001D09KAM </td> <td colspan='1' >  A34XBAIFT02B60 </td> <td colspan='1' >               1 </td> <td colspan='1' > Should advertis </td> <td colspan='1' > First, these sh </td> <td colspan='1' > Title: Should a </td> <td colspan='1' >              78 </td> <td colspan='1' > [-0.01757248118 </td> <tr>
<tr><td colspan='10' style='text-align: left;'>.</td></tr><td colspan='1' >             996 </td> <td colspan='1' >             623 </td> <td colspan='1' >      B0000CFXYA </td> <td colspan='1' >  A3GS4GWPIBV0NT </td> <td colspan='1' >               1 </td> <td colspan='1' > Strange inflamm </td> <td colspan='1' > Truthfully wasn </td> <td colspan='1' > Title: Strange  </td> <td colspan='1' >             110 </td> <td colspan='1' > [0.000110913533 </td> <tr>
<td colspan='1' >             997 </td> <td colspan='1' >             624 </td> <td colspan='1' >      B0001BH5YM </td> <td colspan='1' >   A1BZ3HMAKK0NC </td> <td colspan='1' >               5 </td> <td colspan='1' > My favorite and </td> <td colspan='1' > You've just got </td> <td colspan='1' > Title: My favor </td> <td colspan='1' >              80 </td> <td colspan='1' > [-0.02086931467 </td> <tr>
<td colspan='1' >             998 </td> <td colspan='1' >             625 </td> <td colspan='1' >      B0009ET7TC </td> <td colspan='1' >  A2FSDQY5AI6TNX </td> <td colspan='1' >               5 </td> <td colspan='1' > My furbabies LO </td> <td colspan='1' > Shake the conta </td> <td colspan='1' > Title: My furba </td> <td colspan='1' >              47 </td> <td colspan='1' > [-0.00974910240 </td> <tr>
<td colspan='1' >             999 </td> <td colspan='1' >             619 </td> <td colspan='1' >      B007PA32L2 </td> <td colspan='1' >  A15FF2P7RPKH6G </td> <td colspan='1' >               5 </td> <td colspan='1' > got this for th </td> <td colspan='1' > all i have hear </td> <td colspan='1' > Title: got this </td> <td colspan='1' >              50 </td> <td colspan='1' > [-0.


key: _node_content
string_value:
{"id_": "0e2d9df4-eace-43c7-8392-97bc4be1ed44", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "27b15505-1885-48f9-96f5-a3effb50e0a0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "75a89f006f1de311679cb3df02eccc11236f8ebf85f52e9cfcb503a35e467985", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ddb4e91e-31e9-4eae-a60d-a3e46461caa9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e58befb2223c57c1766a4623659097b15b738722d517d8ff7979910ed7b6c37e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a2b13dc6-4267-44b4-a0e7-0aa131711926", "node_type": "1", "metadata": {}, "hash": "b4315d16f37dcefbee10397aa1bfc1df2ed127ec7827285bbfc01c6dbcb42153", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5752, "end_char_idx": 7916, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
02086931467 </td> <tr>
<td colspan='1' >             998 </td> <td colspan='1' >             625 </td> <td colspan='1' >      B0009ET7TC </td> <td colspan='1' >  A2FSDQY5AI6TNX </td> <td colspan='1' >               5 </td> <td colspan='1' > My furbabies LO </td> <td colspan='1' > Shake the conta </td> <td colspan='1' > Title: My furba </td> <td colspan='1' >              47 </td> <td colspan='1' > [-0.00974910240 </td> <tr>
<td colspan='1' >             999 </td> <td colspan='1' >             619 </td> <td colspan='1' >      B007PA32L2 </td> <td colspan='1' >  A15FF2P7RPKH6G </td> <td colspan='1' >               5 </td> <td colspan='1' > got this for th </td> <td colspan='1' > all i have hear </td> <td colspan='1' > Title: got this </td> <td colspan='1' >              50 </td> <td colspan='1' > [-0.00521062919 </td> <tr>
<td colspan='1' >            1000 </td> <td colspan='1' >             999 </td> <td colspan='1' >      B001EQ5GEO </td> <td colspan='1' >  A3VYU0VO6DYV6I </td> <td colspan='1' >               5 </td> <td colspan='1' > I love Maui Cof </td> <td colspan='1' > My first experi </td> <td colspan='1' > Title: I love M </td> <td colspan='1' >             118 </td> <td colspan='1' > [-0.00605782261 </td> <tr>
<tr>
<td colspan='10' style="text-align: left;"> [1000 rows x 9 columns] </td> <tr>
<tr><td colspan='10' style='text-align: left;'></td></tr><tr><td colspan='10' style='text-align: left;'>*  Use DataGrid.save() to save to disk</td></tr><tr><td colspan='10' style='text-align: left;'>** Use DataGrid.show() to start user interface</td></tr></table>



Now, we create a new DataGrid, converting the numbers into an Embedding:


```python
import ast # to convert string of a list of numbers into a list of numbers

dg = kg.DataGrid(
    name="openai_embeddings",
    columns=data.get_columns(),
    converters={"Score": str},
)
for row in data:
    embedding = ast.literal_eval(row[8])
    row[8] = kg.Embedding(
        embedding, 
        name=str(row[3]), 
        text="%s - %.10s" % (row[3], row[4]),
        projection="umap",
    )
    dg.append(row)
```

The new DataGrid now has an Embedding column with proper datatype.


key: _node_content
string_value:
{"id_": "a2b13dc6-4267-44b4-a0e7-0aa131711926", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "27b15505-1885-48f9-96f5-a3effb50e0a0", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "75a89f006f1de311679cb3df02eccc11236f8ebf85f52e9cfcb503a35e467985", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0e2d9df4-eace-43c7-8392-97bc4be1ed44", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_in_Kangas.ipynb", "title": "## Visualizing the embeddings in Kangas", "tokens": 2958, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "eb963b02af751b225cedd8b827386685a73933f34fcafcebec17432d9f5acd3a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6967, "end_char_idx": 9839, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
00605782261 </td> <tr>
<tr>
<td colspan='10' style="text-align: left;"> [1000 rows x 9 columns] </td> <tr>
<tr><td colspan='10' style='text-align: left;'></td></tr><tr><td colspan='10' style='text-align: left;'>*  Use DataGrid.save() to save to disk</td></tr><tr><td colspan='10' style='text-align: left;'>** Use DataGrid.show() to start user interface</td></tr></table>



Now, we create a new DataGrid, converting the numbers into an Embedding:


```python
import ast # to convert string of a list of numbers into a list of numbers

dg = kg.DataGrid(
    name="openai_embeddings",
    columns=data.get_columns(),
    converters={"Score": str},
)
for row in data:
    embedding = ast.literal_eval(row[8])
    row[8] = kg.Embedding(
        embedding, 
        name=str(row[3]), 
        text="%s - %.10s" % (row[3], row[4]),
        projection="umap",
    )
    dg.append(row)
```

The new DataGrid now has an Embedding column with proper datatype.


```python
dg.info()
```

    DataGrid (in memory)
        Name   : openai_embeddings
        Rows   : 1,000
        Columns: 9
    #   Column                Non-Null Count DataGrid Type       
    --- -------------------- --------------- --------------------
    1   Column 1                       1,000 INTEGER             
    2   ProductId                      1,000 TEXT                
    3   UserId                         1,000 TEXT                
    4   Score                          1,000 TEXT                
    5   Summary                        1,000 TEXT                
    6   Text                           1,000 TEXT                
    7   combined                       1,000 TEXT                
    8   n_tokens                       1,000 INTEGER             
    9   embedding                      1,000 EMBEDDING-ASSET     


We simply save the datagrid, and we're done.


```python
dg.save()
```

### 3. Render 2D Projections

To render the data directly in the notebook, simply show it. Note that each row contains an embedding projection. 

Scroll to far right to see embeddings projection per row.

The color of the point in projection space represents the Score.


```python
dg.show()
```



<iframe
    width="100%"
    height="750px"
    src="http://127.0.1.1:4000/?datagrid=openai_embeddings.datagrid&timestamp=1685559502.7515423"
    frameborder="0"
    allowfullscreen

></iframe>



Group by "Score" to see rows of each group.


```python
dg.show(group="Score", sort="Score", rows=5, select="Score,embedding")
```



<iframe
    width="100%"
    height="750px"
    src="http://127.0.1.1:4000/?datagrid=openai_embeddings.datagrid&timestamp=1685559502.7515423&group=Score&sort=Score&rows=5&select=Score%2Cembedding"
    frameborder="0"
    allowfullscreen

></iframe>



An example of this datagrid is hosted here: https://kangas.comet.com/?datagrid=/data/openai_embeddings.datagrid


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_with_Atlas.ipynb


key: _node_content
string_value:
{"id_": "db14ad13-96cf-4251-a931-0d45420d127d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_with_Atlas.ipynb", "title": "Load the embeddings", "tokens": 776, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "054f16dd-1d58-458d-b3b0-1fb93f20d1c5", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_with_Atlas.ipynb", "title": "Load the embeddings", "tokens": 776, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "995be937ab9d1625166e0acc01b9da3da726385e19c199303d65a3ab24962083", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a5a89cee-03f6-4160-b653-01b6a43beca0", "node_type": "1", "metadata": {}, "hash": "58041455514a08016ed1c06ea7a8cc66a5f924abf18ed23fec08050c5b4f5ae4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 1294, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Visualizing Open AI Embeddings in Atlas

In this example, we will upload food review embeddings to [Atlas](https://atlas.nomic.ai) to visualize the embeddings.

## What is Atlas?

[Atlas](https://atlas.nomic.ai) is a machine learning tool used to visualize massive datasets of embeddings in your web browser. Upload millions of embeddings to Atlas and interact with them in your web browser or jupyter notebook.

### 1. Login to Atlas.



```python
!pip install nomic
```

    


```python
import pandas as pd
import numpy as np
from ast import literal_eval

# Load the embeddings
datafile_path = "data/fine_food_reviews_with_embeddings_1k.csv"
df = pd.read_csv(datafile_path)

# Convert to a list of lists of floats
embeddings = np.array(df.embedding.apply(literal_eval).to_list())
df = df.drop('embedding', axis=1)
df = df.rename(columns={'Unnamed: 0': 'id'})

```


```python
import nomic
from nomic import atlas
nomic.login('7xDPkYXSYDc1_ErdTPIcoAR9RNd8YDlkS3nVNXcVoIMZ6') #demo account

data = df.to_dict('records')
project = atlas.map_embeddings(embeddings=embeddings, data=data,
                               id_field='id',
                               colorable_fields=['Score'])
map = project.maps[0]
```

    

### 2. Interact with your embeddings in Jupyter


```python
map
```


key: _node_content
string_value:
{"id_": "a5a89cee-03f6-4160-b653-01b6a43beca0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_with_Atlas.ipynb", "title": "Load the embeddings", "tokens": 776, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "054f16dd-1d58-458d-b3b0-1fb93f20d1c5", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_with_Atlas.ipynb", "title": "Load the embeddings", "tokens": 776, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "995be937ab9d1625166e0acc01b9da3da726385e19c199303d65a3ab24962083", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "db14ad13-96cf-4251-a931-0d45420d127d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_with_Atlas.ipynb", "title": "Load the embeddings", "tokens": 776, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b58f9e9568ae05ced34c2ee368834e0b53b031b922c4c41dcdb7caa75696a214", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ebae1d26-8bc6-4e0d-bb02-36a56d65b9b5", "node_type": "1", "metadata": {}, "hash": "5b0d7e8666a5b807e3bbea3400b6775cad72d42af45bacc0f92b52f9d959d8db", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2557, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Visualizing Open AI Embeddings in Atlas

In this example, we will upload food review embeddings to [Atlas](https://atlas.nomic.ai) to visualize the embeddings.

## What is Atlas?

[Atlas](https://atlas.nomic.ai) is a machine learning tool used to visualize massive datasets of embeddings in your web browser. Upload millions of embeddings to Atlas and interact with them in your web browser or jupyter notebook.

### 1. Login to Atlas.



```python
!pip install nomic
```

    


```python
import pandas as pd
import numpy as np
from ast import literal_eval

# Load the embeddings
datafile_path = "data/fine_food_reviews_with_embeddings_1k.csv"
df = pd.read_csv(datafile_path)

# Convert to a list of lists of floats
embeddings = np.array(df.embedding.apply(literal_eval).to_list())
df = df.drop('embedding', axis=1)
df = df.rename(columns={'Unnamed: 0': 'id'})

```


```python
import nomic
from nomic import atlas
nomic.login('7xDPkYXSYDc1_ErdTPIcoAR9RNd8YDlkS3nVNXcVoIMZ6') #demo account

data = df.to_dict('records')
project = atlas.map_embeddings(embeddings=embeddings, data=data,
                               id_field='id',
                               colorable_fields=['Score'])
map = project.maps[0]
```

    

### 2. Interact with your embeddings in Jupyter


```python
map
```





    <h3>Project: meek-laborer</h3>
    <script>
    destroy = function() {
        document.getElementById("iframe463f4614-7689-47e4-b55b-1da0cc679559").remove()
    }
</script>

<h4>Projection ID: 463f4614-7689-47e4-b55b-1da0cc679559</h4>
<div class="actions">
    <div id="hide" class="action" onclick="destroy()">Hide embedded project</div>
    <div class="action" id="out">
        <a href="https://atlas.nomic.ai/map/fddc0e07-97c5-477c-827c-96bca44519aa/463f4614-7689-47e4-b55b-1da0cc679559" target="_blank">Explore on atlas.nomic.ai</a>
    </div>
</div>

<iframe class="iframe" id="iframe463f4614-7689-47e4-b55b-1da0cc679559" allow="clipboard-read; clipboard-write" src="https://atlas.nomic.ai/map/fddc0e07-97c5-477c-827c-96bca44519aa/463f4614-7689-47e4-b55b-1da0cc679559">
</iframe>

<style>
    .iframe {
        /* vh can be **very** large in vscode ipynb. */
        height: min(75vh, 66vw);
        width: 100%;
    }
</style>

<style>
    .actions {
      display: block;
    }
    .action {
      min-height: 18px;
      margin: 5px;
      transition: all 500ms ease-in-out;
    }
    .action:hover {
      cursor: pointer;
    }
    #hide:hover::after {
        content: " X";
    }
    #out:hover::after {
        content: "";
    }
</style>


key: _node_content
string_value:
{"id_": "ebae1d26-8bc6-4e0d-bb02-36a56d65b9b5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_with_Atlas.ipynb", "title": "Load the embeddings", "tokens": 776, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "054f16dd-1d58-458d-b3b0-1fb93f20d1c5", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_with_Atlas.ipynb", "title": "Load the embeddings", "tokens": 776, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "995be937ab9d1625166e0acc01b9da3da726385e19c199303d65a3ab24962083", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a5a89cee-03f6-4160-b653-01b6a43beca0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Visualizing_embeddings_with_Atlas.ipynb", "title": "Load the embeddings", "tokens": 776, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "729cb66d6a90668ef647874d4b2d45bedb78243aae102e75538732b69c6eaa99", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2564, "end_char_idx": 2578, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb


key: _node_content
string_value:
{"id_": "0dedac0c-f67a-461c-81bd-2e62400d6045", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1cabd36d-6e46-467d-a775-4d9627eb6fff", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c6c45efa9eb125046624692b393e51c3ff26792f1c25ae7dc0550d744537b171", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ee8ef11d-f246-43c1-85c6-bc4edccbb1f5", "node_type": "1", "metadata": {}, "hash": "fb1e8c8c2c5bec55ae22dc6ecb3ccb784c51d2c593b1464d455ae73ef3fff809", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3123, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Financial Document Analysis with LlamaIndex

In this example notebook, we showcase how to perform financial analysis over [**10-K**](https://en.wikipedia.org/wiki/Form_10-K) documents with the [**LlamaIndex**](https://gpt-index.readthedocs.io/en/latest/) framework with just a few lines of code.

## Notebook Outline
* [Introduction](#Introduction)
* [Setup](#Setup)
* [Data Loading & Indexing](#Data-Loading-and-Indexing)
* [Simple QA](#Simple-QA)
* [Advanced QA - Compare and Contrast](#Advanced-QA---Compare-and-Contrast)


## Introduction

### LLamaIndex
[LlamaIndex](https://gpt-index.readthedocs.io/en/latest/) is a data framework for LLM applications.
You can get started with just a few lines of code and build a retrieval-augmented generation (RAG) system in minutes.
For more advanced users, LlamaIndex offers a rich toolkit for ingesting and indexing your data, modules for retrieval and re-ranking, and composable components for building custom query engines.

See [full documentation](https://gpt-index.readthedocs.io/en/latest/) for more details.

### Financial Analysis over 10-K documents
A key part of a financial analyst's job is to extract information and synthesize insight from long financial documents.
A great example is the 10-K form - an annual report required by the U.S. Securities and Exchange Commission (SEC), that gives a comprehensive summary of a company's financial performance.
These documents typically run hundred of pages in length, and contain domain-specific terminology that makes it challenging for a layperson to digest quickly.


We showcase how LlamaIndex can support a financial analyst in quickly extracting information and synthesize insights **across multiple documents** with very little coding. 

## Setup

To begin, we need to install the llama-index library


```python
!pip install llama-index pypdf
```

Now, we import all modules used in this tutorial


```python
from langchain import OpenAI

from llama_index import SimpleDirectoryReader, ServiceContext, VectorStoreIndex
from llama_index import set_global_service_context
from llama_index.response.pprint_utils import pprint_response
from llama_index.tools import QueryEngineTool, ToolMetadata
from llama_index.query_engine import SubQuestionQueryEngine
```

Before we start, we can configure the LLM provider and model that will power our RAG system.  
Here, we pick `gpt-3.5-turbo-instruct` from OpenAI.  


```python
llm = OpenAI(temperature=0, model_name="gpt-3.5-turbo-instruct", max_tokens=-1)
```

We construct a `ServiceContext` and set it as the global default, so all subsequent operations that depends on LLM calls will use the model we configured here.


```python
service_context = ServiceContext.from_defaults(llm=llm)
set_global_service_context(service_context=service_context)
```

## Data Loading and Indexing

Now, we load and parse 2 PDFs (one for Uber 10-K in 2021 and another for Lyft 10-k in 2021).    
Under the hood, the PDFs are converted to plain text `Document` objects, separate by page.  

> Note: this operation might take a while to run, since each document is more than 100 pages.


key: _node_content
string_value:
{"id_": "ee8ef11d-f246-43c1-85c6-bc4edccbb1f5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1cabd36d-6e46-467d-a775-4d9627eb6fff", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c6c45efa9eb125046624692b393e51c3ff26792f1c25ae7dc0550d744537b171", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0dedac0c-f67a-461c-81bd-2e62400d6045", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "2250595aadff75d1139313acaa93d9c62def9b82f2d44fcc485e3f1c3b81773a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a27bfaee-96f7-435f-b96d-2d013dfaea18", "node_type": "1", "metadata": {}, "hash": "32f63c194005533c3a439a19e2ed545c88309c0df45aa52f20d5c8f903591004", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1576, "end_char_idx": 4689, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We showcase how LlamaIndex can support a financial analyst in quickly extracting information and synthesize insights **across multiple documents** with very little coding. 

## Setup

To begin, we need to install the llama-index library


```python
!pip install llama-index pypdf
```

Now, we import all modules used in this tutorial


```python
from langchain import OpenAI

from llama_index import SimpleDirectoryReader, ServiceContext, VectorStoreIndex
from llama_index import set_global_service_context
from llama_index.response.pprint_utils import pprint_response
from llama_index.tools import QueryEngineTool, ToolMetadata
from llama_index.query_engine import SubQuestionQueryEngine
```

Before we start, we can configure the LLM provider and model that will power our RAG system.  
Here, we pick `gpt-3.5-turbo-instruct` from OpenAI.  


```python
llm = OpenAI(temperature=0, model_name="gpt-3.5-turbo-instruct", max_tokens=-1)
```

We construct a `ServiceContext` and set it as the global default, so all subsequent operations that depends on LLM calls will use the model we configured here.


```python
service_context = ServiceContext.from_defaults(llm=llm)
set_global_service_context(service_context=service_context)
```

## Data Loading and Indexing

Now, we load and parse 2 PDFs (one for Uber 10-K in 2021 and another for Lyft 10-k in 2021).    
Under the hood, the PDFs are converted to plain text `Document` objects, separate by page.  

> Note: this operation might take a while to run, since each document is more than 100 pages.


```python
lyft_docs = SimpleDirectoryReader(input_files=["../data/10k/lyft_2021.pdf"]).load_data()
uber_docs = SimpleDirectoryReader(input_files=["../data/10k/uber_2021.pdf"]).load_data()
```


```python
print(f'Loaded lyft 10-K with {len(lyft_docs)} pages')
print(f'Loaded Uber 10-K with {len(uber_docs)} pages')
```

    Loaded lyft 10-K with 238 pages
    Loaded Uber 10-K with 307 pages


Now, we can build an (in-memory) `VectorStoreIndex` over the documents that we've loaded.  

> Note: this operation might take a while to run, since it calls OpenAI API for computing vector embedding over document chunks.


```python
lyft_index = VectorStoreIndex.from_documents(lyft_docs)
uber_index = VectorStoreIndex.from_documents(uber_docs)
```

## Simple QA

Now we are ready to run some queries against our indices!  
To do so, we first configure a `QueryEngine`, which just captures a set of configurations for how we want to query the underlying index.

For a `VectorStoreIndex`, the most common configuration to adjust is `similarity_top_k` which controls how many document chunks (which we call `Node` objects) are retrieved to use as context for answering our question.


```python
lyft_engine = lyft_index.as_query_engine(similarity_top_k=3)
```


```python
uber_engine = uber_index.as_query_engine(similarity_top_k=3)
```

Let's see some queries in action!


```python
response = await lyft_engine.aquery('What is the revenue of Lyft in 2021? Answer in millions with page reference')
```


```python
print(response)
```

    
    $3,208.3 million (page 63)


key: _node_content
string_value:
{"id_": "a27bfaee-96f7-435f-b96d-2d013dfaea18", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1cabd36d-6e46-467d-a775-4d9627eb6fff", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c6c45efa9eb125046624692b393e51c3ff26792f1c25ae7dc0550d744537b171", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ee8ef11d-f246-43c1-85c6-bc4edccbb1f5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d8ae4af8a52ad9359c05738dcf405e241b747d79b5675f755deb5172f8330a94", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fb9b8d1d-4045-4758-a8fa-0b6122c2b3dc", "node_type": "1", "metadata": {}, "hash": "d5d2afdb820fa2a38202a5e850a5e4ba352e37c2ec00f12ece24f47bf845decb", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3126, "end_char_idx": 5855, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
lyft_docs = SimpleDirectoryReader(input_files=["../data/10k/lyft_2021.pdf"]).load_data()
uber_docs = SimpleDirectoryReader(input_files=["../data/10k/uber_2021.pdf"]).load_data()
```


```python
print(f'Loaded lyft 10-K with {len(lyft_docs)} pages')
print(f'Loaded Uber 10-K with {len(uber_docs)} pages')
```

    Loaded lyft 10-K with 238 pages
    Loaded Uber 10-K with 307 pages


Now, we can build an (in-memory) `VectorStoreIndex` over the documents that we've loaded.  

> Note: this operation might take a while to run, since it calls OpenAI API for computing vector embedding over document chunks.


```python
lyft_index = VectorStoreIndex.from_documents(lyft_docs)
uber_index = VectorStoreIndex.from_documents(uber_docs)
```

## Simple QA

Now we are ready to run some queries against our indices!  
To do so, we first configure a `QueryEngine`, which just captures a set of configurations for how we want to query the underlying index.

For a `VectorStoreIndex`, the most common configuration to adjust is `similarity_top_k` which controls how many document chunks (which we call `Node` objects) are retrieved to use as context for answering our question.


```python
lyft_engine = lyft_index.as_query_engine(similarity_top_k=3)
```


```python
uber_engine = uber_index.as_query_engine(similarity_top_k=3)
```

Let's see some queries in action!


```python
response = await lyft_engine.aquery('What is the revenue of Lyft in 2021? Answer in millions with page reference')
```


```python
print(response)
```

    
    $3,208.3 million (page 63)



```python
response = await uber_engine.aquery('What is the revenue of Uber in 2021? Answer in millions, with page reference')
```


```python
print(response)
```

    
    $17,455 (page 53)


## Advanced QA - Compare and Contrast

For more complex financial analysis, one often needs to reference multiple documents.  

As a example, let's take a look at how to do compare-and-contrast queries over both Lyft and Uber financials.  
For this, we build a `SubQuestionQueryEngine`, which breaks down a complex compare-and-contrast query, into simpler sub-questions to execute on respective sub query engine backed by individual indices.


```python
query_engine_tools = [
    QueryEngineTool(
        query_engine=lyft_engine, 
        metadata=ToolMetadata(name='lyft_10k', description='Provides information about Lyft financials for year 2021')
    ),
    QueryEngineTool(
        query_engine=uber_engine, 
        metadata=ToolMetadata(name='uber_10k', description='Provides information about Uber financials for year 2021')
    ),
]

s_engine = SubQuestionQueryEngine.from_defaults(query_engine_tools=query_engine_tools)
```

Let's see these queries in action!


key: _node_content
string_value:
{"id_": "fb9b8d1d-4045-4758-a8fa-0b6122c2b3dc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1cabd36d-6e46-467d-a775-4d9627eb6fff", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c6c45efa9eb125046624692b393e51c3ff26792f1c25ae7dc0550d744537b171", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a27bfaee-96f7-435f-b96d-2d013dfaea18", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7054c28ed1def82986d85e6f63a25ef6ca56f8c21d9496e3113491a06f663852", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3cb489ff-b78c-426b-9235-6734f4f0c5ac", "node_type": "1", "metadata": {}, "hash": "51a1b374e2f803bfacc0f73c739c33b50edadbd7cf22bfea926744b34de7080f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4303, "end_char_idx": 7987, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
lyft_engine = lyft_index.as_query_engine(similarity_top_k=3)
```


```python
uber_engine = uber_index.as_query_engine(similarity_top_k=3)
```

Let's see some queries in action!


```python
response = await lyft_engine.aquery('What is the revenue of Lyft in 2021? Answer in millions with page reference')
```


```python
print(response)
```

    
    $3,208.3 million (page 63)



```python
response = await uber_engine.aquery('What is the revenue of Uber in 2021? Answer in millions, with page reference')
```


```python
print(response)
```

    
    $17,455 (page 53)


## Advanced QA - Compare and Contrast

For more complex financial analysis, one often needs to reference multiple documents.  

As a example, let's take a look at how to do compare-and-contrast queries over both Lyft and Uber financials.  
For this, we build a `SubQuestionQueryEngine`, which breaks down a complex compare-and-contrast query, into simpler sub-questions to execute on respective sub query engine backed by individual indices.


```python
query_engine_tools = [
    QueryEngineTool(
        query_engine=lyft_engine, 
        metadata=ToolMetadata(name='lyft_10k', description='Provides information about Lyft financials for year 2021')
    ),
    QueryEngineTool(
        query_engine=uber_engine, 
        metadata=ToolMetadata(name='uber_10k', description='Provides information about Uber financials for year 2021')
    ),
]

s_engine = SubQuestionQueryEngine.from_defaults(query_engine_tools=query_engine_tools)
```

Let's see these queries in action!


```python
response = await s_engine.aquery('Compare and contrast the customer segments and geographies that grew the fastest')
```

    Generated 4 sub questions.
    [36;1m[1;3m[uber_10k] Q: What customer segments grew the fastest for Uber
    [0m[36;1m[1;3m[uber_10k] A: in 2021?
    
    The customer segments that grew the fastest for Uber in 2021 were its Mobility Drivers, Couriers, Riders, and Eaters. These segments experienced growth due to the continued stay-at-home order demand related to COVID-19, as well as Uber's introduction of its Uber One, Uber Pass, Eats Pass, and Rides Pass membership programs. Additionally, Uber's marketplace-centric advertising helped to connect merchants and brands with its platform network, further driving growth.
    [0m[33;1m[1;3m[uber_10k] Q: What geographies grew the fastest for Uber
    [0m[33;1m[1;3m[uber_10k] A: 
    Based on the context information, it appears that Uber experienced the most growth in large metropolitan areas, such as Chicago, Miami, New York City, Sao Paulo, and London. Additionally, Uber experienced growth in suburban and rural areas, as well as in countries such as Argentina, Germany, Italy, Japan, South Korea, and Spain.
    [0m[38;5;200m[1;3m[lyft_10k] Q: What customer segments grew the fastest for Lyft
    [0m[38;5;200m[1;3m[lyft_10k] A: 
    The customer segments that grew the fastest for Lyft were ridesharing, light vehicles, and public transit. Ridesharing grew as Lyft was able to predict demand and proactively incentivize drivers to be available for rides in the right place at the right time. Light vehicles grew as users were looking for options that were more active, usually lower-priced, and often more efficient for short trips during heavy traffic. Public transit grew as Lyft integrated third-party public transit data into the Lyft App to offer users a robust view of transportation options around them.
    [0m[32;1m[1;3m[lyft_10k] Q: What geographies grew the fastest for Lyft
    [0m[32;1m[1;3m[lyft_10k] A: 
    It is not possible to answer this question with the given context information.
    [0m


key: _node_content
string_value:
{"id_": "3cb489ff-b78c-426b-9235-6734f4f0c5ac", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1cabd36d-6e46-467d-a775-4d9627eb6fff", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c6c45efa9eb125046624692b393e51c3ff26792f1c25ae7dc0550d744537b171", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fb9b8d1d-4045-4758-a8fa-0b6122c2b3dc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/financial_document_analysis_with_llamaindex.ipynb", "title": "Financial Document Analysis with LlamaIndex", "tokens": 2638, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a71d8cfafbb8a73e35af5b561ae16135a807711966e3e1f2d3170bdca7b40ad3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7990, "end_char_idx": 10896, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
print(response)
```

    
    The customer segments that grew the fastest for Uber in 2021 were its Mobility Drivers, Couriers, Riders, and Eaters. These segments experienced growth due to the continued stay-at-home order demand related to COVID-19, as well as Uber's introduction of its Uber One, Uber Pass, Eats Pass, and Rides Pass membership programs. Additionally, Uber's marketplace-centric advertising helped to connect merchants and brands with its platform network, further driving growth. Uber experienced the most growth in large metropolitan areas, such as Chicago, Miami, New York City, Sao Paulo, and London. Additionally, Uber experienced growth in suburban and rural areas, as well as in countries such as Argentina, Germany, Italy, Japan, South Korea, and Spain.
    
    The customer segments that grew the fastest for Lyft were ridesharing, light vehicles, and public transit. Ridesharing grew as Lyft was able to predict demand and proactively incentivize drivers to be available for rides in the right place at the right time. Light vehicles grew as users were looking for options that were more active, usually lower-priced, and often more efficient for short trips during heavy traffic. Public transit grew as Lyft integrated third-party public transit data into the Lyft App to offer users a robust view of transportation options around them. It is not possible to answer the question of which geographies grew the fastest for Lyft with the given context information.
    
    In summary, Uber and Lyft both experienced growth in customer segments related to mobility, couriers, riders, and eaters. Uber experienced the most growth in large metropolitan areas, as well as in suburban and rural areas, and in countries such as Argentina, Germany, Italy, Japan, South Korea, and Spain. Lyft experienced the most growth in ridesharing, light vehicles, and public transit. It is not possible to answer the question of which geographies grew the fastest for Lyft with the given context information.



```python
response = await s_engine.aquery('Compare revenue growth of Uber and Lyft from 2020 to 2021')
```

    Generated 2 sub questions.
    [36;1m[1;3m[uber_10k] Q: What is the revenue growth of Uber from 2020 to 2021
    [0m[36;1m[1;3m[uber_10k] A: 
    The revenue growth of Uber from 2020 to 2021 was 57%, or 54% on a constant currency basis.
    [0m[33;1m[1;3m[lyft_10k] Q: What is the revenue growth of Lyft from 2020 to 2021
    [0m[33;1m[1;3m[lyft_10k] A: 
    The revenue growth of Lyft from 2020 to 2021 is 36%, increasing from $2,364,681 thousand to $3,208,323 thousand.
    [0m


```python
print(response)
```

    
    The revenue growth of Uber from 2020 to 2021 was 57%, or 54% on a constant currency basis, while the revenue growth of Lyft from 2020 to 2021 was 36%. This means that Uber had a higher revenue growth than Lyft from 2020 to 2021.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb


key: _node_content
string_value:
{"id_": "5a5630fa-3a56-4d7b-a773-1b30b9a5cd14", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d3a645e5-d0d8-44f8-82f6-03f9e2dd4469", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "81fb6405dea1e39e1de71053e5afa8900a2acc089ae79680537cd73079c62598", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0c4290ce-d0e8-4bc4-8602-de99b47db5fe", "node_type": "1", "metadata": {}, "hash": "363a84282b8220a59fa73aa021e726f76ba41f77e326997513e7208d09cc6883", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3031, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# OpenAI API Monitoring with W&B Weave

<img src="http://wandb.me/logo-im-png" width="400" alt="Weights & Biases" />

<!--- @wandbcode{weave_openai_client_qs} -->

<a target="_blank" href="https://colab.research.google.com/github/wandb/weave/blob/master/examples/prompts/llm_monitoring/openai_client_quickstart.ipynb">
  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>

**Note:** you will need an [OpenAI API key](https://platform.openai.com/account/api-keys) to run this colab.

Use the W&B OpenAI integration to monitor OpenAI API calls and understand how your projects and teams are leveraging LLMs.
In this example, we'll generate templated Weave Boards: LLM usage monitoring dashboards which you can explore and customize from the UI.

* automatically track LLM usage and aggregate useful metrics like  cost, latency and throughput across your projects/teams
* dynamically query and derive insights from the logs of all your OpenAI API calls
* iterate visually to slice, aggregate, and explore your data; customize panels to focus on interesting patterns; share progress more easily with your team through an interactive dashboard

<img src="https://raw.githubusercontent.com/wandb/weave/master/docs/assets/full_board_view.png">

[Play with a live version of this Weave Board →](http://wandb.me/llm-monitoring-board)

#### New to Weights & Biases? [-> Sign up for an account here <-](https://wandb.ai/site)

# Step 0: Setup

Install dependencies, login to W&B so you can save and share your work, and authenticate with OpenAI.


```python
# if not already installed
!pip install -qqq weave openai tiktoken wandb
```


```python
import wandb
wandb.login()
```


```python
import weave
import os
WANDB_BASE_URL = "https://api.wandb.ai"
os.environ["WANDB_BASE_URL"] = WANDB_BASE_URL
```


```python
# authenticate with OpenAI
from getpass import getpass

if os.getenv("OPENAI_API_KEY") is None:
  os.environ["OPENAI_API_KEY"] = getpass("Paste your OpenAI key from: https://platform.openai.com/account/api-keys\n")
assert os.getenv("OPENAI_API_KEY", "").startswith("sk-"), "This doesn't look like a valid OpenAI API key"
print("OpenAI API key configured")
```

# Step 1: Configure data streaming and storage in W&B

Set WB_ENTITY to your wandb username or team name. Log in to W&B and navigate to Home Page at [wandb.ai/home](https://wandb.ai/home) to see valid options under your "Profile" and "Teams" in the left sidebar.


```python
WB_ENTITY = "" # set to your wandb username or team name
WB_PROJECT = "weave" # top-level directory for this work
STREAM_NAME = "openai_logs" # record table which stores the logs of OpenAI API calls as they stream in
```

# Step 2: Call init_monitor()

To start monitoring OpenAI API usage, call `init_monitor(<stream>)`, where `<stream>` has the form `<wandb_team_or_user>/<wandb_project>/<stream_name>`. The stream records and stores all the OpenAI API calls.

Running this cell will print out a link to view the current project in the Weave UI.


key: _node_content
string_value:
{"id_": "0c4290ce-d0e8-4bc4-8602-de99b47db5fe", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d3a645e5-d0d8-44f8-82f6-03f9e2dd4469", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "81fb6405dea1e39e1de71053e5afa8900a2acc089ae79680537cd73079c62598", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5a5630fa-3a56-4d7b-a773-1b30b9a5cd14", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e400e76af28ddd58b63efd214cc2a9578c0592268c9a1578a82675b6c47bd75e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "892d587d-7a09-4806-b017-76b433cdc643", "node_type": "1", "metadata": {}, "hash": "396f67fe8962b03eee56e69bfd8b79cfbafc12da60f5d3b5f9c70ac405c03544", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1581, "end_char_idx": 4562, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# if not already installed
!pip install -qqq weave openai tiktoken wandb
```


```python
import wandb
wandb.login()
```


```python
import weave
import os
WANDB_BASE_URL = "https://api.wandb.ai"
os.environ["WANDB_BASE_URL"] = WANDB_BASE_URL
```


```python
# authenticate with OpenAI
from getpass import getpass

if os.getenv("OPENAI_API_KEY") is None:
  os.environ["OPENAI_API_KEY"] = getpass("Paste your OpenAI key from: https://platform.openai.com/account/api-keys\n")
assert os.getenv("OPENAI_API_KEY", "").startswith("sk-"), "This doesn't look like a valid OpenAI API key"
print("OpenAI API key configured")
```

# Step 1: Configure data streaming and storage in W&B

Set WB_ENTITY to your wandb username or team name. Log in to W&B and navigate to Home Page at [wandb.ai/home](https://wandb.ai/home) to see valid options under your "Profile" and "Teams" in the left sidebar.


```python
WB_ENTITY = "" # set to your wandb username or team name
WB_PROJECT = "weave" # top-level directory for this work
STREAM_NAME = "openai_logs" # record table which stores the logs of OpenAI API calls as they stream in
```

# Step 2: Call init_monitor()

To start monitoring OpenAI API usage, call `init_monitor(<stream>)`, where `<stream>` has the form `<wandb_team_or_user>/<wandb_project>/<stream_name>`. The stream records and stores all the OpenAI API calls.

Running this cell will print out a link to view the current project in the Weave UI.


```python
from weave.monitoring import openai, init_monitor
m = init_monitor(f"{WB_ENTITY}/{WB_PROJECT}/{STREAM_NAME}")

# specifying a single model for simplicity
OPENAI_MODEL = 'gpt-3.5-turbo'

# prefill with some sample logs
r = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[{"role": "user", "content": "hello world!"}])
r = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[{"role": "user", "content": "what is 2+2?"}])
```

# Step 3: Preview monitoring dashboard

Click on the link above to preview the data stream, then click "OpenAI Monitor Board" in the right sidebar to create a Weave Board for this data stream.

<img src="https://raw.githubusercontent.com/wandb/weave/master/docs/assets/short_board_attempt.gif" width=75%>

# Step 4: Explore & understand your LLM usage

To save your work, rename the board by clicking on the autogenerated name at the top of the page. To share your board, click \"Publish\" in the top right.

<img src="https://raw.githubusercontent.com/wandb/weave/master/docs/assets/publish_board_short.gif" width=75%>

To visualize your work in real-time as you iterate, you can:
* keep the Board open in a separate tab and refresh to view the latest data
* rename the Board for easier reference at any point and \"Publish\" that version to share a link with others
* find previously saved Boards by navigating to the relevant W&B entity and W&B project name from weave.wandb.ai
* or open a new instance of a Board template to start fresh with all the data accumulated so far


key: _node_content
string_value:
{"id_": "892d587d-7a09-4806-b017-76b433cdc643", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d3a645e5-d0d8-44f8-82f6-03f9e2dd4469", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "81fb6405dea1e39e1de71053e5afa8900a2acc089ae79680537cd73079c62598", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0c4290ce-d0e8-4bc4-8602-de99b47db5fe", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1bd4858a35746f66aff2f418828e38692cc442dffaac25318ee8e5173ee52e48", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "90a6a3da-aa4d-4547-84dc-ab78c8988ff4", "node_type": "1", "metadata": {}, "hash": "12347373e3f59afb296ebc2c75e1ba11ad2c3e89b670fd2ef0e245d794e5230c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3034, "end_char_idx": 6586, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from weave.monitoring import openai, init_monitor
m = init_monitor(f"{WB_ENTITY}/{WB_PROJECT}/{STREAM_NAME}")

# specifying a single model for simplicity
OPENAI_MODEL = 'gpt-3.5-turbo'

# prefill with some sample logs
r = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[{"role": "user", "content": "hello world!"}])
r = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[{"role": "user", "content": "what is 2+2?"}])
```

# Step 3: Preview monitoring dashboard

Click on the link above to preview the data stream, then click "OpenAI Monitor Board" in the right sidebar to create a Weave Board for this data stream.

<img src="https://raw.githubusercontent.com/wandb/weave/master/docs/assets/short_board_attempt.gif" width=75%>

# Step 4: Explore & understand your LLM usage

To save your work, rename the board by clicking on the autogenerated name at the top of the page. To share your board, click \"Publish\" in the top right.

<img src="https://raw.githubusercontent.com/wandb/weave/master/docs/assets/publish_board_short.gif" width=75%>

To visualize your work in real-time as you iterate, you can:
* keep the Board open in a separate tab and refresh to view the latest data
* rename the Board for easier reference at any point and \"Publish\" that version to share a link with others
* find previously saved Boards by navigating to the relevant W&B entity and W&B project name from weave.wandb.ai
* or open a new instance of a Board template to start fresh with all the data accumulated so far


Next we'll illustrate a few ways you could track OpenAI API calls. There are many more possibilities depending on your use case, and we can't wait to see what you create from these starter templates.

# Examples

## Example 0: Log a prompt and its completion

Monitor a ChatCompletion request and print the corresponding response, extracting only the text of the completion.


```python
response = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[
        {"role": "user", "content": f"What is the meaning of life, the universe, and everything?"},
    ])
print(response['choices'][0]['message']['content'])
```

## Example 1: Track relevant parameters as attributes

Factor out parameters of interest and track them as attributes on the logged record.
Here we track the "system prompt" separately from the "prompt template" and the "equation" parameter. This time we'll print the full structured response from the ChatCompletion call.


```python
system_prompt = "you always write in bullet points"
prompt_template = 'solve the following equation step by step: {equation}'
params = {'equation': '4 * (3 - 1)'}
openai.ChatCompletion.create(model=OPENAI_MODEL,
                             messages=[
                                    {"role": "system", "content": system_prompt},
                                    {"role": "user", "content": prompt_template.format(**params)},
                                ],
                             # you can add additional attributes to the logged record
                             # see the monitor_api notebook for more examples
                             monitor_attributes={
                                 'system_prompt': system_prompt,
                                 'prompt_template': prompt_template,
                                 'params': params
                             })
```

## Example 2: Log an ongoing stream of messages

Monitor a stream of messages and log the result as a single record. Note: tokens are not counted in this format.


key: _node_content
string_value:
{"id_": "90a6a3da-aa4d-4547-84dc-ab78c8988ff4", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d3a645e5-d0d8-44f8-82f6-03f9e2dd4469", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "81fb6405dea1e39e1de71053e5afa8900a2acc089ae79680537cd73079c62598", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "892d587d-7a09-4806-b017-76b433cdc643", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "98df897b98cad5fcc09ae7ff45d9331b3424820ee8a16e75991fa0ebc0592d15", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4ee6a028-fdd3-48af-a32a-489d127f2e04", "node_type": "1", "metadata": {}, "hash": "34dfec5596283801792b850a60936b8ebce3d278c76c75954c0abbb19eeefdcf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4565, "end_char_idx": 8061, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Next we'll illustrate a few ways you could track OpenAI API calls. There are many more possibilities depending on your use case, and we can't wait to see what you create from these starter templates.

# Examples

## Example 0: Log a prompt and its completion

Monitor a ChatCompletion request and print the corresponding response, extracting only the text of the completion.


```python
response = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[
        {"role": "user", "content": f"What is the meaning of life, the universe, and everything?"},
    ])
print(response['choices'][0]['message']['content'])
```

## Example 1: Track relevant parameters as attributes

Factor out parameters of interest and track them as attributes on the logged record.
Here we track the "system prompt" separately from the "prompt template" and the "equation" parameter. This time we'll print the full structured response from the ChatCompletion call.


```python
system_prompt = "you always write in bullet points"
prompt_template = 'solve the following equation step by step: {equation}'
params = {'equation': '4 * (3 - 1)'}
openai.ChatCompletion.create(model=OPENAI_MODEL,
                             messages=[
                                    {"role": "system", "content": system_prompt},
                                    {"role": "user", "content": prompt_template.format(**params)},
                                ],
                             # you can add additional attributes to the logged record
                             # see the monitor_api notebook for more examples
                             monitor_attributes={
                                 'system_prompt': system_prompt,
                                 'prompt_template': prompt_template,
                                 'params': params
                             })
```

## Example 2: Log an ongoing stream of messages

Monitor a stream of messages and log the result as a single record. Note: tokens are not counted in this format.


```python
from weave.monitoring.openai import message_from_stream
r = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[
        {"role": "system", "content": "You are a robot and only speak in robot, like beep bloop bop."},
        {"role": "user", "content": "Tell me a 50-word story."},
    ], stream=True)
for s in message_from_stream(r):
    print(s, end='')
```

## Example 3: Structure prompt engineering experiments

Here we compare a few toy options for the system prompt, user question, and intended audience. Try your own experiments and see if any interesting insights emerge as you explore in the Board and group by different parameters.


```python
def explain_math(system_prompt, prompt_template, params):
    openai.ChatCompletion.create(model=OPENAI_MODEL,
                             messages=[
                                    {"role": "system", "content": system_prompt},
                                    {"role": "user", "content": prompt_template.format(**params)},
                                ],
                             # you can add additional attributes to the logged record
                             # see the monitor_api notebook for more examples
                             monitor_attributes={
                                 'system_prompt': system_prompt,
                                 'prompt_template': prompt_template,
                                 'params': params
                             })
```


key: _node_content
string_value:
{"id_": "4ee6a028-fdd3-48af-a32a-489d127f2e04", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "d3a645e5-d0d8-44f8-82f6-03f9e2dd4469", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "81fb6405dea1e39e1de71053e5afa8900a2acc089ae79680537cd73079c62598", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "90a6a3da-aa4d-4547-84dc-ab78c8988ff4", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "13686e93f8ba875da4387aca3d715df0cb6b6d4ecf4faefd96b0f1163898213d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6589, "end_char_idx": 8659, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from weave.monitoring.openai import message_from_stream
r = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[
        {"role": "system", "content": "You are a robot and only speak in robot, like beep bloop bop."},
        {"role": "user", "content": "Tell me a 50-word story."},
    ], stream=True)
for s in message_from_stream(r):
    print(s, end='')
```

## Example 3: Structure prompt engineering experiments

Here we compare a few toy options for the system prompt, user question, and intended audience. Try your own experiments and see if any interesting insights emerge as you explore in the Board and group by different parameters.


```python
def explain_math(system_prompt, prompt_template, params):
    openai.ChatCompletion.create(model=OPENAI_MODEL,
                             messages=[
                                    {"role": "system", "content": system_prompt},
                                    {"role": "user", "content": prompt_template.format(**params)},
                                ],
                             # you can add additional attributes to the logged record
                             # see the monitor_api notebook for more examples
                             monitor_attributes={
                                 'system_prompt': system_prompt,
                                 'prompt_template': prompt_template,
                                 'params': params
                             })
```


```python
# feel free to substitute your own prompts :)
system_prompts = ["you're extremely flowery and poetic", "you're very direct and precise", "balance brevity with insight"]
prompt_template = 'explain the solution of the following to a {audience}: {equation}'
equations = ['x^2 + 4x + 9 = 0', '15 * (2 - 6) / 4']
audience = ["new student", "math genius"]

for system_prompt in system_prompts:
    for equation in equations:
        for person in audience:
            params = {"equation" : equation, "audience" : person}
            explain_math(system_prompt, prompt_template, params)
```


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb


key: _node_content
string_value:
{"id_": "f0fb5af5-272c-4c30-a875-bfad8fd9199f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0ab1df67e41749ec6c62eaf013a9d47d0f1ecdd07fa7855873f263a9bd12438b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b9cf62e7-0c41-4e89-a139-6ca5488c0bc5", "node_type": "1", "metadata": {}, "hash": "2d781cee85ceb8a18b7dbc93b9f0c9e753a9a6314c0b5f46cc41569d0c33ea40", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2737, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to use the DALL·E API

This notebook shows how to use OpenAI's DALL·E image API endpoints.

There are three API endpoints:
- **Generations:** generates an image or images based on an input caption
- **Edits:** edits or extends an existing image
- **Variations:** generates variations of an input image

## Setup

- Import the packages you'll need
- Import your OpenAI API key: You can do this by running \``export OPENAI_API_KEY="your API key"`\` in your terminal.
- Set a directory to save images to


```python
# imports
from openai import OpenAI  # OpenAI Python library to make API calls
import requests  # used to download images
import os  # used to access filepaths
from PIL import Image  # used to print and edit images

# initialize OpenAI client
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))

```


```python
# set a directory to save DALL·E images to
image_dir_name = "images"
image_dir = os.path.join(os.curdir, image_dir_name)

# create the directory if it doesn't yet exist
if not os.path.isdir(image_dir):
    os.mkdir(image_dir)

# print the directory to save to
print(f"{image_dir=}")

```

    image_dir='./images'


## Generations

The generation API endpoint creates an image based on a text prompt. [API Reference](https://platform.openai.com/docs/api-reference/images/create)

**Required inputs:**
- `prompt` (str): A text description of the desired image(s). The maximum length is 1000 characters for dall-e-2 and 4000 characters for dall-e-3.

**Optional inputs:**
- `model` (str): The model to use for image generation. Defaults to dall-e-2
- `n` (int): The number of images to generate. Must be between 1 and 10. Defaults to 1.
- `quality` (str): The quality of the image that will be generated. hd creates images with finer details and greater consistency across the image. This param is only supported for dall-e-3.
- `response_format` (str): The format in which the generated images are returned. Must be one of "url" or "b64_json". Defaults to "url".
- `size` (str): The size of the generated images. Must be one of 256x256, 512x512, or 1024x1024 for dall-e-2. Must be one of 1024x1024, 1792x1024, or 1024x1792 for dall-e-3 models. Defaults to "1024x1024".
- `style`(str | null): The style of the generated images. Must be one of vivid or natural. Vivid causes the model to lean towards generating hyper-real and dramatic images. Natural causes the model to produce more natural, less hyper-real looking images. This param is only supported for dall-e-3.
- `user` (str): A unique identifier representing your end-user, which will help OpenAI to monitor and detect abuse. [Learn more.](https://beta.openai.com/docs/usage-policies/end-user-ids)


key: _node_content
string_value:
{"id_": "b9cf62e7-0c41-4e89-a139-6ca5488c0bc5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0ab1df67e41749ec6c62eaf013a9d47d0f1ecdd07fa7855873f263a9bd12438b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f0fb5af5-272c-4c30-a875-bfad8fd9199f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c1d2f0a793e18b44a725df682a674e6aafdd589f8085d30ff8e27904aa3b3177", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7bd2a930-b8a3-4efc-80a1-0b915f51b282", "node_type": "1", "metadata": {}, "hash": "cae7f3a0656f5991322954cdf529746689f54e953920540a1bdb209958abb39d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2740, "end_char_idx": 4169, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# create an image

# set the prompt
prompt = "A cyberpunk monkey hacker dreaming of a beautiful bunch of bananas, digital art"

# call the OpenAI API
generation_response = client.images.generate(
    model = "dall-e-3",
    prompt=prompt,
    n=1,
    size="1024x1024",
    response_format="url",
)

# print response
print(generation_response)
```

    ImagesResponse(created=1701994117, data=[Image(b64_json=None, revised_prompt=None, url='https://oaidalleapiprodscus.blob.core.windows.net/private/org-9HXYFy8ux4r6aboFyec2OLRf/user-8OA8IvMYkfdAcUZXgzAXHS7d/img-ced13hkOk3lXkccQgW1fAQjm.png?st=2023-12-07T23%3A08%3A37Z&se=2023-12-08T01%3A08%3A37Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2023-12-07T16%3A41%3A48Z&ske=2023-12-08T16%3A41%3A48Z&sks=b&skv=2021-08-06&sig=tcD0iyU0ABOvWAKsY89gp5hLVIYkoSXQnrcmH%2Brkric%3D')])



```python
# save the image
generated_image_name = "generated_image.png"  # any name you like; the filetype should be .png
generated_image_filepath = os.path.join(image_dir, generated_image_name)
generated_image_url = generation_response.data[0].url  # extract image URL from response
generated_image = requests.get(generated_image_url).content  # download the image

with open(generated_image_filepath, "wb") as image_file:
    image_file.write(generated_image)  # write the image to the file
```


key: _node_content
string_value:
{"id_": "7bd2a930-b8a3-4efc-80a1-0b915f51b282", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0ab1df67e41749ec6c62eaf013a9d47d0f1ecdd07fa7855873f263a9bd12438b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b9cf62e7-0c41-4e89-a139-6ca5488c0bc5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ca9863e3282d4ce23b600be97a33d3bae0994bc757d120ace5dda9a37610cb1f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "74a4edab-6576-4453-827e-b5675a2085f0", "node_type": "1", "metadata": {}, "hash": "ea5c5938ad4a3ef80b7ee11a01846346a4617862da6dd403ad94bc1025cc1ca3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3673, "end_char_idx": 5363, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# save the image
generated_image_name = "generated_image.png"  # any name you like; the filetype should be .png
generated_image_filepath = os.path.join(image_dir, generated_image_name)
generated_image_url = generation_response.data[0].url  # extract image URL from response
generated_image = requests.get(generated_image_url).content  # download the image

with open(generated_image_filepath, "wb") as image_file:
    image_file.write(generated_image)  # write the image to the file
```


```python
# print the image
print(generated_image_filepath)
display(Image.open(generated_image_filepath))

```

## Variations

The variations endpoint generates new images (variations) similar to an input image. [API Reference](https://platform.openai.com/docs/api-reference/images/createVariation)

Here we'll generate variations of the image generated above.

**Required inputs:**
- `image` (str): The image to use as the basis for the variation(s). Must be a valid PNG file, less than 4MB, and square.

**Optional inputs:**
- `model` (str): The model to use for image variations. Only dall-e-2 is supported at this time.
- `n` (int): The number of images to generate. Must be between 1 and 10. Defaults to 1.
- `size` (str): The size of the generated images. Must be one of "256x256", "512x512", or "1024x1024". Smaller images are faster. Defaults to "1024x1024".
- `response_format` (str): The format in which the generated images are returned. Must be one of "url" or "b64_json". Defaults to "url".
- `user` (str): A unique identifier representing your end-user, which will help OpenAI to monitor and detect abuse. [Learn more.](https://beta.openai.com/docs/usage-policies/end-user-ids)


key: _node_content
string_value:
{"id_": "74a4edab-6576-4453-827e-b5675a2085f0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0ab1df67e41749ec6c62eaf013a9d47d0f1ecdd07fa7855873f263a9bd12438b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7bd2a930-b8a3-4efc-80a1-0b915f51b282", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "61ddeebe84698449c483f81b29c6a233acb9a00b3dc64cab2474640d833bca38", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "77b745b8-dfcc-4a3d-8c64-c2b685c899a2", "node_type": "1", "metadata": {}, "hash": "becccdac305bcb78ceefb8359364540a75d0fd484b5c57e223ebb1d766a85edc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4172, "end_char_idx": 6795, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# print the image
print(generated_image_filepath)
display(Image.open(generated_image_filepath))

```

## Variations

The variations endpoint generates new images (variations) similar to an input image. [API Reference](https://platform.openai.com/docs/api-reference/images/createVariation)

Here we'll generate variations of the image generated above.

**Required inputs:**
- `image` (str): The image to use as the basis for the variation(s). Must be a valid PNG file, less than 4MB, and square.

**Optional inputs:**
- `model` (str): The model to use for image variations. Only dall-e-2 is supported at this time.
- `n` (int): The number of images to generate. Must be between 1 and 10. Defaults to 1.
- `size` (str): The size of the generated images. Must be one of "256x256", "512x512", or "1024x1024". Smaller images are faster. Defaults to "1024x1024".
- `response_format` (str): The format in which the generated images are returned. Must be one of "url" or "b64_json". Defaults to "url".
- `user` (str): A unique identifier representing your end-user, which will help OpenAI to monitor and detect abuse. [Learn more.](https://beta.openai.com/docs/usage-policies/end-user-ids)



```python
# create variations

# call the OpenAI API, using `create_variation` rather than `create`
variation_response = client.images.create_variation(
    image=generated_image,  # generated_image is the image generated above
    n=2,
    size="1024x1024",
    response_format="url",
)

# print response
print(variation_response)
```

    ImagesResponse(created=1701994139, data=[Image(b64_json=None, revised_prompt=None, url='https://oaidalleapiprodscus.blob.core.windows.net/private/org-9HXYFy8ux4r6aboFyec2OLRf/user-8OA8IvMYkfdAcUZXgzAXHS7d/img-noNRGgwaaotRGIe6Y2GVeSpr.png?st=2023-12-07T23%3A08%3A59Z&se=2023-12-08T01%3A08%3A59Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2023-12-07T16%3A39%3A11Z&ske=2023-12-08T16%3A39%3A11Z&sks=b&skv=2021-08-06&sig=ER5RUglhtIk9LWJXw1DsolorT4bnEmFostfnUjY21ns%3D'), Image(b64_json=None, revised_prompt=None, url='https://oaidalleapiprodscus.blob.core.windows.net/private/org-9HXYFy8ux4r6aboFyec2OLRf/user-8OA8IvMYkfdAcUZXgzAXHS7d/img-oz952tL11FFhf9iXXJVIRUZX.png?st=2023-12-07T23%3A08%3A59Z&se=2023-12-08T01%3A08%3A59Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2023-12-07T16%3A39%3A11Z&ske=2023-12-08T16%3A39%3A11Z&sks=b&skv=2021-08-06&sig=99rJOQwDKsfIeerlMXMHholhAhrHfYaQRFJBF8FKv74%3D')])


key: _node_content
string_value:
{"id_": "77b745b8-dfcc-4a3d-8c64-c2b685c899a2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0ab1df67e41749ec6c62eaf013a9d47d0f1ecdd07fa7855873f263a9bd12438b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "74a4edab-6576-4453-827e-b5675a2085f0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b7c14f199b7dfd341139a1866db80829d2e7c7d1c9713c416fe03c40ecd2211e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3c868ef8-1238-4cf7-a71b-3b3f3cc3b4cd", "node_type": "1", "metadata": {}, "hash": "d146b79fe58b50bc0cf29dbad89ef05d81adc82a9dee84541e043cf912021a08", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6799, "end_char_idx": 9501, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# save the images
variation_urls = [datum.url for datum in variation_response.data]  # extract URLs
variation_images = [requests.get(url).content for url in variation_urls]  # download images
variation_image_names = [f"variation_image_{i}.png" for i in range(len(variation_images))]  # create names
variation_image_filepaths = [os.path.join(image_dir, name) for name in variation_image_names]  # create filepaths
for image, filepath in zip(variation_images, variation_image_filepaths):  # loop through the variations
    with open(filepath, "wb") as image_file:  # open the file
        image_file.write(image)  # write the image to the file
```


```python
# print the original image
print(generated_image_filepath)
display(Image.open(generated_image_filepath))

# print the new variations
for variation_image_filepaths in variation_image_filepaths:
    print(variation_image_filepaths)
    display(Image.open(variation_image_filepaths))

```

## Edits

The edit endpoint uses DALL·E to generate a specified portion of an existing image. Three inputs are needed: the image to edit, a mask specifying the portion to be regenerated, and a prompt describing the desired image. [API Reference](https://platform.openai.com/docs/api-reference/images/createEdit)

**Required inputs:** 
- `image` (str): The image to edit. Must be a valid PNG file, less than 4MB, and square. If mask is not provided, image must have transparency, which will be used as the mask.
- `prompt` (str): A text description of the desired image(s). The maximum length is 1000 characters.

**Optional inputs:**
- `mask` (file): An additional image whose fully transparent areas (e.g. where alpha is zero) indicate where image should be edited. Must be a valid PNG file, less than 4MB, and have the same dimensions as image.
- `model` (str): The model to use for edit image. Only dall-e-2 is supported at this time.
- `n` (int): The number of images to generate. Must be between 1 and 10. Defaults to 1.
- `size` (str): The size of the generated images. Must be one of "256x256", "512x512", or "1024x1024". Smaller images are faster. Defaults to "1024x1024".
- `response_format` (str): The format in which the generated images are returned. Must be one of "url" or "b64_json". Defaults to "url".
- `user` (str): A unique identifier representing your end-user, which will help OpenAI to monitor and detect abuse. [Learn more.](https://beta.openai.com/docs/usage-policies/end-user-ids)


### Set Edit Area

An edit requires a "mask" to specify which portion of the image to regenerate. Any pixel with an alpha of 0 (transparent) will be regenerated. The code below creates a 1024x1024 mask where the bottom half is transparent.


key: _node_content
string_value:
{"id_": "3c868ef8-1238-4cf7-a71b-3b3f3cc3b4cd", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0ab1df67e41749ec6c62eaf013a9d47d0f1ecdd07fa7855873f263a9bd12438b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "77b745b8-dfcc-4a3d-8c64-c2b685c899a2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "039f85b4ba8c1c5af27ed24680cbc2a932b47a226ca57b48a99ec8a540ae6373", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3fbd66a5-b6bd-47c8-b7fd-362b05fd18a1", "node_type": "1", "metadata": {}, "hash": "a9f90653b2f6b2afeecea1e3baea97bdebc00fefe65a847bf3f4757133d59617", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9262, "end_char_idx": 11576, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Set Edit Area

An edit requires a "mask" to specify which portion of the image to regenerate. Any pixel with an alpha of 0 (transparent) will be regenerated. The code below creates a 1024x1024 mask where the bottom half is transparent.


```python
# create a mask
width = 1024
height = 1024
mask = Image.new("RGBA", (width, height), (0, 0, 0, 1))  # create an opaque image mask

# set the bottom half to be transparent
for x in range(width):
    for y in range(height // 2, height):  # only loop over the bottom half of the mask
        # set alpha (A) to zero to turn pixel transparent
        alpha = 0
        mask.putpixel((x, y), (0, 0, 0, alpha))

# save the mask
mask_name = "bottom_half_mask.png"
mask_filepath = os.path.join(image_dir, mask_name)
mask.save(mask_filepath)
```

### Perform Edit

Now we supply our image, caption and mask to the API to get 5 examples of edits to our image


```python
# edit an image

# call the OpenAI API
edit_response = client.images.edit(
    image=open(generated_image_filepath, "rb"),  # from the generation section
    mask=open(mask_filepath, "rb"),  # from right above
    prompt=prompt,  # from the generation section
    n=1,
    size="1024x1024",
    response_format="url",
)

# print response
print(edit_response)
```

    ImagesResponse(created=1701994167, data=[Image(b64_json=None, revised_prompt=None, url='https://oaidalleapiprodscus.blob.core.windows.net/private/org-9HXYFy8ux4r6aboFyec2OLRf/user-8OA8IvMYkfdAcUZXgzAXHS7d/img-9UOVGC7wB8MS2Q7Rwgj0fFBq.png?st=2023-12-07T23%3A09%3A27Z&se=2023-12-08T01%3A09%3A27Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2023-12-07T16%3A40%3A37Z&ske=2023-12-08T16%3A40%3A37Z&sks=b&skv=2021-08-06&sig=MsRMZ1rN434bVdWr%2B9kIoqu9CIrvZypZBfkQPTOhCl4%3D')])



```python
# save the image
edited_image_name = "edited_image.png"  # any name you like; the filetype should be .png
edited_image_filepath = os.path.join(image_dir, edited_image_name)
edited_image_url = edit_response.data[0].url  # extract image URL from response
edited_image = requests.get(edited_image_url).content  # download the image

with open(edited_image_filepath, "wb") as image_file:
    image_file.write(edited_image)  # write the image to the file
```


key: _node_content
string_value:
{"id_": "3fbd66a5-b6bd-47c8-b7fd-362b05fd18a1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "1bec0f40-14a0-47f7-a0ff-5bb8a61bb49b", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0ab1df67e41749ec6c62eaf013a9d47d0f1ecdd07fa7855873f263a9bd12438b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3c868ef8-1238-4cf7-a71b-3b3f3cc3b4cd", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "77807684d9069d61decc78a9f4611e6d5830bbcba09168dcae74cff68a3d76cc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11113, "end_char_idx": 11791, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# save the image
edited_image_name = "edited_image.png"  # any name you like; the filetype should be .png
edited_image_filepath = os.path.join(image_dir, edited_image_name)
edited_image_url = edit_response.data[0].url  # extract image URL from response
edited_image = requests.get(edited_image_url).content  # download the image

with open(edited_image_filepath, "wb") as image_file:
    image_file.write(edited_image)  # write the image to the file
```


```python
# print the original image
print(generated_image_filepath)
display(Image.open(generated_image_filepath))

# print edited image
print(edited_image_filepath)
display(Image.open(edited_image_filepath))
```


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb


key: _node_content
string_value:
{"id_": "9ef051b8-63d6-42bc-bb14-2f7e3cadb37d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4113902b964755cb6485391a9fa1d36013bdb8e97ece3a53909b6fa11bb4c4dd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "502db638-9743-42da-b129-274f9d9d7c53", "node_type": "1", "metadata": {}, "hash": "3f872709cd0ad71669adf9a32846c3719ad168a0d9fe6b7f1d34ff216c84207e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2755, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to create dynamic masks with DALL·E and Segment Anything

Segment Anything is a model from Meta that can be used to select portions of images. Combined with DALL·E's ability to inpaint specified portions of images, you can use Segment Anything to easily select any part of an image you'd like to alter.

In this notebook, we'll use these tools to become fashion designers and dynamically replace our digital models' outfits with tailored, original creations. The notebook follows this flow:
- **Setup:** Initialise your libraries and any location directories.
- **Generate original image:** Make an original image that we'll create dynamic masks from.
- **Generate mask:** Use Segment Anything to create a dynamic mask.
- **Create new image:** Generate a new image with the masked area inpainted with a fresh prompt.

## Setup

To get started we'll need to follow the [instructions](https://github.com/facebookresearch/segment-anything) for using the Segment Anything (SAM) model open-sourced by Meta. As of May 2023, the key steps are:
- Install [Pytorch](https://pytorch.org/get-started/locally/) (version 1.7+).
- Install the library using ```pip install git+https://github.com/facebookresearch/segment-anything.git```.
- Install dependencies using ```pip install opencv-python pycocotools matplotlib onnxruntime onnx```.
- Download a [model checkpoint](https://github.com/facebookresearch/segment-anything#model-checkpoints) to use (default size is 2.4 GB).



```python
!pip install torch torchvision torchaudio
!pip install git+https://github.com/facebookresearch/segment-anything.git
!pip install opencv-python pycocotools matplotlib onnxruntime onnx
!pip install requests
!pip install openai
!pip install numpy
```


```python
!wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth
```


```python
import cv2
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
from matplotlib import rcParams
import numpy as np
from openai import OpenAI
import os
from PIL import Image
import requests
from segment_anything import sam_model_registry, SamAutomaticMaskGenerator, SamPredictor
import torch

# Set directories for generation images and edit images
base_image_dir = os.path.join("images", "01_generations")
mask_dir = os.path.join("images", "02_masks")
edit_image_dir = os.path.join("images", "03_edits")

# Point to your downloaded SAM model
sam_model_filepath = "./sam_vit_h_4b8939.pth"

# Initiate SAM model
sam = sam_model_registry["default"](checkpoint=sam_model_filepath)

# Initiate openAI client
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))

```

## Generate original image

First we'll create an original image which we'll generate masks from.


key: _node_content
string_value:
{"id_": "502db638-9743-42da-b129-274f9d9d7c53", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4113902b964755cb6485391a9fa1d36013bdb8e97ece3a53909b6fa11bb4c4dd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9ef051b8-63d6-42bc-bb14-2f7e3cadb37d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4ddc53453b171b4f334074341e934edcf7591744a638c0009f1759ad6db9ba3f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "53d27639-1aed-4bda-a984-4d46365f1933", "node_type": "1", "metadata": {}, "hash": "150cbdb4185cb59868a7c988f47a1ba89266553763a874c355bb162ad93ba927", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1471, "end_char_idx": 4084, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
!pip install torch torchvision torchaudio
!pip install git+https://github.com/facebookresearch/segment-anything.git
!pip install opencv-python pycocotools matplotlib onnxruntime onnx
!pip install requests
!pip install openai
!pip install numpy
```


```python
!wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth
```


```python
import cv2
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
from matplotlib import rcParams
import numpy as np
from openai import OpenAI
import os
from PIL import Image
import requests
from segment_anything import sam_model_registry, SamAutomaticMaskGenerator, SamPredictor
import torch

# Set directories for generation images and edit images
base_image_dir = os.path.join("images", "01_generations")
mask_dir = os.path.join("images", "02_masks")
edit_image_dir = os.path.join("images", "03_edits")

# Point to your downloaded SAM model
sam_model_filepath = "./sam_vit_h_4b8939.pth"

# Initiate SAM model
sam = sam_model_registry["default"](checkpoint=sam_model_filepath)

# Initiate openAI client
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))

```

## Generate original image

First we'll create an original image which we'll generate masks from.


```python
def process_dalle_images(response, filename, image_dir):
    # save the images
    urls = [datum.url for datum in response.data]  # extract URLs
    images = [requests.get(url).content for url in urls]  # download images
    image_names = [f"{filename}_{i + 1}.png" for i in range(len(images))]  # create names
    filepaths = [os.path.join(image_dir, name) for name in image_names]  # create filepaths
    for image, filepath in zip(images, filepaths):  # loop through the variations
        with open(filepath, "wb") as image_file:  # open the file
            image_file.write(image)  # write the image to the file

    return filepaths
```


```python
dalle_prompt = '''
Full length, zoomed out photo of our premium Lederhosen-inspired jumpsuit.
Showcase the intricate hand-stitched details and high-quality leather, while highlighting the perfect blend of Austrian heritage and modern fashion.
This piece appeals to a sophisticated, trendsetting audience who appreciates cultural fusion and innovative design.
'''
```


```python
# Generate your images
generation_response = client.images.generate(
    model = "dall-e-3",
    prompt=dalle_prompt,
    n=3,
    size="1024x1024",
    response_format="url",
)
```


```python
filepaths = process_dalle_images(generation_response, "generation", base_image_dir)
```


key: _node_content
string_value:
{"id_": "53d27639-1aed-4bda-a984-4d46365f1933", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4113902b964755cb6485391a9fa1d36013bdb8e97ece3a53909b6fa11bb4c4dd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "502db638-9743-42da-b129-274f9d9d7c53", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4e86934f578d8177e2a0ac14fab1c95638fa3de55e169cc798a89faa5d3e9eff", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "04c5afc6-30a4-46e0-9b9a-5f1a5930e0f5", "node_type": "1", "metadata": {}, "hash": "f7f0568eb9192a7ca1476f462fa08aef7b0dfafa43f719ea9763c74f285d8b68", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2758, "end_char_idx": 5595, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def process_dalle_images(response, filename, image_dir):
    # save the images
    urls = [datum.url for datum in response.data]  # extract URLs
    images = [requests.get(url).content for url in urls]  # download images
    image_names = [f"{filename}_{i + 1}.png" for i in range(len(images))]  # create names
    filepaths = [os.path.join(image_dir, name) for name in image_names]  # create filepaths
    for image, filepath in zip(images, filepaths):  # loop through the variations
        with open(filepath, "wb") as image_file:  # open the file
            image_file.write(image)  # write the image to the file

    return filepaths
```


```python
dalle_prompt = '''
Full length, zoomed out photo of our premium Lederhosen-inspired jumpsuit.
Showcase the intricate hand-stitched details and high-quality leather, while highlighting the perfect blend of Austrian heritage and modern fashion.
This piece appeals to a sophisticated, trendsetting audience who appreciates cultural fusion and innovative design.
'''
```


```python
# Generate your images
generation_response = client.images.generate(
    model = "dall-e-3",
    prompt=dalle_prompt,
    n=3,
    size="1024x1024",
    response_format="url",
)
```


```python
filepaths = process_dalle_images(generation_response, "generation", base_image_dir)
```


```python
# print the new generations
for filepath in filepaths:
    print(filepath)
    display(Image.open(filepath))
```

## Generate Mask

Next we'll load up one of our images and generate masks.

For this demonstration we're picking a UX where we "click" on a point on the image to generate masks from. However, there are [example notebooks](https://github.com/facebookresearch/segment-anything/blob/main/notebooks/automatic_mask_generator_example.ipynb) provided by Meta which show how to generate every possible mask for an image, draw a box, and some other useful approaches.


```python
# Pick one of your generated images
chosen_image = "images/01_generations/generation_2.png"
```


```python
# Function to display mask using matplotlib
def show_mask(mask, ax):
    color = np.array([30 / 255, 144 / 255, 255 / 255, 0.6])
    h, w = mask.shape[-2:]
    mask_image = mask.reshape(h, w, 1) * color.reshape(1, 1, -1)
    ax.imshow(mask_image)


# Function to display where we've "clicked"
def show_points(coords, labels, ax, marker_size=375):
    pos_points = coords[labels == 1]
    neg_points = coords[labels == 0]
    ax.scatter(
        pos_points[:, 0],
        pos_points[:, 1],
        color="green",
        marker="*",
        s=marker_size,
        edgecolor="white",
        linewidth=1.25,
    )
    ax.scatter(
        neg_points[:, 0],
        neg_points[:, 1],
        color="red",
        marker="*",
        s=marker_size,
        edgecolor="white",
        linewidth=1.25,
    )

```


key: _node_content
string_value:
{"id_": "04c5afc6-30a4-46e0-9b9a-5f1a5930e0f5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4113902b964755cb6485391a9fa1d36013bdb8e97ece3a53909b6fa11bb4c4dd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "53d27639-1aed-4bda-a984-4d46365f1933", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b110e4ff66400a98b381fd4a4f322e9749517964b58f3727f8dd72bc1b7a2b23", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bb859304-a79c-4087-a156-ba037633dc0d", "node_type": "1", "metadata": {}, "hash": "2c3c5989065f48bf1f980e73e021ea0346f149315b2fcb667411b0f08938bfbd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4087, "end_char_idx": 7011, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# print the new generations
for filepath in filepaths:
    print(filepath)
    display(Image.open(filepath))
```

## Generate Mask

Next we'll load up one of our images and generate masks.

For this demonstration we're picking a UX where we "click" on a point on the image to generate masks from. However, there are [example notebooks](https://github.com/facebookresearch/segment-anything/blob/main/notebooks/automatic_mask_generator_example.ipynb) provided by Meta which show how to generate every possible mask for an image, draw a box, and some other useful approaches.


```python
# Pick one of your generated images
chosen_image = "images/01_generations/generation_2.png"
```


```python
# Function to display mask using matplotlib
def show_mask(mask, ax):
    color = np.array([30 / 255, 144 / 255, 255 / 255, 0.6])
    h, w = mask.shape[-2:]
    mask_image = mask.reshape(h, w, 1) * color.reshape(1, 1, -1)
    ax.imshow(mask_image)


# Function to display where we've "clicked"
def show_points(coords, labels, ax, marker_size=375):
    pos_points = coords[labels == 1]
    neg_points = coords[labels == 0]
    ax.scatter(
        pos_points[:, 0],
        pos_points[:, 1],
        color="green",
        marker="*",
        s=marker_size,
        edgecolor="white",
        linewidth=1.25,
    )
    ax.scatter(
        neg_points[:, 0],
        neg_points[:, 1],
        color="red",
        marker="*",
        s=marker_size,
        edgecolor="white",
        linewidth=1.25,
    )

```


```python
# Load chosen image using opencv
image = cv2.imread(chosen_image)
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

# Display our chosen image
plt.figure(figsize=(10, 10))
plt.imshow(image)
plt.axis("on")
plt.show()
```


    
![png](output_14_0.png)
    



```python
# Set the pixel coordinates for our "click" to assign masks
input_point = np.array([[525, 325]])
input_label = np.array([1])

# Display the point we've clicked on
plt.figure(figsize=(10, 10))
plt.imshow(image)
show_points(input_point, input_label, plt.gca())
plt.axis("on")
plt.show()
```


    
![png](output_15_0.png)
    



```python
# Initiate predictor with Segment Anything model
predictor = SamPredictor(sam)
predictor.set_image(image)

# Use the predictor to gather masks for the point we clicked
masks, scores, logits = predictor.predict(
    point_coords=input_point,
    point_labels=input_label,
    multimask_output=True,
)

# Check the shape - should be three masks of the same dimensions as our image
masks.shape
```




    (3, 1024, 1024)




```python
# Display the possible masks we can select along with their confidence
for i, (mask, score) in enumerate(zip(masks, scores)):
    plt.figure(figsize=(10, 10))
    plt.imshow(image)
    show_mask(mask, plt.gca())
    show_points(input_point, input_label, plt.gca())
    plt.title(f"Mask {i+1}, Score: {score:.3f}", fontsize=18)
    plt.axis("off")
    plt.show()
```


key: _node_content
string_value:
{"id_": "bb859304-a79c-4087-a156-ba037633dc0d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4113902b964755cb6485391a9fa1d36013bdb8e97ece3a53909b6fa11bb4c4dd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "04c5afc6-30a4-46e0-9b9a-5f1a5930e0f5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "be3086ce650e376b704f40498b5eb54d83401987d9e2089deb0beef613d83aa4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e95226ce-9e7c-432d-997e-cbb00cfc7b2b", "node_type": "1", "metadata": {}, "hash": "cdae518d1ba8cdab98b04dbdfd1b6bd19ac151af4ca5491eb795369b603c069a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5598, "end_char_idx": 8235, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Load chosen image using opencv
image = cv2.imread(chosen_image)
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

# Display our chosen image
plt.figure(figsize=(10, 10))
plt.imshow(image)
plt.axis("on")
plt.show()
```


    
![png](output_14_0.png)
    



```python
# Set the pixel coordinates for our "click" to assign masks
input_point = np.array([[525, 325]])
input_label = np.array([1])

# Display the point we've clicked on
plt.figure(figsize=(10, 10))
plt.imshow(image)
show_points(input_point, input_label, plt.gca())
plt.axis("on")
plt.show()
```


    
![png](output_15_0.png)
    



```python
# Initiate predictor with Segment Anything model
predictor = SamPredictor(sam)
predictor.set_image(image)

# Use the predictor to gather masks for the point we clicked
masks, scores, logits = predictor.predict(
    point_coords=input_point,
    point_labels=input_label,
    multimask_output=True,
)

# Check the shape - should be three masks of the same dimensions as our image
masks.shape
```




    (3, 1024, 1024)




```python
# Display the possible masks we can select along with their confidence
for i, (mask, score) in enumerate(zip(masks, scores)):
    plt.figure(figsize=(10, 10))
    plt.imshow(image)
    show_mask(mask, plt.gca())
    show_points(input_point, input_label, plt.gca())
    plt.title(f"Mask {i+1}, Score: {score:.3f}", fontsize=18)
    plt.axis("off")
    plt.show()
```


    
![png](output_17_0.png)
    



    
![png](output_17_1.png)
    



    
![png](output_17_2.png)
    



```python
# Choose which mask you'd like to use
chosen_mask = masks[1]

# We'll now reverse the mask so that it is clear and everything else is white
chosen_mask = chosen_mask.astype("uint8")
chosen_mask[chosen_mask != 0] = 255
chosen_mask[chosen_mask == 0] = 1
chosen_mask[chosen_mask == 255] = 0
chosen_mask[chosen_mask == 1] = 255

```


```python
# create a base blank mask
width = 1024
height = 1024
mask = Image.new("RGBA", (width, height), (0, 0, 0, 1))  # create an opaque image mask

# Convert mask back to pixels to add our mask replacing the third dimension
pix = np.array(mask)
pix[:, :, 3] = chosen_mask

# Convert pixels back to an RGBA image and display
new_mask = Image.fromarray(pix, "RGBA")
new_mask
```




    
![png](output_19_0.png)
    




```python
# We'll save this mask for re-use for our edit
new_mask.save(os.path.join(mask_dir, "new_mask.png"))
```

## Create new image

Now we'll combine our original image with the mask and the Edit endpoint for DALLE to inpaint the transparent area according to a new prompt. (as 0f January 2024 dall-e-2 is the only model that supports edits)


key: _node_content
string_value:
{"id_": "e95226ce-9e7c-432d-997e-cbb00cfc7b2b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "5cb07a00-3915-49b2-a67f-75cefa39c831", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4113902b964755cb6485391a9fa1d36013bdb8e97ece3a53909b6fa11bb4c4dd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bb859304-a79c-4087-a156-ba037633dc0d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "9a231d1a902374e24bfcc4f379204c06bab6ad922fc50e08399ad03bff73be90", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7019, "end_char_idx": 9362, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
![png](output_17_0.png)
    



    
![png](output_17_1.png)
    



    
![png](output_17_2.png)
    



```python
# Choose which mask you'd like to use
chosen_mask = masks[1]

# We'll now reverse the mask so that it is clear and everything else is white
chosen_mask = chosen_mask.astype("uint8")
chosen_mask[chosen_mask != 0] = 255
chosen_mask[chosen_mask == 0] = 1
chosen_mask[chosen_mask == 255] = 0
chosen_mask[chosen_mask == 1] = 255

```


```python
# create a base blank mask
width = 1024
height = 1024
mask = Image.new("RGBA", (width, height), (0, 0, 0, 1))  # create an opaque image mask

# Convert mask back to pixels to add our mask replacing the third dimension
pix = np.array(mask)
pix[:, :, 3] = chosen_mask

# Convert pixels back to an RGBA image and display
new_mask = Image.fromarray(pix, "RGBA")
new_mask
```




    
![png](output_19_0.png)
    




```python
# We'll save this mask for re-use for our edit
new_mask.save(os.path.join(mask_dir, "new_mask.png"))
```

## Create new image

Now we'll combine our original image with the mask and the Edit endpoint for DALLE to inpaint the transparent area according to a new prompt. (as 0f January 2024 dall-e-2 is the only model that supports edits)



```python
# edit an image
edit_response = client.images.edit(
    image=open(chosen_image, "rb"),  # from the generation section
    mask=open(os.path.join(mask_dir, "new_mask.png"), "rb"),  # from right above
    prompt="Brilliant leather Lederhosen with a formal look, detailed, intricate, photorealistic",  # provide a prompt to fill the space
    n=3,
    size="1024x1024",
    response_format="url",
)

edit_filepaths = process_dalle_images(edit_response, "edits", edit_image_dir)

```


```python
# Display your beautiful creations!
%matplotlib inline

# figure size in inches optional
rcParams["figure.figsize"] = 11 ,8

# read images
img_A = mpimg.imread(edit_filepaths[0])
img_B = mpimg.imread(edit_filepaths[1])
img_C = mpimg.imread(edit_filepaths[2])

# display images
fig, ax = plt.subplots(1,3)
[a.axis("off") for a in ax]
ax[0].imshow(img_A)
ax[1].imshow(img_B)
ax[2].imshow(img_C)
```




    <matplotlib.image.AxesImage at 0x791b1f4c58a0>




    
![png](output_23_1.png)
    


Beautiful!

Now you too can easily create dynamic masks to extend your images - enjoy the APIs, and please share what you build!


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb


key: _node_content
string_value:
{"id_": "a6300a41-e6e1-4f44-b1d8-019d1634f0dd", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7b76357f-8c0b-4717-a49c-deacfc601807", "node_type": "1", "metadata": {}, "hash": "95c5cdd8a888a4a50454d7672f9c198c1edabd1ce5a706351e5f15ba5e51e17c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2450, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to evaluate a summarization task

In this notebook we delve into the evaluation techniques for abstractive summarization tasks using a simple example. We explore traditional evaluation methods like [ROUGE](https://aclanthology.org/W04-1013/) and [BERTScore](https://arxiv.org/abs/1904.09675), in addition to showcasing a more novel approach using LLMs as evaluators.

Evaluating the quality of summaries is a time-consuming process, as it involves different quality metrics such as coherence, conciseness, readability and content. Traditional automatic evaluation metrics such as `ROUGE` and `BERTScore` and others are concrete and reliable, but they may not correlate well with the actual quality of summaries. They show relatively low correlation with human judgments, especially for open-ended generation tasks ([Liu et al., 2023](https://arxiv.org/pdf/2303.16634.pdf)). There's a growing need to lean on human evaluations, user feedback, or model-based metrics while being vigilant about potential biases. While human judgment provides invaluable insights, it is often not scalable and can be cost-prohibitive.

In addition to these traditional metrics, we showcase a method ([G-Eval](https://arxiv.org/pdf/2303.16634.pdf)) that leverages Large Language Models (LLMs) as a novel, reference-free metric for assessing abstractive summaries. In this case, we use `gpt-4` to score candidate outputs. `gpt-4` has effectively learned an internal model of language quality that allows it to differentiate between fluent, coherent text and low-quality text. Harnessing this internal scoring mechanism allows auto-evaluation of new candidate outputs generated by an LLM.


## Setup



```python
# Installing necessary packages for the evaluation
# rouge: For evaluating with ROUGE metric
# bert_score: For evaluating with BERTScore
# openai: To interact with OpenAI's API
!pip install rouge --quiet
!pip install bert_score --quiet
!pip install openai --quiet
```


```python
from openai import OpenAI
import os
import re
import pandas as pd

# Python Implementation of the ROUGE Metric
from rouge import Rouge

# BERTScore leverages the pre-trained contextual embeddings from BERT and matches words in candidate and reference sentences by cosine similarity.
from bert_score import BERTScorer

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))

```


    <IPython.core.display.Javascript object>


key: _node_content
string_value:
{"id_": "7b76357f-8c0b-4717-a49c-deacfc601807", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a6300a41-e6e1-4f44-b1d8-019d1634f0dd", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8578004bb41ff667a1b8b98372d06a46f8ea23c2379d6efd592885bc255f93f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ae4f21a9-b32f-4632-a6fe-58e7d0598087", "node_type": "1", "metadata": {}, "hash": "c8f266d2cac970d57cd8bde60d8d84155be9ea038460f0370ee9be4541f6362e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1674, "end_char_idx": 6892, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Setup



```python
# Installing necessary packages for the evaluation
# rouge: For evaluating with ROUGE metric
# bert_score: For evaluating with BERTScore
# openai: To interact with OpenAI's API
!pip install rouge --quiet
!pip install bert_score --quiet
!pip install openai --quiet
```


```python
from openai import OpenAI
import os
import re
import pandas as pd

# Python Implementation of the ROUGE Metric
from rouge import Rouge

# BERTScore leverages the pre-trained contextual embeddings from BERT and matches words in candidate and reference sentences by cosine similarity.
from bert_score import BERTScorer

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))

```


    <IPython.core.display.Javascript object>


## Example task

For the purposes of this notebook we'll use the example summarization below. Notice that we provide two generated summaries to compare, and a reference human-written summary, which evaluation metrics like `ROUGE` and `BERTScore` require.

Excerpt (`excerpt`):

> OpenAI's mission is to ensure that artificial general intelligence (AGI) benefits all of humanity. OpenAI will build safe and beneficial AGI directly, but will also consider its mission fulfilled if its work aids others to achieve this outcome. OpenAI follows several key principles for this purpose. First, broadly distributed benefits - any influence over AGI's deployment will be used for the benefit of all, and to avoid harmful uses or undue concentration of power. Second, long-term safety - OpenAI is committed to doing the research to make AGI safe, and to promote the adoption of such research across the AI community. Third, technical leadership - OpenAI aims to be at the forefront of AI capabilities. Fourth, a cooperative orientation - OpenAI actively cooperates with other research and policy institutions, and seeks to create a global community working together to address AGI's global challenges.

Summaries:

| Reference Summary /`ref_summary` (human generated)                                                                                                                                                                                                                                                                                                                         | Eval Summary 1 / `eval_summary_1` (system generated)                                                                                                                                                                                                                                                                                                                               | Eval Summary 2 / `eval_summary_2` (system generated)                                                                                                                                                                                                                                                   |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| OpenAI aims to ensure artificial general intelligence (AGI) is used for everyone's benefit, avoiding harmful uses or undue power concentration. It is committed to researching AGI safety, promoting such studies among the AI community. OpenAI seeks to lead in AI capabilities and cooperates with global research and policy institutions to address AGI's challenges. | OpenAI aims to AGI benefits all humanity, avoiding harmful uses and power concentration. It pioneers research into safe and beneficial AGI and promotes adoption globally. OpenAI maintains technical leadership in AI while cooperating with global institutions to address AGI challenges. It seeks to lead a collaborative worldwide effort developing AGI for collective good. | OpenAI aims to ensure AGI is for everyone's use, totally avoiding harmful stuff or big power concentration. Committed to researching AGI's safe side, promoting these studies in AI folks. OpenAI wants to be top in AI things and works with worldwide research, policy groups to figure AGI's stuff. |

Take a moment to figure out which summary you'd personally prefer and the one that captures OpenAI's mission really well.


key: _node_content
string_value:
{"id_": "ae4f21a9-b32f-4632-a6fe-58e7d0598087", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7b76357f-8c0b-4717-a49c-deacfc601807", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "bcd5efef4d3e0374c3038373bc10e4bf182c829c53e5f774372c57bde7f1e8c0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e811d8fe-58c1-46e7-bb75-8499e6544374", "node_type": "1", "metadata": {}, "hash": "b4521af41da3037c5cbad4fccbc8f93dac9bc5779bbf896d17fe26f9b4101f53", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6896, "end_char_idx": 10220, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
excerpt = "OpenAI's mission is to ensure that artificial general intelligence (AGI) benefits all of humanity. OpenAI will build safe and beneficial AGI directly, but will also consider its mission fulfilled if its work aids others to achieve this outcome. OpenAI follows several key principles for this purpose. First, broadly distributed benefits - any influence over AGI's deployment will be used for the benefit of all, and to avoid harmful uses or undue concentration of power. Second, long-term safety - OpenAI is committed to doing the research to make AGI safe, and to promote the adoption of such research across the AI community. Third, technical leadership - OpenAI aims to be at the forefront of AI capabilities. Fourth, a cooperative orientation - OpenAI actively cooperates with other research and policy institutions, and seeks to create a global community working together to address AGI's global challenges."
ref_summary = "OpenAI aims to ensure artificial general intelligence (AGI) is used for everyone's benefit, avoiding harmful uses or undue power concentration. It is committed to researching AGI safety, promoting such studies among the AI community. OpenAI seeks to lead in AI capabilities and cooperates with global research and policy institutions to address AGI's challenges."
eval_summary_1 = "OpenAI aims to AGI benefits all humanity, avoiding harmful uses and power concentration. It pioneers research into safe and beneficial AGI and promotes adoption globally. OpenAI maintains technical leadership in AI while cooperating with global institutions to address AGI challenges. It seeks to lead a collaborative worldwide effort developing AGI for collective good."
eval_summary_2 = "OpenAI aims to ensure AGI is for everyone's use, totally avoiding harmful stuff or big power concentration. Committed to researching AGI's safe side, promoting these studies in AI folks. OpenAI wants to be top in AI things and works with worldwide research, policy groups to figure AGI's stuff."
```


    <IPython.core.display.Javascript object>


## Evaluating using ROUGE

[ROUGE](https://aclanthology.org/W04-1013/), which stands for Recall-Oriented Understudy for Gisting Evaluation, primarily gauges the overlap of words between a generated output and a reference text. It's a prevalent metric for evaluating automatic summarization tasks. Among its variants, `ROUGE-L` offers insights into the longest contiguous match between system-generated and reference summaries, gauging how well the system retains the original summary's essence.



```python
# function to calculate the Rouge score
def get_rouge_scores(text1, text2):
    rouge = Rouge()
    return rouge.get_scores(text1, text2)


rouge_scores_out = []

# Calculate the ROUGE scores for both summaries using reference
eval_1_rouge = get_rouge_scores(eval_summary_1, ref_summary)
eval_2_rouge = get_rouge_scores(eval_summary_2, ref_summary)

for metric in ["rouge-1", "rouge-2", "rouge-l"]:
    for label in ["F-Score"]:
        eval_1_score = eval_1_rouge[0][metric][label[0].lower()]
        eval_2_score = eval_2_rouge[0][metric][label[0].lower()]

        row = {
            "Metric": f"{metric} ({label})",
            "Summary 1": eval_1_score,
            "Summary 2": eval_2_score,
        }
        rouge_scores_out.append(row)


key: _node_content
string_value:
{"id_": "e811d8fe-58c1-46e7-bb75-8499e6544374", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ae4f21a9-b32f-4632-a6fe-58e7d0598087", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "efe0754d2c2fb1fa89affb8e8a9e725591765f7528f95a624573f5bbf172737e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c79b6a6c-b173-41f9-b88e-9ad345c6835f", "node_type": "1", "metadata": {}, "hash": "81476445bc57001c5fb5142b26eee5186b5a76ce47defb6e9d76e4d65e7bb780", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8924, "end_char_idx": 10535, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<IPython.core.display.Javascript object>


## Evaluating using ROUGE

[ROUGE](https://aclanthology.org/W04-1013/), which stands for Recall-Oriented Understudy for Gisting Evaluation, primarily gauges the overlap of words between a generated output and a reference text. It's a prevalent metric for evaluating automatic summarization tasks. Among its variants, `ROUGE-L` offers insights into the longest contiguous match between system-generated and reference summaries, gauging how well the system retains the original summary's essence.



```python
# function to calculate the Rouge score
def get_rouge_scores(text1, text2):
    rouge = Rouge()
    return rouge.get_scores(text1, text2)


rouge_scores_out = []

# Calculate the ROUGE scores for both summaries using reference
eval_1_rouge = get_rouge_scores(eval_summary_1, ref_summary)
eval_2_rouge = get_rouge_scores(eval_summary_2, ref_summary)

for metric in ["rouge-1", "rouge-2", "rouge-l"]:
    for label in ["F-Score"]:
        eval_1_score = eval_1_rouge[0][metric][label[0].lower()]
        eval_2_score = eval_2_rouge[0][metric][label[0].lower()]

        row = {
            "Metric": f"{metric} ({label})",
            "Summary 1": eval_1_score,
            "Summary 2": eval_2_score,
        }
        rouge_scores_out.append(row)


def highlight_max(s):
    is_max = s == s.max()
    return [
        "background-color: lightgreen" if v else "background-color: white"
        for v in is_max
    ]


rouge_scores_out = (
    pd.DataFrame(rouge_scores_out)
    .set_index("Metric")
    .style.apply(highlight_max, axis=1)
)

rouge_scores_out
```


key: _node_content
string_value:
{"id_": "c79b6a6c-b173-41f9-b88e-9ad345c6835f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e811d8fe-58c1-46e7-bb75-8499e6544374", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1874c5528d117efd82bc0214e88db51f39ea647bf1f1f20d7caa929dca48ca05", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c1addd0a-8c54-4009-a6a9-040ff886f2c2", "node_type": "1", "metadata": {}, "hash": "eee3558fa20b8944ebcd928b79ba588daac76dc4eed49cb5bec8a0f8df83185d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9465, "end_char_idx": 11966, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# function to calculate the Rouge score
def get_rouge_scores(text1, text2):
    rouge = Rouge()
    return rouge.get_scores(text1, text2)


rouge_scores_out = []

# Calculate the ROUGE scores for both summaries using reference
eval_1_rouge = get_rouge_scores(eval_summary_1, ref_summary)
eval_2_rouge = get_rouge_scores(eval_summary_2, ref_summary)

for metric in ["rouge-1", "rouge-2", "rouge-l"]:
    for label in ["F-Score"]:
        eval_1_score = eval_1_rouge[0][metric][label[0].lower()]
        eval_2_score = eval_2_rouge[0][metric][label[0].lower()]

        row = {
            "Metric": f"{metric} ({label})",
            "Summary 1": eval_1_score,
            "Summary 2": eval_2_score,
        }
        rouge_scores_out.append(row)


def highlight_max(s):
    is_max = s == s.max()
    return [
        "background-color: lightgreen" if v else "background-color: white"
        for v in is_max
    ]


rouge_scores_out = (
    pd.DataFrame(rouge_scores_out)
    .set_index("Metric")
    .style.apply(highlight_max, axis=1)
)

rouge_scores_out
```




<style type="text/css">
#T_7e6ac_row0_col0, #T_7e6ac_row1_col1, #T_7e6ac_row2_col0 {
  background-color: white;
}
#T_7e6ac_row0_col1, #T_7e6ac_row1_col0, #T_7e6ac_row2_col1 {
  background-color: lightgreen;
}
</style>
<table id="T_7e6ac">
  <thead>
    <tr>
      <th class="blank level0" >&nbsp;</th>
      <th id="T_7e6ac_level0_col0" class="col_heading level0 col0" >Summary 1</th>
      <th id="T_7e6ac_level0_col1" class="col_heading level0 col1" >Summary 2</th>
    </tr>
    <tr>
      <th class="index_name level0" >Metric</th>
      <th class="blank col0" >&nbsp;</th>
      <th class="blank col1" >&nbsp;</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th id="T_7e6ac_level0_row0" class="row_heading level0 row0" >rouge-1 (F-Score)</th>
      <td id="T_7e6ac_row0_col0" class="data row0 col0" >0.488889</td>
      <td id="T_7e6ac_row0_col1" class="data row0 col1" >0.511628</td>
    </tr>
    <tr>
      <th id="T_7e6ac_level0_row1" class="row_heading level0 row1" >rouge-2 (F-Score)</th>
      <td id="T_7e6ac_row1_col0" class="data row1 col0" >0.230769</td>
      <td id="T_7e6ac_row1_col1" class="data row1 col1" >0.163265</td>
    </tr>
    <tr>
      <th id="T_7e6ac_level0_row2" class="row_heading level0 row2" >rouge-l (F-Score)</th>
      <td id="T_7e6ac_row2_col0" class="data row2 col0" >0.488889</td>
      <td id="T_7e6ac_row2_col1" class="data row2 col1" >0.511628</td>
    </tr>
  </tbody>
</table>


key: _node_content
string_value:
{"id_": "c1addd0a-8c54-4009-a6a9-040ff886f2c2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c79b6a6c-b173-41f9-b88e-9ad345c6835f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d1e6ae6332f47d3c758e8b71f696c7631e657cab5f88b07bfcc91abd14b1ab7f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5a8ee7d5-25a7-4d7b-b99a-1796e5c52f8a", "node_type": "1", "metadata": {}, "hash": "d01e0636fc392f1e46e62a08904eaaf217b146c4946a8d5c24e556b92a550980", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11976, "end_char_idx": 15206, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<IPython.core.display.Javascript object>


The table shows the `ROUGE` scores for evaluating two different summaries against a reference text. In the case of `rouge-1`, Summary 2 outperforms Summary 1, indicating a better overlap of individual words and for `rouge-l`, Summary 2 has a higher score, implying a closer match in the longest common subsequences, and thus a potentially better overall summarization in capturing the main content and order of the original text. Since Summary 2 has many words and short phrases directly lifted from the excerpt, its overlap with the reference summary would likely be higher, leading to higher `ROUGE` scores.

While `ROUGE` and similar metrics, such as [BLEU](https://aclanthology.org/P02-1040.pdf) and [METEOR](https://www.cs.cmu.edu/~alavie/METEOR/), offer quantitative measures, they often fail to capture the true essence of a well-generated summary. They also correlate worse with human scores. Given the advancements in LLMs, which are adept at producing fluent and coherent summaries, traditional metrics like `ROUGE` may inadvertently penalize these models. This is especially true if the summaries are articulated differently but still encapsulate the core information accurately.


## Evaluating using BERTScore

ROUGE relies on the exact presence of words in both the predicted and reference texts, failing to interpret the underlying semantics. This is where [BERTScore](https://arxiv.org/abs/1904.09675) comes in and leverages the contextual embeddings from the BERT model, aiming to evaluate the similarity between a predicted and a reference sentence in the context of machine-generated text. By comparing embeddings from both sentences, `BERTScore` captures semantic similarities that might be missed by traditional n-gram based metrics.



```python
# Instantiate the BERTScorer object for English language
scorer = BERTScorer(lang="en")

# Calculate BERTScore for the summary 1 against the excerpt
# P1, R1, F1_1 represent Precision, Recall, and F1 Score respectively
P1, R1, F1_1 = scorer.score([eval_summary_1], [ref_summary])

# Calculate BERTScore for summary 2 against the excerpt
# P2, R2, F2_2 represent Precision, Recall, and F1 Score respectively
P2, R2, F2_2 = scorer.score([eval_summary_2], [ref_summary])

print("Summary 1 F1 Score:", F1_1.tolist()[0])
print("Summary 2 F1 Score:", F2_2.tolist()[0])
```

    Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight', 'lm_head.decoder.weight', 'lm_head.bias']
    - This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
    - This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).


    Summary 1 F1 Score: 0.9227314591407776
    Summary 2 F1 Score: 0.9189572930335999



    <IPython.core.display.Javascript object>


key: _node_content
string_value:
{"id_": "5a8ee7d5-25a7-4d7b-b99a-1796e5c52f8a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c1addd0a-8c54-4009-a6a9-040ff886f2c2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1921a14c7b2200582b49bfb9715a0dd8d7139b8372a29ebf1a30d07ea523a431", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ac1580bc-a4ec-430d-93be-2c363bcd3536", "node_type": "1", "metadata": {}, "hash": "1439f8bb4fc7a2a79da913830b7604d9dd81f85ace5d5ffb1ac5bb2d0d6db099", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13777, "end_char_idx": 15737, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Instantiate the BERTScorer object for English language
scorer = BERTScorer(lang="en")

# Calculate BERTScore for the summary 1 against the excerpt
# P1, R1, F1_1 represent Precision, Recall, and F1 Score respectively
P1, R1, F1_1 = scorer.score([eval_summary_1], [ref_summary])

# Calculate BERTScore for summary 2 against the excerpt
# P2, R2, F2_2 represent Precision, Recall, and F1 Score respectively
P2, R2, F2_2 = scorer.score([eval_summary_2], [ref_summary])

print("Summary 1 F1 Score:", F1_1.tolist()[0])
print("Summary 2 F1 Score:", F2_2.tolist()[0])
```

    Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight', 'lm_head.decoder.weight', 'lm_head.bias']
    - This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
    - This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).


    Summary 1 F1 Score: 0.9227314591407776
    Summary 2 F1 Score: 0.9189572930335999



    <IPython.core.display.Javascript object>


The close F1 Scores between the summaries indicate that they may perform similarly in capturing the key information. However, this small difference should be interpreted with caution. Since `BERTScore` may not fully grasp subtleties and high-level concepts that a human evaluator might understand, reliance solely on this metric could lead to misinterpreting the actual quality and nuances of the summary. An integrated approach combining `BERTScore` with human judgment and other metrics could offer a more reliable evaluation.


key: _node_content
string_value:
{"id_": "ac1580bc-a4ec-430d-93be-2c363bcd3536", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5a8ee7d5-25a7-4d7b-b99a-1796e5c52f8a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "9ff7f98557cccf735f7e9dd40ae3397da730c142d8676afed535e5c6e370d010", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b7f809bd-cb1a-4cbe-9ae3-f051066542ba", "node_type": "1", "metadata": {}, "hash": "c7abc59b4b1407c97964e334311db93595655397b1fa1b4054e0aa57e13bb992", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15077, "end_char_idx": 17368, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Summary 1 F1 Score: 0.9227314591407776
    Summary 2 F1 Score: 0.9189572930335999



    <IPython.core.display.Javascript object>


The close F1 Scores between the summaries indicate that they may perform similarly in capturing the key information. However, this small difference should be interpreted with caution. Since `BERTScore` may not fully grasp subtleties and high-level concepts that a human evaluator might understand, reliance solely on this metric could lead to misinterpreting the actual quality and nuances of the summary. An integrated approach combining `BERTScore` with human judgment and other metrics could offer a more reliable evaluation.


## Evaluating using GPT-4

Here we implement an example **reference-free** text evaluator using `gpt-4`, inspired by the [G-Eval]((https://arxiv.org/pdf/2303.16634.pdf)) framework which evaluates the quality of generated text using large language models. Unlike metrics like `ROUGE` or `BERTScore` that rely on comparison to reference summaries, the `gpt-4` based evaluator assesses the quality of generated content based solely on the input prompt and text, without any ground truth references. This makes it applicable to new datasets and tasks where human references are sparse or unavailable. 

Here's an overview of this method:

1. We define four distinct criteria:
    1. **Relevance**: Evaluates if the summary includes only important information and excludes redundancies.
    2. **Coherence**: Assesses the logical flow and organization of the summary.
    3. **Consistency**: Checks if the summary aligns with the facts in the source document.
    4. **Fluency**: Rates the grammar and readability of the summary.
2. We craft prompts for each of these criteria, taking the original document and the summary as inputs, and leveraging chain-of-thought generation and guiding the model to output a numeric score from 1-5 for each criteria. 
3. We generate scores from `gpt-4` with the defined prompts, comparing them across summaries.

In this demonstration, we're using a direct scoring function where `gpt-4` generates a discrete score (1-5) for each metric. Normalizing the scores and taking a weighted sum could result in more robust, continuous scores that better reflect the quality and diversity of the summaries.


key: _node_content
string_value:
{"id_": "b7f809bd-cb1a-4cbe-9ae3-f051066542ba", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ac1580bc-a4ec-430d-93be-2c363bcd3536", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0264a51a7f0892b2fcda8c49649bb0b9ecb60fa210ec8d368479c340f847b5f1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "de7dfc0d-3e15-4725-87b6-6106e46ecd42", "node_type": "1", "metadata": {}, "hash": "23a177c88aa82909084813256b18dd5831fa17f2ed7924b26ca8c4ea6db46ade", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15740, "end_char_idx": 20652, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Evaluating using GPT-4

Here we implement an example **reference-free** text evaluator using `gpt-4`, inspired by the [G-Eval]((https://arxiv.org/pdf/2303.16634.pdf)) framework which evaluates the quality of generated text using large language models. Unlike metrics like `ROUGE` or `BERTScore` that rely on comparison to reference summaries, the `gpt-4` based evaluator assesses the quality of generated content based solely on the input prompt and text, without any ground truth references. This makes it applicable to new datasets and tasks where human references are sparse or unavailable. 

Here's an overview of this method:

1. We define four distinct criteria:
    1. **Relevance**: Evaluates if the summary includes only important information and excludes redundancies.
    2. **Coherence**: Assesses the logical flow and organization of the summary.
    3. **Consistency**: Checks if the summary aligns with the facts in the source document.
    4. **Fluency**: Rates the grammar and readability of the summary.
2. We craft prompts for each of these criteria, taking the original document and the summary as inputs, and leveraging chain-of-thought generation and guiding the model to output a numeric score from 1-5 for each criteria. 
3. We generate scores from `gpt-4` with the defined prompts, comparing them across summaries.

In this demonstration, we're using a direct scoring function where `gpt-4` generates a discrete score (1-5) for each metric. Normalizing the scores and taking a weighted sum could result in more robust, continuous scores that better reflect the quality and diversity of the summaries.


```python
# Evaluation prompt template based on G-Eval
EVALUATION_PROMPT_TEMPLATE = """
You will be given one summary written for an article. Your task is to rate the summary on one metric.
Please make sure you read and understand these instructions very carefully. 
Please keep this document open while reviewing, and refer to it as needed.

Evaluation Criteria:

{criteria}

Evaluation Steps:

{steps}

Example:

Source Text:

{document}

Summary:

{summary}

Evaluation Form (scores ONLY):

- {metric_name}
"""

# Metric 1: Relevance

RELEVANCY_SCORE_CRITERIA = """
Relevance(1-5) - selection of important content from the source. \
The summary should include only important information from the source document. \
Annotators were instructed to penalize summaries which contained redundancies and excess information.
"""

RELEVANCY_SCORE_STEPS = """
1. Read the summary and the source document carefully.
2. Compare the summary to the source document and identify the main points of the article.
3. Assess how well the summary covers the main points of the article, and how much irrelevant or redundant information it contains.
4. Assign a relevance score from 1 to 5.
"""

# Metric 2: Coherence

COHERENCE_SCORE_CRITERIA = """
Coherence(1-5) - the collective quality of all sentences. \
We align this dimension with the DUC quality question of structure and coherence \
whereby "the summary should be well-structured and well-organized. \
The summary should not just be a heap of related information, but should build from sentence to a\
coherent body of information about a topic."
"""

COHERENCE_SCORE_STEPS = """
1. Read the article carefully and identify the main topic and key points.
2. Read the summary and compare it to the article. Check if the summary covers the main topic and key points of the article,
and if it presents them in a clear and logical order.
3. Assign a score for coherence on a scale of 1 to 5, where 1 is the lowest and 5 is the highest based on the Evaluation Criteria.
"""

# Metric 3: Consistency

CONSISTENCY_SCORE_CRITERIA = """
Consistency(1-5) - the factual alignment between the summary and the summarized source. \
A factually consistent summary contains only statements that are entailed by the source document. \
Annotators were also asked to penalize summaries that contained hallucinated facts.
"""

CONSISTENCY_SCORE_STEPS = """
1. Read the article carefully and identify the main facts and details it presents.
2. Read the summary and compare it to the article. Check if the summary contains any factual errors that are not supported by the article.
3. Assign a score for consistency based on the Evaluation Criteria.
"""

# Metric 4: Fluency

FLUENCY_SCORE_CRITERIA = """
Fluency(1-3): the quality of the summary in terms of grammar, spelling, punctuation, word choice, and sentence structure.
1: Poor. The summary has many errors that make it hard to understand or sound unnatural.
2: Fair. The summary has some errors that affect the clarity or smoothness of the text, but the main points are still comprehensible.
3: Good. The summary has few or no errors and is easy to read and follow.
"""

FLUENCY_SCORE_STEPS = """
Read the summary and evaluate its fluency based on the given criteria. Assign a fluency score from 1 to 3.
"""


key: _node_content
string_value:
{"id_": "de7dfc0d-3e15-4725-87b6-6106e46ecd42", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b7f809bd-cb1a-4cbe-9ae3-f051066542ba", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "cca993148068c170e12e6ef7c89d1af0b4bc69c2e57cae4514119ec691c061df", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2b61792e-b10e-41a7-818d-d8baa65d00c7", "node_type": "1", "metadata": {}, "hash": "3ea6e35c127f1e8581148d8689708c8b5e08709865a2b0e091e648c8e0b35793", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20655, "end_char_idx": 22257, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
def get_geval_score(
    criteria: str, steps: str, document: str, summary: str, metric_name: str
):
    prompt = EVALUATION_PROMPT_TEMPLATE.format(
        criteria=criteria,
        steps=steps,
        metric_name=metric_name,
        document=document,
        summary=summary,
    )
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        max_tokens=5,
        top_p=1,
        frequency_penalty=0,
        presence_penalty=0,
    )
    return response.choices[0].message.content


evaluation_metrics = {
    "Relevance": (RELEVANCY_SCORE_CRITERIA, RELEVANCY_SCORE_STEPS),
    "Coherence": (COHERENCE_SCORE_CRITERIA, COHERENCE_SCORE_STEPS),
    "Consistency": (CONSISTENCY_SCORE_CRITERIA, CONSISTENCY_SCORE_STEPS),
    "Fluency": (FLUENCY_SCORE_CRITERIA, FLUENCY_SCORE_STEPS),
}

summaries = {"Summary 1": eval_summary_1, "Summary 2": eval_summary_2}

data = {"Evaluation Type": [], "Summary Type": [], "Score": []}

for eval_type, (criteria, steps) in evaluation_metrics.items():
    for summ_type, summary in summaries.items():
        data["Evaluation Type"].append(eval_type)
        data["Summary Type"].append(summ_type)
        result = get_geval_score(criteria, steps, excerpt, summary, eval_type)
        score_num = int(result.strip())
        data["Score"].append(score_num)

pivot_df = pd.DataFrame(data, index=None).pivot(
    index="Evaluation Type", columns="Summary Type", values="Score"
)
styled_pivot_df = pivot_df.style.apply(highlight_max, axis=1)
display(styled_pivot_df)
```


key: _node_content
string_value:
{"id_": "2b61792e-b10e-41a7-818d-d8baa65d00c7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "de7dfc0d-3e15-4725-87b6-6106e46ecd42", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7d01dbd435f8cf90e8365309a81d16b57f3165bcb272d3633c1739109db2e15d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d3e99123-8fa4-4d23-a1b1-3e744915a2f0", "node_type": "1", "metadata": {}, "hash": "88d39cc6f73eb420872c3104aa885fca7521be5c7395c9bcdd5bf0637f81b1e2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21243, "end_char_idx": 23910, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


