# embedding_metadata

key: chroma:document
string_value:
```python
print(response)
```

    
    The customer segments that grew the fastest for Uber in 2021 were its Mobility Drivers, Couriers, Riders, and Eaters. These segments experienced growth due to the continued stay-at-home order demand related to COVID-19, as well as Uber's introduction of its Uber One, Uber Pass, Eats Pass, and Rides Pass membership programs. Additionally, Uber's marketplace-centric advertising helped to connect merchants and brands with its platform network, further driving growth. Uber experienced the most growth in large metropolitan areas, such as Chicago, Miami, New York City, Sao Paulo, and London. Additionally, Uber experienced growth in suburban and rural areas, as well as in countries such as Argentina, Germany, Italy, Japan, South Korea, and Spain.
    
    The customer segments that grew the fastest for Lyft were ridesharing, light vehicles, and public transit. Ridesharing grew as Lyft was able to predict demand and proactively incentivize drivers to be available for rides in the right place at the right time. Light vehicles grew as users were looking for options that were more active, usually lower-priced, and often more efficient for short trips during heavy traffic. Public transit grew as Lyft integrated third-party public transit data into the Lyft App to offer users a robust view of transportation options around them. It is not possible to answer the question of which geographies grew the fastest for Lyft with the given context information.
    
    In summary, Uber and Lyft both experienced growth in customer segments related to mobility, couriers, riders, and eaters. Uber experienced the most growth in large metropolitan areas, as well as in suburban and rural areas, and in countries such as Argentina, Germany, Italy, Japan, South Korea, and Spain. Lyft experienced the most growth in ridesharing, light vehicles, and public transit. It is not possible to answer the question of which geographies grew the fastest for Lyft with the given context information.



```python
response = await s_engine.aquery('Compare revenue growth of Uber and Lyft from 2020 to 2021')
```

    Generated 2 sub questions.
    [36;1m[1;3m[uber_10k] Q: What is the revenue growth of Uber from 2020 to 2021
    [0m[36;1m[1;3m[uber_10k] A: 
    The revenue growth of Uber from 2020 to 2021 was 57%, or 54% on a constant currency basis.
    [0m[33;1m[1;3m[lyft_10k] Q: What is the revenue growth of Lyft from 2020 to 2021
    [0m[33;1m[1;3m[lyft_10k] A: 
    The revenue growth of Lyft from 2020 to 2021 is 36%, increasing from $2,364,681 thousand to $3,208,323 thousand.
    [0m


```python
print(response)
```

    
    The revenue growth of Uber from 2020 to 2021 was 57%, or 54% on a constant currency basis, while the revenue growth of Lyft from 2020 to 2021 was 36%. This means that Uber had a higher revenue growth than Lyft from 2020 to 2021.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb


key: _node_content
string_value:
{"id_": "07536e1d-a389-4d20-9d44-6f8d26b73a7a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "76c17e88-717b-48cf-ba55-6649d4a8aab2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "81fb6405dea1e39e1de71053e5afa8900a2acc089ae79680537cd73079c62598", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "148a261e-6cb4-46f0-a6a7-837bccd4cfb9", "node_type": "1", "metadata": {}, "hash": "363a84282b8220a59fa73aa021e726f76ba41f77e326997513e7208d09cc6883", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3031, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# OpenAI API Monitoring with W&B Weave

<img src="http://wandb.me/logo-im-png" width="400" alt="Weights & Biases" />

<!--- @wandbcode{weave_openai_client_qs} -->

<a target="_blank" href="https://colab.research.google.com/github/wandb/weave/blob/master/examples/prompts/llm_monitoring/openai_client_quickstart.ipynb">
  <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/>
</a>

**Note:** you will need an [OpenAI API key](https://platform.openai.com/account/api-keys) to run this colab.

Use the W&B OpenAI integration to monitor OpenAI API calls and understand how your projects and teams are leveraging LLMs.
In this example, we'll generate templated Weave Boards: LLM usage monitoring dashboards which you can explore and customize from the UI.

* automatically track LLM usage and aggregate useful metrics like  cost, latency and throughput across your projects/teams
* dynamically query and derive insights from the logs of all your OpenAI API calls
* iterate visually to slice, aggregate, and explore your data; customize panels to focus on interesting patterns; share progress more easily with your team through an interactive dashboard

<img src="https://raw.githubusercontent.com/wandb/weave/master/docs/assets/full_board_view.png">

[Play with a live version of this Weave Board →](http://wandb.me/llm-monitoring-board)

#### New to Weights & Biases? [-> Sign up for an account here <-](https://wandb.ai/site)

# Step 0: Setup

Install dependencies, login to W&B so you can save and share your work, and authenticate with OpenAI.


```python
# if not already installed
!pip install -qqq weave openai tiktoken wandb
```


```python
import wandb
wandb.login()
```


```python
import weave
import os
WANDB_BASE_URL = "https://api.wandb.ai"
os.environ["WANDB_BASE_URL"] = WANDB_BASE_URL
```


```python
# authenticate with OpenAI
from getpass import getpass

if os.getenv("OPENAI_API_KEY") is None:
  os.environ["OPENAI_API_KEY"] = getpass("Paste your OpenAI key from: https://platform.openai.com/account/api-keys\n")
assert os.getenv("OPENAI_API_KEY", "").startswith("sk-"), "This doesn't look like a valid OpenAI API key"
print("OpenAI API key configured")
```

# Step 1: Configure data streaming and storage in W&B

Set WB_ENTITY to your wandb username or team name. Log in to W&B and navigate to Home Page at [wandb.ai/home](https://wandb.ai/home) to see valid options under your "Profile" and "Teams" in the left sidebar.


```python
WB_ENTITY = "" # set to your wandb username or team name
WB_PROJECT = "weave" # top-level directory for this work
STREAM_NAME = "openai_logs" # record table which stores the logs of OpenAI API calls as they stream in
```

# Step 2: Call init_monitor()

To start monitoring OpenAI API usage, call `init_monitor(<stream>)`, where `<stream>` has the form `<wandb_team_or_user>/<wandb_project>/<stream_name>`. The stream records and stores all the OpenAI API calls.

Running this cell will print out a link to view the current project in the Weave UI.


key: _node_content
string_value:
{"id_": "148a261e-6cb4-46f0-a6a7-837bccd4cfb9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "76c17e88-717b-48cf-ba55-6649d4a8aab2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "81fb6405dea1e39e1de71053e5afa8900a2acc089ae79680537cd73079c62598", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "07536e1d-a389-4d20-9d44-6f8d26b73a7a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "e400e76af28ddd58b63efd214cc2a9578c0592268c9a1578a82675b6c47bd75e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aa5bc539-e418-4813-af5c-f1140bc5cff1", "node_type": "1", "metadata": {}, "hash": "396f67fe8962b03eee56e69bfd8b79cfbafc12da60f5d3b5f9c70ac405c03544", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1581, "end_char_idx": 4562, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# if not already installed
!pip install -qqq weave openai tiktoken wandb
```


```python
import wandb
wandb.login()
```


```python
import weave
import os
WANDB_BASE_URL = "https://api.wandb.ai"
os.environ["WANDB_BASE_URL"] = WANDB_BASE_URL
```


```python
# authenticate with OpenAI
from getpass import getpass

if os.getenv("OPENAI_API_KEY") is None:
  os.environ["OPENAI_API_KEY"] = getpass("Paste your OpenAI key from: https://platform.openai.com/account/api-keys\n")
assert os.getenv("OPENAI_API_KEY", "").startswith("sk-"), "This doesn't look like a valid OpenAI API key"
print("OpenAI API key configured")
```

# Step 1: Configure data streaming and storage in W&B

Set WB_ENTITY to your wandb username or team name. Log in to W&B and navigate to Home Page at [wandb.ai/home](https://wandb.ai/home) to see valid options under your "Profile" and "Teams" in the left sidebar.


```python
WB_ENTITY = "" # set to your wandb username or team name
WB_PROJECT = "weave" # top-level directory for this work
STREAM_NAME = "openai_logs" # record table which stores the logs of OpenAI API calls as they stream in
```

# Step 2: Call init_monitor()

To start monitoring OpenAI API usage, call `init_monitor(<stream>)`, where `<stream>` has the form `<wandb_team_or_user>/<wandb_project>/<stream_name>`. The stream records and stores all the OpenAI API calls.

Running this cell will print out a link to view the current project in the Weave UI.


```python
from weave.monitoring import openai, init_monitor
m = init_monitor(f"{WB_ENTITY}/{WB_PROJECT}/{STREAM_NAME}")

# specifying a single model for simplicity
OPENAI_MODEL = 'gpt-3.5-turbo'

# prefill with some sample logs
r = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[{"role": "user", "content": "hello world!"}])
r = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[{"role": "user", "content": "what is 2+2?"}])
```

# Step 3: Preview monitoring dashboard

Click on the link above to preview the data stream, then click "OpenAI Monitor Board" in the right sidebar to create a Weave Board for this data stream.

<img src="https://raw.githubusercontent.com/wandb/weave/master/docs/assets/short_board_attempt.gif" width=75%>

# Step 4: Explore & understand your LLM usage

To save your work, rename the board by clicking on the autogenerated name at the top of the page. To share your board, click \"Publish\" in the top right.

<img src="https://raw.githubusercontent.com/wandb/weave/master/docs/assets/publish_board_short.gif" width=75%>

To visualize your work in real-time as you iterate, you can:
* keep the Board open in a separate tab and refresh to view the latest data
* rename the Board for easier reference at any point and \"Publish\" that version to share a link with others
* find previously saved Boards by navigating to the relevant W&B entity and W&B project name from weave.wandb.ai
* or open a new instance of a Board template to start fresh with all the data accumulated so far


key: _node_content
string_value:
{"id_": "aa5bc539-e418-4813-af5c-f1140bc5cff1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "76c17e88-717b-48cf-ba55-6649d4a8aab2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "81fb6405dea1e39e1de71053e5afa8900a2acc089ae79680537cd73079c62598", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "148a261e-6cb4-46f0-a6a7-837bccd4cfb9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1bd4858a35746f66aff2f418828e38692cc442dffaac25318ee8e5173ee52e48", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2a470129-48ad-44cd-ab26-2429ee41783c", "node_type": "1", "metadata": {}, "hash": "12347373e3f59afb296ebc2c75e1ba11ad2c3e89b670fd2ef0e245d794e5230c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3034, "end_char_idx": 6586, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from weave.monitoring import openai, init_monitor
m = init_monitor(f"{WB_ENTITY}/{WB_PROJECT}/{STREAM_NAME}")

# specifying a single model for simplicity
OPENAI_MODEL = 'gpt-3.5-turbo'

# prefill with some sample logs
r = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[{"role": "user", "content": "hello world!"}])
r = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[{"role": "user", "content": "what is 2+2?"}])
```

# Step 3: Preview monitoring dashboard

Click on the link above to preview the data stream, then click "OpenAI Monitor Board" in the right sidebar to create a Weave Board for this data stream.

<img src="https://raw.githubusercontent.com/wandb/weave/master/docs/assets/short_board_attempt.gif" width=75%>

# Step 4: Explore & understand your LLM usage

To save your work, rename the board by clicking on the autogenerated name at the top of the page. To share your board, click \"Publish\" in the top right.

<img src="https://raw.githubusercontent.com/wandb/weave/master/docs/assets/publish_board_short.gif" width=75%>

To visualize your work in real-time as you iterate, you can:
* keep the Board open in a separate tab and refresh to view the latest data
* rename the Board for easier reference at any point and \"Publish\" that version to share a link with others
* find previously saved Boards by navigating to the relevant W&B entity and W&B project name from weave.wandb.ai
* or open a new instance of a Board template to start fresh with all the data accumulated so far


Next we'll illustrate a few ways you could track OpenAI API calls. There are many more possibilities depending on your use case, and we can't wait to see what you create from these starter templates.

# Examples

## Example 0: Log a prompt and its completion

Monitor a ChatCompletion request and print the corresponding response, extracting only the text of the completion.


```python
response = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[
        {"role": "user", "content": f"What is the meaning of life, the universe, and everything?"},
    ])
print(response['choices'][0]['message']['content'])
```

## Example 1: Track relevant parameters as attributes

Factor out parameters of interest and track them as attributes on the logged record.
Here we track the "system prompt" separately from the "prompt template" and the "equation" parameter. This time we'll print the full structured response from the ChatCompletion call.


```python
system_prompt = "you always write in bullet points"
prompt_template = 'solve the following equation step by step: {equation}'
params = {'equation': '4 * (3 - 1)'}
openai.ChatCompletion.create(model=OPENAI_MODEL,
                             messages=[
                                    {"role": "system", "content": system_prompt},
                                    {"role": "user", "content": prompt_template.format(**params)},
                                ],
                             # you can add additional attributes to the logged record
                             # see the monitor_api notebook for more examples
                             monitor_attributes={
                                 'system_prompt': system_prompt,
                                 'prompt_template': prompt_template,
                                 'params': params
                             })
```

## Example 2: Log an ongoing stream of messages

Monitor a stream of messages and log the result as a single record. Note: tokens are not counted in this format.


key: _node_content
string_value:
{"id_": "2a470129-48ad-44cd-ab26-2429ee41783c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "76c17e88-717b-48cf-ba55-6649d4a8aab2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "81fb6405dea1e39e1de71053e5afa8900a2acc089ae79680537cd73079c62598", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aa5bc539-e418-4813-af5c-f1140bc5cff1", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "98df897b98cad5fcc09ae7ff45d9331b3424820ee8a16e75991fa0ebc0592d15", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9f56faf8-7f29-44af-b7e6-4cb56893059d", "node_type": "1", "metadata": {}, "hash": "34dfec5596283801792b850a60936b8ebce3d278c76c75954c0abbb19eeefdcf", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4565, "end_char_idx": 8061, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Next we'll illustrate a few ways you could track OpenAI API calls. There are many more possibilities depending on your use case, and we can't wait to see what you create from these starter templates.

# Examples

## Example 0: Log a prompt and its completion

Monitor a ChatCompletion request and print the corresponding response, extracting only the text of the completion.


```python
response = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[
        {"role": "user", "content": f"What is the meaning of life, the universe, and everything?"},
    ])
print(response['choices'][0]['message']['content'])
```

## Example 1: Track relevant parameters as attributes

Factor out parameters of interest and track them as attributes on the logged record.
Here we track the "system prompt" separately from the "prompt template" and the "equation" parameter. This time we'll print the full structured response from the ChatCompletion call.


```python
system_prompt = "you always write in bullet points"
prompt_template = 'solve the following equation step by step: {equation}'
params = {'equation': '4 * (3 - 1)'}
openai.ChatCompletion.create(model=OPENAI_MODEL,
                             messages=[
                                    {"role": "system", "content": system_prompt},
                                    {"role": "user", "content": prompt_template.format(**params)},
                                ],
                             # you can add additional attributes to the logged record
                             # see the monitor_api notebook for more examples
                             monitor_attributes={
                                 'system_prompt': system_prompt,
                                 'prompt_template': prompt_template,
                                 'params': params
                             })
```

## Example 2: Log an ongoing stream of messages

Monitor a stream of messages and log the result as a single record. Note: tokens are not counted in this format.


```python
from weave.monitoring.openai import message_from_stream
r = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[
        {"role": "system", "content": "You are a robot and only speak in robot, like beep bloop bop."},
        {"role": "user", "content": "Tell me a 50-word story."},
    ], stream=True)
for s in message_from_stream(r):
    print(s, end='')
```

## Example 3: Structure prompt engineering experiments

Here we compare a few toy options for the system prompt, user question, and intended audience. Try your own experiments and see if any interesting insights emerge as you explore in the Board and group by different parameters.


```python
def explain_math(system_prompt, prompt_template, params):
    openai.ChatCompletion.create(model=OPENAI_MODEL,
                             messages=[
                                    {"role": "system", "content": system_prompt},
                                    {"role": "user", "content": prompt_template.format(**params)},
                                ],
                             # you can add additional attributes to the logged record
                             # see the monitor_api notebook for more examples
                             monitor_attributes={
                                 'system_prompt': system_prompt,
                                 'prompt_template': prompt_template,
                                 'params': params
                             })
```


key: _node_content
string_value:
{"id_": "9f56faf8-7f29-44af-b7e6-4cb56893059d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "76c17e88-717b-48cf-ba55-6649d4a8aab2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "81fb6405dea1e39e1de71053e5afa8900a2acc089ae79680537cd73079c62598", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2a470129-48ad-44cd-ab26-2429ee41783c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/third_party/Openai_monitoring_with_wandb_weave.ipynb", "title": "OpenAI API Monitoring with W&B Weave", "tokens": 1924, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "13686e93f8ba875da4387aca3d715df0cb6b6d4ecf4faefd96b0f1163898213d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6589, "end_char_idx": 8659, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from weave.monitoring.openai import message_from_stream
r = openai.ChatCompletion.create(model=OPENAI_MODEL, messages=[
        {"role": "system", "content": "You are a robot and only speak in robot, like beep bloop bop."},
        {"role": "user", "content": "Tell me a 50-word story."},
    ], stream=True)
for s in message_from_stream(r):
    print(s, end='')
```

## Example 3: Structure prompt engineering experiments

Here we compare a few toy options for the system prompt, user question, and intended audience. Try your own experiments and see if any interesting insights emerge as you explore in the Board and group by different parameters.


```python
def explain_math(system_prompt, prompt_template, params):
    openai.ChatCompletion.create(model=OPENAI_MODEL,
                             messages=[
                                    {"role": "system", "content": system_prompt},
                                    {"role": "user", "content": prompt_template.format(**params)},
                                ],
                             # you can add additional attributes to the logged record
                             # see the monitor_api notebook for more examples
                             monitor_attributes={
                                 'system_prompt': system_prompt,
                                 'prompt_template': prompt_template,
                                 'params': params
                             })
```


```python
# feel free to substitute your own prompts :)
system_prompts = ["you're extremely flowery and poetic", "you're very direct and precise", "balance brevity with insight"]
prompt_template = 'explain the solution of the following to a {audience}: {equation}'
equations = ['x^2 + 4x + 9 = 0', '15 * (2 - 6) / 4']
audience = ["new student", "math genius"]

for system_prompt in system_prompts:
    for equation in equations:
        for person in audience:
            params = {"equation" : equation, "audience" : person}
            explain_math(system_prompt, prompt_template, params)
```


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb


key: _node_content
string_value:
{"id_": "b6747a93-8c84-4f01-9bc7-3d6f78e8254c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "07c27b30-7f6c-4ed1-9a60-f8d90a97cd7e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0ab1df67e41749ec6c62eaf013a9d47d0f1ecdd07fa7855873f263a9bd12438b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "49f7ec3d-7b2d-4a28-bf83-f59618d8f5b1", "node_type": "1", "metadata": {}, "hash": "2d781cee85ceb8a18b7dbc93b9f0c9e753a9a6314c0b5f46cc41569d0c33ea40", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2737, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to use the DALL·E API

This notebook shows how to use OpenAI's DALL·E image API endpoints.

There are three API endpoints:
- **Generations:** generates an image or images based on an input caption
- **Edits:** edits or extends an existing image
- **Variations:** generates variations of an input image

## Setup

- Import the packages you'll need
- Import your OpenAI API key: You can do this by running \``export OPENAI_API_KEY="your API key"`\` in your terminal.
- Set a directory to save images to


```python
# imports
from openai import OpenAI  # OpenAI Python library to make API calls
import requests  # used to download images
import os  # used to access filepaths
from PIL import Image  # used to print and edit images

# initialize OpenAI client
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))

```


```python
# set a directory to save DALL·E images to
image_dir_name = "images"
image_dir = os.path.join(os.curdir, image_dir_name)

# create the directory if it doesn't yet exist
if not os.path.isdir(image_dir):
    os.mkdir(image_dir)

# print the directory to save to
print(f"{image_dir=}")

```

    image_dir='./images'


## Generations

The generation API endpoint creates an image based on a text prompt. [API Reference](https://platform.openai.com/docs/api-reference/images/create)

**Required inputs:**
- `prompt` (str): A text description of the desired image(s). The maximum length is 1000 characters for dall-e-2 and 4000 characters for dall-e-3.

**Optional inputs:**
- `model` (str): The model to use for image generation. Defaults to dall-e-2
- `n` (int): The number of images to generate. Must be between 1 and 10. Defaults to 1.
- `quality` (str): The quality of the image that will be generated. hd creates images with finer details and greater consistency across the image. This param is only supported for dall-e-3.
- `response_format` (str): The format in which the generated images are returned. Must be one of "url" or "b64_json". Defaults to "url".
- `size` (str): The size of the generated images. Must be one of 256x256, 512x512, or 1024x1024 for dall-e-2. Must be one of 1024x1024, 1792x1024, or 1024x1792 for dall-e-3 models. Defaults to "1024x1024".
- `style`(str | null): The style of the generated images. Must be one of vivid or natural. Vivid causes the model to lean towards generating hyper-real and dramatic images. Natural causes the model to produce more natural, less hyper-real looking images. This param is only supported for dall-e-3.
- `user` (str): A unique identifier representing your end-user, which will help OpenAI to monitor and detect abuse. [Learn more.](https://beta.openai.com/docs/usage-policies/end-user-ids)


key: _node_content
string_value:
{"id_": "49f7ec3d-7b2d-4a28-bf83-f59618d8f5b1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "07c27b30-7f6c-4ed1-9a60-f8d90a97cd7e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0ab1df67e41749ec6c62eaf013a9d47d0f1ecdd07fa7855873f263a9bd12438b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b6747a93-8c84-4f01-9bc7-3d6f78e8254c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c1d2f0a793e18b44a725df682a674e6aafdd589f8085d30ff8e27904aa3b3177", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "47624edc-33b3-47ff-afd9-1b8fb85ec943", "node_type": "1", "metadata": {}, "hash": "cae7f3a0656f5991322954cdf529746689f54e953920540a1bdb209958abb39d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2740, "end_char_idx": 4169, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# create an image

# set the prompt
prompt = "A cyberpunk monkey hacker dreaming of a beautiful bunch of bananas, digital art"

# call the OpenAI API
generation_response = client.images.generate(
    model = "dall-e-3",
    prompt=prompt,
    n=1,
    size="1024x1024",
    response_format="url",
)

# print response
print(generation_response)
```

    ImagesResponse(created=1701994117, data=[Image(b64_json=None, revised_prompt=None, url='https://oaidalleapiprodscus.blob.core.windows.net/private/org-9HXYFy8ux4r6aboFyec2OLRf/user-8OA8IvMYkfdAcUZXgzAXHS7d/img-ced13hkOk3lXkccQgW1fAQjm.png?st=2023-12-07T23%3A08%3A37Z&se=2023-12-08T01%3A08%3A37Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2023-12-07T16%3A41%3A48Z&ske=2023-12-08T16%3A41%3A48Z&sks=b&skv=2021-08-06&sig=tcD0iyU0ABOvWAKsY89gp5hLVIYkoSXQnrcmH%2Brkric%3D')])



```python
# save the image
generated_image_name = "generated_image.png"  # any name you like; the filetype should be .png
generated_image_filepath = os.path.join(image_dir, generated_image_name)
generated_image_url = generation_response.data[0].url  # extract image URL from response
generated_image = requests.get(generated_image_url).content  # download the image

with open(generated_image_filepath, "wb") as image_file:
    image_file.write(generated_image)  # write the image to the file
```


key: _node_content
string_value:
{"id_": "47624edc-33b3-47ff-afd9-1b8fb85ec943", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "07c27b30-7f6c-4ed1-9a60-f8d90a97cd7e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0ab1df67e41749ec6c62eaf013a9d47d0f1ecdd07fa7855873f263a9bd12438b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "49f7ec3d-7b2d-4a28-bf83-f59618d8f5b1", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ca9863e3282d4ce23b600be97a33d3bae0994bc757d120ace5dda9a37610cb1f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "01ce5576-e318-4e10-acbc-d26bfa76f0cb", "node_type": "1", "metadata": {}, "hash": "ea5c5938ad4a3ef80b7ee11a01846346a4617862da6dd403ad94bc1025cc1ca3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3673, "end_char_idx": 5363, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# save the image
generated_image_name = "generated_image.png"  # any name you like; the filetype should be .png
generated_image_filepath = os.path.join(image_dir, generated_image_name)
generated_image_url = generation_response.data[0].url  # extract image URL from response
generated_image = requests.get(generated_image_url).content  # download the image

with open(generated_image_filepath, "wb") as image_file:
    image_file.write(generated_image)  # write the image to the file
```


```python
# print the image
print(generated_image_filepath)
display(Image.open(generated_image_filepath))

```

## Variations

The variations endpoint generates new images (variations) similar to an input image. [API Reference](https://platform.openai.com/docs/api-reference/images/createVariation)

Here we'll generate variations of the image generated above.

**Required inputs:**
- `image` (str): The image to use as the basis for the variation(s). Must be a valid PNG file, less than 4MB, and square.

**Optional inputs:**
- `model` (str): The model to use for image variations. Only dall-e-2 is supported at this time.
- `n` (int): The number of images to generate. Must be between 1 and 10. Defaults to 1.
- `size` (str): The size of the generated images. Must be one of "256x256", "512x512", or "1024x1024". Smaller images are faster. Defaults to "1024x1024".
- `response_format` (str): The format in which the generated images are returned. Must be one of "url" or "b64_json". Defaults to "url".
- `user` (str): A unique identifier representing your end-user, which will help OpenAI to monitor and detect abuse. [Learn more.](https://beta.openai.com/docs/usage-policies/end-user-ids)


key: _node_content
string_value:
{"id_": "01ce5576-e318-4e10-acbc-d26bfa76f0cb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "07c27b30-7f6c-4ed1-9a60-f8d90a97cd7e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0ab1df67e41749ec6c62eaf013a9d47d0f1ecdd07fa7855873f263a9bd12438b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "47624edc-33b3-47ff-afd9-1b8fb85ec943", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "61ddeebe84698449c483f81b29c6a233acb9a00b3dc64cab2474640d833bca38", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cacc9243-9838-4146-a0fd-a88fe4494c8e", "node_type": "1", "metadata": {}, "hash": "becccdac305bcb78ceefb8359364540a75d0fd484b5c57e223ebb1d766a85edc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4172, "end_char_idx": 6795, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# print the image
print(generated_image_filepath)
display(Image.open(generated_image_filepath))

```

## Variations

The variations endpoint generates new images (variations) similar to an input image. [API Reference](https://platform.openai.com/docs/api-reference/images/createVariation)

Here we'll generate variations of the image generated above.

**Required inputs:**
- `image` (str): The image to use as the basis for the variation(s). Must be a valid PNG file, less than 4MB, and square.

**Optional inputs:**
- `model` (str): The model to use for image variations. Only dall-e-2 is supported at this time.
- `n` (int): The number of images to generate. Must be between 1 and 10. Defaults to 1.
- `size` (str): The size of the generated images. Must be one of "256x256", "512x512", or "1024x1024". Smaller images are faster. Defaults to "1024x1024".
- `response_format` (str): The format in which the generated images are returned. Must be one of "url" or "b64_json". Defaults to "url".
- `user` (str): A unique identifier representing your end-user, which will help OpenAI to monitor and detect abuse. [Learn more.](https://beta.openai.com/docs/usage-policies/end-user-ids)



```python
# create variations

# call the OpenAI API, using `create_variation` rather than `create`
variation_response = client.images.create_variation(
    image=generated_image,  # generated_image is the image generated above
    n=2,
    size="1024x1024",
    response_format="url",
)

# print response
print(variation_response)
```

    ImagesResponse(created=1701994139, data=[Image(b64_json=None, revised_prompt=None, url='https://oaidalleapiprodscus.blob.core.windows.net/private/org-9HXYFy8ux4r6aboFyec2OLRf/user-8OA8IvMYkfdAcUZXgzAXHS7d/img-noNRGgwaaotRGIe6Y2GVeSpr.png?st=2023-12-07T23%3A08%3A59Z&se=2023-12-08T01%3A08%3A59Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2023-12-07T16%3A39%3A11Z&ske=2023-12-08T16%3A39%3A11Z&sks=b&skv=2021-08-06&sig=ER5RUglhtIk9LWJXw1DsolorT4bnEmFostfnUjY21ns%3D'), Image(b64_json=None, revised_prompt=None, url='https://oaidalleapiprodscus.blob.core.windows.net/private/org-9HXYFy8ux4r6aboFyec2OLRf/user-8OA8IvMYkfdAcUZXgzAXHS7d/img-oz952tL11FFhf9iXXJVIRUZX.png?st=2023-12-07T23%3A08%3A59Z&se=2023-12-08T01%3A08%3A59Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2023-12-07T16%3A39%3A11Z&ske=2023-12-08T16%3A39%3A11Z&sks=b&skv=2021-08-06&sig=99rJOQwDKsfIeerlMXMHholhAhrHfYaQRFJBF8FKv74%3D')])


key: _node_content
string_value:
{"id_": "cacc9243-9838-4146-a0fd-a88fe4494c8e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "07c27b30-7f6c-4ed1-9a60-f8d90a97cd7e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0ab1df67e41749ec6c62eaf013a9d47d0f1ecdd07fa7855873f263a9bd12438b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "01ce5576-e318-4e10-acbc-d26bfa76f0cb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b7c14f199b7dfd341139a1866db80829d2e7c7d1c9713c416fe03c40ecd2211e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fd905f2e-7b28-465a-9d4f-3162c5ada312", "node_type": "1", "metadata": {}, "hash": "d146b79fe58b50bc0cf29dbad89ef05d81adc82a9dee84541e043cf912021a08", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6799, "end_char_idx": 9501, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# save the images
variation_urls = [datum.url for datum in variation_response.data]  # extract URLs
variation_images = [requests.get(url).content for url in variation_urls]  # download images
variation_image_names = [f"variation_image_{i}.png" for i in range(len(variation_images))]  # create names
variation_image_filepaths = [os.path.join(image_dir, name) for name in variation_image_names]  # create filepaths
for image, filepath in zip(variation_images, variation_image_filepaths):  # loop through the variations
    with open(filepath, "wb") as image_file:  # open the file
        image_file.write(image)  # write the image to the file
```


```python
# print the original image
print(generated_image_filepath)
display(Image.open(generated_image_filepath))

# print the new variations
for variation_image_filepaths in variation_image_filepaths:
    print(variation_image_filepaths)
    display(Image.open(variation_image_filepaths))

```

## Edits

The edit endpoint uses DALL·E to generate a specified portion of an existing image. Three inputs are needed: the image to edit, a mask specifying the portion to be regenerated, and a prompt describing the desired image. [API Reference](https://platform.openai.com/docs/api-reference/images/createEdit)

**Required inputs:** 
- `image` (str): The image to edit. Must be a valid PNG file, less than 4MB, and square. If mask is not provided, image must have transparency, which will be used as the mask.
- `prompt` (str): A text description of the desired image(s). The maximum length is 1000 characters.

**Optional inputs:**
- `mask` (file): An additional image whose fully transparent areas (e.g. where alpha is zero) indicate where image should be edited. Must be a valid PNG file, less than 4MB, and have the same dimensions as image.
- `model` (str): The model to use for edit image. Only dall-e-2 is supported at this time.
- `n` (int): The number of images to generate. Must be between 1 and 10. Defaults to 1.
- `size` (str): The size of the generated images. Must be one of "256x256", "512x512", or "1024x1024". Smaller images are faster. Defaults to "1024x1024".
- `response_format` (str): The format in which the generated images are returned. Must be one of "url" or "b64_json". Defaults to "url".
- `user` (str): A unique identifier representing your end-user, which will help OpenAI to monitor and detect abuse. [Learn more.](https://beta.openai.com/docs/usage-policies/end-user-ids)


### Set Edit Area

An edit requires a "mask" to specify which portion of the image to regenerate. Any pixel with an alpha of 0 (transparent) will be regenerated. The code below creates a 1024x1024 mask where the bottom half is transparent.


key: _node_content
string_value:
{"id_": "fd905f2e-7b28-465a-9d4f-3162c5ada312", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "07c27b30-7f6c-4ed1-9a60-f8d90a97cd7e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0ab1df67e41749ec6c62eaf013a9d47d0f1ecdd07fa7855873f263a9bd12438b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cacc9243-9838-4146-a0fd-a88fe4494c8e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "039f85b4ba8c1c5af27ed24680cbc2a932b47a226ca57b48a99ec8a540ae6373", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7c7d95de-7981-4c80-b960-6bd6ad12dcdb", "node_type": "1", "metadata": {}, "hash": "a9f90653b2f6b2afeecea1e3baea97bdebc00fefe65a847bf3f4757133d59617", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9262, "end_char_idx": 11576, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
### Set Edit Area

An edit requires a "mask" to specify which portion of the image to regenerate. Any pixel with an alpha of 0 (transparent) will be regenerated. The code below creates a 1024x1024 mask where the bottom half is transparent.


```python
# create a mask
width = 1024
height = 1024
mask = Image.new("RGBA", (width, height), (0, 0, 0, 1))  # create an opaque image mask

# set the bottom half to be transparent
for x in range(width):
    for y in range(height // 2, height):  # only loop over the bottom half of the mask
        # set alpha (A) to zero to turn pixel transparent
        alpha = 0
        mask.putpixel((x, y), (0, 0, 0, alpha))

# save the mask
mask_name = "bottom_half_mask.png"
mask_filepath = os.path.join(image_dir, mask_name)
mask.save(mask_filepath)
```

### Perform Edit

Now we supply our image, caption and mask to the API to get 5 examples of edits to our image


```python
# edit an image

# call the OpenAI API
edit_response = client.images.edit(
    image=open(generated_image_filepath, "rb"),  # from the generation section
    mask=open(mask_filepath, "rb"),  # from right above
    prompt=prompt,  # from the generation section
    n=1,
    size="1024x1024",
    response_format="url",
)

# print response
print(edit_response)
```

    ImagesResponse(created=1701994167, data=[Image(b64_json=None, revised_prompt=None, url='https://oaidalleapiprodscus.blob.core.windows.net/private/org-9HXYFy8ux4r6aboFyec2OLRf/user-8OA8IvMYkfdAcUZXgzAXHS7d/img-9UOVGC7wB8MS2Q7Rwgj0fFBq.png?st=2023-12-07T23%3A09%3A27Z&se=2023-12-08T01%3A09%3A27Z&sp=r&sv=2021-08-06&sr=b&rscd=inline&rsct=image/png&skoid=6aaadede-4fb3-4698-a8f6-684d7786b067&sktid=a48cca56-e6da-484e-a814-9c849652bcb3&skt=2023-12-07T16%3A40%3A37Z&ske=2023-12-08T16%3A40%3A37Z&sks=b&skv=2021-08-06&sig=MsRMZ1rN434bVdWr%2B9kIoqu9CIrvZypZBfkQPTOhCl4%3D')])



```python
# save the image
edited_image_name = "edited_image.png"  # any name you like; the filetype should be .png
edited_image_filepath = os.path.join(image_dir, edited_image_name)
edited_image_url = edit_response.data[0].url  # extract image URL from response
edited_image = requests.get(edited_image_url).content  # download the image

with open(edited_image_filepath, "wb") as image_file:
    image_file.write(edited_image)  # write the image to the file
```


key: _node_content
string_value:
{"id_": "7c7d95de-7981-4c80-b960-6bd6ad12dcdb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "07c27b30-7f6c-4ed1-9a60-f8d90a97cd7e", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0ab1df67e41749ec6c62eaf013a9d47d0f1ecdd07fa7855873f263a9bd12438b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fd905f2e-7b28-465a-9d4f-3162c5ada312", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/Image_generations_edits_and_variations_with_DALL-E.ipynb", "title": "How to use the DALL\u00b7E API", "tokens": 3601, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "77807684d9069d61decc78a9f4611e6d5830bbcba09168dcae74cff68a3d76cc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11113, "end_char_idx": 11791, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# save the image
edited_image_name = "edited_image.png"  # any name you like; the filetype should be .png
edited_image_filepath = os.path.join(image_dir, edited_image_name)
edited_image_url = edit_response.data[0].url  # extract image URL from response
edited_image = requests.get(edited_image_url).content  # download the image

with open(edited_image_filepath, "wb") as image_file:
    image_file.write(edited_image)  # write the image to the file
```


```python
# print the original image
print(generated_image_filepath)
display(Image.open(generated_image_filepath))

# print edited image
print(edited_image_filepath)
display(Image.open(edited_image_filepath))
```


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb


key: _node_content
string_value:
{"id_": "fa9b435b-3921-41d3-af1c-2cecd68a9f44", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ee647f98-6796-4ace-9fa1-781cc8ca2627", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4113902b964755cb6485391a9fa1d36013bdb8e97ece3a53909b6fa11bb4c4dd", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0ac2c008-d7a3-4715-8b61-d4a673214aac", "node_type": "1", "metadata": {}, "hash": "3f872709cd0ad71669adf9a32846c3719ad168a0d9fe6b7f1d34ff216c84207e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2755, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to create dynamic masks with DALL·E and Segment Anything

Segment Anything is a model from Meta that can be used to select portions of images. Combined with DALL·E's ability to inpaint specified portions of images, you can use Segment Anything to easily select any part of an image you'd like to alter.

In this notebook, we'll use these tools to become fashion designers and dynamically replace our digital models' outfits with tailored, original creations. The notebook follows this flow:
- **Setup:** Initialise your libraries and any location directories.
- **Generate original image:** Make an original image that we'll create dynamic masks from.
- **Generate mask:** Use Segment Anything to create a dynamic mask.
- **Create new image:** Generate a new image with the masked area inpainted with a fresh prompt.

## Setup

To get started we'll need to follow the [instructions](https://github.com/facebookresearch/segment-anything) for using the Segment Anything (SAM) model open-sourced by Meta. As of May 2023, the key steps are:
- Install [Pytorch](https://pytorch.org/get-started/locally/) (version 1.7+).
- Install the library using ```pip install git+https://github.com/facebookresearch/segment-anything.git```.
- Install dependencies using ```pip install opencv-python pycocotools matplotlib onnxruntime onnx```.
- Download a [model checkpoint](https://github.com/facebookresearch/segment-anything#model-checkpoints) to use (default size is 2.4 GB).



```python
!pip install torch torchvision torchaudio
!pip install git+https://github.com/facebookresearch/segment-anything.git
!pip install opencv-python pycocotools matplotlib onnxruntime onnx
!pip install requests
!pip install openai
!pip install numpy
```


```python
!wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth
```


```python
import cv2
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
from matplotlib import rcParams
import numpy as np
from openai import OpenAI
import os
from PIL import Image
import requests
from segment_anything import sam_model_registry, SamAutomaticMaskGenerator, SamPredictor
import torch

# Set directories for generation images and edit images
base_image_dir = os.path.join("images", "01_generations")
mask_dir = os.path.join("images", "02_masks")
edit_image_dir = os.path.join("images", "03_edits")

# Point to your downloaded SAM model
sam_model_filepath = "./sam_vit_h_4b8939.pth"

# Initiate SAM model
sam = sam_model_registry["default"](checkpoint=sam_model_filepath)

# Initiate openAI client
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))

```

## Generate original image

First we'll create an original image which we'll generate masks from.


key: _node_content
string_value:
{"id_": "0ac2c008-d7a3-4715-8b61-d4a673214aac", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ee647f98-6796-4ace-9fa1-781cc8ca2627", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4113902b964755cb6485391a9fa1d36013bdb8e97ece3a53909b6fa11bb4c4dd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fa9b435b-3921-41d3-af1c-2cecd68a9f44", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4ddc53453b171b4f334074341e934edcf7591744a638c0009f1759ad6db9ba3f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f1d59f05-c25d-4b75-8ab8-0cbd9b14c51b", "node_type": "1", "metadata": {}, "hash": "150cbdb4185cb59868a7c988f47a1ba89266553763a874c355bb162ad93ba927", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1471, "end_char_idx": 4084, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
!pip install torch torchvision torchaudio
!pip install git+https://github.com/facebookresearch/segment-anything.git
!pip install opencv-python pycocotools matplotlib onnxruntime onnx
!pip install requests
!pip install openai
!pip install numpy
```


```python
!wget https://dl.fbaipublicfiles.com/segment_anything/sam_vit_h_4b8939.pth
```


```python
import cv2
import matplotlib.pyplot as plt
import matplotlib.image as mpimg
from matplotlib import rcParams
import numpy as np
from openai import OpenAI
import os
from PIL import Image
import requests
from segment_anything import sam_model_registry, SamAutomaticMaskGenerator, SamPredictor
import torch

# Set directories for generation images and edit images
base_image_dir = os.path.join("images", "01_generations")
mask_dir = os.path.join("images", "02_masks")
edit_image_dir = os.path.join("images", "03_edits")

# Point to your downloaded SAM model
sam_model_filepath = "./sam_vit_h_4b8939.pth"

# Initiate SAM model
sam = sam_model_registry["default"](checkpoint=sam_model_filepath)

# Initiate openAI client
client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))

```

## Generate original image

First we'll create an original image which we'll generate masks from.


```python
def process_dalle_images(response, filename, image_dir):
    # save the images
    urls = [datum.url for datum in response.data]  # extract URLs
    images = [requests.get(url).content for url in urls]  # download images
    image_names = [f"{filename}_{i + 1}.png" for i in range(len(images))]  # create names
    filepaths = [os.path.join(image_dir, name) for name in image_names]  # create filepaths
    for image, filepath in zip(images, filepaths):  # loop through the variations
        with open(filepath, "wb") as image_file:  # open the file
            image_file.write(image)  # write the image to the file

    return filepaths
```


```python
dalle_prompt = '''
Full length, zoomed out photo of our premium Lederhosen-inspired jumpsuit.
Showcase the intricate hand-stitched details and high-quality leather, while highlighting the perfect blend of Austrian heritage and modern fashion.
This piece appeals to a sophisticated, trendsetting audience who appreciates cultural fusion and innovative design.
'''
```


```python
# Generate your images
generation_response = client.images.generate(
    model = "dall-e-3",
    prompt=dalle_prompt,
    n=3,
    size="1024x1024",
    response_format="url",
)
```


```python
filepaths = process_dalle_images(generation_response, "generation", base_image_dir)
```


key: _node_content
string_value:
{"id_": "f1d59f05-c25d-4b75-8ab8-0cbd9b14c51b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ee647f98-6796-4ace-9fa1-781cc8ca2627", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4113902b964755cb6485391a9fa1d36013bdb8e97ece3a53909b6fa11bb4c4dd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0ac2c008-d7a3-4715-8b61-d4a673214aac", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4e86934f578d8177e2a0ac14fab1c95638fa3de55e169cc798a89faa5d3e9eff", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e8c88507-0df9-433a-86b8-7a00219ee04e", "node_type": "1", "metadata": {}, "hash": "f7f0568eb9192a7ca1476f462fa08aef7b0dfafa43f719ea9763c74f285d8b68", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2758, "end_char_idx": 5595, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def process_dalle_images(response, filename, image_dir):
    # save the images
    urls = [datum.url for datum in response.data]  # extract URLs
    images = [requests.get(url).content for url in urls]  # download images
    image_names = [f"{filename}_{i + 1}.png" for i in range(len(images))]  # create names
    filepaths = [os.path.join(image_dir, name) for name in image_names]  # create filepaths
    for image, filepath in zip(images, filepaths):  # loop through the variations
        with open(filepath, "wb") as image_file:  # open the file
            image_file.write(image)  # write the image to the file

    return filepaths
```


```python
dalle_prompt = '''
Full length, zoomed out photo of our premium Lederhosen-inspired jumpsuit.
Showcase the intricate hand-stitched details and high-quality leather, while highlighting the perfect blend of Austrian heritage and modern fashion.
This piece appeals to a sophisticated, trendsetting audience who appreciates cultural fusion and innovative design.
'''
```


```python
# Generate your images
generation_response = client.images.generate(
    model = "dall-e-3",
    prompt=dalle_prompt,
    n=3,
    size="1024x1024",
    response_format="url",
)
```


```python
filepaths = process_dalle_images(generation_response, "generation", base_image_dir)
```


```python
# print the new generations
for filepath in filepaths:
    print(filepath)
    display(Image.open(filepath))
```

## Generate Mask

Next we'll load up one of our images and generate masks.

For this demonstration we're picking a UX where we "click" on a point on the image to generate masks from. However, there are [example notebooks](https://github.com/facebookresearch/segment-anything/blob/main/notebooks/automatic_mask_generator_example.ipynb) provided by Meta which show how to generate every possible mask for an image, draw a box, and some other useful approaches.


```python
# Pick one of your generated images
chosen_image = "images/01_generations/generation_2.png"
```


```python
# Function to display mask using matplotlib
def show_mask(mask, ax):
    color = np.array([30 / 255, 144 / 255, 255 / 255, 0.6])
    h, w = mask.shape[-2:]
    mask_image = mask.reshape(h, w, 1) * color.reshape(1, 1, -1)
    ax.imshow(mask_image)


# Function to display where we've "clicked"
def show_points(coords, labels, ax, marker_size=375):
    pos_points = coords[labels == 1]
    neg_points = coords[labels == 0]
    ax.scatter(
        pos_points[:, 0],
        pos_points[:, 1],
        color="green",
        marker="*",
        s=marker_size,
        edgecolor="white",
        linewidth=1.25,
    )
    ax.scatter(
        neg_points[:, 0],
        neg_points[:, 1],
        color="red",
        marker="*",
        s=marker_size,
        edgecolor="white",
        linewidth=1.25,
    )

```


key: _node_content
string_value:
{"id_": "e8c88507-0df9-433a-86b8-7a00219ee04e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ee647f98-6796-4ace-9fa1-781cc8ca2627", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4113902b964755cb6485391a9fa1d36013bdb8e97ece3a53909b6fa11bb4c4dd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f1d59f05-c25d-4b75-8ab8-0cbd9b14c51b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b110e4ff66400a98b381fd4a4f322e9749517964b58f3727f8dd72bc1b7a2b23", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a66dd4a7-3cad-4769-b70b-56892b1cc4c7", "node_type": "1", "metadata": {}, "hash": "2c3c5989065f48bf1f980e73e021ea0346f149315b2fcb667411b0f08938bfbd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4087, "end_char_idx": 7011, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# print the new generations
for filepath in filepaths:
    print(filepath)
    display(Image.open(filepath))
```

## Generate Mask

Next we'll load up one of our images and generate masks.

For this demonstration we're picking a UX where we "click" on a point on the image to generate masks from. However, there are [example notebooks](https://github.com/facebookresearch/segment-anything/blob/main/notebooks/automatic_mask_generator_example.ipynb) provided by Meta which show how to generate every possible mask for an image, draw a box, and some other useful approaches.


```python
# Pick one of your generated images
chosen_image = "images/01_generations/generation_2.png"
```


```python
# Function to display mask using matplotlib
def show_mask(mask, ax):
    color = np.array([30 / 255, 144 / 255, 255 / 255, 0.6])
    h, w = mask.shape[-2:]
    mask_image = mask.reshape(h, w, 1) * color.reshape(1, 1, -1)
    ax.imshow(mask_image)


# Function to display where we've "clicked"
def show_points(coords, labels, ax, marker_size=375):
    pos_points = coords[labels == 1]
    neg_points = coords[labels == 0]
    ax.scatter(
        pos_points[:, 0],
        pos_points[:, 1],
        color="green",
        marker="*",
        s=marker_size,
        edgecolor="white",
        linewidth=1.25,
    )
    ax.scatter(
        neg_points[:, 0],
        neg_points[:, 1],
        color="red",
        marker="*",
        s=marker_size,
        edgecolor="white",
        linewidth=1.25,
    )

```


```python
# Load chosen image using opencv
image = cv2.imread(chosen_image)
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

# Display our chosen image
plt.figure(figsize=(10, 10))
plt.imshow(image)
plt.axis("on")
plt.show()
```


    
![png](output_14_0.png)
    



```python
# Set the pixel coordinates for our "click" to assign masks
input_point = np.array([[525, 325]])
input_label = np.array([1])

# Display the point we've clicked on
plt.figure(figsize=(10, 10))
plt.imshow(image)
show_points(input_point, input_label, plt.gca())
plt.axis("on")
plt.show()
```


    
![png](output_15_0.png)
    



```python
# Initiate predictor with Segment Anything model
predictor = SamPredictor(sam)
predictor.set_image(image)

# Use the predictor to gather masks for the point we clicked
masks, scores, logits = predictor.predict(
    point_coords=input_point,
    point_labels=input_label,
    multimask_output=True,
)

# Check the shape - should be three masks of the same dimensions as our image
masks.shape
```




    (3, 1024, 1024)




```python
# Display the possible masks we can select along with their confidence
for i, (mask, score) in enumerate(zip(masks, scores)):
    plt.figure(figsize=(10, 10))
    plt.imshow(image)
    show_mask(mask, plt.gca())
    show_points(input_point, input_label, plt.gca())
    plt.title(f"Mask {i+1}, Score: {score:.3f}", fontsize=18)
    plt.axis("off")
    plt.show()
```


key: _node_content
string_value:
{"id_": "a66dd4a7-3cad-4769-b70b-56892b1cc4c7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ee647f98-6796-4ace-9fa1-781cc8ca2627", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4113902b964755cb6485391a9fa1d36013bdb8e97ece3a53909b6fa11bb4c4dd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e8c88507-0df9-433a-86b8-7a00219ee04e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "be3086ce650e376b704f40498b5eb54d83401987d9e2089deb0beef613d83aa4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "24652c26-49e9-4fdd-8e15-204994e7287c", "node_type": "1", "metadata": {}, "hash": "cdae518d1ba8cdab98b04dbdfd1b6bd19ac151af4ca5491eb795369b603c069a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5598, "end_char_idx": 8235, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Load chosen image using opencv
image = cv2.imread(chosen_image)
image = cv2.cvtColor(image, cv2.COLOR_BGR2RGB)

# Display our chosen image
plt.figure(figsize=(10, 10))
plt.imshow(image)
plt.axis("on")
plt.show()
```


    
![png](output_14_0.png)
    



```python
# Set the pixel coordinates for our "click" to assign masks
input_point = np.array([[525, 325]])
input_label = np.array([1])

# Display the point we've clicked on
plt.figure(figsize=(10, 10))
plt.imshow(image)
show_points(input_point, input_label, plt.gca())
plt.axis("on")
plt.show()
```


    
![png](output_15_0.png)
    



```python
# Initiate predictor with Segment Anything model
predictor = SamPredictor(sam)
predictor.set_image(image)

# Use the predictor to gather masks for the point we clicked
masks, scores, logits = predictor.predict(
    point_coords=input_point,
    point_labels=input_label,
    multimask_output=True,
)

# Check the shape - should be three masks of the same dimensions as our image
masks.shape
```




    (3, 1024, 1024)




```python
# Display the possible masks we can select along with their confidence
for i, (mask, score) in enumerate(zip(masks, scores)):
    plt.figure(figsize=(10, 10))
    plt.imshow(image)
    show_mask(mask, plt.gca())
    show_points(input_point, input_label, plt.gca())
    plt.title(f"Mask {i+1}, Score: {score:.3f}", fontsize=18)
    plt.axis("off")
    plt.show()
```


    
![png](output_17_0.png)
    



    
![png](output_17_1.png)
    



    
![png](output_17_2.png)
    



```python
# Choose which mask you'd like to use
chosen_mask = masks[1]

# We'll now reverse the mask so that it is clear and everything else is white
chosen_mask = chosen_mask.astype("uint8")
chosen_mask[chosen_mask != 0] = 255
chosen_mask[chosen_mask == 0] = 1
chosen_mask[chosen_mask == 255] = 0
chosen_mask[chosen_mask == 1] = 255

```


```python
# create a base blank mask
width = 1024
height = 1024
mask = Image.new("RGBA", (width, height), (0, 0, 0, 1))  # create an opaque image mask

# Convert mask back to pixels to add our mask replacing the third dimension
pix = np.array(mask)
pix[:, :, 3] = chosen_mask

# Convert pixels back to an RGBA image and display
new_mask = Image.fromarray(pix, "RGBA")
new_mask
```




    
![png](output_19_0.png)
    




```python
# We'll save this mask for re-use for our edit
new_mask.save(os.path.join(mask_dir, "new_mask.png"))
```

## Create new image

Now we'll combine our original image with the mask and the Edit endpoint for DALLE to inpaint the transparent area according to a new prompt. (as 0f January 2024 dall-e-2 is the only model that supports edits)


key: _node_content
string_value:
{"id_": "24652c26-49e9-4fdd-8e15-204994e7287c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "ee647f98-6796-4ace-9fa1-781cc8ca2627", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4113902b964755cb6485391a9fa1d36013bdb8e97ece3a53909b6fa11bb4c4dd", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a66dd4a7-3cad-4769-b70b-56892b1cc4c7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/dalle/How_to_create_dynamic_masks_with_DALL-E_and_Segment_Anything.ipynb", "title": "How to create dynamic masks with DALL\u00b7E and Segment Anything", "tokens": 2392, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "9a231d1a902374e24bfcc4f379204c06bab6ad922fc50e08399ad03bff73be90", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7019, "end_char_idx": 9362, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
![png](output_17_0.png)
    



    
![png](output_17_1.png)
    



    
![png](output_17_2.png)
    



```python
# Choose which mask you'd like to use
chosen_mask = masks[1]

# We'll now reverse the mask so that it is clear and everything else is white
chosen_mask = chosen_mask.astype("uint8")
chosen_mask[chosen_mask != 0] = 255
chosen_mask[chosen_mask == 0] = 1
chosen_mask[chosen_mask == 255] = 0
chosen_mask[chosen_mask == 1] = 255

```


```python
# create a base blank mask
width = 1024
height = 1024
mask = Image.new("RGBA", (width, height), (0, 0, 0, 1))  # create an opaque image mask

# Convert mask back to pixels to add our mask replacing the third dimension
pix = np.array(mask)
pix[:, :, 3] = chosen_mask

# Convert pixels back to an RGBA image and display
new_mask = Image.fromarray(pix, "RGBA")
new_mask
```




    
![png](output_19_0.png)
    




```python
# We'll save this mask for re-use for our edit
new_mask.save(os.path.join(mask_dir, "new_mask.png"))
```

## Create new image

Now we'll combine our original image with the mask and the Edit endpoint for DALLE to inpaint the transparent area according to a new prompt. (as 0f January 2024 dall-e-2 is the only model that supports edits)



```python
# edit an image
edit_response = client.images.edit(
    image=open(chosen_image, "rb"),  # from the generation section
    mask=open(os.path.join(mask_dir, "new_mask.png"), "rb"),  # from right above
    prompt="Brilliant leather Lederhosen with a formal look, detailed, intricate, photorealistic",  # provide a prompt to fill the space
    n=3,
    size="1024x1024",
    response_format="url",
)

edit_filepaths = process_dalle_images(edit_response, "edits", edit_image_dir)

```


```python
# Display your beautiful creations!
%matplotlib inline

# figure size in inches optional
rcParams["figure.figsize"] = 11 ,8

# read images
img_A = mpimg.imread(edit_filepaths[0])
img_B = mpimg.imread(edit_filepaths[1])
img_C = mpimg.imread(edit_filepaths[2])

# display images
fig, ax = plt.subplots(1,3)
[a.axis("off") for a in ax]
ax[0].imshow(img_A)
ax[1].imshow(img_B)
ax[2].imshow(img_C)
```




    <matplotlib.image.AxesImage at 0x791b1f4c58a0>




    
![png](output_23_1.png)
    


Beautiful!

Now you too can easily create dynamic masks to extend your images - enjoy the APIs, and please share what you build!


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb


key: _node_content
string_value:
{"id_": "b0c8fe1a-9ef3-4b57-aad3-a9bbfa4bbbf4", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8ce7fce6-5e88-446a-b528-7116a7e010d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "694aa4c2-746f-4722-b6da-6f8fa777f58f", "node_type": "1", "metadata": {}, "hash": "95c5cdd8a888a4a50454d7672f9c198c1edabd1ce5a706351e5f15ba5e51e17c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2450, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# How to evaluate a summarization task

In this notebook we delve into the evaluation techniques for abstractive summarization tasks using a simple example. We explore traditional evaluation methods like [ROUGE](https://aclanthology.org/W04-1013/) and [BERTScore](https://arxiv.org/abs/1904.09675), in addition to showcasing a more novel approach using LLMs as evaluators.

Evaluating the quality of summaries is a time-consuming process, as it involves different quality metrics such as coherence, conciseness, readability and content. Traditional automatic evaluation metrics such as `ROUGE` and `BERTScore` and others are concrete and reliable, but they may not correlate well with the actual quality of summaries. They show relatively low correlation with human judgments, especially for open-ended generation tasks ([Liu et al., 2023](https://arxiv.org/pdf/2303.16634.pdf)). There's a growing need to lean on human evaluations, user feedback, or model-based metrics while being vigilant about potential biases. While human judgment provides invaluable insights, it is often not scalable and can be cost-prohibitive.

In addition to these traditional metrics, we showcase a method ([G-Eval](https://arxiv.org/pdf/2303.16634.pdf)) that leverages Large Language Models (LLMs) as a novel, reference-free metric for assessing abstractive summaries. In this case, we use `gpt-4` to score candidate outputs. `gpt-4` has effectively learned an internal model of language quality that allows it to differentiate between fluent, coherent text and low-quality text. Harnessing this internal scoring mechanism allows auto-evaluation of new candidate outputs generated by an LLM.


## Setup



```python
# Installing necessary packages for the evaluation
# rouge: For evaluating with ROUGE metric
# bert_score: For evaluating with BERTScore
# openai: To interact with OpenAI's API
!pip install rouge --quiet
!pip install bert_score --quiet
!pip install openai --quiet
```


```python
from openai import OpenAI
import os
import re
import pandas as pd

# Python Implementation of the ROUGE Metric
from rouge import Rouge

# BERTScore leverages the pre-trained contextual embeddings from BERT and matches words in candidate and reference sentences by cosine similarity.
from bert_score import BERTScorer

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))

```


    <IPython.core.display.Javascript object>


key: _node_content
string_value:
{"id_": "694aa4c2-746f-4722-b6da-6f8fa777f58f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8ce7fce6-5e88-446a-b528-7116a7e010d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b0c8fe1a-9ef3-4b57-aad3-a9bbfa4bbbf4", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a8578004bb41ff667a1b8b98372d06a46f8ea23c2379d6efd592885bc255f93f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b21febf7-02cb-4e32-befb-ec74b44f54e9", "node_type": "1", "metadata": {}, "hash": "c8f266d2cac970d57cd8bde60d8d84155be9ea038460f0370ee9be4541f6362e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1674, "end_char_idx": 6892, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Setup



```python
# Installing necessary packages for the evaluation
# rouge: For evaluating with ROUGE metric
# bert_score: For evaluating with BERTScore
# openai: To interact with OpenAI's API
!pip install rouge --quiet
!pip install bert_score --quiet
!pip install openai --quiet
```


```python
from openai import OpenAI
import os
import re
import pandas as pd

# Python Implementation of the ROUGE Metric
from rouge import Rouge

# BERTScore leverages the pre-trained contextual embeddings from BERT and matches words in candidate and reference sentences by cosine similarity.
from bert_score import BERTScorer

client = OpenAI(api_key=os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as env var>"))

```


    <IPython.core.display.Javascript object>


## Example task

For the purposes of this notebook we'll use the example summarization below. Notice that we provide two generated summaries to compare, and a reference human-written summary, which evaluation metrics like `ROUGE` and `BERTScore` require.

Excerpt (`excerpt`):

> OpenAI's mission is to ensure that artificial general intelligence (AGI) benefits all of humanity. OpenAI will build safe and beneficial AGI directly, but will also consider its mission fulfilled if its work aids others to achieve this outcome. OpenAI follows several key principles for this purpose. First, broadly distributed benefits - any influence over AGI's deployment will be used for the benefit of all, and to avoid harmful uses or undue concentration of power. Second, long-term safety - OpenAI is committed to doing the research to make AGI safe, and to promote the adoption of such research across the AI community. Third, technical leadership - OpenAI aims to be at the forefront of AI capabilities. Fourth, a cooperative orientation - OpenAI actively cooperates with other research and policy institutions, and seeks to create a global community working together to address AGI's global challenges.

Summaries:

| Reference Summary /`ref_summary` (human generated)                                                                                                                                                                                                                                                                                                                         | Eval Summary 1 / `eval_summary_1` (system generated)                                                                                                                                                                                                                                                                                                                               | Eval Summary 2 / `eval_summary_2` (system generated)                                                                                                                                                                                                                                                   |
| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |
| OpenAI aims to ensure artificial general intelligence (AGI) is used for everyone's benefit, avoiding harmful uses or undue power concentration. It is committed to researching AGI safety, promoting such studies among the AI community. OpenAI seeks to lead in AI capabilities and cooperates with global research and policy institutions to address AGI's challenges. | OpenAI aims to AGI benefits all humanity, avoiding harmful uses and power concentration. It pioneers research into safe and beneficial AGI and promotes adoption globally. OpenAI maintains technical leadership in AI while cooperating with global institutions to address AGI challenges. It seeks to lead a collaborative worldwide effort developing AGI for collective good. | OpenAI aims to ensure AGI is for everyone's use, totally avoiding harmful stuff or big power concentration. Committed to researching AGI's safe side, promoting these studies in AI folks. OpenAI wants to be top in AI things and works with worldwide research, policy groups to figure AGI's stuff. |

Take a moment to figure out which summary you'd personally prefer and the one that captures OpenAI's mission really well.


key: _node_content
string_value:
{"id_": "b21febf7-02cb-4e32-befb-ec74b44f54e9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8ce7fce6-5e88-446a-b528-7116a7e010d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "694aa4c2-746f-4722-b6da-6f8fa777f58f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "bcd5efef4d3e0374c3038373bc10e4bf182c829c53e5f774372c57bde7f1e8c0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a243aa88-bad4-4a7c-a3c8-a9e19608269f", "node_type": "1", "metadata": {}, "hash": "b4521af41da3037c5cbad4fccbc8f93dac9bc5779bbf896d17fe26f9b4101f53", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6896, "end_char_idx": 10220, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
excerpt = "OpenAI's mission is to ensure that artificial general intelligence (AGI) benefits all of humanity. OpenAI will build safe and beneficial AGI directly, but will also consider its mission fulfilled if its work aids others to achieve this outcome. OpenAI follows several key principles for this purpose. First, broadly distributed benefits - any influence over AGI's deployment will be used for the benefit of all, and to avoid harmful uses or undue concentration of power. Second, long-term safety - OpenAI is committed to doing the research to make AGI safe, and to promote the adoption of such research across the AI community. Third, technical leadership - OpenAI aims to be at the forefront of AI capabilities. Fourth, a cooperative orientation - OpenAI actively cooperates with other research and policy institutions, and seeks to create a global community working together to address AGI's global challenges."
ref_summary = "OpenAI aims to ensure artificial general intelligence (AGI) is used for everyone's benefit, avoiding harmful uses or undue power concentration. It is committed to researching AGI safety, promoting such studies among the AI community. OpenAI seeks to lead in AI capabilities and cooperates with global research and policy institutions to address AGI's challenges."
eval_summary_1 = "OpenAI aims to AGI benefits all humanity, avoiding harmful uses and power concentration. It pioneers research into safe and beneficial AGI and promotes adoption globally. OpenAI maintains technical leadership in AI while cooperating with global institutions to address AGI challenges. It seeks to lead a collaborative worldwide effort developing AGI for collective good."
eval_summary_2 = "OpenAI aims to ensure AGI is for everyone's use, totally avoiding harmful stuff or big power concentration. Committed to researching AGI's safe side, promoting these studies in AI folks. OpenAI wants to be top in AI things and works with worldwide research, policy groups to figure AGI's stuff."
```


    <IPython.core.display.Javascript object>


## Evaluating using ROUGE

[ROUGE](https://aclanthology.org/W04-1013/), which stands for Recall-Oriented Understudy for Gisting Evaluation, primarily gauges the overlap of words between a generated output and a reference text. It's a prevalent metric for evaluating automatic summarization tasks. Among its variants, `ROUGE-L` offers insights into the longest contiguous match between system-generated and reference summaries, gauging how well the system retains the original summary's essence.



```python
# function to calculate the Rouge score
def get_rouge_scores(text1, text2):
    rouge = Rouge()
    return rouge.get_scores(text1, text2)


rouge_scores_out = []

# Calculate the ROUGE scores for both summaries using reference
eval_1_rouge = get_rouge_scores(eval_summary_1, ref_summary)
eval_2_rouge = get_rouge_scores(eval_summary_2, ref_summary)

for metric in ["rouge-1", "rouge-2", "rouge-l"]:
    for label in ["F-Score"]:
        eval_1_score = eval_1_rouge[0][metric][label[0].lower()]
        eval_2_score = eval_2_rouge[0][metric][label[0].lower()]

        row = {
            "Metric": f"{metric} ({label})",
            "Summary 1": eval_1_score,
            "Summary 2": eval_2_score,
        }
        rouge_scores_out.append(row)


key: _node_content
string_value:
{"id_": "a243aa88-bad4-4a7c-a3c8-a9e19608269f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8ce7fce6-5e88-446a-b528-7116a7e010d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b21febf7-02cb-4e32-befb-ec74b44f54e9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "efe0754d2c2fb1fa89affb8e8a9e725591765f7528f95a624573f5bbf172737e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "77566eb4-3d97-4c14-b438-06828b735f83", "node_type": "1", "metadata": {}, "hash": "81476445bc57001c5fb5142b26eee5186b5a76ce47defb6e9d76e4d65e7bb780", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8924, "end_char_idx": 10535, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<IPython.core.display.Javascript object>


## Evaluating using ROUGE

[ROUGE](https://aclanthology.org/W04-1013/), which stands for Recall-Oriented Understudy for Gisting Evaluation, primarily gauges the overlap of words between a generated output and a reference text. It's a prevalent metric for evaluating automatic summarization tasks. Among its variants, `ROUGE-L` offers insights into the longest contiguous match between system-generated and reference summaries, gauging how well the system retains the original summary's essence.



```python
# function to calculate the Rouge score
def get_rouge_scores(text1, text2):
    rouge = Rouge()
    return rouge.get_scores(text1, text2)


rouge_scores_out = []

# Calculate the ROUGE scores for both summaries using reference
eval_1_rouge = get_rouge_scores(eval_summary_1, ref_summary)
eval_2_rouge = get_rouge_scores(eval_summary_2, ref_summary)

for metric in ["rouge-1", "rouge-2", "rouge-l"]:
    for label in ["F-Score"]:
        eval_1_score = eval_1_rouge[0][metric][label[0].lower()]
        eval_2_score = eval_2_rouge[0][metric][label[0].lower()]

        row = {
            "Metric": f"{metric} ({label})",
            "Summary 1": eval_1_score,
            "Summary 2": eval_2_score,
        }
        rouge_scores_out.append(row)


def highlight_max(s):
    is_max = s == s.max()
    return [
        "background-color: lightgreen" if v else "background-color: white"
        for v in is_max
    ]


rouge_scores_out = (
    pd.DataFrame(rouge_scores_out)
    .set_index("Metric")
    .style.apply(highlight_max, axis=1)
)

rouge_scores_out
```


key: _node_content
string_value:
{"id_": "77566eb4-3d97-4c14-b438-06828b735f83", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8ce7fce6-5e88-446a-b528-7116a7e010d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a243aa88-bad4-4a7c-a3c8-a9e19608269f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1874c5528d117efd82bc0214e88db51f39ea647bf1f1f20d7caa929dca48ca05", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "86e73283-1edf-4e33-9065-ae3749dc7ee2", "node_type": "1", "metadata": {}, "hash": "eee3558fa20b8944ebcd928b79ba588daac76dc4eed49cb5bec8a0f8df83185d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9465, "end_char_idx": 11966, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# function to calculate the Rouge score
def get_rouge_scores(text1, text2):
    rouge = Rouge()
    return rouge.get_scores(text1, text2)


rouge_scores_out = []

# Calculate the ROUGE scores for both summaries using reference
eval_1_rouge = get_rouge_scores(eval_summary_1, ref_summary)
eval_2_rouge = get_rouge_scores(eval_summary_2, ref_summary)

for metric in ["rouge-1", "rouge-2", "rouge-l"]:
    for label in ["F-Score"]:
        eval_1_score = eval_1_rouge[0][metric][label[0].lower()]
        eval_2_score = eval_2_rouge[0][metric][label[0].lower()]

        row = {
            "Metric": f"{metric} ({label})",
            "Summary 1": eval_1_score,
            "Summary 2": eval_2_score,
        }
        rouge_scores_out.append(row)


def highlight_max(s):
    is_max = s == s.max()
    return [
        "background-color: lightgreen" if v else "background-color: white"
        for v in is_max
    ]


rouge_scores_out = (
    pd.DataFrame(rouge_scores_out)
    .set_index("Metric")
    .style.apply(highlight_max, axis=1)
)

rouge_scores_out
```




<style type="text/css">
#T_7e6ac_row0_col0, #T_7e6ac_row1_col1, #T_7e6ac_row2_col0 {
  background-color: white;
}
#T_7e6ac_row0_col1, #T_7e6ac_row1_col0, #T_7e6ac_row2_col1 {
  background-color: lightgreen;
}
</style>
<table id="T_7e6ac">
  <thead>
    <tr>
      <th class="blank level0" >&nbsp;</th>
      <th id="T_7e6ac_level0_col0" class="col_heading level0 col0" >Summary 1</th>
      <th id="T_7e6ac_level0_col1" class="col_heading level0 col1" >Summary 2</th>
    </tr>
    <tr>
      <th class="index_name level0" >Metric</th>
      <th class="blank col0" >&nbsp;</th>
      <th class="blank col1" >&nbsp;</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th id="T_7e6ac_level0_row0" class="row_heading level0 row0" >rouge-1 (F-Score)</th>
      <td id="T_7e6ac_row0_col0" class="data row0 col0" >0.488889</td>
      <td id="T_7e6ac_row0_col1" class="data row0 col1" >0.511628</td>
    </tr>
    <tr>
      <th id="T_7e6ac_level0_row1" class="row_heading level0 row1" >rouge-2 (F-Score)</th>
      <td id="T_7e6ac_row1_col0" class="data row1 col0" >0.230769</td>
      <td id="T_7e6ac_row1_col1" class="data row1 col1" >0.163265</td>
    </tr>
    <tr>
      <th id="T_7e6ac_level0_row2" class="row_heading level0 row2" >rouge-l (F-Score)</th>
      <td id="T_7e6ac_row2_col0" class="data row2 col0" >0.488889</td>
      <td id="T_7e6ac_row2_col1" class="data row2 col1" >0.511628</td>
    </tr>
  </tbody>
</table>


key: _node_content
string_value:
{"id_": "86e73283-1edf-4e33-9065-ae3749dc7ee2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8ce7fce6-5e88-446a-b528-7116a7e010d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "77566eb4-3d97-4c14-b438-06828b735f83", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d1e6ae6332f47d3c758e8b71f696c7631e657cab5f88b07bfcc91abd14b1ab7f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c43eb4f8-b310-49af-8746-d903e0067150", "node_type": "1", "metadata": {}, "hash": "d01e0636fc392f1e46e62a08904eaaf217b146c4946a8d5c24e556b92a550980", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11976, "end_char_idx": 15206, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<IPython.core.display.Javascript object>


The table shows the `ROUGE` scores for evaluating two different summaries against a reference text. In the case of `rouge-1`, Summary 2 outperforms Summary 1, indicating a better overlap of individual words and for `rouge-l`, Summary 2 has a higher score, implying a closer match in the longest common subsequences, and thus a potentially better overall summarization in capturing the main content and order of the original text. Since Summary 2 has many words and short phrases directly lifted from the excerpt, its overlap with the reference summary would likely be higher, leading to higher `ROUGE` scores.

While `ROUGE` and similar metrics, such as [BLEU](https://aclanthology.org/P02-1040.pdf) and [METEOR](https://www.cs.cmu.edu/~alavie/METEOR/), offer quantitative measures, they often fail to capture the true essence of a well-generated summary. They also correlate worse with human scores. Given the advancements in LLMs, which are adept at producing fluent and coherent summaries, traditional metrics like `ROUGE` may inadvertently penalize these models. This is especially true if the summaries are articulated differently but still encapsulate the core information accurately.


## Evaluating using BERTScore

ROUGE relies on the exact presence of words in both the predicted and reference texts, failing to interpret the underlying semantics. This is where [BERTScore](https://arxiv.org/abs/1904.09675) comes in and leverages the contextual embeddings from the BERT model, aiming to evaluate the similarity between a predicted and a reference sentence in the context of machine-generated text. By comparing embeddings from both sentences, `BERTScore` captures semantic similarities that might be missed by traditional n-gram based metrics.



```python
# Instantiate the BERTScorer object for English language
scorer = BERTScorer(lang="en")

# Calculate BERTScore for the summary 1 against the excerpt
# P1, R1, F1_1 represent Precision, Recall, and F1 Score respectively
P1, R1, F1_1 = scorer.score([eval_summary_1], [ref_summary])

# Calculate BERTScore for summary 2 against the excerpt
# P2, R2, F2_2 represent Precision, Recall, and F1 Score respectively
P2, R2, F2_2 = scorer.score([eval_summary_2], [ref_summary])

print("Summary 1 F1 Score:", F1_1.tolist()[0])
print("Summary 2 F1 Score:", F2_2.tolist()[0])
```

    Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight', 'lm_head.decoder.weight', 'lm_head.bias']
    - This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
    - This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).


    Summary 1 F1 Score: 0.9227314591407776
    Summary 2 F1 Score: 0.9189572930335999



    <IPython.core.display.Javascript object>


key: _node_content
string_value:
{"id_": "c43eb4f8-b310-49af-8746-d903e0067150", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8ce7fce6-5e88-446a-b528-7116a7e010d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "86e73283-1edf-4e33-9065-ae3749dc7ee2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "1921a14c7b2200582b49bfb9715a0dd8d7139b8372a29ebf1a30d07ea523a431", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5ffe76b7-9d08-435a-a269-452f68883986", "node_type": "1", "metadata": {}, "hash": "1439f8bb4fc7a2a79da913830b7604d9dd81f85ace5d5ffb1ac5bb2d0d6db099", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13777, "end_char_idx": 15737, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Instantiate the BERTScorer object for English language
scorer = BERTScorer(lang="en")

# Calculate BERTScore for the summary 1 against the excerpt
# P1, R1, F1_1 represent Precision, Recall, and F1 Score respectively
P1, R1, F1_1 = scorer.score([eval_summary_1], [ref_summary])

# Calculate BERTScore for summary 2 against the excerpt
# P2, R2, F2_2 represent Precision, Recall, and F1 Score respectively
P2, R2, F2_2 = scorer.score([eval_summary_2], [ref_summary])

print("Summary 1 F1 Score:", F1_1.tolist()[0])
print("Summary 2 F1 Score:", F2_2.tolist()[0])
```

    Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight', 'lm_head.decoder.weight', 'lm_head.bias']
    - This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
    - This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).


    Summary 1 F1 Score: 0.9227314591407776
    Summary 2 F1 Score: 0.9189572930335999



    <IPython.core.display.Javascript object>


The close F1 Scores between the summaries indicate that they may perform similarly in capturing the key information. However, this small difference should be interpreted with caution. Since `BERTScore` may not fully grasp subtleties and high-level concepts that a human evaluator might understand, reliance solely on this metric could lead to misinterpreting the actual quality and nuances of the summary. An integrated approach combining `BERTScore` with human judgment and other metrics could offer a more reliable evaluation.


key: _node_content
string_value:
{"id_": "5ffe76b7-9d08-435a-a269-452f68883986", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8ce7fce6-5e88-446a-b528-7116a7e010d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c43eb4f8-b310-49af-8746-d903e0067150", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "9ff7f98557cccf735f7e9dd40ae3397da730c142d8676afed535e5c6e370d010", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a9b32e7d-3c10-4a11-b5af-4c8c97bfdc28", "node_type": "1", "metadata": {}, "hash": "c7abc59b4b1407c97964e334311db93595655397b1fa1b4054e0aa57e13bb992", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15077, "end_char_idx": 17368, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Summary 1 F1 Score: 0.9227314591407776
    Summary 2 F1 Score: 0.9189572930335999



    <IPython.core.display.Javascript object>


The close F1 Scores between the summaries indicate that they may perform similarly in capturing the key information. However, this small difference should be interpreted with caution. Since `BERTScore` may not fully grasp subtleties and high-level concepts that a human evaluator might understand, reliance solely on this metric could lead to misinterpreting the actual quality and nuances of the summary. An integrated approach combining `BERTScore` with human judgment and other metrics could offer a more reliable evaluation.


## Evaluating using GPT-4

Here we implement an example **reference-free** text evaluator using `gpt-4`, inspired by the [G-Eval]((https://arxiv.org/pdf/2303.16634.pdf)) framework which evaluates the quality of generated text using large language models. Unlike metrics like `ROUGE` or `BERTScore` that rely on comparison to reference summaries, the `gpt-4` based evaluator assesses the quality of generated content based solely on the input prompt and text, without any ground truth references. This makes it applicable to new datasets and tasks where human references are sparse or unavailable. 

Here's an overview of this method:

1. We define four distinct criteria:
    1. **Relevance**: Evaluates if the summary includes only important information and excludes redundancies.
    2. **Coherence**: Assesses the logical flow and organization of the summary.
    3. **Consistency**: Checks if the summary aligns with the facts in the source document.
    4. **Fluency**: Rates the grammar and readability of the summary.
2. We craft prompts for each of these criteria, taking the original document and the summary as inputs, and leveraging chain-of-thought generation and guiding the model to output a numeric score from 1-5 for each criteria. 
3. We generate scores from `gpt-4` with the defined prompts, comparing them across summaries.

In this demonstration, we're using a direct scoring function where `gpt-4` generates a discrete score (1-5) for each metric. Normalizing the scores and taking a weighted sum could result in more robust, continuous scores that better reflect the quality and diversity of the summaries.


key: _node_content
string_value:
{"id_": "a9b32e7d-3c10-4a11-b5af-4c8c97bfdc28", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8ce7fce6-5e88-446a-b528-7116a7e010d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5ffe76b7-9d08-435a-a269-452f68883986", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0264a51a7f0892b2fcda8c49649bb0b9ecb60fa210ec8d368479c340f847b5f1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2f9588d7-1ff9-4cc5-89af-85ec168b1c32", "node_type": "1", "metadata": {}, "hash": "23a177c88aa82909084813256b18dd5831fa17f2ed7924b26ca8c4ea6db46ade", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15740, "end_char_idx": 20652, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
## Evaluating using GPT-4

Here we implement an example **reference-free** text evaluator using `gpt-4`, inspired by the [G-Eval]((https://arxiv.org/pdf/2303.16634.pdf)) framework which evaluates the quality of generated text using large language models. Unlike metrics like `ROUGE` or `BERTScore` that rely on comparison to reference summaries, the `gpt-4` based evaluator assesses the quality of generated content based solely on the input prompt and text, without any ground truth references. This makes it applicable to new datasets and tasks where human references are sparse or unavailable. 

Here's an overview of this method:

1. We define four distinct criteria:
    1. **Relevance**: Evaluates if the summary includes only important information and excludes redundancies.
    2. **Coherence**: Assesses the logical flow and organization of the summary.
    3. **Consistency**: Checks if the summary aligns with the facts in the source document.
    4. **Fluency**: Rates the grammar and readability of the summary.
2. We craft prompts for each of these criteria, taking the original document and the summary as inputs, and leveraging chain-of-thought generation and guiding the model to output a numeric score from 1-5 for each criteria. 
3. We generate scores from `gpt-4` with the defined prompts, comparing them across summaries.

In this demonstration, we're using a direct scoring function where `gpt-4` generates a discrete score (1-5) for each metric. Normalizing the scores and taking a weighted sum could result in more robust, continuous scores that better reflect the quality and diversity of the summaries.


```python
# Evaluation prompt template based on G-Eval
EVALUATION_PROMPT_TEMPLATE = """
You will be given one summary written for an article. Your task is to rate the summary on one metric.
Please make sure you read and understand these instructions very carefully. 
Please keep this document open while reviewing, and refer to it as needed.

Evaluation Criteria:

{criteria}

Evaluation Steps:

{steps}

Example:

Source Text:

{document}

Summary:

{summary}

Evaluation Form (scores ONLY):

- {metric_name}
"""

# Metric 1: Relevance

RELEVANCY_SCORE_CRITERIA = """
Relevance(1-5) - selection of important content from the source. \
The summary should include only important information from the source document. \
Annotators were instructed to penalize summaries which contained redundancies and excess information.
"""

RELEVANCY_SCORE_STEPS = """
1. Read the summary and the source document carefully.
2. Compare the summary to the source document and identify the main points of the article.
3. Assess how well the summary covers the main points of the article, and how much irrelevant or redundant information it contains.
4. Assign a relevance score from 1 to 5.
"""

# Metric 2: Coherence

COHERENCE_SCORE_CRITERIA = """
Coherence(1-5) - the collective quality of all sentences. \
We align this dimension with the DUC quality question of structure and coherence \
whereby "the summary should be well-structured and well-organized. \
The summary should not just be a heap of related information, but should build from sentence to a\
coherent body of information about a topic."
"""

COHERENCE_SCORE_STEPS = """
1. Read the article carefully and identify the main topic and key points.
2. Read the summary and compare it to the article. Check if the summary covers the main topic and key points of the article,
and if it presents them in a clear and logical order.
3. Assign a score for coherence on a scale of 1 to 5, where 1 is the lowest and 5 is the highest based on the Evaluation Criteria.
"""

# Metric 3: Consistency

CONSISTENCY_SCORE_CRITERIA = """
Consistency(1-5) - the factual alignment between the summary and the summarized source. \
A factually consistent summary contains only statements that are entailed by the source document. \
Annotators were also asked to penalize summaries that contained hallucinated facts.
"""

CONSISTENCY_SCORE_STEPS = """
1. Read the article carefully and identify the main facts and details it presents.
2. Read the summary and compare it to the article. Check if the summary contains any factual errors that are not supported by the article.
3. Assign a score for consistency based on the Evaluation Criteria.
"""

# Metric 4: Fluency

FLUENCY_SCORE_CRITERIA = """
Fluency(1-3): the quality of the summary in terms of grammar, spelling, punctuation, word choice, and sentence structure.
1: Poor. The summary has many errors that make it hard to understand or sound unnatural.
2: Fair. The summary has some errors that affect the clarity or smoothness of the text, but the main points are still comprehensible.
3: Good. The summary has few or no errors and is easy to read and follow.
"""

FLUENCY_SCORE_STEPS = """
Read the summary and evaluate its fluency based on the given criteria. Assign a fluency score from 1 to 3.
"""


key: _node_content
string_value:
{"id_": "2f9588d7-1ff9-4cc5-89af-85ec168b1c32", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8ce7fce6-5e88-446a-b528-7116a7e010d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a9b32e7d-3c10-4a11-b5af-4c8c97bfdc28", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "cca993148068c170e12e6ef7c89d1af0b4bc69c2e57cae4514119ec691c061df", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c24905c8-b852-47a4-a74f-c51958d5007a", "node_type": "1", "metadata": {}, "hash": "3ea6e35c127f1e8581148d8689708c8b5e08709865a2b0e091e648c8e0b35793", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20655, "end_char_idx": 22257, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
def get_geval_score(
    criteria: str, steps: str, document: str, summary: str, metric_name: str
):
    prompt = EVALUATION_PROMPT_TEMPLATE.format(
        criteria=criteria,
        steps=steps,
        metric_name=metric_name,
        document=document,
        summary=summary,
    )
    response = client.chat.completions.create(
        model="gpt-4",
        messages=[{"role": "user", "content": prompt}],
        temperature=0,
        max_tokens=5,
        top_p=1,
        frequency_penalty=0,
        presence_penalty=0,
    )
    return response.choices[0].message.content


evaluation_metrics = {
    "Relevance": (RELEVANCY_SCORE_CRITERIA, RELEVANCY_SCORE_STEPS),
    "Coherence": (COHERENCE_SCORE_CRITERIA, COHERENCE_SCORE_STEPS),
    "Consistency": (CONSISTENCY_SCORE_CRITERIA, CONSISTENCY_SCORE_STEPS),
    "Fluency": (FLUENCY_SCORE_CRITERIA, FLUENCY_SCORE_STEPS),
}

summaries = {"Summary 1": eval_summary_1, "Summary 2": eval_summary_2}

data = {"Evaluation Type": [], "Summary Type": [], "Score": []}

for eval_type, (criteria, steps) in evaluation_metrics.items():
    for summ_type, summary in summaries.items():
        data["Evaluation Type"].append(eval_type)
        data["Summary Type"].append(summ_type)
        result = get_geval_score(criteria, steps, excerpt, summary, eval_type)
        score_num = int(result.strip())
        data["Score"].append(score_num)

pivot_df = pd.DataFrame(data, index=None).pivot(
    index="Evaluation Type", columns="Summary Type", values="Score"
)
styled_pivot_df = pivot_df.style.apply(highlight_max, axis=1)
display(styled_pivot_df)
```


key: _node_content
string_value:
{"id_": "c24905c8-b852-47a4-a74f-c51958d5007a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8ce7fce6-5e88-446a-b528-7116a7e010d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2f9588d7-1ff9-4cc5-89af-85ec168b1c32", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7d01dbd435f8cf90e8365309a81d16b57f3165bcb272d3633c1739109db2e15d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "893a25a0-1921-45c5-ad6b-6283d0226098", "node_type": "1", "metadata": {}, "hash": "88d39cc6f73eb420872c3104aa885fca7521be5c7395c9bcdd5bf0637f81b1e2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 21243, "end_char_idx": 23910, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
evaluation_metrics = {
    "Relevance": (RELEVANCY_SCORE_CRITERIA, RELEVANCY_SCORE_STEPS),
    "Coherence": (COHERENCE_SCORE_CRITERIA, COHERENCE_SCORE_STEPS),
    "Consistency": (CONSISTENCY_SCORE_CRITERIA, CONSISTENCY_SCORE_STEPS),
    "Fluency": (FLUENCY_SCORE_CRITERIA, FLUENCY_SCORE_STEPS),
}

summaries = {"Summary 1": eval_summary_1, "Summary 2": eval_summary_2}

data = {"Evaluation Type": [], "Summary Type": [], "Score": []}

for eval_type, (criteria, steps) in evaluation_metrics.items():
    for summ_type, summary in summaries.items():
        data["Evaluation Type"].append(eval_type)
        data["Summary Type"].append(summ_type)
        result = get_geval_score(criteria, steps, excerpt, summary, eval_type)
        score_num = int(result.strip())
        data["Score"].append(score_num)

pivot_df = pd.DataFrame(data, index=None).pivot(
    index="Evaluation Type", columns="Summary Type", values="Score"
)
styled_pivot_df = pivot_df.style.apply(highlight_max, axis=1)
display(styled_pivot_df)
```


<style type="text/css">
#T_94fab_row0_col0, #T_94fab_row1_col0, #T_94fab_row1_col1, #T_94fab_row2_col0, #T_94fab_row3_col0 {
  background-color: lightgreen;
}
#T_94fab_row0_col1, #T_94fab_row2_col1, #T_94fab_row3_col1 {
  background-color: white;
}
</style>
<table id="T_94fab">
  <thead>
    <tr>
      <th class="index_name level0" >Summary Type</th>
      <th id="T_94fab_level0_col0" class="col_heading level0 col0" >Summary 1</th>
      <th id="T_94fab_level0_col1" class="col_heading level0 col1" >Summary 2</th>
    </tr>
    <tr>
      <th class="index_name level0" >Evaluation Type</th>
      <th class="blank col0" >&nbsp;</th>
      <th class="blank col1" >&nbsp;</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th id="T_94fab_level0_row0" class="row_heading level0 row0" >Coherence</th>
      <td id="T_94fab_row0_col0" class="data row0 col0" >5</td>
      <td id="T_94fab_row0_col1" class="data row0 col1" >3</td>
    </tr>
    <tr>
      <th id="T_94fab_level0_row1" class="row_heading level0 row1" >Consistency</th>
      <td id="T_94fab_row1_col0" class="data row1 col0" >5</td>
      <td id="T_94fab_row1_col1" class="data row1 col1" >5</td>
    </tr>
    <tr>
      <th id="T_94fab_level0_row2" class="row_heading level0 row2" >Fluency</th>
      <td id="T_94fab_row2_col0" class="data row2 col0" >3</td>
      <td id="T_94fab_row2_col1" class="data row2 col1" >2</td>
    </tr>
    <tr>
      <th id="T_94fab_level0_row3" class="row_heading level0 row3" >Relevance</th>
      <td id="T_94fab_row3_col0" class="data row3 col0" >5</td>
      <td id="T_94fab_row3_col1" class="data row3 col1" >4</td>
    </tr>
  </tbody>
</table>


key: _node_content
string_value:
{"id_": "893a25a0-1921-45c5-ad6b-6283d0226098", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "8ce7fce6-5e88-446a-b528-7116a7e010d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c24905c8-b852-47a4-a74f-c51958d5007a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "23452416588d307b2144d8a52eef91571b5a5aaf3dc9488a1a87cf5043dfba1c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23919, "end_char_idx": 26471, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<IPython.core.display.Javascript object>



Overall, the Summary 1 appears to outperform Summary 2 in three of the four categories (Coherence, Relevance and Fluency). Both summaries are found to be consistent with each other. The result might suggest that Summary 1 is generally preferable based on the given evaluation criteria.

### Limitations

Note that LLM-based metrics could have a bias towards preferring LLM-generated texts over human-written texts. Additionally LLM based metrics are sensitive to system messages/prompts. We recommend experimenting with other techniques that can help improve performance and/or get consistent scores, striking the right balance between high-quality expensive evaluation and automated evaluations. It is also worth noting that this scoring methodology is currently limited by `gpt-4`'s context window.

## Conclusion

Evaluating abstractive summarization remains an open area for further improvement. Traditional metrics like `ROUGE`, `BLEU`, and `BERTScore` provide useful automatic evaluation but have limitations in capturing semantic similarity and nuanced aspects of summarization quality. Moreover, they require reference outputs which can be expensive to collect/label. LLM-based metrics offer promise as a reference-free method of evaluating coherence, fluency, and relevance. However, they too have potential biases favoring text generated by LLMs. Ultimately, a combination of automatic metrics and human evaluation is ideal for reliably assessing abstractive summarization systems. While human evaluation is indispensable for gaining a comprehensive understanding of summary quality, it should be complemented with automated evaluation to enable efficient, large-scale testing. The field will continue to evolve more robust evaluation techniques, balancing quality, scalability, and fairness. Advancing evaluation methods is crucial for driving progress in production applications.

## References

- [G-EVAL: NLG Evaluation Using GPT-4 with Better Human Alignment](https://arxiv.org/pdf/2303.16634.pdf) - Liu Y, Iter D, Xu Y, Wang S, Xu R, Zhu C. Published May, 2023.
- [BERTScore: Evaluating Text Generation with BERT](https://arxiv.org/abs/1904.09675) - Zhang T, Kishore V, Wu F, Weinberger KQ, Artzi Y. Published online February, 2020.
- [ROUGE: A Package for Automatic Evaluation of Summaries](https://aclanthology.org/W04-1013/) - Lin CY. Published July, 2004.
- [SummEval: Re-evaluating Summarization Evaluation](https://aclanthology.org/2021.tacl-1.24) - Fabbri et al. Published April, 2021.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb


key: _node_content
string_value:
{"id_": "2567e2c8-3eab-4afc-bc7a-0c8f59604ffc", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8e279319-1b53-4520-94b7-920079f6b92c", "node_type": "1", "metadata": {}, "hash": "ab4c3631d8abc55b383c576a8b460c6c41062f442df7d0e09a6ead8c1240672b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3413, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Getting Started with OpenAI Evals

The [OpenAI Evals](https://github.com/openai/evals/tree/main) framework consists of
1. A framework to evaluate an LLM (large language model) or a system built on top of an LLM.
2. An open-source registry of challenging evals

This notebook will cover:
* Introduction to Evaluation and the [OpenAI Evals](https://github.com/openai/evals/tree/main) library
* Building an Eval
* Running an Eval

#### What are evaluations/ `evals`?

Evaluation is the process of validating and testing the outputs that your LLM applications are producing. Having strong evaluations ("evals") will mean a more stable, reliable application that is resilient to code and model changes. An eval is a task used to measure the quality of the output of an LLM or LLM system. Given an input prompt, an output is generated. We evaluate this output with a set of ideal answers and find the quality of the LLM system.

#### Importance of Evaluations

If you are building with foundational models like `GPT-4`, creating high quality evals is one of the most impactful things you can do. Developing AI solutions involves an iterative design process. [Without evals, it can be very difficult and time intensive to understand](https://youtu.be/XGJNo8TpuVA?feature=shared&t=1089) how different model versions and prompts might affect your use case.

With OpenAI’s [continuous model upgrades](https://platform.openai.com/docs/models/continuous-model-upgrades), evals allow you to efficiently test model performance for your use cases in a standardized way. Developing a suite of evals customized to your objectives will help you quickly and effectively understand how new models may perform for your use cases. You can also make evals a part of your CI/CD pipeline to make sure you achieve the desired accuracy before deploying.

#### Types of evals

There are two main ways we can evaluate/grade completions: writing some validation logic in code
or using the model itself to inspect the answer. We’ll introduce each with some examples.

**Writing logic for answer checking**

The simplest and most common type of eval has an input and an ideal response or answer. For example,
we can have an eval sample where the input is "What year was Obama elected president for the first
time?" and the ideal answer is "2008". We feed the input to a model and get the completion. If the model
says "2008", it is then graded as correct. We can write a string match to check if the completion includes the phrase "2008". If it does, we consider it correct.

Consider another eval where the input is to generate valid JSON: We can write some code that
attempts to parse the completion as JSON and then considers the completion correct if it is
parsable.

**Model grading: A two stage process where the model first answers the question, then we ask a
model to look at the response to check if it’s correct.**

Consider an input that asks the model to write a funny joke. The model then generates a
completion. We then create a new input to the model to answer the question: "Is this following
joke funny? First reason step by step, then answer yes or no" that includes the completion." We
finally consider the original completion correct if the new model completion ends with "yes".

Model grading works best with the latest, most powerful models like `GPT-4` and if we give them the ability
to reason before making a judgment.


key: _node_content
string_value:
{"id_": "8e279319-1b53-4520-94b7-920079f6b92c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2567e2c8-3eab-4afc-bc7a-0c8f59604ffc", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "38029a5b18c45432a1b640575a06a0261f607c874da72bab845523d1d2ab7152", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c5f8d067-d7d7-46cc-ad15-e0d24048b356", "node_type": "1", "metadata": {}, "hash": "0fc158e077f087629de75301c3a8ad9ef30edf09d88ff0e6a6816311e60c4f1a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1711, "end_char_idx": 4725, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can also make evals a part of your CI/CD pipeline to make sure you achieve the desired accuracy before deploying.

#### Types of evals

There are two main ways we can evaluate/grade completions: writing some validation logic in code
or using the model itself to inspect the answer. We’ll introduce each with some examples.

**Writing logic for answer checking**

The simplest and most common type of eval has an input and an ideal response or answer. For example,
we can have an eval sample where the input is "What year was Obama elected president for the first
time?" and the ideal answer is "2008". We feed the input to a model and get the completion. If the model
says "2008", it is then graded as correct. We can write a string match to check if the completion includes the phrase "2008". If it does, we consider it correct.

Consider another eval where the input is to generate valid JSON: We can write some code that
attempts to parse the completion as JSON and then considers the completion correct if it is
parsable.

**Model grading: A two stage process where the model first answers the question, then we ask a
model to look at the response to check if it’s correct.**

Consider an input that asks the model to write a funny joke. The model then generates a
completion. We then create a new input to the model to answer the question: "Is this following
joke funny? First reason step by step, then answer yes or no" that includes the completion." We
finally consider the original completion correct if the new model completion ends with "yes".

Model grading works best with the latest, most powerful models like `GPT-4` and if we give them the ability
to reason before making a judgment. Model grading will have an error rate, so it is important to validate
the performance with human evaluation before running the evals at scale. For best results, it makes
sense to use a different model to do grading from the one that did the completion, like using `GPT-4` to
grade `GPT-3.5` answers.

#### OpenAI Eval Templates

In using evals, we have discovered several "templates" that accommodate many different benchmarks. We have implemented these templates in the OpenAI Evals library to simplify the development of new evals. For example, we have defined 2 types of eval templates that can be used out of the box:

* **Basic Eval Templates**: These contain deterministic functions to compare the output to the ideal_answers. In cases where the desired model response has very little variation, such as answering multiple choice questions or simple questions with a straightforward answer, we have found this following templates to be useful.

* **Model-Graded Templates**: These contain functions where an LLM compares the output to the ideal_answers and attempts to judge the accuracy. In cases where the desired model response can contain significant variation, such as answering an open-ended question, we have found that using the model to grade itself is a viable strategy for automated evaluation.


key: _node_content
string_value:
{"id_": "c5f8d067-d7d7-46cc-ad15-e0d24048b356", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8e279319-1b53-4520-94b7-920079f6b92c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "cb89f44b483d26f496649d703364a80a49127560da5172618a243c0bb6ca2e30", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1c5eddf3-cb78-43ba-a4d7-b9e47761549a", "node_type": "1", "metadata": {}, "hash": "ef15426fa16a5af3c64c0c7b38afeb6eb2d83277228ce31476ef3c04a0a2e8e7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2892, "end_char_idx": 6042, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
**

Consider an input that asks the model to write a funny joke. The model then generates a
completion. We then create a new input to the model to answer the question: "Is this following
joke funny? First reason step by step, then answer yes or no" that includes the completion." We
finally consider the original completion correct if the new model completion ends with "yes".

Model grading works best with the latest, most powerful models like `GPT-4` and if we give them the ability
to reason before making a judgment. Model grading will have an error rate, so it is important to validate
the performance with human evaluation before running the evals at scale. For best results, it makes
sense to use a different model to do grading from the one that did the completion, like using `GPT-4` to
grade `GPT-3.5` answers.

#### OpenAI Eval Templates

In using evals, we have discovered several "templates" that accommodate many different benchmarks. We have implemented these templates in the OpenAI Evals library to simplify the development of new evals. For example, we have defined 2 types of eval templates that can be used out of the box:

* **Basic Eval Templates**: These contain deterministic functions to compare the output to the ideal_answers. In cases where the desired model response has very little variation, such as answering multiple choice questions or simple questions with a straightforward answer, we have found this following templates to be useful.

* **Model-Graded Templates**: These contain functions where an LLM compares the output to the ideal_answers and attempts to judge the accuracy. In cases where the desired model response can contain significant variation, such as answering an open-ended question, we have found that using the model to grade itself is a viable strategy for automated evaluation.


### Getting Setup

First, go to [github.com/openai/evals](https://github.com/openai/evals), clone the repository with `git clone git@github.com:openai/evals.git` and go through the [setup instructions](https://github.com/openai/evals). 

To run evals later in this notebook, you will need to set up and specify your OpenAI API key. After you obtain an API key, specify it using the `OPENAI_API_KEY` environment variable. 

Please be aware of the costs associated with using the API when running evals.


```python
from openai import OpenAI
import pandas as pd

client = OpenAI()
```

## Building an evaluation for OpenAI Evals framework

At its core, an eval is a dataset and an eval class that is defined in a YAML file. To start creating an eval, we need

1. The test dataset in the `jsonl` format.
2. The eval template to be used

### Creating the eval dataset
Lets create a dataset for a use case where we are evaluating the model's ability to generate syntactically correct SQL. In this use case, we have a series of tables that are related to car manufacturing

First we will need to create a system prompt that we would like to evaluate. We will pass in instructions for the model as well as an overview of the table structure:
```
"TASK: Answer the following question with syntactically correct SQLite SQL.


key: _node_content
string_value:
{"id_": "1c5eddf3-cb78-43ba-a4d7-b9e47761549a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c5f8d067-d7d7-46cc-ad15-e0d24048b356", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "574d841494e4d3462013d80195796b1eb123d1177ea85cc8d7c5e7d0de4e56b6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fd3b21de-d0f2-4574-a9ad-481995d3187f", "node_type": "1", "metadata": {}, "hash": "098fa12002a403201c2dc3ce999656ad9eb8764397035ddb0aa8c39930dd9500", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4365, "end_char_idx": 7321, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
* **Model-Graded Templates**: These contain functions where an LLM compares the output to the ideal_answers and attempts to judge the accuracy. In cases where the desired model response can contain significant variation, such as answering an open-ended question, we have found that using the model to grade itself is a viable strategy for automated evaluation.


### Getting Setup

First, go to [github.com/openai/evals](https://github.com/openai/evals), clone the repository with `git clone git@github.com:openai/evals.git` and go through the [setup instructions](https://github.com/openai/evals). 

To run evals later in this notebook, you will need to set up and specify your OpenAI API key. After you obtain an API key, specify it using the `OPENAI_API_KEY` environment variable. 

Please be aware of the costs associated with using the API when running evals.


```python
from openai import OpenAI
import pandas as pd

client = OpenAI()
```

## Building an evaluation for OpenAI Evals framework

At its core, an eval is a dataset and an eval class that is defined in a YAML file. To start creating an eval, we need

1. The test dataset in the `jsonl` format.
2. The eval template to be used

### Creating the eval dataset
Lets create a dataset for a use case where we are evaluating the model's ability to generate syntactically correct SQL. In this use case, we have a series of tables that are related to car manufacturing

First we will need to create a system prompt that we would like to evaluate. We will pass in instructions for the model as well as an overview of the table structure:
```
"TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]"
```

For this prompt, we can ask a specific question:
```
"Q: how many car makers are their in germany?"
```

And we have an expected answer:
```
"A: SELECT count ( * )  FROM CAR_MAKERS AS T1 JOIN COUNTRIES AS T2 ON T1.Country   =   T2.CountryId WHERE T2.CountryName   =   'germany'"
```

The dataset needs to be in the following format:
```
"input": [{"role": "system", "content": "<input prompt>"}, {"role": "user", "content": <user input>}, "ideal": "correct answer"]
```

Putting it all together, we get:
```
{"input": [{"role": "system", "content": "TASK: Answer the following question with syntactically correct SQLite SQL.


key: _node_content
string_value:
{"id_": "fd3b21de-d0f2-4574-a9ad-481995d3187f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1c5eddf3-cb78-43ba-a4d7-b9e47761549a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "e19ffa91b3260dd2e52de284c303fb26d26e6a0dd2ff0c0a3b46a2bfded16f56", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a3e2f604-2514-4bc5-97a7-eed051d6eddb", "node_type": "1", "metadata": {}, "hash": "a7179d32f86a295a13789a06d4f3696e44fedf34b12da732b4b2e2b6d6a01d54", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5873, "end_char_idx": 8791, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We will pass in instructions for the model as well as an overview of the table structure:
```
"TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]"
```

For this prompt, we can ask a specific question:
```
"Q: how many car makers are their in germany?"
```

And we have an expected answer:
```
"A: SELECT count ( * )  FROM CAR_MAKERS AS T1 JOIN COUNTRIES AS T2 ON T1.Country   =   T2.CountryId WHERE T2.CountryName   =   'germany'"
```

The dataset needs to be in the following format:
```
"input": [{"role": "system", "content": "<input prompt>"}, {"role": "user", "content": <user input>}, "ideal": "correct answer"]
```

Putting it all together, we get:
```
{"input": [{"role": "system", "content": "TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]\n"}, {"role": "system", "content": "Q: how many car makers are their in germany"}, "ideal": ["A: SELECT count ( * )  FROM CAR_MAKERS AS T1 JOIN COUNTRIES AS T2 ON T1.Country   =   T2.CountryId WHERE T2.CountryName   =   'germany'"]}
```


One way to speed up the process of building eval datasets, is to use `GPT-4` to generate synthetic data


```python
## Use GPT-4 to generate synthetic data
# Define the system prompt and user input (these should be filled as per the specific use case)
system_prompt = """You are a helpful assistant that can ask questions about a database table and write SQL queries to answer the question.
    A user will pass in a table schema and your job is to return a question answer pairing. The question should relevant to the schema of the table,
    and you can speculate on its contents.


key: _node_content
string_value:
{"id_": "a3e2f604-2514-4bc5-97a7-eed051d6eddb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fd3b21de-d0f2-4574-a9ad-481995d3187f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8356451ce944a77314e2b0abb1aa8dbd35df75048f2e97a5ad8ce240eba918e0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1efd2a70-e498-4dee-8aa0-479a84c403a6", "node_type": "1", "metadata": {}, "hash": "944dd52fafe091ffa8cacc3ed42efca11af9cbbaae0e63d1dd6f8567fc92b130", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7322, "end_char_idx": 9957, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]\n"}, {"role": "system", "content": "Q: how many car makers are their in germany"}, "ideal": ["A: SELECT count ( * )  FROM CAR_MAKERS AS T1 JOIN COUNTRIES AS T2 ON T1.Country   =   T2.CountryId WHERE T2.CountryName   =   'germany'"]}
```


One way to speed up the process of building eval datasets, is to use `GPT-4` to generate synthetic data


```python
## Use GPT-4 to generate synthetic data
# Define the system prompt and user input (these should be filled as per the specific use case)
system_prompt = """You are a helpful assistant that can ask questions about a database table and write SQL queries to answer the question.
    A user will pass in a table schema and your job is to return a question answer pairing. The question should relevant to the schema of the table,
    and you can speculate on its contents. You will then have to generate a SQL query to answer the question. Below are some examples of what this should look like.

    Example 1
    ```````````
    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\n
    Assistant Response:
    Q: How many visitors have visited the museum with the most staff?
    A: SELECT count ( * )  FROM VISIT AS T1 JOIN MUSEUM AS T2 ON T1.Museum_ID   =   T2.Museum_ID WHERE T2.Num_of_Staff   =   ( SELECT max ( Num_of_Staff )  FROM MUSEUM ) 
    ```````````

    Example 2
    ```````````
    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\n
    Assistant Response:
    Q: What are the names who have a membership level higher than 4?


key: _node_content
string_value:
{"id_": "1efd2a70-e498-4dee-8aa0-479a84c403a6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a3e2f604-2514-4bc5-97a7-eed051d6eddb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "eeec75448ce3ead6304e2bb70d57889ac18db5235cfedca50b487978f2195d6a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "aa37f1c1-619f-4740-bf58-6aeeb8330305", "node_type": "1", "metadata": {}, "hash": "b20c3a7abd7d0dd5b2d836298bfb85360eeb3498b6d85f8a6af5dc2f9a7883e0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8604, "end_char_idx": 10452, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
A user will pass in a table schema and your job is to return a question answer pairing. The question should relevant to the schema of the table,
    and you can speculate on its contents. You will then have to generate a SQL query to answer the question. Below are some examples of what this should look like.

    Example 1
    ```````````
    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\n
    Assistant Response:
    Q: How many visitors have visited the museum with the most staff?
    A: SELECT count ( * )  FROM VISIT AS T1 JOIN MUSEUM AS T2 ON T1.Museum_ID   =   T2.Museum_ID WHERE T2.Num_of_Staff   =   ( SELECT max ( Num_of_Staff )  FROM MUSEUM ) 
    ```````````

    Example 2
    ```````````
    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\n
    Assistant Response:
    Q: What are the names who have a membership level higher than 4?
    A: SELECT Name   FROM VISITOR AS T1 WHERE T1.Level_of_membership   >   4 
    ```````````

    Example 3
    ```````````
    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\n
    Assistant Response:
    Q: How many tickets of customer id 5?


key: _node_content
string_value:
{"id_": "aa37f1c1-619f-4740-bf58-6aeeb8330305", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1efd2a70-e498-4dee-8aa0-479a84c403a6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "5a4a99c1dfca9ed8ad0352124a203601e14d8b94ec13612adde14aa4d4eee072", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f3504532-298c-4ee5-886a-81188d6fbd42", "node_type": "1", "metadata": {}, "hash": "36b266f7fb045846f1558880bb533a39b93c3c4c5d4c17b4d9c8482a210e7bf2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9347, "end_char_idx": 12032, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
A: SELECT count ( * )  FROM VISIT AS T1 JOIN MUSEUM AS T2 ON T1.Museum_ID   =   T2.Museum_ID WHERE T2.Num_of_Staff   =   ( SELECT max ( Num_of_Staff )  FROM MUSEUM ) 
    ```````````

    Example 2
    ```````````
    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\n
    Assistant Response:
    Q: What are the names who have a membership level higher than 4?
    A: SELECT Name   FROM VISITOR AS T1 WHERE T1.Level_of_membership   >   4 
    ```````````

    Example 3
    ```````````
    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\n
    Assistant Response:
    Q: How many tickets of customer id 5?
    A: SELECT count ( * )  FROM VISIT AS T1 JOIN VISITOR AS T2 ON T1.visitor_ID   =   T2.ID WHERE T2.ID   =   5 
    ```````````
    """

user_input = "Table car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]"

messages = [{
        "role": "system",
        "content": system_prompt
    },
    {
        "role": "user",
        "content": user_input
    }
]

completion = client.chat.completions.create(
    model="gpt-4-turbo-preview",
    messages=messages,
    temperature=0.7,
    n=5
)

for choice in completion.choices:
    print(choice.message.content + "\n")

```

    Q: What is the average horsepower for cars made in Europe?
    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'
    
    Q: What is the average horsepower for cars made in the USA?


key: _node_content
string_value:
{"id_": "f3504532-298c-4ee5-886a-81188d6fbd42", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "aa37f1c1-619f-4740-bf58-6aeeb8330305", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "85b4c5371c3bb17dbc8099bd6eaa26b3a14eff3e793ef654b45c38fe7e84f368", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3fd15893-b9d1-4841-bf1d-435d65cd61b1", "node_type": "1", "metadata": {}, "hash": "b11368cbd5e605f39b7c48292cbf7cce0831e71d58e51a566ce46279fe839d9b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11601, "end_char_idx": 13747, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'
    
    Q: What is the average horsepower for cars made in the USA?
    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'
    
    Q: What is the average horsepower for cars produced in countries from the continent with the id '3'?
    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.ContId = '3'
    
    Q: What is the average horsepower for cars made by makers from Europe?
    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'
    
    Q: What is the average horsepower for cars made in the USA?
    
    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'
    


Once we have the synthetic data, we need to convert it to match the format of the eval dataset.


```python
eval_data = []
input_prompt = "TASK: Answer the following question with syntactically correct SQLite SQL.


key: _node_content
string_value:
{"id_": "3fd15893-b9d1-4841-bf1d-435d65cd61b1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f3504532-298c-4ee5-886a-81188d6fbd42", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0006e3efd6d39948546e561c9e61c61cd06b236878d5d6f4e43191a77590f4ef", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "53856a56-a0dd-4293-b7b4-75a33d1d148c", "node_type": "1", "metadata": {}, "hash": "2b450c9ac0bbdae406b9891bad4de9e27a58db3b165a5b0f405ba1c9eb765058", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12037, "end_char_idx": 14979, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'
    
    Q: What is the average horsepower for cars produced in countries from the continent with the id '3'?
    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.ContId = '3'
    
    Q: What is the average horsepower for cars made by makers from Europe?
    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'
    
    Q: What is the average horsepower for cars made in the USA?
    
    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'
    


Once we have the synthetic data, we need to convert it to match the format of the eval dataset.


```python
eval_data = []
input_prompt = "TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]"

for choice in completion.choices:
    question = choice.message.content.split("Q: ")[1].split("\n")[0]  # Extracting the question
    answer = choice.message.content.split("\nA: ")[1].split("\n")[0]  # Extracting the answer
    eval_data.append({
        "input": [
            {"role": "system", "content": input_prompt},
            {"role": "user", "content": question},
        ],
        "ideal": answer
    })

for item in eval_data:
    print(item)
```

    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL.


key: _node_content
string_value:
{"id_": "53856a56-a0dd-4293-b7b4-75a33d1d148c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3fd15893-b9d1-4841-bf1d-435d65cd61b1", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4ecc462842f51fa1d62c7b941ef819b649c02517db7cc39dcda6c4af175edb64", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0d591b21-c643-4af2-a8d4-a95faa9b4da7", "node_type": "1", "metadata": {}, "hash": "5ac2d4cdb10c785db2011cd46194ded7b48ac4d8a3a3804c2a043b8e3ba77aea", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13534, "end_char_idx": 16212, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Once we have the synthetic data, we need to convert it to match the format of the eval dataset.


```python
eval_data = []
input_prompt = "TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]"

for choice in completion.choices:
    question = choice.message.content.split("Q: ")[1].split("\n")[0]  # Extracting the question
    answer = choice.message.content.split("\nA: ")[1].split("\n")[0]  # Extracting the answer
    eval_data.append({
        "input": [
            {"role": "system", "content": input_prompt},
            {"role": "user", "content": question},
        ],
        "ideal": answer
    })

for item in eval_data:
    print(item)
```

    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in Europe?'}], 'ideal': "SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'"}
    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL.


key: _node_content
string_value:
{"id_": "0d591b21-c643-4af2-a8d4-a95faa9b4da7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "53856a56-a0dd-4293-b7b4-75a33d1d148c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "17f2f92d33fb1019cb102f93322912962622a079e9ed77b3e14c07722c20a845", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8defaecf-2d13-464d-9608-caf787e60763", "node_type": "1", "metadata": {}, "hash": "0759faa0866775bc7fd049a62d7ebf20e2e96bcb9f6eb5e0036a653002f4b05d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14980, "end_char_idx": 17331, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in Europe?'}], 'ideal': "SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'"}
    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in the USA?'}], 'ideal': "SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'"}
    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL.


key: _node_content
string_value:
{"id_": "8defaecf-2d13-464d-9608-caf787e60763", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0d591b21-c643-4af2-a8d4-a95faa9b4da7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "e5930df01517ce71a58f03767cf0aa626db1a5ed0b356aac526e6c24108b816c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f9c7dba1-5ce6-49b5-9d0d-2bdddcc1311a", "node_type": "1", "metadata": {}, "hash": "feedee288015d6864fb34882ead95610e365e753a7191adbaffdf52914fa3c2a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16213, "end_char_idx": 18598, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in the USA?'}], 'ideal': "SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'"}
    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': "What is the average horsepower for cars produced in countries from the continent with the id '3'?"}], 'ideal': "SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.ContId = '3'"}
    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL.


key: _node_content
string_value:
{"id_": "f9c7dba1-5ce6-49b5-9d0d-2bdddcc1311a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8defaecf-2d13-464d-9608-caf787e60763", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4137499b1e28ebd3cc5afd3c435fa2cce5bc70d8077928be5c94edfcd56f88d7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c76a37ab-efad-4764-af98-9602329aeea7", "node_type": "1", "metadata": {}, "hash": "2eeb961643986f96eaa89dfd6bcce8bc5c156ce5ebd6299b46993006b5d6c333", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17332, "end_char_idx": 19843, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': "What is the average horsepower for cars produced in countries from the continent with the id '3'?"}], 'ideal': "SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.ContId = '3'"}
    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made by makers from Europe?'}], 'ideal': "SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'"}
    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL.


key: _node_content
string_value:
{"id_": "c76a37ab-efad-4764-af98-9602329aeea7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f9c7dba1-5ce6-49b5-9d0d-2bdddcc1311a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "ed1f78af58ea25d5906f91b13cda30ca05735ebedb6b278d04580cca00235d5b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f10e4292-02c7-4327-abe0-2292c0ff3a55", "node_type": "1", "metadata": {}, "hash": "f4101e8d4f54af493be4f6b69555c21d9f985f5a3be8fbfb485ad596ca6c88e0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18599, "end_char_idx": 21288, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made by makers from Europe?'}], 'ideal': "SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'"}
    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in the USA?'}], 'ideal': "SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'"}


Next we need to create the eval registry to run it in the framework.

The evals framework requires a `.yaml` file structured with the following properties:
* `id` - An identifier for your eval
* `description` - A short description of your eval
* `disclaimer` - An additional notes about your eval
* `metrics` - There are three types of eval metrics we can choose from: match, includes, fuzzyMatch

For our eval, we will configure the following:


key: _node_content
string_value:
{"id_": "f10e4292-02c7-4327-abe0-2292c0ff3a55", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c76a37ab-efad-4764-af98-9602329aeea7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c0bab94aea84fbc8e14088a14d648be4d5e01de3895b0e280bdc9e79d6213c1a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bb8a833d-7d0e-4a54-a01f-f16cfaf19abe", "node_type": "1", "metadata": {}, "hash": "056e66da09a52a90c9b9f0370fe26e54477be31ae1e67d339493a333e7063741", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19844, "end_char_idx": 22364, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in the USA?'}], 'ideal': "SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'"}


Next we need to create the eval registry to run it in the framework.

The evals framework requires a `.yaml` file structured with the following properties:
* `id` - An identifier for your eval
* `description` - A short description of your eval
* `disclaimer` - An additional notes about your eval
* `metrics` - There are three types of eval metrics we can choose from: match, includes, fuzzyMatch

For our eval, we will configure the following:


```python
"""
spider-sql:
  id: spider-sql.dev.v0
  metrics: [accuracy]
  description: Eval that scores SQL code from 194 examples in the Spider Text-to-SQL test dataset. The problems are selected by taking the first 10 problems for each database that appears in the test set.
    Yu, Tao, et al. \"Spider; A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task.\" Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, https://doi.org/10.18653/v1/d18-1425.
  disclaimer: Problems are solved zero-shot with no prompting other than the schema; performance may improve with training examples, fine tuning, or a different schema format. Evaluation is currently done through model-grading, where SQL code is not actually executed; the model may judge correct SQL to be incorrect, or vice-versa.
spider-sql.dev.v0:
  class: evals.elsuite.modelgraded.classify:ModelBasedClassify
  args:
    samples_jsonl: sql/spider_sql.jsonl
    eval_type: cot_classify
    modelgraded_spec: sql
  """""
```


key: _node_content
string_value:
{"id_": "bb8a833d-7d0e-4a54-a01f-f16cfaf19abe", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f10e4292-02c7-4327-abe0-2292c0ff3a55", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "7dd70446cef1ed5aae510c0af2023c004dd7f661f1d09966a7cba2f1ae78bdae", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "70a9c89f-4d30-482c-8939-78f5535ad9e7", "node_type": "1", "metadata": {}, "hash": "916cae0985b0e2eb687e7d89eed355ea3537abfedd50a5ef0e6545f213fbf667", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20844, "end_char_idx": 23437, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Next we need to create the eval registry to run it in the framework.

The evals framework requires a `.yaml` file structured with the following properties:
* `id` - An identifier for your eval
* `description` - A short description of your eval
* `disclaimer` - An additional notes about your eval
* `metrics` - There are three types of eval metrics we can choose from: match, includes, fuzzyMatch

For our eval, we will configure the following:


```python
"""
spider-sql:
  id: spider-sql.dev.v0
  metrics: [accuracy]
  description: Eval that scores SQL code from 194 examples in the Spider Text-to-SQL test dataset. The problems are selected by taking the first 10 problems for each database that appears in the test set.
    Yu, Tao, et al. \"Spider; A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task.\" Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, https://doi.org/10.18653/v1/d18-1425.
  disclaimer: Problems are solved zero-shot with no prompting other than the schema; performance may improve with training examples, fine tuning, or a different schema format. Evaluation is currently done through model-grading, where SQL code is not actually executed; the model may judge correct SQL to be incorrect, or vice-versa.
spider-sql.dev.v0:
  class: evals.elsuite.modelgraded.classify:ModelBasedClassify
  args:
    samples_jsonl: sql/spider_sql.jsonl
    eval_type: cot_classify
    modelgraded_spec: sql
  """""
```




    '\nspider-sql:\n  id: spider-sql.dev.v0\n  metrics: [accuracy]\n  description: Eval that scores SQL code from 194 examples in the Spider Text-to-SQL test dataset. The problems are selected by taking the first 10 problems for each database that appears in the test set.\n    Yu, Tao, et al. "Spider; A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task." Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, https://doi.org/10.18653/v1/d18-1425.\n  disclaimer: Problems are solved zero-shot with no prompting other than the schema; performance may improve with training examples, fine tuning, or a different schema format. Evaluation is currently done through model-grading, where SQL code is not actually executed; the model may judge correct SQL to be incorrect, or vice-versa.\nspider-sql.dev.v0:\n  class: evals.elsuite.modelgraded.classify:ModelBasedClassify\n  args:\n    samples_jsonl: sql/spider_sql.jsonl\n    eval_type: cot_classify\n    modelgraded_spec: sql\n  '


key: _node_content
string_value:
{"id_": "70a9c89f-4d30-482c-8939-78f5535ad9e7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bb8a833d-7d0e-4a54-a01f-f16cfaf19abe", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "73f9b4e79a92391ad24e05b60f3af73836c371131cb0f0bed589bf2f1268e78f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d3dc48c6-5477-4ab2-b491-8c1898df04d0", "node_type": "1", "metadata": {}, "hash": "c88731d5079dd1904859263f84017b8c722a22fd922d538a3b3b88189dd88dbc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 22373, "end_char_idx": 25130, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
'\nspider-sql:\n  id: spider-sql.dev.v0\n  metrics: [accuracy]\n  description: Eval that scores SQL code from 194 examples in the Spider Text-to-SQL test dataset. The problems are selected by taking the first 10 problems for each database that appears in the test set.\n    Yu, Tao, et al. "Spider; A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task." Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, https://doi.org/10.18653/v1/d18-1425.\n  disclaimer: Problems are solved zero-shot with no prompting other than the schema; performance may improve with training examples, fine tuning, or a different schema format. Evaluation is currently done through model-grading, where SQL code is not actually executed; the model may judge correct SQL to be incorrect, or vice-versa.\nspider-sql.dev.v0:\n  class: evals.elsuite.modelgraded.classify:ModelBasedClassify\n  args:\n    samples_jsonl: sql/spider_sql.jsonl\n    eval_type: cot_classify\n    modelgraded_spec: sql\n  '



## Running an evaluation

We can run this eval using the `oaieval` CLI. To get setup, install the library: `pip install .` (if you are running the [OpenAI Evals library](github.com/openai/evals) locally) or `pip install oaieval` if you are running an existing eval.

Then, run the eval using the CLI: `oaieval gpt-3.5-turbo spider-sql`

This command expects a model name and an eval set name. Note that we provide two command line interfaces (CLIs): `oaieval` for running a single eval and `oaievalset` for running a set of evals. The valid eval names are specified in the YAML files under `evals/registry/evals` and their corresponding implementations can be found in `evals/elsuite`.


```python
!pip install evals --quiet
```

The `oaieval` CLI can accept various flags to modify the default behavior. You can run `oaieval --help` to see a full list of CLI options. 

After running that command, you’ll see the final report of accuracy printed to the console, as well as a file path to a temporary file that contains the full report.

These CLIs can accept various flags to modify their default behavior. You can run `oaieval --help` to see a full list of CLI options. 

`oaieval` will search for the `spider-sql` eval YAML file in the `evals/registry/evals` directory, following the format specified in cell 4 above. The path to the eval dataset is specified in the eval YAML file under the args: parameter as `samples_jsonl: sql/spider_sql.jsonl`, with the file content in JSONL format (as generated in step 3 above).

After running that command, you’ll see the final report of accuracy printed to the console, as well as a file path to a temporary file that contains the full report.


key: _node_content
string_value:
{"id_": "d3dc48c6-5477-4ab2-b491-8c1898df04d0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "70a9c89f-4d30-482c-8939-78f5535ad9e7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "6cb52d561c1c1f6423e78ae5ef41ce2c8e8983dcd7391c47aff9f6bd4bedb4f4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "22a9c5d8-0e1e-4a83-afde-99da74f7eccd", "node_type": "1", "metadata": {}, "hash": "0cafe37e6073396b41848498fbb85c6ebd20393fd8494a86f449b6b5ace42e5c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24129, "end_char_idx": 26532, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
!pip install evals --quiet
```

The `oaieval` CLI can accept various flags to modify the default behavior. You can run `oaieval --help` to see a full list of CLI options. 

After running that command, you’ll see the final report of accuracy printed to the console, as well as a file path to a temporary file that contains the full report.

These CLIs can accept various flags to modify their default behavior. You can run `oaieval --help` to see a full list of CLI options. 

`oaieval` will search for the `spider-sql` eval YAML file in the `evals/registry/evals` directory, following the format specified in cell 4 above. The path to the eval dataset is specified in the eval YAML file under the args: parameter as `samples_jsonl: sql/spider_sql.jsonl`, with the file content in JSONL format (as generated in step 3 above).

After running that command, you’ll see the final report of accuracy printed to the console, as well as a file path to a temporary file that contains the full report.


```python
!oaieval gpt-3.5-turbo spider-sql --max_samples 25
```

    [2024-03-26 19:44:39,836] [registry.py:257] Loading registry from /Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry/evals
    [2024-03-26 19:44:43,623] [registry.py:257] Loading registry from /Users/shyamal/.evals/evals
    [2024-03-26 19:44:43,635] [oaieval.py:189] [1;35mRun started: 240327024443FACXGMKA[0m
    [2024-03-26 19:44:43,663] [registry.py:257] Loading registry from /Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry/modelgraded
    [2024-03-26 19:44:43,851] [registry.py:257] Loading registry from /Users/shyamal/.evals/modelgraded
    [2024-03-26 19:44:43,853] [data.py:90] Fetching /Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry/data/sql/spider_sql.jsonl
    [2024-03-26 19:44:43,878] [eval.py:36] Evaluating 25 samples
    [2024-03-26 19:44:43,952] [eval.py:144] Running in threaded mode with 10 threads!
      0%|                                                    | 0/25 [00:00<?, ?it/s][2024-03-26 19:44:44,810] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:44,829] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:44,991] [_client.py:1026] HTTP Request: POST https://api.


key: _node_content
string_value:
{"id_": "22a9c5d8-0e1e-4a83-afde-99da74f7eccd", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d3dc48c6-5477-4ab2-b491-8c1898df04d0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "96f3c442809380861c315b93b399e78554300a75b29623c9a19337f61df53596", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a057d48b-e079-4eea-b89b-61c1076640bb", "node_type": "1", "metadata": {}, "hash": "081235968d0e797ca990c2399a521982c7e437b5f730de75a4af4662fb794ee5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 26123, "end_char_idx": 28011, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0%|                                                    | 0/25 [00:00<?, ?it/s][2024-03-26 19:44:44,810] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:44,829] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:44,991] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:45,090] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:45,145] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:45,971] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:46,040] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:46,069] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:46,378] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:46,587] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:47,412] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
      4%|█▊                                          | 1/25 [00:03<01:23,  3.46s/it][2024-03-26 19:44:47,714] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
      8%|███▌                                        | 2/25 [00:03<00:36,  1.60s/it][2024-03-26 19:44:47,947] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.


key: _node_content
string_value:
{"id_": "a057d48b-e079-4eea-b89b-61c1076640bb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "22a9c5d8-0e1e-4a83-afde-99da74f7eccd", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "11c6361da9251d3cd2ba3282aa24d66b0ced13e87bbd0da3075692a89caaa2ec", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "978de80f-e0a5-4a49-b9c8-19064d666150", "node_type": "1", "metadata": {}, "hash": "db366f425162c8b26059e702eeec766dcdd8d5f223f0f4222a143afb335fa1f8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27044, "end_char_idx": 28926, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:46,069] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:46,378] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:46,587] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:47,412] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
      4%|█▊                                          | 1/25 [00:03<01:23,  3.46s/it][2024-03-26 19:44:47,714] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
      8%|███▌                                        | 2/25 [00:03<00:36,  1.60s/it][2024-03-26 19:44:47,947] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     12%|█████▎                                      | 3/25 [00:03<00:21,  1.02it/s][2024-03-26 19:44:48,413] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:48,643] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     16%|███████                                     | 4/25 [00:04<00:18,  1.15it/s][2024-03-26 19:44:48,909] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     20%|████████▊                                   | 5/25 [00:04<00:12,  1.54it/s][2024-03-26 19:44:49,131] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:49,500] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:49,


key: _node_content
string_value:
{"id_": "978de80f-e0a5-4a49-b9c8-19064d666150", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a057d48b-e079-4eea-b89b-61c1076640bb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f8ce2ab67d930b434fb8e8540ec8fa019a319b00d39a9967f8a340969a8d2b54", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3e0feefa-8c9b-4c32-af4b-ad68d4159a0a", "node_type": "1", "metadata": {}, "hash": "57a75db204086cf2fa4fe1c71627c2c33d8915580d7201ceb300d6824ea63b00", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27972, "end_char_idx": 29854, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     12%|█████▎                                      | 3/25 [00:03<00:21,  1.02it/s][2024-03-26 19:44:48,413] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:48,643] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     16%|███████                                     | 4/25 [00:04<00:18,  1.15it/s][2024-03-26 19:44:48,909] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     20%|████████▊                                   | 5/25 [00:04<00:12,  1.54it/s][2024-03-26 19:44:49,131] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:49,500] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:49,530] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     24%|██████████▌                                 | 6/25 [00:05<00:12,  1.56it/s][2024-03-26 19:44:49,962] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:49,964] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:49,967] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     28%|████████████▎                               | 7/25 [00:06<00:10,  1.73it/s][2024-03-26 19:44:50,577] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:50,602] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:50,


key: _node_content
string_value:
{"id_": "3e0feefa-8c9b-4c32-af4b-ad68d4159a0a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "978de80f-e0a5-4a49-b9c8-19064d666150", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "117aad98c637725c9d4a82276ba547a06c92f677ef9f87951c0ab4efbaa38f65", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "88859d62-d2d1-458c-9949-85eb9abf2bd0", "node_type": "1", "metadata": {}, "hash": "51de4367ae40c6512905d30f1b3b99adce02a0a291ab512d027c14ca99febed4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 28891, "end_char_idx": 30747, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
1 200 OK"
    [2024-03-26 19:44:49,530] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     24%|██████████▌                                 | 6/25 [00:05<00:12,  1.56it/s][2024-03-26 19:44:49,962] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:49,964] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:49,967] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     28%|████████████▎                               | 7/25 [00:06<00:10,  1.73it/s][2024-03-26 19:44:50,577] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:50,602] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:50,634] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:50,862] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:51,503] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:51,608] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     40%|█████████████████▏                         | 10/25 [00:07<00:08,  1.79it/s][2024-03-26 19:44:51,801] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     44%|██████████████████▉                        | 11/25 [00:07<00:06,  2.09it/s][2024-03-26 19:44:51,856] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.


key: _node_content
string_value:
{"id_": "88859d62-d2d1-458c-9949-85eb9abf2bd0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3e0feefa-8c9b-4c32-af4b-ad68d4159a0a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "70afdefdc12f4a25bffe98095dc610d0afa19c92b566d94a26e8c90869710122", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a652e4cc-52aa-4c04-acad-c8eec6629fa5", "node_type": "1", "metadata": {}, "hash": "f8306eec8985f16e0a9a4ee47cd9a2ccd16d9aff13d9cbd82a093be64dd1e450", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29780, "end_char_idx": 31636, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:50,634] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:50,862] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:51,503] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:51,608] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     40%|█████████████████▏                         | 10/25 [00:07<00:08,  1.79it/s][2024-03-26 19:44:51,801] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     44%|██████████████████▉                        | 11/25 [00:07<00:06,  2.09it/s][2024-03-26 19:44:51,856] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:51,969] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:52,227] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     52%|██████████████████████▎                    | 13/25 [00:08<00:04,  2.65it/s][2024-03-26 19:44:52,450] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:52,526] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:52,615] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     56%|████████████████████████                   | 14/25 [00:08<00:04,  2.64it/s][2024-03-26 19:44:52,625] [_client.py:1026] HTTP Request: POST https://api.


key: _node_content
string_value:
{"id_": "a652e4cc-52aa-4c04-acad-c8eec6629fa5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "88859d62-d2d1-458c-9949-85eb9abf2bd0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "e72051191e6ef2aa1deb6f9c5d5a8fa74ed70bd12f05786c8ad5f3d119a83f4c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3c7e9a10-cf24-4986-9ee5-38954a534246", "node_type": "1", "metadata": {}, "hash": "7f901a8ee5ed0a6397597abe944a5b433ddd0c7cc86e71208e4597435b045557", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30668, "end_char_idx": 32510, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:51,969] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:52,227] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     52%|██████████████████████▎                    | 13/25 [00:08<00:04,  2.65it/s][2024-03-26 19:44:52,450] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:52,526] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:52,615] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     56%|████████████████████████                   | 14/25 [00:08<00:04,  2.64it/s][2024-03-26 19:44:52,625] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:52,777] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:53,653] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     60%|█████████████████████████▊                 | 15/25 [00:09<00:05,  1.87it/s][2024-03-26 19:44:53,670] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:54,028] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     68%|█████████████████████████████▏             | 17/25 [00:10<00:03,  2.54it/s][2024-03-26 19:44:54,388] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:54,


key: _node_content
string_value:
{"id_": "3c7e9a10-cf24-4986-9ee5-38954a534246", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a652e4cc-52aa-4c04-acad-c8eec6629fa5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "fa8c0206957e6eae8970e32ae1468b8ec5d06669c321b0910b282255130b2bc9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "84a15dcd-980e-4137-8b25-7be916d9734d", "node_type": "1", "metadata": {}, "hash": "202c11a22b026984c25fa6e1b160ff245da11a8bbfe0356ecfc1b605cd17d31d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 31582, "end_char_idx": 33355, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
625] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:52,777] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:53,653] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     60%|█████████████████████████▊                 | 15/25 [00:09<00:05,  1.87it/s][2024-03-26 19:44:53,670] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:54,028] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     68%|█████████████████████████████▏             | 17/25 [00:10<00:03,  2.54it/s][2024-03-26 19:44:54,388] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:54,396] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     72%|██████████████████████████████▉            | 18/25 [00:10<00:02,  2.58it/s][2024-03-26 19:44:54,529] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:54,585] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     76%|████████████████████████████████▋          | 19/25 [00:10<00:02,  2.94it/s][2024-03-26 19:44:54,980] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     80%|██████████████████████████████████▍        | 20/25 [00:11<00:01,  2.82it/s][2024-03-26 19:44:55,152] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.


key: _node_content
string_value:
{"id_": "84a15dcd-980e-4137-8b25-7be916d9734d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3c7e9a10-cf24-4986-9ee5-38954a534246", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "efab0118c51042a335009b590a6955420d360abfde5c8627b6edb800ec14f3e2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "899c1474-b87d-45b4-959c-35bc105b6faf", "node_type": "1", "metadata": {}, "hash": "b5e494782afe4346ea938d8e0449b6bfb661e6650f10622756d2e34a12b0ebc0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 32436, "end_char_idx": 34187, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:54,396] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     72%|██████████████████████████████▉            | 18/25 [00:10<00:02,  2.58it/s][2024-03-26 19:44:54,529] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:54,585] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     76%|████████████████████████████████▋          | 19/25 [00:10<00:02,  2.94it/s][2024-03-26 19:44:54,980] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     80%|██████████████████████████████████▍        | 20/25 [00:11<00:01,  2.82it/s][2024-03-26 19:44:55,152] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     84%|████████████████████████████████████       | 21/25 [00:11<00:01,  3.27it/s][2024-03-26 19:44:56,420] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     88%|█████████████████████████████████████▊     | 22/25 [00:12<00:01,  1.75it/s][2024-03-26 19:44:56,984] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     92%|███████████████████████████████████████▌   | 23/25 [00:13<00:01,  1.76it/s][2024-03-26 19:44:57,370] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     96%|█████████████████████████████████████████▎ | 24/25 [00:13<00:00,  1.94it/s][2024-03-26 19:44:59,589] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.


key: _node_content
string_value:
{"id_": "899c1474-b87d-45b4-959c-35bc105b6faf", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "84a15dcd-980e-4137-8b25-7be916d9734d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c130447072542527b6981999946bfdecec4a0d9ba19faae2f9cb50138feeef6e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "79398d17-c6c4-4ffe-bec1-e34cbbf9e2ff", "node_type": "1", "metadata": {}, "hash": "d68329986a0dde70d712bd8b52a4af48e59c7b1f51214cff0084da94d46b8da1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 33262, "end_char_idx": 34892, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
152] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     84%|████████████████████████████████████       | 21/25 [00:11<00:01,  3.27it/s][2024-03-26 19:44:56,420] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     88%|█████████████████████████████████████▊     | 22/25 [00:12<00:01,  1.75it/s][2024-03-26 19:44:56,984] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     92%|███████████████████████████████████████▌   | 23/25 [00:13<00:01,  1.76it/s][2024-03-26 19:44:57,370] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     96%|█████████████████████████████████████████▎ | 24/25 [00:13<00:00,  1.94it/s][2024-03-26 19:44:59,589] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    100%|███████████████████████████████████████████| 25/25 [00:15<00:00,  1.60it/s]
    [2024-03-26 19:44:59,607] [record.py:360] Final report: {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}. Logged to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl
    [2024-03-26 19:44:59,608] [oaieval.py:229] Final report:
    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Correct: 20
    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Incorrect: 5
    [2024-03-26 19:44:59,608] [oaieval.py:231] score: 0.8
    [2024-03-26 19:44:59,640] [record.py:349] Logged 75 rows of events to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl: insert_time=27.915ms


key: _node_content
string_value:
{"id_": "79398d17-c6c4-4ffe-bec1-e34cbbf9e2ff", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "899c1474-b87d-45b4-959c-35bc105b6faf", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "261b701cd3da7888fbe90afbe22213be287e37f57e05c2372ed69737f59ac1a4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6c294eee-8c42-460d-b998-8aaba1728ccd", "node_type": "1", "metadata": {}, "hash": "c76d41267ae2a9651a44e6c00718bfcfba021096ff7b65c123e3641062e87d15", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34064, "end_char_idx": 35381, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
1.94it/s][2024-03-26 19:44:59,589] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    100%|███████████████████████████████████████████| 25/25 [00:15<00:00,  1.60it/s]
    [2024-03-26 19:44:59,607] [record.py:360] Final report: {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}. Logged to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl
    [2024-03-26 19:44:59,608] [oaieval.py:229] Final report:
    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Correct: 20
    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Incorrect: 5
    [2024-03-26 19:44:59,608] [oaieval.py:231] score: 0.8
    [2024-03-26 19:44:59,640] [record.py:349] Logged 75 rows of events to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl: insert_time=27.915ms


`oaievalset` expects a model name and an eval set name, for which the valid options are specified in the YAML files under `evals/registry/eval_sets`.

### Going through eval logs

The eval logs are located at `/tmp/evallogs` and different log files are created for each evaluation run. 


```python
log_name = '240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl' # "EDIT THIS" - copy from above
events = f"/tmp/evallogs/{log_name}"
display(pd.read_json(events, lines=True).head(5))
```


key: _node_content
string_value:
{"id_": "6c294eee-8c42-460d-b998-8aaba1728ccd", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "79398d17-c6c4-4ffe-bec1-e34cbbf9e2ff", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0789b1ccbe72dec2dcc31388d0b9f2df2fb905e9fb9c4ae493b33814081ce585", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7f323cf3-6159-44f7-9a66-ed1fa5714d2b", "node_type": "1", "metadata": {}, "hash": "df8d0a1d4d691c842a849196abc154b427afc611a470efec6e5c4ec9d7ad1557", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34307, "end_char_idx": 37478, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
607] [record.py:360] Final report: {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}. Logged to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl
    [2024-03-26 19:44:59,608] [oaieval.py:229] Final report:
    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Correct: 20
    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Incorrect: 5
    [2024-03-26 19:44:59,608] [oaieval.py:231] score: 0.8
    [2024-03-26 19:44:59,640] [record.py:349] Logged 75 rows of events to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl: insert_time=27.915ms


`oaievalset` expects a model name and an eval set name, for which the valid options are specified in the YAML files under `evals/registry/eval_sets`.

### Going through eval logs

The eval logs are located at `/tmp/evallogs` and different log files are created for each evaluation run. 


```python
log_name = '240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl' # "EDIT THIS" - copy from above
events = f"/tmp/evallogs/{log_name}"
display(pd.read_json(events, lines=True).head(5))
```


<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>spec</th>
      <th>final_report</th>
      <th>run_id</th>
      <th>event_id</th>
      <th>sample_id</th>
      <th>type</th>
      <th>data</th>
      <th>created_by</th>
      <th>created_at</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>{'completion_fns': ['gpt-3.5-turbo'], 'eval_name': 'spider-sql.dev.v0', 'base_eval': 'spider-sql', 'split': 'dev', 'run_config': {'completion_fns': ['gpt-3.5-turbo'], 'eval_spec': {'cls': 'evals.elsuite.modelgraded.classify:ModelBasedClassify', 'registry_path': '/Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry', 'args': {'samples_jsonl': 'sql/spider_sql.jsonl', 'eval_type': 'cot_classify', 'modelgraded_spec': 'sql'}, 'key': 'spider-sql.dev.v0', 'group': 'sql'}, 'seed': 20220722, 'max_samples': 25, 'command': '/Users/shyamal/.virtualenvs/openai/bin/oaieval gpt-3.5-turbo spider-sql --max_samples 25', 'initial_settings': {'visible': False}}, 'created_by': '', 'run_id': '240327024443FACXGMKA', 'created_at': '2024-03-27 02:44:43.626043'}</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaT</td>
    </tr>
    <tr>
      <th>1</th>
      <td>NaN</td>
      <td>{'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaT</td>
    </tr>
    <tr>
      <th>2</th>
      <td>NaN</td>
      <td>NaN</td>
      <td>240327024443FACXGMKA</td>
      <td>0.0</td>
      <td>spider-sql.dev.88</td>
      <td>sampling</td>
      <td>{'prompt': [{'content': 'Answer the following question with syntactically correct SQLite SQL.


key: _node_content
string_value:
{"id_": "7f323cf3-6159-44f7-9a66-ed1fa5714d2b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6c294eee-8c42-460d-b998-8aaba1728ccd", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "6653a273dcd5ea8f7c8b99df5e5dc9da02b8fa2281a5448872f66faccbb384eb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9accb52e-d497-4edc-a458-93557a533450", "node_type": "1", "metadata": {}, "hash": "8fd03c3cad930b21286a3d08a17e04d228d35937cc112ef200190f164b83a650", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 37479, "end_char_idx": 40052, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Be creative but the SQL must be correct.
Use only the following tables and columns:
Table: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)
Table: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)
Table: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)

Question: Find the average rank of winners in all matches.
', 'role': 'system'}], 'sampled': ['SELECT AVG(winner_rank) AS average_rank_of_winners
FROM matches;']}</td>
      <td></td>
      <td>2024-03-27 02:44:44.821110+00:00</td>
    </tr>
    <tr>
      <th>3</th>
      <td>NaN</td>
      <td>NaN</td>
      <td>240327024443FACXGMKA</td>
      <td>1.0</td>
      <td>spider-sql.dev.82</td>
      <td>sampling</td>
      <td>{'prompt': [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.
Use only the following tables and columns:
Table: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)
Table: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)
Table: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)

Question: Find the total number of matches.


key: _node_content
string_value:
{"id_": "9accb52e-d497-4edc-a458-93557a533450", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7f323cf3-6159-44f7-9a66-ed1fa5714d2b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8e0cc7c740d25642b408b58fe3bc7fd3ec8cfa7f75b0e496cc34483c8da52c94", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "60497603-8976-48fe-a53b-1f14a84f1df4", "node_type": "1", "metadata": {}, "hash": "0ec710541b81819af80b8e7dbcb45936042d6b4ae46b4f832dc9a5bc72cec976", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 39005, "end_char_idx": 41789, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Be creative but the SQL must be correct.
Use only the following tables and columns:
Table: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)
Table: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)
Table: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)

Question: Find the total number of matches.
', 'role': 'system'}], 'sampled': ['SELECT COUNT(*) AS total_matches
FROM matches;']}</td>
      <td></td>
      <td>2024-03-27 02:44:44.831848+00:00</td>
    </tr>
    <tr>
      <th>4</th>
      <td>NaN</td>
      <td>NaN</td>
      <td>240327024443FACXGMKA</td>
      <td>2.0</td>
      <td>spider-sql.dev.25</td>
      <td>sampling</td>
      <td>{'prompt': [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.
Use only the following tables and columns:
Table: continents. Columns: ContId (number), Continent (text)
Table: countries. Columns: CountryId (number), CountryName (text), Continent (number)
Table: car_makers. Columns: Id (number), Maker (text), FullName (text), Country (text)
Table: model_list. Columns: ModelId (number), Maker (number), Model (text)
Table: car_names. Columns: MakeId (number), Model (text), Make (text)
Table: cars_data. Columns: Id (number), MPG (text), Cylinders (number), Edispl (number), Horsepower (text), Weight (number), Accelerate (number), Year (number)

Question: How many countries exist?
', 'role': 'system'}], 'sampled': ['SELECT COUNT(*) AS TotalCountries
FROM countries;']}</td>
      <td></td>
      <td>2024-03-27 02:44:44.996647+00:00</td>
    </tr>
  </tbody>
</table>
</div>



```python
# processing the log events generated by oaieval

with open(events, "r") as f:
    events_df = pd.read_json(f, lines=True)
```

This file will contain structured logs of the evaluation. The first entry provides a detailed specification of the evaluation, including the completion functions, evaluation name, run configuration, creator’s name, run ID, and creation timestamp.


```python
display(events_df.iloc[0].spec)
```


key: _node_content
string_value:
{"id_": "60497603-8976-48fe-a53b-1f14a84f1df4", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9accb52e-d497-4edc-a458-93557a533450", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0aa72862556fd4d1087ed1e904559866eb7b1ed33010f83d36c963b3f7b2ba7a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "96870e4d-bd53-4c8c-8559-e5fcb28b238e", "node_type": "1", "metadata": {}, "hash": "2ec1f596b22aaf971bfc5a8f8ff999c198dc0be20e825f80bc68689950cad9b3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 40498, "end_char_idx": 43047, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Be creative but the SQL must be correct.
Use only the following tables and columns:
Table: continents. Columns: ContId (number), Continent (text)
Table: countries. Columns: CountryId (number), CountryName (text), Continent (number)
Table: car_makers. Columns: Id (number), Maker (text), FullName (text), Country (text)
Table: model_list. Columns: ModelId (number), Maker (number), Model (text)
Table: car_names. Columns: MakeId (number), Model (text), Make (text)
Table: cars_data. Columns: Id (number), MPG (text), Cylinders (number), Edispl (number), Horsepower (text), Weight (number), Accelerate (number), Year (number)

Question: How many countries exist?
', 'role': 'system'}], 'sampled': ['SELECT COUNT(*) AS TotalCountries
FROM countries;']}</td>
      <td></td>
      <td>2024-03-27 02:44:44.996647+00:00</td>
    </tr>
  </tbody>
</table>
</div>



```python
# processing the log events generated by oaieval

with open(events, "r") as f:
    events_df = pd.read_json(f, lines=True)
```

This file will contain structured logs of the evaluation. The first entry provides a detailed specification of the evaluation, including the completion functions, evaluation name, run configuration, creator’s name, run ID, and creation timestamp.


```python
display(events_df.iloc[0].spec)
```


    {'completion_fns': ['gpt-3.5-turbo'],
     'eval_name': 'spider-sql.dev.v0',
     'base_eval': 'spider-sql',
     'split': 'dev',
     'run_config': {'completion_fns': ['gpt-3.5-turbo'],
      'eval_spec': {'cls': 'evals.elsuite.modelgraded.classify:ModelBasedClassify',
       'registry_path': '/Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry',
       'args': {'samples_jsonl': 'sql/spider_sql.jsonl',
        'eval_type': 'cot_classify',
        'modelgraded_spec': 'sql'},
       'key': 'spider-sql.dev.v0',
       'group': 'sql'},
      'seed': 20220722,
      'max_samples': 25,
      'command': '/Users/shyamal/.virtualenvs/openai/bin/oaieval gpt-3.5-turbo spider-sql --max_samples 25',
      'initial_settings': {'visible': False}},
     'created_by': '',
     'run_id': '240327024443FACXGMKA',
     'created_at': '2024-03-27 02:44:43.626043'}


Let's also look at the entry which provides the final report of the evaluation.


```python
display(events_df.dropna(subset=['final_report']).iloc[0]['final_report'])
```


    {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}


We can also review individual evaluation events that provide specific samples (`sample_id`), results, event types, and metadata.


key: _node_content
string_value:
{"id_": "96870e4d-bd53-4c8c-8559-e5fcb28b238e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "60497603-8976-48fe-a53b-1f14a84f1df4", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8bf7e16f58920fd038da862bae8d9519166af984e5674ee69520d94badf1102b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f5d5c879-27bc-468c-b624-7523315f5850", "node_type": "1", "metadata": {}, "hash": "8b13c7ab4b4dfd52de72c9adbe24ae0c4bcfb57d878aff96a2106c02a6abfea0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 41744, "end_char_idx": 43229, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
display(events_df.iloc[0].spec)
```


    {'completion_fns': ['gpt-3.5-turbo'],
     'eval_name': 'spider-sql.dev.v0',
     'base_eval': 'spider-sql',
     'split': 'dev',
     'run_config': {'completion_fns': ['gpt-3.5-turbo'],
      'eval_spec': {'cls': 'evals.elsuite.modelgraded.classify:ModelBasedClassify',
       'registry_path': '/Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry',
       'args': {'samples_jsonl': 'sql/spider_sql.jsonl',
        'eval_type': 'cot_classify',
        'modelgraded_spec': 'sql'},
       'key': 'spider-sql.dev.v0',
       'group': 'sql'},
      'seed': 20220722,
      'max_samples': 25,
      'command': '/Users/shyamal/.virtualenvs/openai/bin/oaieval gpt-3.5-turbo spider-sql --max_samples 25',
      'initial_settings': {'visible': False}},
     'created_by': '',
     'run_id': '240327024443FACXGMKA',
     'created_at': '2024-03-27 02:44:43.626043'}


Let's also look at the entry which provides the final report of the evaluation.


```python
display(events_df.dropna(subset=['final_report']).iloc[0]['final_report'])
```


    {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}


We can also review individual evaluation events that provide specific samples (`sample_id`), results, event types, and metadata.


```python
pd.set_option('display.max_colwidth', None)  # None means no truncation
display(events_df.iloc[2][['run_id', 'event_id', 'sample_id', 'type', 'data', 'created_at']])
```


key: _node_content
string_value:
{"id_": "f5d5c879-27bc-468c-b624-7523315f5850", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "96870e4d-bd53-4c8c-8559-e5fcb28b238e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c51f334d524e91795754f0f48a6969035394eb4997035234fc6564420e9b915c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "71fb12ed-d055-4e22-a2c2-c09d6f0a0234", "node_type": "1", "metadata": {}, "hash": "1c4d59968cf169d38bc2b3a2d85d03618c48f1467be680f6a4ebb8c74c7a023e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 42680, "end_char_idx": 51567, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Let's also look at the entry which provides the final report of the evaluation.


```python
display(events_df.dropna(subset=['final_report']).iloc[0]['final_report'])
```


    {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}


We can also review individual evaluation events that provide specific samples (`sample_id`), results, event types, and metadata.


```python
pd.set_option('display.max_colwidth', None)  # None means no truncation
display(events_df.iloc[2][['run_id', 'event_id', 'sample_id', 'type', 'data', 'created_at']])
```


    run_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                240327024443FACXGMKA
    event_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               0.0
    sample_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                spider-sql.dev.88
    type                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              sampling
    data          {'prompt': [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.
    Use only the following tables and columns:
    Table: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)
    Table: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)
    Table: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)
    
    Question: Find the average rank of winners in all matches.
    ', 'role': 'system'}], 'sampled': ['SELECT AVG(winner_rank) AS average_rank_of_winners
    FROM matches;']}
    created_at                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                2024-03-27 02:44:44.821110+00:00
    Name: 2, dtype: object



```python
# Inspect samples
for i, row in events_df[events_df['type'] == 'sampling'].head(5).iterrows():
    data = pd.json_normalize(row['data'])
    print(f"Prompt: {data['prompt'].iloc[0]}")
    print(f"Sampled: {data['sampled'].iloc[0]}")
    print("-" * 10)
```

    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\nTable: matches.


key: _node_content
string_value:
{"id_": "71fb12ed-d055-4e22-a2c2-c09d6f0a0234", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f5d5c879-27bc-468c-b624-7523315f5850", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "6f579e2bac6958c8aa7072094195da82d0df93353548b4f117052193c79f76d9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d5ac2edf-283f-4d36-8996-e7e2e69b09aa", "node_type": "1", "metadata": {}, "hash": "d448681c028549f6f2c1d6c21014c4d15631cfa5b7cf63b8cf6cca01c6042648", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 50968, "end_char_idx": 53972, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Inspect samples
for i, row in events_df[events_df['type'] == 'sampling'].head(5).iterrows():
    data = pd.json_normalize(row['data'])
    print(f"Prompt: {data['prompt'].iloc[0]}")
    print(f"Sampled: {data['sampled'].iloc[0]}")
    print("-" * 10)
```

    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\nTable: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)\nTable: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)\n\nQuestion: Find the average rank of winners in all matches.\n', 'role': 'system'}]
    Sampled: ['SELECT AVG(winner_rank) AS average_rank_of_winners\nFROM matches;']
    ----------
    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\nTable: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)\nTable: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)\n\nQuestion: Find the total number of matches.\n', 'role': 'system'}]
    Sampled: ['SELECT COUNT(*) AS total_matches\nFROM matches;']
    ----------
    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: continents.


key: _node_content
string_value:
{"id_": "d5ac2edf-283f-4d36-8996-e7e2e69b09aa", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "71fb12ed-d055-4e22-a2c2-c09d6f0a0234", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "efe845d6cd0b8d48286c17dbf185fb910817e6237a605a9429e948392dee6c8c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "29d818f9-3fa8-46e5-9be1-21988d8f0af5", "node_type": "1", "metadata": {}, "hash": "8081cfa1c0edb27d083aaa3a7baaaeaa758676d384ae77990b417c063565201a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 52616, "end_char_idx": 55242, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\nTable: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)\nTable: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)\n\nQuestion: Find the total number of matches.\n', 'role': 'system'}]
    Sampled: ['SELECT COUNT(*) AS total_matches\nFROM matches;']
    ----------
    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: continents. Columns: ContId (number), Continent (text)\nTable: countries. Columns: CountryId (number), CountryName (text), Continent (number)\nTable: car_makers. Columns: Id (number), Maker (text), FullName (text), Country (text)\nTable: model_list. Columns: ModelId (number), Maker (number), Model (text)\nTable: car_names. Columns: MakeId (number), Model (text), Make (text)\nTable: cars_data. Columns: Id (number), MPG (text), Cylinders (number), Edispl (number), Horsepower (text), Weight (number), Accelerate (number), Year (number)\n\nQuestion: How many countries exist?\n', 'role': 'system'}]
    Sampled: ['SELECT COUNT(*) AS TotalCountries\nFROM countries;']
    ----------
    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: TV_Channel. Columns: id (text), series_name (text), Country (text), Language (text), Content (text), Pixel_aspect_ratio_PAR (text), Hight_definition_TV (text), Pay_per_view_PPV (text), Package_Option (text)\nTable: TV_series. Columns: id (number), Episode (text), Air_Date (text), Rating (text), Share (number), 18_49_Rating_Share (text), Viewers_m (text), Weekly_Rank (number), Channel (text)\nTable: Cartoon.


key: _node_content
string_value:
{"id_": "29d818f9-3fa8-46e5-9be1-21988d8f0af5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d5ac2edf-283f-4d36-8996-e7e2e69b09aa", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "2dac83b1bce5b1d9821abe9fdf90e512ba52cf62cf03ab3b89a73c7a7b827cb3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "675c2e9d-27ff-4029-9451-ffd2318b46c0", "node_type": "1", "metadata": {}, "hash": "dad931d4882b288500dbac7b74da622f9de656e29875e150bb5538b7803bf14a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 53868, "end_char_idx": 56584, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: continents. Columns: ContId (number), Continent (text)\nTable: countries. Columns: CountryId (number), CountryName (text), Continent (number)\nTable: car_makers. Columns: Id (number), Maker (text), FullName (text), Country (text)\nTable: model_list. Columns: ModelId (number), Maker (number), Model (text)\nTable: car_names. Columns: MakeId (number), Model (text), Make (text)\nTable: cars_data. Columns: Id (number), MPG (text), Cylinders (number), Edispl (number), Horsepower (text), Weight (number), Accelerate (number), Year (number)\n\nQuestion: How many countries exist?\n', 'role': 'system'}]
    Sampled: ['SELECT COUNT(*) AS TotalCountries\nFROM countries;']
    ----------
    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: TV_Channel. Columns: id (text), series_name (text), Country (text), Language (text), Content (text), Pixel_aspect_ratio_PAR (text), Hight_definition_TV (text), Pay_per_view_PPV (text), Package_Option (text)\nTable: TV_series. Columns: id (number), Episode (text), Air_Date (text), Rating (text), Share (number), 18_49_Rating_Share (text), Viewers_m (text), Weekly_Rank (number), Channel (text)\nTable: Cartoon. Columns: id (number), Title (text), Directed_by (text), Written_by (text), Original_air_date (text), Production_code (number), Channel (text)\n\nQuestion: What is the name and directors of all the cartoons that are ordered by air date?\n', 'role': 'system'}]
    Sampled: ['SELECT Title, Directed_by\nFROM Cartoon\nORDER BY Original_air_date;']
    ----------
    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: stadium. Columns: Stadium_ID (number), Location (text), Name (text), Capacity (number), Highest (number), Lowest (number), Average (number)\nTable: singer. Columns: Singer_ID (number), Name (text), Country (text), Song_Name (text), Song_release_year (text), Age (number), Is_male (others)\nTable: concert. Columns: concert_ID (number), concert_Name (text), Theme (text), Stadium_ID (text), Year (text)\nTable: singer_in_concert. Columns: concert_ID (number), Singer_ID (text)\n\nQuestion: Show the name and the release year of the song by the youngest singer.\n', 'role': 'system'}]
    Sampled: ['```sql\nSELECT s.Name, s.Song_release_year\nFROM singer s\nWHERE s.Age = (SELECT MIN(Age) FROM singer)\n```']
    ----------


Let's review our failures to understand which tests did not succeed.


key: _node_content
string_value:
{"id_": "675c2e9d-27ff-4029-9451-ffd2318b46c0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "29d818f9-3fa8-46e5-9be1-21988d8f0af5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "b759c9da2bf2f2536d413c263443f979c9667c39c375ef49fa37fac02e2c86a9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "63d4bb56-51bd-4555-b6b7-1fdc0e5437a9", "node_type": "1", "metadata": {}, "hash": "2b257916321e80c1f8c4b87ad84d80904cdb749c8f3349e8efb261dd3cf7dc6e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 55058, "end_char_idx": 58210, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Columns: id (number), Episode (text), Air_Date (text), Rating (text), Share (number), 18_49_Rating_Share (text), Viewers_m (text), Weekly_Rank (number), Channel (text)\nTable: Cartoon. Columns: id (number), Title (text), Directed_by (text), Written_by (text), Original_air_date (text), Production_code (number), Channel (text)\n\nQuestion: What is the name and directors of all the cartoons that are ordered by air date?\n', 'role': 'system'}]
    Sampled: ['SELECT Title, Directed_by\nFROM Cartoon\nORDER BY Original_air_date;']
    ----------
    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: stadium. Columns: Stadium_ID (number), Location (text), Name (text), Capacity (number), Highest (number), Lowest (number), Average (number)\nTable: singer. Columns: Singer_ID (number), Name (text), Country (text), Song_Name (text), Song_release_year (text), Age (number), Is_male (others)\nTable: concert. Columns: concert_ID (number), concert_Name (text), Theme (text), Stadium_ID (text), Year (text)\nTable: singer_in_concert. Columns: concert_ID (number), Singer_ID (text)\n\nQuestion: Show the name and the release year of the song by the youngest singer.\n', 'role': 'system'}]
    Sampled: ['```sql\nSELECT s.Name, s.Song_release_year\nFROM singer s\nWHERE s.Age = (SELECT MIN(Age) FROM singer)\n```']
    ----------


Let's review our failures to understand which tests did not succeed.


```python
def pretty_print_text(prompt):
    # Define markers for the sections
    markers = {
        "question": "[Question]:",
        "expert": "[Expert]:",
        "submission": "[Submission]:",
        "end": "[END DATA]"
    }
    
    # Function to extract text between markers
    def extract_text(start_marker, end_marker):
        start = prompt.find(start_marker) + len(start_marker)
        end = prompt.find(end_marker)
        text = prompt[start:end].strip()
        if start_marker == markers["question"]:
            text = text.split("\n\nQuestion:")[-1].strip() if "\n\nQuestion:" in text else text
        elif start_marker == markers["submission"]:
            text = text.replace("```sql", "").replace("```", "").strip()
        return text
    
    # Extracting text for each section
    question_text = extract_text(markers["question"], markers["expert"])
    expert_text = extract_text(markers["expert"], markers["submission"])
    submission_text = extract_text(markers["submission"], markers["end"])
    
    # HTML color codes and formatting
    colors = {
        "question": '<span style="color: #0000FF;">QUESTION:<br>', 
        "expert": '<span style="color: #008000;">EXPECTED:<br>',  
        "submission": '<span style="color: #FFA500;">SUBMISSION:<br>' 
    }
    color_end = '</span>'
    
    # Display each section with color
    from IPython.display import display, HTML
    display(HTML(f"{colors['question']}{question_text}{color_end}"))
    display(HTML(f"{colors['expert']}{expert_text}{color_end}"))
    display(HTML(f"{colors['submission']}{submission_text}{color_end}"))
```


key: _node_content
string_value:
{"id_": "63d4bb56-51bd-4555-b6b7-1fdc0e5437a9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "675c2e9d-27ff-4029-9451-ffd2318b46c0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "57266b63ea63aeababf4f2c80f873e1339a5029ffb143149a18054eb9f8dc622", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2ba9a58c-a81e-4de5-9053-3448226586b5", "node_type": "1", "metadata": {}, "hash": "9e9f577507627bfa8df69233d7f66af92445e05915055a1bce054270fc75cce6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 56516, "end_char_idx": 59647, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Let's review our failures to understand which tests did not succeed.


```python
def pretty_print_text(prompt):
    # Define markers for the sections
    markers = {
        "question": "[Question]:",
        "expert": "[Expert]:",
        "submission": "[Submission]:",
        "end": "[END DATA]"
    }
    
    # Function to extract text between markers
    def extract_text(start_marker, end_marker):
        start = prompt.find(start_marker) + len(start_marker)
        end = prompt.find(end_marker)
        text = prompt[start:end].strip()
        if start_marker == markers["question"]:
            text = text.split("\n\nQuestion:")[-1].strip() if "\n\nQuestion:" in text else text
        elif start_marker == markers["submission"]:
            text = text.replace("```sql", "").replace("```", "").strip()
        return text
    
    # Extracting text for each section
    question_text = extract_text(markers["question"], markers["expert"])
    expert_text = extract_text(markers["expert"], markers["submission"])
    submission_text = extract_text(markers["submission"], markers["end"])
    
    # HTML color codes and formatting
    colors = {
        "question": '<span style="color: #0000FF;">QUESTION:<br>', 
        "expert": '<span style="color: #008000;">EXPECTED:<br>',  
        "submission": '<span style="color: #FFA500;">SUBMISSION:<br>' 
    }
    color_end = '</span>'
    
    # Display each section with color
    from IPython.display import display, HTML
    display(HTML(f"{colors['question']}{question_text}{color_end}"))
    display(HTML(f"{colors['expert']}{expert_text}{color_end}"))
    display(HTML(f"{colors['submission']}{submission_text}{color_end}"))
```


```python
# Inspect metrics where choice is made and print only the prompt, result, and expected result if the choice is incorrect
for i, row in events_df[events_df['type'] == 'metrics'].iterrows():
    if row['data']['choice'] == 'Incorrect':
        # Get the previous row's data, which contains the prompt and the expected result
        prev_row = events_df.iloc[i-1]
        prompt = prev_row['data']['prompt'][0]['content'] if 'prompt' in prev_row['data'] and len(prev_row['data']['prompt']) > 0 else "Prompt not available"
        expected_result = prev_row['data'].get('ideal', 'Expected result not provided')
        
        # Current row's data will be the actual result
        result = row['data'].get('result', 'Actual result not provided')
        
        pretty_print_text(prompt)
        print("-" * 40)
```


<span style="color: #0000FF;">QUESTION:<br>How many countries have a republic as their form of government?

************</span>



<span style="color: #008000;">EXPECTED:<br>SELECT count(*) FROM country WHERE GovernmentForm  =  "Republic"
************</span>



<span style="color: #FFA500;">SUBMISSION:<br>SELECT COUNT(*) 
FROM country 
WHERE GovernmentForm LIKE '%Republic%'

************</span>


    ----------------------------------------



<span style="color: #0000FF;">QUESTION:<br>Return the document id, template id, and description for the document with the name Robbin CV.

************</span>


key: _node_content
string_value:
{"id_": "2ba9a58c-a81e-4de5-9053-3448226586b5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "63d4bb56-51bd-4555-b6b7-1fdc0e5437a9", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "bac33913ada2e5360bf72540fb18a424adeee8410530d3cfbcad0f4b23840045", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f0bf8ebc-fdf9-49e0-b67f-b872a11e3428", "node_type": "1", "metadata": {}, "hash": "42e338f1df2f76b100b9cbf663bcd88ce512a131678cd3692062fe336d720aea", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 58213, "end_char_idx": 61303, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Inspect metrics where choice is made and print only the prompt, result, and expected result if the choice is incorrect
for i, row in events_df[events_df['type'] == 'metrics'].iterrows():
    if row['data']['choice'] == 'Incorrect':
        # Get the previous row's data, which contains the prompt and the expected result
        prev_row = events_df.iloc[i-1]
        prompt = prev_row['data']['prompt'][0]['content'] if 'prompt' in prev_row['data'] and len(prev_row['data']['prompt']) > 0 else "Prompt not available"
        expected_result = prev_row['data'].get('ideal', 'Expected result not provided')
        
        # Current row's data will be the actual result
        result = row['data'].get('result', 'Actual result not provided')
        
        pretty_print_text(prompt)
        print("-" * 40)
```


<span style="color: #0000FF;">QUESTION:<br>How many countries have a republic as their form of government?

************</span>



<span style="color: #008000;">EXPECTED:<br>SELECT count(*) FROM country WHERE GovernmentForm  =  "Republic"
************</span>



<span style="color: #FFA500;">SUBMISSION:<br>SELECT COUNT(*) 
FROM country 
WHERE GovernmentForm LIKE '%Republic%'

************</span>


    ----------------------------------------



<span style="color: #0000FF;">QUESTION:<br>Return the document id, template id, and description for the document with the name Robbin CV.

************</span>



<span style="color: #008000;">EXPECTED:<br>SELECT document_id ,  template_id ,  Document_Description FROM Documents WHERE document_name  =  "Robbin CV"
************</span>



<span style="color: #FFA500;">SUBMISSION:<br>SELECT Documents.Document_ID, Documents.Template_ID, Documents.Document_Description
FROM Documents
JOIN Templates ON Documents.Template_ID = Templates.Template_ID
WHERE Documents.Document_Name = 'Robbin CV';

************</span>


    ----------------------------------------



<span style="color: #0000FF;">QUESTION:<br>Which professionals live in the state of Indiana or have done treatment on more than 2 treatments? List his or her id, last name and cell phone.

************</span>



<span style="color: #008000;">EXPECTED:<br>SELECT professional_id ,  last_name ,  cell_number FROM Professionals WHERE state  =  'Indiana' UNION SELECT T1.professional_id ,  T1.last_name ,  T1.cell_number FROM Professionals AS T1 JOIN Treatments AS T2 ON T1.professional_id  =  T2.professional_id GROUP BY T1.professional_id HAVING count(*)  >  2
************</span>



<span style="color: #FFA500;">SUBMISSION:<br>SELECT professional_id, last_name, cell_number
FROM Professionals
WHERE state = 'Indiana'
OR professional_id IN (
    SELECT professional_id
    FROM Treatments
    GROUP BY professional_id
    HAVING COUNT(*) > 2
);
************</span>


    ----------------------------------------



<span style="color: #0000FF;">QUESTION:<br>What is the continent name which Anguilla belongs to?

************</span>



<span style="color: #008000;">EXPECTED:<br>SELECT Continent FROM country WHERE Name  =  "Anguilla"
************</span>


key: _node_content
string_value:
{"id_": "f0bf8ebc-fdf9-49e0-b67f-b872a11e3428", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2ba9a58c-a81e-4de5-9053-3448226586b5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "59cd05af2fba430ae2c191492e8cc7f531aefe1341d9053270aa4ce8dfd55449", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e3fbf0e1-3feb-459d-a910-52057990d9e9", "node_type": "1", "metadata": {}, "hash": "d93fb102df7799c5f49af1b446e5f57ad73eee8ef49a7ad1a8e6aaa7035ae59f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 59651, "end_char_idx": 61846, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<span style="color: #008000;">EXPECTED:<br>SELECT document_id ,  template_id ,  Document_Description FROM Documents WHERE document_name  =  "Robbin CV"
************</span>



<span style="color: #FFA500;">SUBMISSION:<br>SELECT Documents.Document_ID, Documents.Template_ID, Documents.Document_Description
FROM Documents
JOIN Templates ON Documents.Template_ID = Templates.Template_ID
WHERE Documents.Document_Name = 'Robbin CV';

************</span>


    ----------------------------------------



<span style="color: #0000FF;">QUESTION:<br>Which professionals live in the state of Indiana or have done treatment on more than 2 treatments? List his or her id, last name and cell phone.

************</span>



<span style="color: #008000;">EXPECTED:<br>SELECT professional_id ,  last_name ,  cell_number FROM Professionals WHERE state  =  'Indiana' UNION SELECT T1.professional_id ,  T1.last_name ,  T1.cell_number FROM Professionals AS T1 JOIN Treatments AS T2 ON T1.professional_id  =  T2.professional_id GROUP BY T1.professional_id HAVING count(*)  >  2
************</span>



<span style="color: #FFA500;">SUBMISSION:<br>SELECT professional_id, last_name, cell_number
FROM Professionals
WHERE state = 'Indiana'
OR professional_id IN (
    SELECT professional_id
    FROM Treatments
    GROUP BY professional_id
    HAVING COUNT(*) > 2
);
************</span>


    ----------------------------------------



<span style="color: #0000FF;">QUESTION:<br>What is the continent name which Anguilla belongs to?

************</span>



<span style="color: #008000;">EXPECTED:<br>SELECT Continent FROM country WHERE Name  =  "Anguilla"
************</span>



<span style="color: #FFA500;">SUBMISSION:<br>SELECT c.Continent
FROM country c
WHERE c.Code = 'AIA';

************</span>


    ----------------------------------------



<span style="color: #0000FF;">QUESTION:<br>How many airlines do we have?

************</span>



<span style="color: #008000;">EXPECTED:<br>SELECT count(*) FROM AIRLINES
************</span>



<span style="color: #FFA500;">SUBMISSION:<br>SELECT COUNT(DISTINCT Airline) AS TotalAirlines
FROM airlines;
************</span>


    ----------------------------------------


key: _node_content
string_value:
{"id_": "e3fbf0e1-3feb-459d-a910-52057990d9e9", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "e3848c26-661e-4e5a-9309-bade5dadeb70", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f0bf8ebc-fdf9-49e0-b67f-b872a11e3428", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "fff891177ef2fbb3dfbc338898f05a9ac776770b836b585d37c049a8cde026c7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 60362, "end_char_idx": 63108, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<span style="color: #008000;">EXPECTED:<br>SELECT professional_id ,  last_name ,  cell_number FROM Professionals WHERE state  =  'Indiana' UNION SELECT T1.professional_id ,  T1.last_name ,  T1.cell_number FROM Professionals AS T1 JOIN Treatments AS T2 ON T1.professional_id  =  T2.professional_id GROUP BY T1.professional_id HAVING count(*)  >  2
************</span>



<span style="color: #FFA500;">SUBMISSION:<br>SELECT professional_id, last_name, cell_number
FROM Professionals
WHERE state = 'Indiana'
OR professional_id IN (
    SELECT professional_id
    FROM Treatments
    GROUP BY professional_id
    HAVING COUNT(*) > 2
);
************</span>


    ----------------------------------------



<span style="color: #0000FF;">QUESTION:<br>What is the continent name which Anguilla belongs to?

************</span>



<span style="color: #008000;">EXPECTED:<br>SELECT Continent FROM country WHERE Name  =  "Anguilla"
************</span>



<span style="color: #FFA500;">SUBMISSION:<br>SELECT c.Continent
FROM country c
WHERE c.Code = 'AIA';

************</span>


    ----------------------------------------



<span style="color: #0000FF;">QUESTION:<br>How many airlines do we have?

************</span>



<span style="color: #008000;">EXPECTED:<br>SELECT count(*) FROM AIRLINES
************</span>



<span style="color: #FFA500;">SUBMISSION:<br>SELECT COUNT(DISTINCT Airline) AS TotalAirlines
FROM airlines;
************</span>


    ----------------------------------------


Reviewing some of the failures we see the following:
* The second incorrect answer had an unnecessary join with the 'Templates' table. Our eval was able to accurately identify this and flag this as incorrect. 
* Few other answers have minor syntax differences that caused the answers to get flagged.
  * In situations like this, it would be worthwhile exploring whether we should continue iterating on the prompt to ensure certain stylistic choices, or if we should modify the evaluation suite to capture this variation.
  * This type of failure hints at the potential need for model-graded evals as a way to ensure accuracy in grading the results

# Conclusion

Building out effective evals is a core part of the development cycle of LLM-based applications. The OpenAI Evals framework provides the core structure of building evals out of the box, and allows you to quickly spin up new tests for your various use cases. In this guide, we demonstrated step-by-step how to create an eval, run it, and analyze the results.

The example shown in this guide represent a straightfoward use case for evals. As you continue to explore this framework, we recommend you explore creating more complex model-graded evals for actual production use cases. Happy evaluating!


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb


key: _node_content
string_value:
{"id_": "b0e6af70-a6a4-442d-b2e9-9028ca38ed77", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f8307df-f435-40fc-91fd-e52c0970b1d2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3d87226cf3e1ecbc42d0bed8b11ca11fa5c935369f6c9e45c85efd4d1eec2978", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f91f077d-a4c0-4834-8e0b-dfa7996c9bc5", "node_type": "1", "metadata": {}, "hash": "4987a94b9964c66fd52540d6ed08d64f33ec1c0cc7725cb5e845dbc12a104f94", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3549, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Evaluate RAG with LlamaIndex

In this notebook we will look into building an RAG pipeline and evaluating it with LlamaIndex. It has following 3 sections.

1. Understanding Retrieval Augmented Generation (RAG).
2. Building RAG with LlamaIndex.
3. Evaluating RAG with LlamaIndex.

**Retrieval Augmented Generation (RAG)**

LLMs are trained on vast datasets, but these will not include your specific data. Retrieval-Augmented Generation (RAG) addresses this by dynamically incorporating your data during the generation process. This is done not by altering the training data of LLMs, but by allowing the model to access and utilize your data in real-time to provide more tailored and contextually relevant responses.

In RAG, your data is loaded and and prepared for queries or “indexed”. User queries act on the index, which filters your data down to the most relevant context. This context and your query then go to the LLM along with a prompt, and the LLM provides a response.

Even if what you’re building is a chatbot or an agent, you’ll want to know RAG techniques for getting data into your application.

![RAG Overview](../../images/llamaindex_rag_overview.png)

**Stages within RAG**

There are five key stages within RAG, which in turn will be a part of any larger application you build. These are:

**Loading:** this refers to getting your data from where it lives – whether it’s text files, PDFs, another website, a database, or an API – into your pipeline. LlamaHub provides hundreds of connectors to choose from.

**Indexing:** this means creating a data structure that allows for querying the data. For LLMs this nearly always means creating vector embeddings, numerical representations of the meaning of your data, as well as numerous other metadata strategies to make it easy to accurately find contextually relevant data.

**Storing:** Once your data is indexed, you will want to store your index, along with any other metadata, to avoid the need to re-index it.

**Querying:** for any given indexing strategy there are many ways you can utilize LLMs and LlamaIndex data structures to query, including sub-queries, multi-step queries and hybrid strategies.

**Evaluation:** a critical step in any pipeline is checking how effective it is relative to other strategies, or when you make changes. Evaluation provides objective measures of how accurate, faithful and fast your responses to queries are.

## Build RAG system.

Now that we have understood the significance of RAG system, let's build a simple RAG pipeline.


```python
!pip install llama-index
```


```python
# The nest_asyncio module enables the nesting of asynchronous functions within an already running async loop.
# This is necessary because Jupyter notebooks inherently operate in an asynchronous loop.
# By applying nest_asyncio, we can run additional async functions within this existing loop without conflicts.
import nest_asyncio

nest_asyncio.apply()

from llama_index.evaluation import generate_question_context_pairs
from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext
from llama_index.node_parser import SimpleNodeParser
from llama_index.evaluation import generate_question_context_pairs
from llama_index.evaluation import RetrieverEvaluator
from llama_index.llms import OpenAI

import os
import pandas as pd
```

Set Your OpenAI API Key


```python
os.environ['OPENAI_API_KEY'] = 'YOUR OPENAI API KEY'
```

Let's use [Paul Graham Essay text](https://www.paulgraham.com/worked.html) for building RAG pipeline.

#### Download Data


key: _node_content
string_value:
{"id_": "f91f077d-a4c0-4834-8e0b-dfa7996c9bc5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f8307df-f435-40fc-91fd-e52c0970b1d2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3d87226cf3e1ecbc42d0bed8b11ca11fa5c935369f6c9e45c85efd4d1eec2978", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b0e6af70-a6a4-442d-b2e9-9028ca38ed77", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0faf228e5bc8a80222528f722a0c78d455d584894cd1b845a2c73dc0ab710031", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c1439fd1-b58c-4bf0-a289-39dda2aea7bb", "node_type": "1", "metadata": {}, "hash": "61da286d96fb963babd58cdeeadd7f9bc332f4579fcbc66c3f1f1b17c7cb20ea", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2536, "end_char_idx": 5052, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
!pip install llama-index
```


```python
# The nest_asyncio module enables the nesting of asynchronous functions within an already running async loop.
# This is necessary because Jupyter notebooks inherently operate in an asynchronous loop.
# By applying nest_asyncio, we can run additional async functions within this existing loop without conflicts.
import nest_asyncio

nest_asyncio.apply()

from llama_index.evaluation import generate_question_context_pairs
from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext
from llama_index.node_parser import SimpleNodeParser
from llama_index.evaluation import generate_question_context_pairs
from llama_index.evaluation import RetrieverEvaluator
from llama_index.llms import OpenAI

import os
import pandas as pd
```

Set Your OpenAI API Key


```python
os.environ['OPENAI_API_KEY'] = 'YOUR OPENAI API KEY'
```

Let's use [Paul Graham Essay text](https://www.paulgraham.com/worked.html) for building RAG pipeline.

#### Download Data


```python
!mkdir -p 'data/paul_graham/'
!curl 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -o 'data/paul_graham/paul_graham_essay.txt'
```

      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100 75042  100 75042    0     0   190k      0 --:--:-- --:--:-- --:--:--  190k--:--  0:00:03 24586


#### Load Data and Build Index.


```python
documents = SimpleDirectoryReader("./data/paul_graham/").load_data()

# Define an LLM
llm = OpenAI(model="gpt-4")

# Build index with a chunk_size of 512
node_parser = SimpleNodeParser.from_defaults(chunk_size=512)
nodes = node_parser.get_nodes_from_documents(documents)
vector_index = VectorStoreIndex(nodes)
```

Build a QueryEngine and start querying.


```python
query_engine = vector_index.as_query_engine()
```


```python
response_vector = query_engine.query("What did the author do growing up?")
```

Check response.


```python
response_vector.response
```




    'The author wrote short stories and worked on programming, specifically on an IBM 1401 computer using an early version of Fortran.'



By default it retrieves `two` similar nodes/ chunks. You can modify that in `vector_index.as_query_engine(similarity_top_k=k)`.

Let's check the text in each of these retrieved nodes.


```python
# First retrieved node
response_vector.source_nodes[0].get_text()
```


key: _node_content
string_value:
{"id_": "c1439fd1-b58c-4bf0-a289-39dda2aea7bb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f8307df-f435-40fc-91fd-e52c0970b1d2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3d87226cf3e1ecbc42d0bed8b11ca11fa5c935369f6c9e45c85efd4d1eec2978", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f91f077d-a4c0-4834-8e0b-dfa7996c9bc5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4776de8bd34aa62c6ac9e6f8020b6b188b3d52d21a6e2cd3601f82e94aebe17f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a30202ad-e9f9-408e-a579-5f2924c9ec0d", "node_type": "1", "metadata": {}, "hash": "81bd1a393b6cb99db4709af65021450a8d4448932336a3cef5fe84a52cdd8971", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4034, "end_char_idx": 7154, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
#### Load Data and Build Index.


```python
documents = SimpleDirectoryReader("./data/paul_graham/").load_data()

# Define an LLM
llm = OpenAI(model="gpt-4")

# Build index with a chunk_size of 512
node_parser = SimpleNodeParser.from_defaults(chunk_size=512)
nodes = node_parser.get_nodes_from_documents(documents)
vector_index = VectorStoreIndex(nodes)
```

Build a QueryEngine and start querying.


```python
query_engine = vector_index.as_query_engine()
```


```python
response_vector = query_engine.query("What did the author do growing up?")
```

Check response.


```python
response_vector.response
```




    'The author wrote short stories and worked on programming, specifically on an IBM 1401 computer using an early version of Fortran.'



By default it retrieves `two` similar nodes/ chunks. You can modify that in `vector_index.as_query_engine(similarity_top_k=k)`.

Let's check the text in each of these retrieved nodes.


```python
# First retrieved node
response_vector.source_nodes[0].get_text()
```




    'What I Worked On\n\nFebruary 2021\n\nBefore college the two main things I worked on, outside of school, were writing and programming. I didn\'t write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.\n\nThe first programs I tried writing were on the IBM 1401 that our school district used for what was then called "data processing." This was in 9th grade, so I was 13 or 14. The school district\'s 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain\'s lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights.\n\nThe language we used was an early version of Fortran. You had to type programs on punch cards, then stack them in the card reader and press a button to load the program into memory and run it. The result would ordinarily be to print something on the spectacularly loud printer.\n\nI was puzzled by the 1401. I couldn\'t figure out what to do with it. And in retrospect there\'s not much I could have done with it. The only form of input to programs was data stored on punched cards, and I didn\'t have any data stored on punched cards. The only other option was to do things that didn\'t rely on any input, like calculate approximations of pi, but I didn\'t know enough math to do anything interesting of that type. So I\'m not surprised I can\'t remember any programs I wrote, because they can\'t have done much. My clearest memory is of the moment I learned it was possible for programs not to terminate, when one of mine didn\'t. On a machine without time-sharing, this was a social as well as a technical error, as the data center manager\'s expression made clear.\n\nWith microcomputers, everything changed.'




```python
# Second retrieved node
response_vector.source_nodes[1].get_text()
```


key: _node_content
string_value:
{"id_": "a30202ad-e9f9-408e-a579-5f2924c9ec0d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f8307df-f435-40fc-91fd-e52c0970b1d2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3d87226cf3e1ecbc42d0bed8b11ca11fa5c935369f6c9e45c85efd4d1eec2978", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c1439fd1-b58c-4bf0-a289-39dda2aea7bb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "41469b2004099464480bf9ddf983ad852d923e2c5b371f2a0807b4061e4c58e0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "44eae43a-e440-4e18-ae6f-15e99bcbf5b6", "node_type": "1", "metadata": {}, "hash": "c04fa934ab1372695a643ad2181d320704d196e8cbcd866226719624c27f30ca", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7074, "end_char_idx": 9296, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Second retrieved node
response_vector.source_nodes[1].get_text()
```




    "It felt like I was doing life right. I remember that because I was slightly dismayed at how novel it felt. The good news is that I had more moments like this over the next few years.\n\nIn the summer of 2016 we moved to England. We wanted our kids to see what it was like living in another country, and since I was a British citizen by birth, that seemed the obvious choice. We only meant to stay for a year, but we liked it so much that we still live there. So most of Bel was written in England.\n\nIn the fall of 2019, Bel was finally finished. Like McCarthy's original Lisp, it's a spec rather than an implementation, although like McCarthy's Lisp it's a spec expressed as code.\n\nNow that I could write essays again, I wrote a bunch about topics I'd had stacked up. I kept writing essays through 2020, but I also started to think about other things I could work on. How should I choose what to do? Well, how had I chosen what to work on in the past? I wrote an essay for myself to answer that question, and I was surprised how long and messy the answer turned out to be. If this surprised me, who'd lived it, then I thought perhaps it would be interesting to other people, and encouraging to those with similarly messy lives. So I wrote a more detailed version for others to read, and this is the last sentence of it.\n\n\n\n\n\n\n\n\n\nNotes\n\n[1] My experience skipped a step in the evolution of computers: time-sharing machines with interactive OSes. I went straight from batch processing to microcomputers, which made microcomputers seem all the more exciting.\n\n[2] Italian words for abstract concepts can nearly always be predicted from their English cognates (except for occasional traps like polluzione). It's the everyday words that differ. So if you string together a lot of abstract concepts with a few simple verbs, you can make a little Italian go a long way.\n\n[3] I lived at Piazza San Felice 4, so my walk to the Accademia went straight down the spine of old Florence: past the Pitti, across the bridge, past Orsanmichele, between the Duomo and the Baptistery, and then up Via Ricasoli to Piazza San Marco."


key: _node_content
string_value:
{"id_": "44eae43a-e440-4e18-ae6f-15e99bcbf5b6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f8307df-f435-40fc-91fd-e52c0970b1d2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3d87226cf3e1ecbc42d0bed8b11ca11fa5c935369f6c9e45c85efd4d1eec2978", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a30202ad-e9f9-408e-a579-5f2924c9ec0d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "6f011ed0db493d5c4e165df089c4b780d3302efa11ac001c3de1418cedf7b846", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2d5fa9bd-53f7-4696-8efc-7f78df476a70", "node_type": "1", "metadata": {}, "hash": "b73f12652763c10651caedcb96949757445bf1aea680b1f95df7293ebc735e5f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9300, "end_char_idx": 12800, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We have built a RAG pipeline and now need to evaluate its performance. We can assess our RAG system/query engine using LlamaIndex's core evaluation modules. Let's examine how to leverage these tools to quantify the quality of our retrieval-augmented generation system.

## Evaluation

Evaluation should serve as the primary metric for assessing your RAG application. It determines whether the pipeline will produce accurate responses based on the data sources and a range of queries.

While it's beneficial to examine individual queries and responses at the start, this approach may become impractical as the volume of edge cases and failures increases. Instead, it may be more effective to establish a suite of summary metrics or automated evaluations. These tools can provide insights into overall system performance and indicate specific areas that may require closer scrutiny.

In a RAG system, evaluation focuses on two critical aspects:

*   **Retrieval Evaluation:** This assesses the accuracy and relevance of the information retrieved by the system.
*   **Response Evaluation:** This measures the quality and appropriateness of the responses generated by the system based on the retrieved information.

#### Question-Context Pair Generation:

For the evaluation of a RAG system, it's essential to have queries that can fetch the correct context and subsequently generate an appropriate response. `LlamaIndex` offers a `generate_question_context_pairs` module specifically for crafting questions and context pairs which can be used in the assessment of the RAG system of both Retrieval and Response Evaluation. For more details on Question Generation, please refer to the [documentation](https://docs.llamaindex.ai/en/stable/examples/evaluation/QuestionGeneration.html).


```python
qa_dataset = generate_question_context_pairs(
    nodes,
    llm=llm,
    num_questions_per_chunk=2
)
```

    100%|██████████| 58/58 [06:26<00:00,  6.67s/it]


#### Retrieval Evaluation:

We are now prepared to conduct our retrieval evaluations. We will execute our `RetrieverEvaluator` using the evaluation dataset we have generated.

We first create the `Retriever` and then define two functions: `get_eval_results`, which operates our retriever on the dataset, and `display_results`, which presents the outcomes of the evaluation.

Let's create the retriever.


```python
retriever = vector_index.as_retriever(similarity_top_k=2)
```

Define `RetrieverEvaluator`. We use **Hit Rate** and **MRR** metrics to evaluate our Retriever.

**Hit Rate:**

Hit rate calculates the fraction of queries where the correct answer is found within the top-k retrieved documents. In simpler terms, it’s about how often our system gets it right within the top few guesses.

**Mean Reciprocal Rank (MRR):**

For each query, MRR evaluates the system’s accuracy by looking at the rank of the highest-placed relevant document. Specifically, it’s the average of the reciprocals of these ranks across all the queries. So, if the first relevant document is the top result, the reciprocal rank is 1; if it’s second, the reciprocal rank is 1/2, and so on.

Let's check these metrics to check the performance of out retriever.


```python
retriever_evaluator = RetrieverEvaluator.from_metric_names(
    ["mrr", "hit_rate"], retriever=retriever
)
```


```python
# Evaluate
eval_results = await retriever_evaluator.aevaluate_dataset(qa_dataset)
```

Let's define a function to display the Retrieval evaluation results in table format.


key: _node_content
string_value:
{"id_": "2d5fa9bd-53f7-4696-8efc-7f78df476a70", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f8307df-f435-40fc-91fd-e52c0970b1d2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3d87226cf3e1ecbc42d0bed8b11ca11fa5c935369f6c9e45c85efd4d1eec2978", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "44eae43a-e440-4e18-ae6f-15e99bcbf5b6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7944b33bcec19cba256b20f74f99120037af9ad27b8eb2ca5160f4d21b219b29", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "86317943-ef96-462e-91de-cad208639fdb", "node_type": "1", "metadata": {}, "hash": "f0c32c09851d0ec78ec39e7758ebb04c7f7983d69d4700d7061f846cfd345133", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11252, "end_char_idx": 13969, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
#### Retrieval Evaluation:

We are now prepared to conduct our retrieval evaluations. We will execute our `RetrieverEvaluator` using the evaluation dataset we have generated.

We first create the `Retriever` and then define two functions: `get_eval_results`, which operates our retriever on the dataset, and `display_results`, which presents the outcomes of the evaluation.

Let's create the retriever.


```python
retriever = vector_index.as_retriever(similarity_top_k=2)
```

Define `RetrieverEvaluator`. We use **Hit Rate** and **MRR** metrics to evaluate our Retriever.

**Hit Rate:**

Hit rate calculates the fraction of queries where the correct answer is found within the top-k retrieved documents. In simpler terms, it’s about how often our system gets it right within the top few guesses.

**Mean Reciprocal Rank (MRR):**

For each query, MRR evaluates the system’s accuracy by looking at the rank of the highest-placed relevant document. Specifically, it’s the average of the reciprocals of these ranks across all the queries. So, if the first relevant document is the top result, the reciprocal rank is 1; if it’s second, the reciprocal rank is 1/2, and so on.

Let's check these metrics to check the performance of out retriever.


```python
retriever_evaluator = RetrieverEvaluator.from_metric_names(
    ["mrr", "hit_rate"], retriever=retriever
)
```


```python
# Evaluate
eval_results = await retriever_evaluator.aevaluate_dataset(qa_dataset)
```

Let's define a function to display the Retrieval evaluation results in table format.


```python
def display_results(name, eval_results):
    """Display results from evaluate."""

    metric_dicts = []
    for eval_result in eval_results:
        metric_dict = eval_result.metric_vals_dict
        metric_dicts.append(metric_dict)

    full_df = pd.DataFrame(metric_dicts)

    hit_rate = full_df["hit_rate"].mean()
    mrr = full_df["mrr"].mean()

    metric_df = pd.DataFrame(
        {"Retriever Name": [name], "Hit Rate": [hit_rate], "MRR": [mrr]}
    )

    return metric_df
```


```python
display_results("OpenAI Embedding Retriever", eval_results)
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Retriever Name</th>
      <th>Hit Rate</th>
      <th>MRR</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>OpenAI Embedding Retriever</td>
      <td>0.758621</td>
      <td>0.62069</td>
    </tr>
  </tbody>
</table>
</div>


key: _node_content
string_value:
{"id_": "86317943-ef96-462e-91de-cad208639fdb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f8307df-f435-40fc-91fd-e52c0970b1d2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3d87226cf3e1ecbc42d0bed8b11ca11fa5c935369f6c9e45c85efd4d1eec2978", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2d5fa9bd-53f7-4696-8efc-7f78df476a70", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "51f9e6e0f79a9aa24393431eadcff2f8124fd66df028e16c46e2706035e69a48", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e23f31aa-5ac8-4f25-9b22-72f4cb413670", "node_type": "1", "metadata": {}, "hash": "549106f8cb76e64f06462e529b22bf76d9967224215f05a7db7b7fad0c367dee", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12619, "end_char_idx": 15282, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Evaluate
eval_results = await retriever_evaluator.aevaluate_dataset(qa_dataset)
```

Let's define a function to display the Retrieval evaluation results in table format.


```python
def display_results(name, eval_results):
    """Display results from evaluate."""

    metric_dicts = []
    for eval_result in eval_results:
        metric_dict = eval_result.metric_vals_dict
        metric_dicts.append(metric_dict)

    full_df = pd.DataFrame(metric_dicts)

    hit_rate = full_df["hit_rate"].mean()
    mrr = full_df["mrr"].mean()

    metric_df = pd.DataFrame(
        {"Retriever Name": [name], "Hit Rate": [hit_rate], "MRR": [mrr]}
    )

    return metric_df
```


```python
display_results("OpenAI Embedding Retriever", eval_results)
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Retriever Name</th>
      <th>Hit Rate</th>
      <th>MRR</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>OpenAI Embedding Retriever</td>
      <td>0.758621</td>
      <td>0.62069</td>
    </tr>
  </tbody>
</table>
</div>



#### Observation:

The Retriever with OpenAI Embedding  demonstrates a performance with a hit rate of `0.7586`, while the MRR, at `0.6206`, suggests there's room for improvement in ensuring the most relevant results appear at the top. The observation that MRR is less than the hit rate indicates that the top-ranking results aren't always the most relevant. Enhancing MRR could involve the use of rerankers, which refine the order of retrieved documents. For a deeper understanding of how rerankers can optimize retrieval metrics, refer to the detailed discussion in our [blog post](https://blog.llamaindex.ai/boosting-rag-picking-the-best-embedding-reranker-models-42d079022e83).

#### Response Evaluation:

1. FaithfulnessEvaluator: Measures if the response from a query engine matches any source nodes which is useful for measuring if the response is hallucinated.
2. Relevancy Evaluator: Measures if the response + source nodes match the query.


```python
# Get the list of queries from the above created dataset

queries = list(qa_dataset.queries.values())
```

#### Faithfulness Evaluator

Let's start with FaithfulnessEvaluator.

We will use `gpt-3.5-turbo` for generating response for a given query and `gpt-4` for evaluation.

Let's create service_context seperately for `gpt-3.5-turbo` and `gpt-4`.


key: _node_content
string_value:
{"id_": "e23f31aa-5ac8-4f25-9b22-72f4cb413670", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f8307df-f435-40fc-91fd-e52c0970b1d2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3d87226cf3e1ecbc42d0bed8b11ca11fa5c935369f6c9e45c85efd4d1eec2978", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "86317943-ef96-462e-91de-cad208639fdb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "5eb97467bb193a103c8e6a850a974b9c4fcc3c3b1cfe2ee03fef586f40a5a743", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dd6ccc8c-f3c6-42c0-8f4f-fce6f727cf31", "node_type": "1", "metadata": {}, "hash": "4c2c05e99bc0e5d6ed09c4eb42b8b5e8d04fb4adfcd0cbafe1145c8497b96755", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13973, "end_char_idx": 17109, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
#### Observation:

The Retriever with OpenAI Embedding  demonstrates a performance with a hit rate of `0.7586`, while the MRR, at `0.6206`, suggests there's room for improvement in ensuring the most relevant results appear at the top. The observation that MRR is less than the hit rate indicates that the top-ranking results aren't always the most relevant. Enhancing MRR could involve the use of rerankers, which refine the order of retrieved documents. For a deeper understanding of how rerankers can optimize retrieval metrics, refer to the detailed discussion in our [blog post](https://blog.llamaindex.ai/boosting-rag-picking-the-best-embedding-reranker-models-42d079022e83).

#### Response Evaluation:

1. FaithfulnessEvaluator: Measures if the response from a query engine matches any source nodes which is useful for measuring if the response is hallucinated.
2. Relevancy Evaluator: Measures if the response + source nodes match the query.


```python
# Get the list of queries from the above created dataset

queries = list(qa_dataset.queries.values())
```

#### Faithfulness Evaluator

Let's start with FaithfulnessEvaluator.

We will use `gpt-3.5-turbo` for generating response for a given query and `gpt-4` for evaluation.

Let's create service_context seperately for `gpt-3.5-turbo` and `gpt-4`.


```python
# gpt-3.5-turbo
gpt35 = OpenAI(temperature=0, model="gpt-3.5-turbo")
service_context_gpt35 = ServiceContext.from_defaults(llm=gpt35)

# gpt-4
gpt4 = OpenAI(temperature=0, model="gpt-4")
service_context_gpt4 = ServiceContext.from_defaults(llm=gpt4)
```

Create a `QueryEngine` with `gpt-3.5-turbo` service_context to generate response for the query.


```python
vector_index = VectorStoreIndex(nodes, service_context = service_context_gpt35)
query_engine = vector_index.as_query_engine()
```

Create a  FaithfulnessEvaluator.





```python
from llama_index.evaluation import FaithfulnessEvaluator
faithfulness_gpt4 = FaithfulnessEvaluator(service_context=service_context_gpt4)
```

Let's evaluate on one question.


```python
eval_query = queries[10]

eval_query
```




    "Based on the author's experience and observations, why did he consider the AI practices during his first year of grad school as a hoax? Provide specific examples from the text to support your answer."



Generate response first and use faithfull evaluator.


```python
response_vector = query_engine.query(eval_query)
```


```python
# Compute faithfulness evaluation

eval_result = faithfulness_gpt4.evaluate_response(response=response_vector)
```


```python
# You can check passing parameter in eval_result if it passed the evaluation.
eval_result.passing
```




    True



#### Relevancy Evaluator

RelevancyEvaluator is useful to measure if the response and source nodes (retrieved context) match the query. Useful to see if response actually answers the query.

Instantiate `RelevancyEvaluator` for relevancy evaluation with `gpt-4`


```python
from llama_index.evaluation import RelevancyEvaluator

relevancy_gpt4 = RelevancyEvaluator(service_context=service_context_gpt4)
```

Let's do relevancy evaluation for one of the query.


key: _node_content
string_value:
{"id_": "dd6ccc8c-f3c6-42c0-8f4f-fce6f727cf31", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f8307df-f435-40fc-91fd-e52c0970b1d2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3d87226cf3e1ecbc42d0bed8b11ca11fa5c935369f6c9e45c85efd4d1eec2978", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e23f31aa-5ac8-4f25-9b22-72f4cb413670", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "512b06e91036f3f8068628ddfbd58047555177c8911aa6f1414a643ec3489b9f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7e873bb4-be80-460f-889b-5c241daacace", "node_type": "1", "metadata": {}, "hash": "029bba2fa3cccf385dc9a81d85edb398af99c2381665be7e9461197e76b7cbc1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15646, "end_char_idx": 18901, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
vector_index = VectorStoreIndex(nodes, service_context = service_context_gpt35)
query_engine = vector_index.as_query_engine()
```

Create a  FaithfulnessEvaluator.





```python
from llama_index.evaluation import FaithfulnessEvaluator
faithfulness_gpt4 = FaithfulnessEvaluator(service_context=service_context_gpt4)
```

Let's evaluate on one question.


```python
eval_query = queries[10]

eval_query
```




    "Based on the author's experience and observations, why did he consider the AI practices during his first year of grad school as a hoax? Provide specific examples from the text to support your answer."



Generate response first and use faithfull evaluator.


```python
response_vector = query_engine.query(eval_query)
```


```python
# Compute faithfulness evaluation

eval_result = faithfulness_gpt4.evaluate_response(response=response_vector)
```


```python
# You can check passing parameter in eval_result if it passed the evaluation.
eval_result.passing
```




    True



#### Relevancy Evaluator

RelevancyEvaluator is useful to measure if the response and source nodes (retrieved context) match the query. Useful to see if response actually answers the query.

Instantiate `RelevancyEvaluator` for relevancy evaluation with `gpt-4`


```python
from llama_index.evaluation import RelevancyEvaluator

relevancy_gpt4 = RelevancyEvaluator(service_context=service_context_gpt4)
```

Let's do relevancy evaluation for one of the query.


```python
# Pick a query
query = queries[10]

query
```




    "Based on the author's experience and observations, why did he consider the AI practices during his first year of grad school as a hoax? Provide specific examples from the text to support your answer."




```python
# Generate response.
# response_vector has response and source nodes (retrieved context)
response_vector = query_engine.query(query)

# Relevancy evaluation
eval_result = relevancy_gpt4.evaluate_response(
    query=query, response=response_vector
)
```


```python
# You can check passing parameter in eval_result if it passed the evaluation.
eval_result.passing
```




    True




```python
# You can get the feedback for the evaluation.
eval_result.feedback
```




    'YES'



#### Batch Evaluator:

Now that we have done FaithFulness and Relevancy Evaluation independently. LlamaIndex has `BatchEvalRunner` to compute multiple evaluations in batch wise manner.


```python
from llama_index.evaluation import BatchEvalRunner

# Let's pick top 10 queries to do evaluation
batch_eval_queries = queries[:10]

# Initiate BatchEvalRunner to compute FaithFulness and Relevancy Evaluation.
runner = BatchEvalRunner(
    {"faithfulness": faithfulness_gpt4, "relevancy": relevancy_gpt4},
    workers=8,
)

# Compute evaluation
eval_results = await runner.aevaluate_queries(
    query_engine, queries=batch_eval_queries
)
```


```python
# Let's get faithfulness score

faithfulness_score = sum(result.passing for result in eval_results['faithfulness']) / len(eval_results['faithfulness'])

faithfulness_score
```




    1.0




```python
# Let's get relevancy score

relevancy_score = sum(result.passing for result in eval_results['relevancy']) / len(eval_results['relevancy'])

relevancy_score

```




    1.0


key: _node_content
string_value:
{"id_": "7e873bb4-be80-460f-889b-5c241daacace", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6f8307df-f435-40fc-91fd-e52c0970b1d2", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3d87226cf3e1ecbc42d0bed8b11ca11fa5c935369f6c9e45c85efd4d1eec2978", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dd6ccc8c-f3c6-42c0-8f4f-fce6f727cf31", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ce531dd10450efe578a6e37a674d623bea6daa1ef24f646395d62a410b313f26", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17176, "end_char_idx": 19579, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"Based on the author's experience and observations, why did he consider the AI practices during his first year of grad school as a hoax? Provide specific examples from the text to support your answer."




```python
# Generate response.
# response_vector has response and source nodes (retrieved context)
response_vector = query_engine.query(query)

# Relevancy evaluation
eval_result = relevancy_gpt4.evaluate_response(
    query=query, response=response_vector
)
```


```python
# You can check passing parameter in eval_result if it passed the evaluation.
eval_result.passing
```




    True




```python
# You can get the feedback for the evaluation.
eval_result.feedback
```




    'YES'



#### Batch Evaluator:

Now that we have done FaithFulness and Relevancy Evaluation independently. LlamaIndex has `BatchEvalRunner` to compute multiple evaluations in batch wise manner.


```python
from llama_index.evaluation import BatchEvalRunner

# Let's pick top 10 queries to do evaluation
batch_eval_queries = queries[:10]

# Initiate BatchEvalRunner to compute FaithFulness and Relevancy Evaluation.
runner = BatchEvalRunner(
    {"faithfulness": faithfulness_gpt4, "relevancy": relevancy_gpt4},
    workers=8,
)

# Compute evaluation
eval_results = await runner.aevaluate_queries(
    query_engine, queries=batch_eval_queries
)
```


```python
# Let's get faithfulness score

faithfulness_score = sum(result.passing for result in eval_results['faithfulness']) / len(eval_results['faithfulness'])

faithfulness_score
```




    1.0




```python
# Let's get relevancy score

relevancy_score = sum(result.passing for result in eval_results['relevancy']) / len(eval_results['relevancy'])

relevancy_score

```




    1.0



#### Observation:

Faithfulness score of `1.0` signifies that the generated answers contain no hallucinations and are entirely based on retrieved context.

Relevancy score of `1.0` suggests that the answers generated are consistently aligned with the retrieved context and the queries.

## Conclusion

In this notebook, we have explored how to build and evaluate a RAG pipeline using LlamaIndex, with a specific focus on evaluating the retrieval system and generated responses within the pipeline. 

LlamaIndex offers a variety of other evaluation modules as well, which you can explore further [here](https://docs.llamaindex.ai/en/stable/module_guides/evaluating/root.html)


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb


key: _node_content
string_value:
{"id_": "7d6db1fc-41d6-40bf-829f-4cb4be61d03a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7f5fa63-a6d8-42f0-be9f-6da64870c366", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "2ac92a14208add6de440c8322277d55e8ca59151af9a73c7605b06cb2aa6327b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "689ebe18-fac3-49c0-87a2-6b08c88c5f4c", "node_type": "1", "metadata": {}, "hash": "55eed429aec6c27c9bdf5797d80594800ff9cf6a85932198ef18202f8ac12339", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2064, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT

This notebook provides step by step instuctions on using Azure AI Search (f.k.a Azure Cognitive Search) as a vector database with OpenAI embeddings, then creating an Azure Function on top to plug into a Custom GPT in ChatGPT. 

This can be a solution for customers looking to set up RAG infrastructure contained within Azure, and exposing it as an endpoint to integrate that with other platforms such as ChatGPT.

Azure AI Search is a cloud search service that gives developers infrastructure, APIs, and tools for building a rich search experience over private, heterogeneous content in web, mobile, and enterprise applications. 

Azure Functions is a serverless compute service that runs event-driven code, automatically managing infrastructure, scaling, and integrating with other Azure services.

## Prerequisites:
For the purposes of this exercise you must have the following:
- Azure user with permission to create [Azure AI Search Service](https://learn.microsoft.com/azure/search/) and Azure Function Apps
- Azure subscription ID and a resource group.
- [OpenAI Key](https://platform.openai.com/account/api-keys) 

# Architecture
Below is a diagram of the architecture of this solution, which we'll walk through step-by-step.

![azure-rag-architecture.png](../../../../images/azure-rag-architecture.png)


> Note: This architecture pattern of vector data store + serverless functions can be extrapolated to other vector data stores. For example, if you would want to use something like Postgres within Azure, you'd change the [Configure Azure AI Search Settings](#configure-azure-ai-search-settings) step to set-up the requirements for Postgres, you'd modify the [Create Azure AI Vector Search](#create-azure-ai-vector-search) to create the database and table in Postgres instead, and you'd update the `function_app.py` code in this repository to query Postgres instead of Azure AI Search. The data preparation and creation of the Azure Function would stay consistent.


key: _node_content
string_value:
{"id_": "689ebe18-fac3-49c0-87a2-6b08c88c5f4c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7f5fa63-a6d8-42f0-be9f-6da64870c366", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "2ac92a14208add6de440c8322277d55e8ca59151af9a73c7605b06cb2aa6327b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7d6db1fc-41d6-40bf-829f-4cb4be61d03a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "5ba85cea8ed56930c58a15b0a691baf4b4558f36f0510b22fea47a413019b210", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9f277bf4-7a78-4140-83b1-6f487458d2ff", "node_type": "1", "metadata": {}, "hash": "d5e5ff2e1959591ad2c62a929fa8af3de32e3f5cc13cb74fbd3d787c163e437e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1402, "end_char_idx": 4481, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
> Note: This architecture pattern of vector data store + serverless functions can be extrapolated to other vector data stores. For example, if you would want to use something like Postgres within Azure, you'd change the [Configure Azure AI Search Settings](#configure-azure-ai-search-settings) step to set-up the requirements for Postgres, you'd modify the [Create Azure AI Vector Search](#create-azure-ai-vector-search) to create the database and table in Postgres instead, and you'd update the `function_app.py` code in this repository to query Postgres instead of Azure AI Search. The data preparation and creation of the Azure Function would stay consistent. 


# Table of Contents:

1. **[Setup of Environment](#set-up-environment)**
    Setup environment by installing and importing the required libraries and configuring our Azure settings. Includes:
     - [Install and Import Required Libraries](#install-and-import-required-libraries)
     - [Configure OpenAI Settings](#configure-openai-settings)
     - [Configure Azure AI Search Settings](#configure-azure-ai-search-settings)
 

2. **[Prepare Data](#prepare-data)** Prepare the data for uploading by embedding the documents, as well as capturing additional metadata. We will use a subset of OpenAI's docs as example data for this.
 
3. **[Create Azure AI Vector Search](#create-azure-ai-vector-search)** Create an Azure AI Vector Search and upload the data we've prepared. Includes:
     - [Create Index](#create-index): Steps to create an index in Azure AI Search.
     - [Upload Data](#upload-data): Instructions to upload data to Azure AI Search.
     - [Test Search](#test-search): Steps to test the search functionality.
 
4. **[Create Azure Function](#create-azure-function)** Create an Azure Function to interact with the Azure AI Vector Search. Includes:
     - [Create Storage Account](#create-storage-account): Steps to create a storage account for the Azure Function.
     - [Create Function App](#create-function-app): Instructions to create a function app in Azure.
 
5. **[Input in a Custom GPT in ChatGPT](#input-in-a-custom-gpt-in-chatgpt)** Integrate the Azure Function with a Custom GPT in ChatGPT. Includes:
     - [Create OpenAPI Spec](#create-openapi-spec): Steps to create an OpenAPI specification for the Azure Function.
     - [Create GPT Instructions](#create-gpt-instructions): Instructions to create GPT-specific instructions for the integration.




# Set up environment
We'll set up our environment by importing the required libraries and configuring our Azure settings.

## Install and import required libraries
We categorize these libraries into standard Python libraries, third-party libraries, and Azure-related libraries for readability.


```python
! pip install -q wget
! pip install -q azure-search-documents 
! pip install -q azure-identity
! pip install -q openai
! pip install -q azure-mgmt-search
! pip install -q pandas
! pip install -q azure-mgmt-resource 
! pip install -q azure-mgmt-storage
! pip install -q pyperclip
! pip install -q PyPDF2
! pip install -q tiktoken
```


key: _node_content
string_value:
{"id_": "9f277bf4-7a78-4140-83b1-6f487458d2ff", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7f5fa63-a6d8-42f0-be9f-6da64870c366", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "2ac92a14208add6de440c8322277d55e8ca59151af9a73c7605b06cb2aa6327b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "689ebe18-fac3-49c0-87a2-6b08c88c5f4c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c891640ca6e94ee5afb409917d4b535b88dd3ddd7374e8514432cccdb675375d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "940d7f2f-f521-49b0-ae66-d873e2d2679d", "node_type": "1", "metadata": {}, "hash": "fac34c7bfe3727c21a1238c42e099bb83f0148da3f07d7da94308f4af2a37770", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3843, "end_char_idx": 6713, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Set up environment
We'll set up our environment by importing the required libraries and configuring our Azure settings.

## Install and import required libraries
We categorize these libraries into standard Python libraries, third-party libraries, and Azure-related libraries for readability.


```python
! pip install -q wget
! pip install -q azure-search-documents 
! pip install -q azure-identity
! pip install -q openai
! pip install -q azure-mgmt-search
! pip install -q pandas
! pip install -q azure-mgmt-resource 
! pip install -q azure-mgmt-storage
! pip install -q pyperclip
! pip install -q PyPDF2
! pip install -q tiktoken
```


```python
# Standard Libraries
import json  
import os
import platform
import subprocess
import csv
from itertools import islice
import uuid
import shutil
import concurrent.futures

# Third-Party Libraries
import pandas as pd
from PyPDF2 import PdfReader
import tiktoken
from dotenv import load_dotenv
import pyperclip

# OpenAI Libraries (note we use OpenAI directly here, but you can replace with Azure OpenAI as needed)
from openai import OpenAI

# Azure Identity and Credentials
from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential
from azure.core.credentials import AzureKeyCredential  
from azure.core.exceptions import HttpResponseError

# Azure Search Documents
from azure.search.documents import SearchClient, SearchIndexingBufferedSender  
from azure.search.documents.indexes import SearchIndexClient  
from azure.search.documents.models import (
    VectorizedQuery
)
from azure.search.documents.indexes.models import (
    HnswAlgorithmConfiguration,
    HnswParameters,
    SearchField,
    SearchableField,
    SearchFieldDataType,
    SearchIndex,
    SimpleField,
    VectorSearch,
    VectorSearchAlgorithmKind,
    VectorSearchAlgorithmMetric,
    VectorSearchProfile,
)

# Azure Management Clients
from azure.mgmt.search import SearchManagementClient
from azure.mgmt.resource import ResourceManagementClient, SubscriptionClient
from azure.mgmt.storage import StorageManagementClient
```

## Configure OpenAI settings

Before going through this section, make sure you have your OpenAI API key.



```python
openai_api_key = os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as an env var>") # Saving this as a variable to reference in function app in later step
openai_client = OpenAI(api_key=openai_api_key)
embeddings_model = "text-embedding-3-small" # We'll use this by default, but you can change to your text-embedding-3-large if desired
```

## Configure Azure AI Search Settings
You can locate your Azure AI Search service details in the Azure Portal or programmatically via the [Search Management SDK](https://learn.microsoft.com/rest/api/searchmanagement/).


#### Prerequisites:
- Subscription ID from Azure
- Resource Group name from Azure
- Region in Azure


key: _node_content
string_value:
{"id_": "940d7f2f-f521-49b0-ae66-d873e2d2679d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7f5fa63-a6d8-42f0-be9f-6da64870c366", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "2ac92a14208add6de440c8322277d55e8ca59151af9a73c7605b06cb2aa6327b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9f277bf4-7a78-4140-83b1-6f487458d2ff", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "53a4f03c09ddefb4bacf7c45a5411bb838ad586e33f33ef017eaddd095b4cef9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f9e7d49c-7885-4d77-ae23-f0abd71defe7", "node_type": "1", "metadata": {}, "hash": "8b072004b85a152337416fbe4fe776eee7fd5b9e4dad835a6a57bf19c37fd374", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6026, "end_char_idx": 9267, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
openai_api_key = os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as an env var>") # Saving this as a variable to reference in function app in later step
openai_client = OpenAI(api_key=openai_api_key)
embeddings_model = "text-embedding-3-small" # We'll use this by default, but you can change to your text-embedding-3-large if desired
```

## Configure Azure AI Search Settings
You can locate your Azure AI Search service details in the Azure Portal or programmatically via the [Search Management SDK](https://learn.microsoft.com/rest/api/searchmanagement/).


#### Prerequisites:
- Subscription ID from Azure
- Resource Group name from Azure
- Region in Azure


```python
# Update the below with your values
subscription_id="<enter_your_subscription_id>"
resource_group="<enter_your_resource_group>"

## Make sure to choose a region that supports the proper products. We've defaulted to "eastus" below. https://azure.microsoft.com/en-us/explore/global-infrastructure/products-by-region/#products-by-region_tab5
region = "eastus"
credential = InteractiveBrowserCredential()
subscription_client = SubscriptionClient(credential)
subscription = next(subscription_client.subscriptions.list())
```

#### Create and Configure Azure AI Search Service
Below we'll generate a unique name for the search service, set up the service properties, and create the search service.


```python
# Initialize the SearchManagementClient with the provided credentials and subscription ID
search_management_client = SearchManagementClient(
    credential=credential,
    subscription_id=subscription_id,
)

# Generate a unique name for the search service using UUID, but you can change this if you'd like.
generated_uuid = str(uuid.uuid4())
search_service_name = "search-service-gpt-demo" + generated_uuid
## The below is the default endpoint structure that is created when you create a search service. This may differ based on your Azure settings.
search_service_endpoint = 'https://'+search_service_name+'.search.windows.net'

# Create or update the search service with the specified parameters
response = search_management_client.services.begin_create_or_update(
    resource_group_name=resource_group,
    search_service_name=search_service_name,
    service={
        "location": region,
        "properties": {"hostingMode": "default", "partitionCount": 1, "replicaCount": 1},
        # We are using the free pricing tier for this demo. You are only allowed one free search service per subscription.
        "sku": {"name": "free"},
        "tags": {"app-name": "Search service demo"},
    },
).result()

# Convert the response to a dictionary and then to a pretty-printed JSON string
response_dict = response.as_dict()
response_json = json.dumps(response_dict, indent=4)

print(response_json)
print("Search Service Name:" + search_service_name)
print("Search Service Endpoint:" + search_service_endpoint)

```

#### Get the Search Service API Key
Now that we have the search service up and running, we need the [Search Service API Key](https://learn.microsoft.com/en-us/azure/search/search-security-api-keys?tabs=rest-use,portal-find,portal-query), which we'll use to initiate the index creation, and later to execute the search.


key: _node_content
string_value:
{"id_": "f9e7d49c-7885-4d77-ae23-f0abd71defe7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "c7f5fa63-a6d8-42f0-be9f-6da64870c366", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "2ac92a14208add6de440c8322277d55e8ca59151af9a73c7605b06cb2aa6327b", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "940d7f2f-f521-49b0-ae66-d873e2d2679d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "b745a46e58552127728ef862a0063e9b5ab79e2189c4d20bbf083c10189220a8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6b53a416-db83-436e-8865-7b7f6cf44eee", "node_type": "1", "metadata": {}, "hash": "43ae46e9a96cffdd57e67dd4216650cd12021baa719055bc26b55c64691191a2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7420, "end_char_idx": 10245, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Initialize the SearchManagementClient with the provided credentials and subscription ID
search_management_client = SearchManagementClient(
    credential=credential,
    subscription_id=subscription_id,
)

# Generate a unique name for the search service using UUID, but you can change this if you'd like.
generated_uuid = str(uuid.uuid4())
search_service_name = "search-service-gpt-demo" + generated_uuid
## The below is the default endpoint structure that is created when you create a search service. This may differ based on your Azure settings.
search_service_endpoint = 'https://'+search_service_name+'.search.windows.net'

# Create or update the search service with the specified parameters
response = search_management_client.services.begin_create_or_update(
    resource_group_name=resource_group,
    search_service_name=search_service_name,
    service={
        "location": region,
        "properties": {"hostingMode": "default", "partitionCount": 1, "replicaCount": 1},
        # We are using the free pricing tier for this demo. You are only allowed one free search service per subscription.
        "sku": {"name": "free"},
        "tags": {"app-name": "Search service demo"},
    },
).result()

# Convert the response to a dictionary and then to a pretty-printed JSON string
response_dict = response.as_dict()
response_json = json.dumps(response_dict, indent=4)

print(response_json)
print("Search Service Name:" + search_service_name)
print("Search Service Endpoint:" + search_service_endpoint)

```

#### Get the Search Service API Key
Now that we have the search service up and running, we need the [Search Service API Key](https://learn.microsoft.com/en-us/azure/search/search-security-api-keys?tabs=rest-use,portal-find,portal-query), which we'll use to initiate the index creation, and later to execute the search.


```python
# Retrieve the admin keys for the search service
try:
    response = search_management_client.admin_keys.get(
        resource_group_name=resource_group,
        search_service_name=search_service_name,
    )
    # Extract the primary API key from the response and save as a variable to be used later
    search_service_api_key = response.primary_key
    print("Successfully retrieved the API key.")
except Exception as e:
    print(f"Failed to retrieve the API key: {e}")
```

# Prepare data
We're going to embed and store a few pages of the OpenAI docs in the oai_docs folder. We'll first embed each, add it to a CSV, and then use that CSV to upload to the index.

In order to handle longer text files beyond the context of 8191 tokens, we can either use the chunk embeddings separately, or combine them in some way, such as averaging (weighted by the size of each chunk).

We will take a function from Python's own cookbook that breaks up a sequence into chunks.


