# embedding_metadata

key: chroma:document
string_value:
evaluation_metrics = {
    "Relevance": (RELEVANCY_SCORE_CRITERIA, RELEVANCY_SCORE_STEPS),
    "Coherence": (COHERENCE_SCORE_CRITERIA, COHERENCE_SCORE_STEPS),
    "Consistency": (CONSISTENCY_SCORE_CRITERIA, CONSISTENCY_SCORE_STEPS),
    "Fluency": (FLUENCY_SCORE_CRITERIA, FLUENCY_SCORE_STEPS),
}

summaries = {"Summary 1": eval_summary_1, "Summary 2": eval_summary_2}

data = {"Evaluation Type": [], "Summary Type": [], "Score": []}

for eval_type, (criteria, steps) in evaluation_metrics.items():
    for summ_type, summary in summaries.items():
        data["Evaluation Type"].append(eval_type)
        data["Summary Type"].append(summ_type)
        result = get_geval_score(criteria, steps, excerpt, summary, eval_type)
        score_num = int(result.strip())
        data["Score"].append(score_num)

pivot_df = pd.DataFrame(data, index=None).pivot(
    index="Evaluation Type", columns="Summary Type", values="Score"
)
styled_pivot_df = pivot_df.style.apply(highlight_max, axis=1)
display(styled_pivot_df)
```


<style type="text/css">
#T_94fab_row0_col0, #T_94fab_row1_col0, #T_94fab_row1_col1, #T_94fab_row2_col0, #T_94fab_row3_col0 {
  background-color: lightgreen;
}
#T_94fab_row0_col1, #T_94fab_row2_col1, #T_94fab_row3_col1 {
  background-color: white;
}
</style>
<table id="T_94fab">
  <thead>
    <tr>
      <th class="index_name level0" >Summary Type</th>
      <th id="T_94fab_level0_col0" class="col_heading level0 col0" >Summary 1</th>
      <th id="T_94fab_level0_col1" class="col_heading level0 col1" >Summary 2</th>
    </tr>
    <tr>
      <th class="index_name level0" >Evaluation Type</th>
      <th class="blank col0" >&nbsp;</th>
      <th class="blank col1" >&nbsp;</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th id="T_94fab_level0_row0" class="row_heading level0 row0" >Coherence</th>
      <td id="T_94fab_row0_col0" class="data row0 col0" >5</td>
      <td id="T_94fab_row0_col1" class="data row0 col1" >3</td>
    </tr>
    <tr>
      <th id="T_94fab_level0_row1" class="row_heading level0 row1" >Consistency</th>
      <td id="T_94fab_row1_col0" class="data row1 col0" >5</td>
      <td id="T_94fab_row1_col1" class="data row1 col1" >5</td>
    </tr>
    <tr>
      <th id="T_94fab_level0_row2" class="row_heading level0 row2" >Fluency</th>
      <td id="T_94fab_row2_col0" class="data row2 col0" >3</td>
      <td id="T_94fab_row2_col1" class="data row2 col1" >2</td>
    </tr>
    <tr>
      <th id="T_94fab_level0_row3" class="row_heading level0 row3" >Relevance</th>
      <td id="T_94fab_row3_col0" class="data row3 col0" >5</td>
      <td id="T_94fab_row3_col1" class="data row3 col1" >4</td>
    </tr>
  </tbody>
</table>


key: _node_content
string_value:
{"id_": "d3e99123-8fa4-4d23-a1b1-3e744915a2f0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "dab62be3-40f5-4f6c-8c90-010816f48e4d", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "36739c1ae56ea864ddf9ac51bad53d5d7786811e58e4930e10c082fac39cd24d", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2b61792e-b10e-41a7-818d-d8baa65d00c7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "23452416588d307b2144d8a52eef91571b5a5aaf3dc9488a1a87cf5043dfba1c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23919, "end_char_idx": 26471, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<IPython.core.display.Javascript object>



Overall, the Summary 1 appears to outperform Summary 2 in three of the four categories (Coherence, Relevance and Fluency). Both summaries are found to be consistent with each other. The result might suggest that Summary 1 is generally preferable based on the given evaluation criteria.

### Limitations

Note that LLM-based metrics could have a bias towards preferring LLM-generated texts over human-written texts. Additionally LLM based metrics are sensitive to system messages/prompts. We recommend experimenting with other techniques that can help improve performance and/or get consistent scores, striking the right balance between high-quality expensive evaluation and automated evaluations. It is also worth noting that this scoring methodology is currently limited by `gpt-4`'s context window.

## Conclusion

Evaluating abstractive summarization remains an open area for further improvement. Traditional metrics like `ROUGE`, `BLEU`, and `BERTScore` provide useful automatic evaluation but have limitations in capturing semantic similarity and nuanced aspects of summarization quality. Moreover, they require reference outputs which can be expensive to collect/label. LLM-based metrics offer promise as a reference-free method of evaluating coherence, fluency, and relevance. However, they too have potential biases favoring text generated by LLMs. Ultimately, a combination of automatic metrics and human evaluation is ideal for reliably assessing abstractive summarization systems. While human evaluation is indispensable for gaining a comprehensive understanding of summary quality, it should be complemented with automated evaluation to enable efficient, large-scale testing. The field will continue to evolve more robust evaluation techniques, balancing quality, scalability, and fairness. Advancing evaluation methods is crucial for driving progress in production applications.

## References

- [G-EVAL: NLG Evaluation Using GPT-4 with Better Human Alignment](https://arxiv.org/pdf/2303.16634.pdf) - Liu Y, Iter D, Xu Y, Wang S, Xu R, Zhu C. Published May, 2023.
- [BERTScore: Evaluating Text Generation with BERT](https://arxiv.org/abs/1904.09675) - Zhang T, Kishore V, Wu F, Weinberger KQ, Artzi Y. Published online February, 2020.
- [ROUGE: A Package for Automatic Evaluation of Summaries](https://aclanthology.org/W04-1013/) - Lin CY. Published July, 2004.
- [SummEval: Re-evaluating Summarization Evaluation](https://aclanthology.org/2021.tacl-1.24) - Fabbri et al. Published April, 2021.


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb


key: _node_content
string_value:
{"id_": "bc340776-f54a-47e6-960e-542283aa0dc7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "23fc111e-8df6-433c-9a2e-2b76c9871f25", "node_type": "1", "metadata": {}, "hash": "ab4c3631d8abc55b383c576a8b460c6c41062f442df7d0e09a6ead8c1240672b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3413, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Getting Started with OpenAI Evals

The [OpenAI Evals](https://github.com/openai/evals/tree/main) framework consists of
1. A framework to evaluate an LLM (large language model) or a system built on top of an LLM.
2. An open-source registry of challenging evals

This notebook will cover:
* Introduction to Evaluation and the [OpenAI Evals](https://github.com/openai/evals/tree/main) library
* Building an Eval
* Running an Eval

#### What are evaluations/ `evals`?

Evaluation is the process of validating and testing the outputs that your LLM applications are producing. Having strong evaluations ("evals") will mean a more stable, reliable application that is resilient to code and model changes. An eval is a task used to measure the quality of the output of an LLM or LLM system. Given an input prompt, an output is generated. We evaluate this output with a set of ideal answers and find the quality of the LLM system.

#### Importance of Evaluations

If you are building with foundational models like `GPT-4`, creating high quality evals is one of the most impactful things you can do. Developing AI solutions involves an iterative design process. [Without evals, it can be very difficult and time intensive to understand](https://youtu.be/XGJNo8TpuVA?feature=shared&t=1089) how different model versions and prompts might affect your use case.

With OpenAI’s [continuous model upgrades](https://platform.openai.com/docs/models/continuous-model-upgrades), evals allow you to efficiently test model performance for your use cases in a standardized way. Developing a suite of evals customized to your objectives will help you quickly and effectively understand how new models may perform for your use cases. You can also make evals a part of your CI/CD pipeline to make sure you achieve the desired accuracy before deploying.

#### Types of evals

There are two main ways we can evaluate/grade completions: writing some validation logic in code
or using the model itself to inspect the answer. We’ll introduce each with some examples.

**Writing logic for answer checking**

The simplest and most common type of eval has an input and an ideal response or answer. For example,
we can have an eval sample where the input is "What year was Obama elected president for the first
time?" and the ideal answer is "2008". We feed the input to a model and get the completion. If the model
says "2008", it is then graded as correct. We can write a string match to check if the completion includes the phrase "2008". If it does, we consider it correct.

Consider another eval where the input is to generate valid JSON: We can write some code that
attempts to parse the completion as JSON and then considers the completion correct if it is
parsable.

**Model grading: A two stage process where the model first answers the question, then we ask a
model to look at the response to check if it’s correct.**

Consider an input that asks the model to write a funny joke. The model then generates a
completion. We then create a new input to the model to answer the question: "Is this following
joke funny? First reason step by step, then answer yes or no" that includes the completion." We
finally consider the original completion correct if the new model completion ends with "yes".

Model grading works best with the latest, most powerful models like `GPT-4` and if we give them the ability
to reason before making a judgment.


key: _node_content
string_value:
{"id_": "23fc111e-8df6-433c-9a2e-2b76c9871f25", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bc340776-f54a-47e6-960e-542283aa0dc7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "38029a5b18c45432a1b640575a06a0261f607c874da72bab845523d1d2ab7152", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4c967fc9-fd7d-47e7-888a-a35ed1bd8f30", "node_type": "1", "metadata": {}, "hash": "0fc158e077f087629de75301c3a8ad9ef30edf09d88ff0e6a6816311e60c4f1a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1711, "end_char_idx": 4725, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
You can also make evals a part of your CI/CD pipeline to make sure you achieve the desired accuracy before deploying.

#### Types of evals

There are two main ways we can evaluate/grade completions: writing some validation logic in code
or using the model itself to inspect the answer. We’ll introduce each with some examples.

**Writing logic for answer checking**

The simplest and most common type of eval has an input and an ideal response or answer. For example,
we can have an eval sample where the input is "What year was Obama elected president for the first
time?" and the ideal answer is "2008". We feed the input to a model and get the completion. If the model
says "2008", it is then graded as correct. We can write a string match to check if the completion includes the phrase "2008". If it does, we consider it correct.

Consider another eval where the input is to generate valid JSON: We can write some code that
attempts to parse the completion as JSON and then considers the completion correct if it is
parsable.

**Model grading: A two stage process where the model first answers the question, then we ask a
model to look at the response to check if it’s correct.**

Consider an input that asks the model to write a funny joke. The model then generates a
completion. We then create a new input to the model to answer the question: "Is this following
joke funny? First reason step by step, then answer yes or no" that includes the completion." We
finally consider the original completion correct if the new model completion ends with "yes".

Model grading works best with the latest, most powerful models like `GPT-4` and if we give them the ability
to reason before making a judgment. Model grading will have an error rate, so it is important to validate
the performance with human evaluation before running the evals at scale. For best results, it makes
sense to use a different model to do grading from the one that did the completion, like using `GPT-4` to
grade `GPT-3.5` answers.

#### OpenAI Eval Templates

In using evals, we have discovered several "templates" that accommodate many different benchmarks. We have implemented these templates in the OpenAI Evals library to simplify the development of new evals. For example, we have defined 2 types of eval templates that can be used out of the box:

* **Basic Eval Templates**: These contain deterministic functions to compare the output to the ideal_answers. In cases where the desired model response has very little variation, such as answering multiple choice questions or simple questions with a straightforward answer, we have found this following templates to be useful.

* **Model-Graded Templates**: These contain functions where an LLM compares the output to the ideal_answers and attempts to judge the accuracy. In cases where the desired model response can contain significant variation, such as answering an open-ended question, we have found that using the model to grade itself is a viable strategy for automated evaluation.


key: _node_content
string_value:
{"id_": "4c967fc9-fd7d-47e7-888a-a35ed1bd8f30", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "23fc111e-8df6-433c-9a2e-2b76c9871f25", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "cb89f44b483d26f496649d703364a80a49127560da5172618a243c0bb6ca2e30", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d88cec30-e3f7-48d6-97dd-466844bd54d6", "node_type": "1", "metadata": {}, "hash": "ef15426fa16a5af3c64c0c7b38afeb6eb2d83277228ce31476ef3c04a0a2e8e7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2892, "end_char_idx": 6042, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
**

Consider an input that asks the model to write a funny joke. The model then generates a
completion. We then create a new input to the model to answer the question: "Is this following
joke funny? First reason step by step, then answer yes or no" that includes the completion." We
finally consider the original completion correct if the new model completion ends with "yes".

Model grading works best with the latest, most powerful models like `GPT-4` and if we give them the ability
to reason before making a judgment. Model grading will have an error rate, so it is important to validate
the performance with human evaluation before running the evals at scale. For best results, it makes
sense to use a different model to do grading from the one that did the completion, like using `GPT-4` to
grade `GPT-3.5` answers.

#### OpenAI Eval Templates

In using evals, we have discovered several "templates" that accommodate many different benchmarks. We have implemented these templates in the OpenAI Evals library to simplify the development of new evals. For example, we have defined 2 types of eval templates that can be used out of the box:

* **Basic Eval Templates**: These contain deterministic functions to compare the output to the ideal_answers. In cases where the desired model response has very little variation, such as answering multiple choice questions or simple questions with a straightforward answer, we have found this following templates to be useful.

* **Model-Graded Templates**: These contain functions where an LLM compares the output to the ideal_answers and attempts to judge the accuracy. In cases where the desired model response can contain significant variation, such as answering an open-ended question, we have found that using the model to grade itself is a viable strategy for automated evaluation.


### Getting Setup

First, go to [github.com/openai/evals](https://github.com/openai/evals), clone the repository with `git clone git@github.com:openai/evals.git` and go through the [setup instructions](https://github.com/openai/evals). 

To run evals later in this notebook, you will need to set up and specify your OpenAI API key. After you obtain an API key, specify it using the `OPENAI_API_KEY` environment variable. 

Please be aware of the costs associated with using the API when running evals.


```python
from openai import OpenAI
import pandas as pd

client = OpenAI()
```

## Building an evaluation for OpenAI Evals framework

At its core, an eval is a dataset and an eval class that is defined in a YAML file. To start creating an eval, we need

1. The test dataset in the `jsonl` format.
2. The eval template to be used

### Creating the eval dataset
Lets create a dataset for a use case where we are evaluating the model's ability to generate syntactically correct SQL. In this use case, we have a series of tables that are related to car manufacturing

First we will need to create a system prompt that we would like to evaluate. We will pass in instructions for the model as well as an overview of the table structure:
```
"TASK: Answer the following question with syntactically correct SQLite SQL.


key: _node_content
string_value:
{"id_": "d88cec30-e3f7-48d6-97dd-466844bd54d6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4c967fc9-fd7d-47e7-888a-a35ed1bd8f30", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "574d841494e4d3462013d80195796b1eb123d1177ea85cc8d7c5e7d0de4e56b6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "12ea236f-05b4-4e2f-8468-dc59603f7a14", "node_type": "1", "metadata": {}, "hash": "098fa12002a403201c2dc3ce999656ad9eb8764397035ddb0aa8c39930dd9500", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4365, "end_char_idx": 7321, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
* **Model-Graded Templates**: These contain functions where an LLM compares the output to the ideal_answers and attempts to judge the accuracy. In cases where the desired model response can contain significant variation, such as answering an open-ended question, we have found that using the model to grade itself is a viable strategy for automated evaluation.


### Getting Setup

First, go to [github.com/openai/evals](https://github.com/openai/evals), clone the repository with `git clone git@github.com:openai/evals.git` and go through the [setup instructions](https://github.com/openai/evals). 

To run evals later in this notebook, you will need to set up and specify your OpenAI API key. After you obtain an API key, specify it using the `OPENAI_API_KEY` environment variable. 

Please be aware of the costs associated with using the API when running evals.


```python
from openai import OpenAI
import pandas as pd

client = OpenAI()
```

## Building an evaluation for OpenAI Evals framework

At its core, an eval is a dataset and an eval class that is defined in a YAML file. To start creating an eval, we need

1. The test dataset in the `jsonl` format.
2. The eval template to be used

### Creating the eval dataset
Lets create a dataset for a use case where we are evaluating the model's ability to generate syntactically correct SQL. In this use case, we have a series of tables that are related to car manufacturing

First we will need to create a system prompt that we would like to evaluate. We will pass in instructions for the model as well as an overview of the table structure:
```
"TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]"
```

For this prompt, we can ask a specific question:
```
"Q: how many car makers are their in germany?"
```

And we have an expected answer:
```
"A: SELECT count ( * )  FROM CAR_MAKERS AS T1 JOIN COUNTRIES AS T2 ON T1.Country   =   T2.CountryId WHERE T2.CountryName   =   'germany'"
```

The dataset needs to be in the following format:
```
"input": [{"role": "system", "content": "<input prompt>"}, {"role": "user", "content": <user input>}, "ideal": "correct answer"]
```

Putting it all together, we get:
```
{"input": [{"role": "system", "content": "TASK: Answer the following question with syntactically correct SQLite SQL.


key: _node_content
string_value:
{"id_": "12ea236f-05b4-4e2f-8468-dc59603f7a14", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d88cec30-e3f7-48d6-97dd-466844bd54d6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "e19ffa91b3260dd2e52de284c303fb26d26e6a0dd2ff0c0a3b46a2bfded16f56", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cbbed732-83f1-40fd-8167-955b684f896e", "node_type": "1", "metadata": {}, "hash": "a7179d32f86a295a13789a06d4f3696e44fedf34b12da732b4b2e2b6d6a01d54", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5873, "end_char_idx": 8791, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We will pass in instructions for the model as well as an overview of the table structure:
```
"TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]"
```

For this prompt, we can ask a specific question:
```
"Q: how many car makers are their in germany?"
```

And we have an expected answer:
```
"A: SELECT count ( * )  FROM CAR_MAKERS AS T1 JOIN COUNTRIES AS T2 ON T1.Country   =   T2.CountryId WHERE T2.CountryName   =   'germany'"
```

The dataset needs to be in the following format:
```
"input": [{"role": "system", "content": "<input prompt>"}, {"role": "user", "content": <user input>}, "ideal": "correct answer"]
```

Putting it all together, we get:
```
{"input": [{"role": "system", "content": "TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]\n"}, {"role": "system", "content": "Q: how many car makers are their in germany"}, "ideal": ["A: SELECT count ( * )  FROM CAR_MAKERS AS T1 JOIN COUNTRIES AS T2 ON T1.Country   =   T2.CountryId WHERE T2.CountryName   =   'germany'"]}
```


One way to speed up the process of building eval datasets, is to use `GPT-4` to generate synthetic data


```python
## Use GPT-4 to generate synthetic data
# Define the system prompt and user input (these should be filled as per the specific use case)
system_prompt = """You are a helpful assistant that can ask questions about a database table and write SQL queries to answer the question.
    A user will pass in a table schema and your job is to return a question answer pairing. The question should relevant to the schema of the table,
    and you can speculate on its contents.


key: _node_content
string_value:
{"id_": "cbbed732-83f1-40fd-8167-955b684f896e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "12ea236f-05b4-4e2f-8468-dc59603f7a14", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8356451ce944a77314e2b0abb1aa8dbd35df75048f2e97a5ad8ce240eba918e0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6b8d4de4-86d6-478c-9e69-84829cc22308", "node_type": "1", "metadata": {}, "hash": "944dd52fafe091ffa8cacc3ed42efca11af9cbbaae0e63d1dd6f8567fc92b130", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7322, "end_char_idx": 9957, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]\n"}, {"role": "system", "content": "Q: how many car makers are their in germany"}, "ideal": ["A: SELECT count ( * )  FROM CAR_MAKERS AS T1 JOIN COUNTRIES AS T2 ON T1.Country   =   T2.CountryId WHERE T2.CountryName   =   'germany'"]}
```


One way to speed up the process of building eval datasets, is to use `GPT-4` to generate synthetic data


```python
## Use GPT-4 to generate synthetic data
# Define the system prompt and user input (these should be filled as per the specific use case)
system_prompt = """You are a helpful assistant that can ask questions about a database table and write SQL queries to answer the question.
    A user will pass in a table schema and your job is to return a question answer pairing. The question should relevant to the schema of the table,
    and you can speculate on its contents. You will then have to generate a SQL query to answer the question. Below are some examples of what this should look like.

    Example 1
    ```````````
    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\n
    Assistant Response:
    Q: How many visitors have visited the museum with the most staff?
    A: SELECT count ( * )  FROM VISIT AS T1 JOIN MUSEUM AS T2 ON T1.Museum_ID   =   T2.Museum_ID WHERE T2.Num_of_Staff   =   ( SELECT max ( Num_of_Staff )  FROM MUSEUM ) 
    ```````````

    Example 2
    ```````````
    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\n
    Assistant Response:
    Q: What are the names who have a membership level higher than 4?


key: _node_content
string_value:
{"id_": "6b8d4de4-86d6-478c-9e69-84829cc22308", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cbbed732-83f1-40fd-8167-955b684f896e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "eeec75448ce3ead6304e2bb70d57889ac18db5235cfedca50b487978f2195d6a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b1b4b880-25ec-4307-ba60-f1304aed71e0", "node_type": "1", "metadata": {}, "hash": "b20c3a7abd7d0dd5b2d836298bfb85360eeb3498b6d85f8a6af5dc2f9a7883e0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8604, "end_char_idx": 10452, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
A user will pass in a table schema and your job is to return a question answer pairing. The question should relevant to the schema of the table,
    and you can speculate on its contents. You will then have to generate a SQL query to answer the question. Below are some examples of what this should look like.

    Example 1
    ```````````
    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\n
    Assistant Response:
    Q: How many visitors have visited the museum with the most staff?
    A: SELECT count ( * )  FROM VISIT AS T1 JOIN MUSEUM AS T2 ON T1.Museum_ID   =   T2.Museum_ID WHERE T2.Num_of_Staff   =   ( SELECT max ( Num_of_Staff )  FROM MUSEUM ) 
    ```````````

    Example 2
    ```````````
    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\n
    Assistant Response:
    Q: What are the names who have a membership level higher than 4?
    A: SELECT Name   FROM VISITOR AS T1 WHERE T1.Level_of_membership   >   4 
    ```````````

    Example 3
    ```````````
    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\n
    Assistant Response:
    Q: How many tickets of customer id 5?


key: _node_content
string_value:
{"id_": "b1b4b880-25ec-4307-ba60-f1304aed71e0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6b8d4de4-86d6-478c-9e69-84829cc22308", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "5a4a99c1dfca9ed8ad0352124a203601e14d8b94ec13612adde14aa4d4eee072", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ffee6f55-2362-4906-aae9-a7e497df2526", "node_type": "1", "metadata": {}, "hash": "36b266f7fb045846f1558880bb533a39b93c3c4c5d4c17b4d9c8482a210e7bf2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9347, "end_char_idx": 12032, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
A: SELECT count ( * )  FROM VISIT AS T1 JOIN MUSEUM AS T2 ON T1.Museum_ID   =   T2.Museum_ID WHERE T2.Num_of_Staff   =   ( SELECT max ( Num_of_Staff )  FROM MUSEUM ) 
    ```````````

    Example 2
    ```````````
    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\n
    Assistant Response:
    Q: What are the names who have a membership level higher than 4?
    A: SELECT Name   FROM VISITOR AS T1 WHERE T1.Level_of_membership   >   4 
    ```````````

    Example 3
    ```````````
    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\n
    Assistant Response:
    Q: How many tickets of customer id 5?
    A: SELECT count ( * )  FROM VISIT AS T1 JOIN VISITOR AS T2 ON T1.visitor_ID   =   T2.ID WHERE T2.ID   =   5 
    ```````````
    """

user_input = "Table car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]"

messages = [{
        "role": "system",
        "content": system_prompt
    },
    {
        "role": "user",
        "content": user_input
    }
]

completion = client.chat.completions.create(
    model="gpt-4-turbo-preview",
    messages=messages,
    temperature=0.7,
    n=5
)

for choice in completion.choices:
    print(choice.message.content + "\n")

```

    Q: What is the average horsepower for cars made in Europe?
    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'
    
    Q: What is the average horsepower for cars made in the USA?


key: _node_content
string_value:
{"id_": "ffee6f55-2362-4906-aae9-a7e497df2526", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b1b4b880-25ec-4307-ba60-f1304aed71e0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "85b4c5371c3bb17dbc8099bd6eaa26b3a14eff3e793ef654b45c38fe7e84f368", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d8fbaeb5-9146-4d9e-8e82-2b81f75fa4de", "node_type": "1", "metadata": {}, "hash": "b11368cbd5e605f39b7c48292cbf7cce0831e71d58e51a566ce46279fe839d9b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11601, "end_char_idx": 13747, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'
    
    Q: What is the average horsepower for cars made in the USA?
    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'
    
    Q: What is the average horsepower for cars produced in countries from the continent with the id '3'?
    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.ContId = '3'
    
    Q: What is the average horsepower for cars made by makers from Europe?
    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'
    
    Q: What is the average horsepower for cars made in the USA?
    
    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'
    


Once we have the synthetic data, we need to convert it to match the format of the eval dataset.


```python
eval_data = []
input_prompt = "TASK: Answer the following question with syntactically correct SQLite SQL.


key: _node_content
string_value:
{"id_": "d8fbaeb5-9146-4d9e-8e82-2b81f75fa4de", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ffee6f55-2362-4906-aae9-a7e497df2526", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0006e3efd6d39948546e561c9e61c61cd06b236878d5d6f4e43191a77590f4ef", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ca912825-4eeb-4b31-bafc-8d9341952498", "node_type": "1", "metadata": {}, "hash": "2b450c9ac0bbdae406b9891bad4de9e27a58db3b165a5b0f405ba1c9eb765058", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12037, "end_char_idx": 14979, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'
    
    Q: What is the average horsepower for cars produced in countries from the continent with the id '3'?
    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.ContId = '3'
    
    Q: What is the average horsepower for cars made by makers from Europe?
    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'
    
    Q: What is the average horsepower for cars made in the USA?
    
    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'
    


Once we have the synthetic data, we need to convert it to match the format of the eval dataset.


```python
eval_data = []
input_prompt = "TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]"

for choice in completion.choices:
    question = choice.message.content.split("Q: ")[1].split("\n")[0]  # Extracting the question
    answer = choice.message.content.split("\nA: ")[1].split("\n")[0]  # Extracting the answer
    eval_data.append({
        "input": [
            {"role": "system", "content": input_prompt},
            {"role": "user", "content": question},
        ],
        "ideal": answer
    })

for item in eval_data:
    print(item)
```

    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL.


key: _node_content
string_value:
{"id_": "ca912825-4eeb-4b31-bafc-8d9341952498", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d8fbaeb5-9146-4d9e-8e82-2b81f75fa4de", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4ecc462842f51fa1d62c7b941ef819b649c02517db7cc39dcda6c4af175edb64", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9050baf6-de93-4d88-969c-ea8a95c859f3", "node_type": "1", "metadata": {}, "hash": "5ac2d4cdb10c785db2011cd46194ded7b48ac4d8a3a3804c2a043b8e3ba77aea", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13534, "end_char_idx": 16212, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Once we have the synthetic data, we need to convert it to match the format of the eval dataset.


```python
eval_data = []
input_prompt = "TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]"

for choice in completion.choices:
    question = choice.message.content.split("Q: ")[1].split("\n")[0]  # Extracting the question
    answer = choice.message.content.split("\nA: ")[1].split("\n")[0]  # Extracting the answer
    eval_data.append({
        "input": [
            {"role": "system", "content": input_prompt},
            {"role": "user", "content": question},
        ],
        "ideal": answer
    })

for item in eval_data:
    print(item)
```

    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in Europe?'}], 'ideal': "SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'"}
    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL.


key: _node_content
string_value:
{"id_": "9050baf6-de93-4d88-969c-ea8a95c859f3", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ca912825-4eeb-4b31-bafc-8d9341952498", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "17f2f92d33fb1019cb102f93322912962622a079e9ed77b3e14c07722c20a845", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c4c45bd2-42f4-41df-b807-8337df817acb", "node_type": "1", "metadata": {}, "hash": "0759faa0866775bc7fd049a62d7ebf20e2e96bcb9f6eb5e0036a653002f4b05d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 14980, "end_char_idx": 17331, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in Europe?'}], 'ideal': "SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'"}
    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in the USA?'}], 'ideal': "SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'"}
    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL.


key: _node_content
string_value:
{"id_": "c4c45bd2-42f4-41df-b807-8337df817acb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9050baf6-de93-4d88-969c-ea8a95c859f3", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "e5930df01517ce71a58f03767cf0aa626db1a5ed0b356aac526e6c24108b816c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "65c324e3-1965-446a-b52a-a13aad4af440", "node_type": "1", "metadata": {}, "hash": "feedee288015d6864fb34882ead95610e365e753a7191adbaffdf52914fa3c2a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 16213, "end_char_idx": 18598, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in the USA?'}], 'ideal': "SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'"}
    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': "What is the average horsepower for cars produced in countries from the continent with the id '3'?"}], 'ideal': "SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.ContId = '3'"}
    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL.


key: _node_content
string_value:
{"id_": "65c324e3-1965-446a-b52a-a13aad4af440", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c4c45bd2-42f4-41df-b807-8337df817acb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4137499b1e28ebd3cc5afd3c435fa2cce5bc70d8077928be5c94edfcd56f88d7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bbada030-e7cc-4cce-b619-b8f615b33985", "node_type": "1", "metadata": {}, "hash": "2eeb961643986f96eaa89dfd6bcce8bc5c156ce5ebd6299b46993006b5d6c333", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17332, "end_char_idx": 19843, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': "What is the average horsepower for cars produced in countries from the continent with the id '3'?"}], 'ideal': "SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.ContId = '3'"}
    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made by makers from Europe?'}], 'ideal': "SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'"}
    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL.


key: _node_content
string_value:
{"id_": "bbada030-e7cc-4cce-b619-b8f615b33985", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "65c324e3-1965-446a-b52a-a13aad4af440", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "ed1f78af58ea25d5906f91b13cda30ca05735ebedb6b278d04580cca00235d5b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "490c577a-fd2a-450f-8990-ac4dc95ee8ec", "node_type": "1", "metadata": {}, "hash": "f4101e8d4f54af493be4f6b69555c21d9f985f5a3be8fbfb485ad596ca6c88e0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 18599, "end_char_idx": 21288, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made by makers from Europe?'}], 'ideal': "SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'"}
    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in the USA?'}], 'ideal': "SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'"}


Next we need to create the eval registry to run it in the framework.

The evals framework requires a `.yaml` file structured with the following properties:
* `id` - An identifier for your eval
* `description` - A short description of your eval
* `disclaimer` - An additional notes about your eval
* `metrics` - There are three types of eval metrics we can choose from: match, includes, fuzzyMatch

For our eval, we will configure the following:


key: _node_content
string_value:
{"id_": "490c577a-fd2a-450f-8990-ac4dc95ee8ec", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bbada030-e7cc-4cce-b619-b8f615b33985", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c0bab94aea84fbc8e14088a14d648be4d5e01de3895b0e280bdc9e79d6213c1a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "21ed9e72-8f48-4163-95e8-c86469109912", "node_type": "1", "metadata": {}, "hash": "056e66da09a52a90c9b9f0370fe26e54477be31ae1e67d339493a333e7063741", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19844, "end_char_idx": 22364, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
The SQL should be correct and be in context of the previous question-answer pairs.\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\nTable car_names, columns = [*,MakeId,Model,Make]\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\nTable continents, columns = [*,ContId,Continent]\nTable countries, columns = [*,CountryId,CountryName,Continent]\nTable model_list, columns = [*,ModelId,Maker,Model]\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in the USA?'}], 'ideal': "SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'"}


Next we need to create the eval registry to run it in the framework.

The evals framework requires a `.yaml` file structured with the following properties:
* `id` - An identifier for your eval
* `description` - A short description of your eval
* `disclaimer` - An additional notes about your eval
* `metrics` - There are three types of eval metrics we can choose from: match, includes, fuzzyMatch

For our eval, we will configure the following:


```python
"""
spider-sql:
  id: spider-sql.dev.v0
  metrics: [accuracy]
  description: Eval that scores SQL code from 194 examples in the Spider Text-to-SQL test dataset. The problems are selected by taking the first 10 problems for each database that appears in the test set.
    Yu, Tao, et al. \"Spider; A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task.\" Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, https://doi.org/10.18653/v1/d18-1425.
  disclaimer: Problems are solved zero-shot with no prompting other than the schema; performance may improve with training examples, fine tuning, or a different schema format. Evaluation is currently done through model-grading, where SQL code is not actually executed; the model may judge correct SQL to be incorrect, or vice-versa.
spider-sql.dev.v0:
  class: evals.elsuite.modelgraded.classify:ModelBasedClassify
  args:
    samples_jsonl: sql/spider_sql.jsonl
    eval_type: cot_classify
    modelgraded_spec: sql
  """""
```


key: _node_content
string_value:
{"id_": "21ed9e72-8f48-4163-95e8-c86469109912", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "490c577a-fd2a-450f-8990-ac4dc95ee8ec", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "7dd70446cef1ed5aae510c0af2023c004dd7f661f1d09966a7cba2f1ae78bdae", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "58096600-d1f9-4f21-b7fa-def0ad48b1f1", "node_type": "1", "metadata": {}, "hash": "916cae0985b0e2eb687e7d89eed355ea3537abfedd50a5ef0e6545f213fbf667", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20844, "end_char_idx": 23437, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Next we need to create the eval registry to run it in the framework.

The evals framework requires a `.yaml` file structured with the following properties:
* `id` - An identifier for your eval
* `description` - A short description of your eval
* `disclaimer` - An additional notes about your eval
* `metrics` - There are three types of eval metrics we can choose from: match, includes, fuzzyMatch

For our eval, we will configure the following:


```python
"""
spider-sql:
  id: spider-sql.dev.v0
  metrics: [accuracy]
  description: Eval that scores SQL code from 194 examples in the Spider Text-to-SQL test dataset. The problems are selected by taking the first 10 problems for each database that appears in the test set.
    Yu, Tao, et al. \"Spider; A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task.\" Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, https://doi.org/10.18653/v1/d18-1425.
  disclaimer: Problems are solved zero-shot with no prompting other than the schema; performance may improve with training examples, fine tuning, or a different schema format. Evaluation is currently done through model-grading, where SQL code is not actually executed; the model may judge correct SQL to be incorrect, or vice-versa.
spider-sql.dev.v0:
  class: evals.elsuite.modelgraded.classify:ModelBasedClassify
  args:
    samples_jsonl: sql/spider_sql.jsonl
    eval_type: cot_classify
    modelgraded_spec: sql
  """""
```




    '\nspider-sql:\n  id: spider-sql.dev.v0\n  metrics: [accuracy]\n  description: Eval that scores SQL code from 194 examples in the Spider Text-to-SQL test dataset. The problems are selected by taking the first 10 problems for each database that appears in the test set.\n    Yu, Tao, et al. "Spider; A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task." Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, https://doi.org/10.18653/v1/d18-1425.\n  disclaimer: Problems are solved zero-shot with no prompting other than the schema; performance may improve with training examples, fine tuning, or a different schema format. Evaluation is currently done through model-grading, where SQL code is not actually executed; the model may judge correct SQL to be incorrect, or vice-versa.\nspider-sql.dev.v0:\n  class: evals.elsuite.modelgraded.classify:ModelBasedClassify\n  args:\n    samples_jsonl: sql/spider_sql.jsonl\n    eval_type: cot_classify\n    modelgraded_spec: sql\n  '


key: _node_content
string_value:
{"id_": "58096600-d1f9-4f21-b7fa-def0ad48b1f1", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "21ed9e72-8f48-4163-95e8-c86469109912", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "73f9b4e79a92391ad24e05b60f3af73836c371131cb0f0bed589bf2f1268e78f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "faad21f6-4e87-4232-ba9e-c08e26d73b43", "node_type": "1", "metadata": {}, "hash": "c88731d5079dd1904859263f84017b8c722a22fd922d538a3b3b88189dd88dbc", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 22373, "end_char_idx": 25130, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
'\nspider-sql:\n  id: spider-sql.dev.v0\n  metrics: [accuracy]\n  description: Eval that scores SQL code from 194 examples in the Spider Text-to-SQL test dataset. The problems are selected by taking the first 10 problems for each database that appears in the test set.\n    Yu, Tao, et al. "Spider; A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task." Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, https://doi.org/10.18653/v1/d18-1425.\n  disclaimer: Problems are solved zero-shot with no prompting other than the schema; performance may improve with training examples, fine tuning, or a different schema format. Evaluation is currently done through model-grading, where SQL code is not actually executed; the model may judge correct SQL to be incorrect, or vice-versa.\nspider-sql.dev.v0:\n  class: evals.elsuite.modelgraded.classify:ModelBasedClassify\n  args:\n    samples_jsonl: sql/spider_sql.jsonl\n    eval_type: cot_classify\n    modelgraded_spec: sql\n  '



## Running an evaluation

We can run this eval using the `oaieval` CLI. To get setup, install the library: `pip install .` (if you are running the [OpenAI Evals library](github.com/openai/evals) locally) or `pip install oaieval` if you are running an existing eval.

Then, run the eval using the CLI: `oaieval gpt-3.5-turbo spider-sql`

This command expects a model name and an eval set name. Note that we provide two command line interfaces (CLIs): `oaieval` for running a single eval and `oaievalset` for running a set of evals. The valid eval names are specified in the YAML files under `evals/registry/evals` and their corresponding implementations can be found in `evals/elsuite`.


```python
!pip install evals --quiet
```

The `oaieval` CLI can accept various flags to modify the default behavior. You can run `oaieval --help` to see a full list of CLI options. 

After running that command, you’ll see the final report of accuracy printed to the console, as well as a file path to a temporary file that contains the full report.

These CLIs can accept various flags to modify their default behavior. You can run `oaieval --help` to see a full list of CLI options. 

`oaieval` will search for the `spider-sql` eval YAML file in the `evals/registry/evals` directory, following the format specified in cell 4 above. The path to the eval dataset is specified in the eval YAML file under the args: parameter as `samples_jsonl: sql/spider_sql.jsonl`, with the file content in JSONL format (as generated in step 3 above).

After running that command, you’ll see the final report of accuracy printed to the console, as well as a file path to a temporary file that contains the full report.


key: _node_content
string_value:
{"id_": "faad21f6-4e87-4232-ba9e-c08e26d73b43", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "58096600-d1f9-4f21-b7fa-def0ad48b1f1", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "6cb52d561c1c1f6423e78ae5ef41ce2c8e8983dcd7391c47aff9f6bd4bedb4f4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fbc0c41e-25f5-4748-a681-8feb1ce0eb73", "node_type": "1", "metadata": {}, "hash": "0cafe37e6073396b41848498fbb85c6ebd20393fd8494a86f449b6b5ace42e5c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 24129, "end_char_idx": 26532, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
!pip install evals --quiet
```

The `oaieval` CLI can accept various flags to modify the default behavior. You can run `oaieval --help` to see a full list of CLI options. 

After running that command, you’ll see the final report of accuracy printed to the console, as well as a file path to a temporary file that contains the full report.

These CLIs can accept various flags to modify their default behavior. You can run `oaieval --help` to see a full list of CLI options. 

`oaieval` will search for the `spider-sql` eval YAML file in the `evals/registry/evals` directory, following the format specified in cell 4 above. The path to the eval dataset is specified in the eval YAML file under the args: parameter as `samples_jsonl: sql/spider_sql.jsonl`, with the file content in JSONL format (as generated in step 3 above).

After running that command, you’ll see the final report of accuracy printed to the console, as well as a file path to a temporary file that contains the full report.


```python
!oaieval gpt-3.5-turbo spider-sql --max_samples 25
```

    [2024-03-26 19:44:39,836] [registry.py:257] Loading registry from /Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry/evals
    [2024-03-26 19:44:43,623] [registry.py:257] Loading registry from /Users/shyamal/.evals/evals
    [2024-03-26 19:44:43,635] [oaieval.py:189] [1;35mRun started: 240327024443FACXGMKA[0m
    [2024-03-26 19:44:43,663] [registry.py:257] Loading registry from /Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry/modelgraded
    [2024-03-26 19:44:43,851] [registry.py:257] Loading registry from /Users/shyamal/.evals/modelgraded
    [2024-03-26 19:44:43,853] [data.py:90] Fetching /Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry/data/sql/spider_sql.jsonl
    [2024-03-26 19:44:43,878] [eval.py:36] Evaluating 25 samples
    [2024-03-26 19:44:43,952] [eval.py:144] Running in threaded mode with 10 threads!
      0%|                                                    | 0/25 [00:00<?, ?it/s][2024-03-26 19:44:44,810] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:44,829] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:44,991] [_client.py:1026] HTTP Request: POST https://api.


key: _node_content
string_value:
{"id_": "fbc0c41e-25f5-4748-a681-8feb1ce0eb73", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "faad21f6-4e87-4232-ba9e-c08e26d73b43", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "96f3c442809380861c315b93b399e78554300a75b29623c9a19337f61df53596", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d51222c7-d468-4d0e-81ae-b44b0d27ed13", "node_type": "1", "metadata": {}, "hash": "081235968d0e797ca990c2399a521982c7e437b5f730de75a4af4662fb794ee5", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 26123, "end_char_idx": 28011, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
0%|                                                    | 0/25 [00:00<?, ?it/s][2024-03-26 19:44:44,810] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:44,829] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:44,991] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:45,090] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:45,145] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:45,971] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:46,040] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:46,069] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:46,378] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:46,587] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:47,412] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
      4%|█▊                                          | 1/25 [00:03<01:23,  3.46s/it][2024-03-26 19:44:47,714] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
      8%|███▌                                        | 2/25 [00:03<00:36,  1.60s/it][2024-03-26 19:44:47,947] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.


key: _node_content
string_value:
{"id_": "d51222c7-d468-4d0e-81ae-b44b0d27ed13", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fbc0c41e-25f5-4748-a681-8feb1ce0eb73", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "11c6361da9251d3cd2ba3282aa24d66b0ced13e87bbd0da3075692a89caaa2ec", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3360e5d6-ec02-4ae9-ba85-51b94b6ee184", "node_type": "1", "metadata": {}, "hash": "db366f425162c8b26059e702eeec766dcdd8d5f223f0f4222a143afb335fa1f8", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27044, "end_char_idx": 28926, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:46,069] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:46,378] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:46,587] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:47,412] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
      4%|█▊                                          | 1/25 [00:03<01:23,  3.46s/it][2024-03-26 19:44:47,714] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
      8%|███▌                                        | 2/25 [00:03<00:36,  1.60s/it][2024-03-26 19:44:47,947] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     12%|█████▎                                      | 3/25 [00:03<00:21,  1.02it/s][2024-03-26 19:44:48,413] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:48,643] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     16%|███████                                     | 4/25 [00:04<00:18,  1.15it/s][2024-03-26 19:44:48,909] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     20%|████████▊                                   | 5/25 [00:04<00:12,  1.54it/s][2024-03-26 19:44:49,131] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:49,500] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:49,


key: _node_content
string_value:
{"id_": "3360e5d6-ec02-4ae9-ba85-51b94b6ee184", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d51222c7-d468-4d0e-81ae-b44b0d27ed13", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "f8ce2ab67d930b434fb8e8540ec8fa019a319b00d39a9967f8a340969a8d2b54", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fa23da19-833a-4999-a27c-2cb257a75f8f", "node_type": "1", "metadata": {}, "hash": "57a75db204086cf2fa4fe1c71627c2c33d8915580d7201ceb300d6824ea63b00", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27972, "end_char_idx": 29854, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     12%|█████▎                                      | 3/25 [00:03<00:21,  1.02it/s][2024-03-26 19:44:48,413] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:48,643] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     16%|███████                                     | 4/25 [00:04<00:18,  1.15it/s][2024-03-26 19:44:48,909] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     20%|████████▊                                   | 5/25 [00:04<00:12,  1.54it/s][2024-03-26 19:44:49,131] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:49,500] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:49,530] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     24%|██████████▌                                 | 6/25 [00:05<00:12,  1.56it/s][2024-03-26 19:44:49,962] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:49,964] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:49,967] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     28%|████████████▎                               | 7/25 [00:06<00:10,  1.73it/s][2024-03-26 19:44:50,577] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:50,602] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:50,


key: _node_content
string_value:
{"id_": "fa23da19-833a-4999-a27c-2cb257a75f8f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3360e5d6-ec02-4ae9-ba85-51b94b6ee184", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "117aad98c637725c9d4a82276ba547a06c92f677ef9f87951c0ab4efbaa38f65", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6c76b0d3-0e9a-4888-954f-19ebc1eeea09", "node_type": "1", "metadata": {}, "hash": "51de4367ae40c6512905d30f1b3b99adce02a0a291ab512d027c14ca99febed4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 28891, "end_char_idx": 30747, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
1 200 OK"
    [2024-03-26 19:44:49,530] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     24%|██████████▌                                 | 6/25 [00:05<00:12,  1.56it/s][2024-03-26 19:44:49,962] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:49,964] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:49,967] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     28%|████████████▎                               | 7/25 [00:06<00:10,  1.73it/s][2024-03-26 19:44:50,577] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:50,602] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:50,634] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:50,862] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:51,503] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:51,608] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     40%|█████████████████▏                         | 10/25 [00:07<00:08,  1.79it/s][2024-03-26 19:44:51,801] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     44%|██████████████████▉                        | 11/25 [00:07<00:06,  2.09it/s][2024-03-26 19:44:51,856] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.


key: _node_content
string_value:
{"id_": "6c76b0d3-0e9a-4888-954f-19ebc1eeea09", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fa23da19-833a-4999-a27c-2cb257a75f8f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "70afdefdc12f4a25bffe98095dc610d0afa19c92b566d94a26e8c90869710122", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "a1d015f1-7e8a-4221-9241-0d99ba9be9f2", "node_type": "1", "metadata": {}, "hash": "f8306eec8985f16e0a9a4ee47cd9a2ccd16d9aff13d9cbd82a093be64dd1e450", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 29780, "end_char_idx": 31636, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:50,634] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:50,862] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:51,503] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:51,608] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     40%|█████████████████▏                         | 10/25 [00:07<00:08,  1.79it/s][2024-03-26 19:44:51,801] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     44%|██████████████████▉                        | 11/25 [00:07<00:06,  2.09it/s][2024-03-26 19:44:51,856] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:51,969] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:52,227] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     52%|██████████████████████▎                    | 13/25 [00:08<00:04,  2.65it/s][2024-03-26 19:44:52,450] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:52,526] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:52,615] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     56%|████████████████████████                   | 14/25 [00:08<00:04,  2.64it/s][2024-03-26 19:44:52,625] [_client.py:1026] HTTP Request: POST https://api.


key: _node_content
string_value:
{"id_": "a1d015f1-7e8a-4221-9241-0d99ba9be9f2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "6c76b0d3-0e9a-4888-954f-19ebc1eeea09", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "e72051191e6ef2aa1deb6f9c5d5a8fa74ed70bd12f05786c8ad5f3d119a83f4c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e7b3091a-e59c-489b-935e-d8f45d36dabb", "node_type": "1", "metadata": {}, "hash": "7f901a8ee5ed0a6397597abe944a5b433ddd0c7cc86e71208e4597435b045557", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 30668, "end_char_idx": 32510, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:51,969] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:52,227] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     52%|██████████████████████▎                    | 13/25 [00:08<00:04,  2.65it/s][2024-03-26 19:44:52,450] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:52,526] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:52,615] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     56%|████████████████████████                   | 14/25 [00:08<00:04,  2.64it/s][2024-03-26 19:44:52,625] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:52,777] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:53,653] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     60%|█████████████████████████▊                 | 15/25 [00:09<00:05,  1.87it/s][2024-03-26 19:44:53,670] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:54,028] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     68%|█████████████████████████████▏             | 17/25 [00:10<00:03,  2.54it/s][2024-03-26 19:44:54,388] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:54,


key: _node_content
string_value:
{"id_": "e7b3091a-e59c-489b-935e-d8f45d36dabb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "a1d015f1-7e8a-4221-9241-0d99ba9be9f2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "fa8c0206957e6eae8970e32ae1468b8ec5d06669c321b0910b282255130b2bc9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "bc298cf6-d6bd-49e6-8ad5-f75c6094042c", "node_type": "1", "metadata": {}, "hash": "202c11a22b026984c25fa6e1b160ff245da11a8bbfe0356ecfc1b605cd17d31d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 31582, "end_char_idx": 33355, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
625] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:52,777] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:53,653] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     60%|█████████████████████████▊                 | 15/25 [00:09<00:05,  1.87it/s][2024-03-26 19:44:53,670] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:54,028] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     68%|█████████████████████████████▏             | 17/25 [00:10<00:03,  2.54it/s][2024-03-26 19:44:54,388] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:54,396] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     72%|██████████████████████████████▉            | 18/25 [00:10<00:02,  2.58it/s][2024-03-26 19:44:54,529] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:54,585] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     76%|████████████████████████████████▋          | 19/25 [00:10<00:02,  2.94it/s][2024-03-26 19:44:54,980] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     80%|██████████████████████████████████▍        | 20/25 [00:11<00:01,  2.82it/s][2024-03-26 19:44:55,152] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.


key: _node_content
string_value:
{"id_": "bc298cf6-d6bd-49e6-8ad5-f75c6094042c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e7b3091a-e59c-489b-935e-d8f45d36dabb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "efab0118c51042a335009b590a6955420d360abfde5c8627b6edb800ec14f3e2", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3b7aaed0-0821-4e35-86ea-7cc8452c1d8e", "node_type": "1", "metadata": {}, "hash": "b5e494782afe4346ea938d8e0449b6bfb661e6650f10622756d2e34a12b0ebc0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 32436, "end_char_idx": 34187, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:54,396] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     72%|██████████████████████████████▉            | 18/25 [00:10<00:02,  2.58it/s][2024-03-26 19:44:54,529] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    [2024-03-26 19:44:54,585] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     76%|████████████████████████████████▋          | 19/25 [00:10<00:02,  2.94it/s][2024-03-26 19:44:54,980] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     80%|██████████████████████████████████▍        | 20/25 [00:11<00:01,  2.82it/s][2024-03-26 19:44:55,152] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     84%|████████████████████████████████████       | 21/25 [00:11<00:01,  3.27it/s][2024-03-26 19:44:56,420] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     88%|█████████████████████████████████████▊     | 22/25 [00:12<00:01,  1.75it/s][2024-03-26 19:44:56,984] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     92%|███████████████████████████████████████▌   | 23/25 [00:13<00:01,  1.76it/s][2024-03-26 19:44:57,370] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     96%|█████████████████████████████████████████▎ | 24/25 [00:13<00:00,  1.94it/s][2024-03-26 19:44:59,589] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.


key: _node_content
string_value:
{"id_": "3b7aaed0-0821-4e35-86ea-7cc8452c1d8e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "bc298cf6-d6bd-49e6-8ad5-f75c6094042c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c130447072542527b6981999946bfdecec4a0d9ba19faae2f9cb50138feeef6e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8dc6f031-0777-467d-8f2b-92a73a58a082", "node_type": "1", "metadata": {}, "hash": "d68329986a0dde70d712bd8b52a4af48e59c7b1f51214cff0084da94d46b8da1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 33262, "end_char_idx": 34892, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
152] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     84%|████████████████████████████████████       | 21/25 [00:11<00:01,  3.27it/s][2024-03-26 19:44:56,420] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     88%|█████████████████████████████████████▊     | 22/25 [00:12<00:01,  1.75it/s][2024-03-26 19:44:56,984] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     92%|███████████████████████████████████████▌   | 23/25 [00:13<00:01,  1.76it/s][2024-03-26 19:44:57,370] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
     96%|█████████████████████████████████████████▎ | 24/25 [00:13<00:00,  1.94it/s][2024-03-26 19:44:59,589] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    100%|███████████████████████████████████████████| 25/25 [00:15<00:00,  1.60it/s]
    [2024-03-26 19:44:59,607] [record.py:360] Final report: {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}. Logged to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl
    [2024-03-26 19:44:59,608] [oaieval.py:229] Final report:
    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Correct: 20
    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Incorrect: 5
    [2024-03-26 19:44:59,608] [oaieval.py:231] score: 0.8
    [2024-03-26 19:44:59,640] [record.py:349] Logged 75 rows of events to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl: insert_time=27.915ms


key: _node_content
string_value:
{"id_": "8dc6f031-0777-467d-8f2b-92a73a58a082", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3b7aaed0-0821-4e35-86ea-7cc8452c1d8e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "261b701cd3da7888fbe90afbe22213be287e37f57e05c2372ed69737f59ac1a4", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "417fb6de-d5a5-4a36-90b5-f47bf9fd58c8", "node_type": "1", "metadata": {}, "hash": "c76d41267ae2a9651a44e6c00718bfcfba021096ff7b65c123e3641062e87d15", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34064, "end_char_idx": 35381, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
1.94it/s][2024-03-26 19:44:59,589] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
    100%|███████████████████████████████████████████| 25/25 [00:15<00:00,  1.60it/s]
    [2024-03-26 19:44:59,607] [record.py:360] Final report: {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}. Logged to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl
    [2024-03-26 19:44:59,608] [oaieval.py:229] Final report:
    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Correct: 20
    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Incorrect: 5
    [2024-03-26 19:44:59,608] [oaieval.py:231] score: 0.8
    [2024-03-26 19:44:59,640] [record.py:349] Logged 75 rows of events to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl: insert_time=27.915ms


`oaievalset` expects a model name and an eval set name, for which the valid options are specified in the YAML files under `evals/registry/eval_sets`.

### Going through eval logs

The eval logs are located at `/tmp/evallogs` and different log files are created for each evaluation run. 


```python
log_name = '240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl' # "EDIT THIS" - copy from above
events = f"/tmp/evallogs/{log_name}"
display(pd.read_json(events, lines=True).head(5))
```


key: _node_content
string_value:
{"id_": "417fb6de-d5a5-4a36-90b5-f47bf9fd58c8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8dc6f031-0777-467d-8f2b-92a73a58a082", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0789b1ccbe72dec2dcc31388d0b9f2df2fb905e9fb9c4ae493b33814081ce585", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "afd87d68-113c-4437-ab27-c6f502ec63a8", "node_type": "1", "metadata": {}, "hash": "df8d0a1d4d691c842a849196abc154b427afc611a470efec6e5c4ec9d7ad1557", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34307, "end_char_idx": 37478, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
607] [record.py:360] Final report: {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}. Logged to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl
    [2024-03-26 19:44:59,608] [oaieval.py:229] Final report:
    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Correct: 20
    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Incorrect: 5
    [2024-03-26 19:44:59,608] [oaieval.py:231] score: 0.8
    [2024-03-26 19:44:59,640] [record.py:349] Logged 75 rows of events to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl: insert_time=27.915ms


`oaievalset` expects a model name and an eval set name, for which the valid options are specified in the YAML files under `evals/registry/eval_sets`.

### Going through eval logs

The eval logs are located at `/tmp/evallogs` and different log files are created for each evaluation run. 


```python
log_name = '240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl' # "EDIT THIS" - copy from above
events = f"/tmp/evallogs/{log_name}"
display(pd.read_json(events, lines=True).head(5))
```


<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>spec</th>
      <th>final_report</th>
      <th>run_id</th>
      <th>event_id</th>
      <th>sample_id</th>
      <th>type</th>
      <th>data</th>
      <th>created_by</th>
      <th>created_at</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>{'completion_fns': ['gpt-3.5-turbo'], 'eval_name': 'spider-sql.dev.v0', 'base_eval': 'spider-sql', 'split': 'dev', 'run_config': {'completion_fns': ['gpt-3.5-turbo'], 'eval_spec': {'cls': 'evals.elsuite.modelgraded.classify:ModelBasedClassify', 'registry_path': '/Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry', 'args': {'samples_jsonl': 'sql/spider_sql.jsonl', 'eval_type': 'cot_classify', 'modelgraded_spec': 'sql'}, 'key': 'spider-sql.dev.v0', 'group': 'sql'}, 'seed': 20220722, 'max_samples': 25, 'command': '/Users/shyamal/.virtualenvs/openai/bin/oaieval gpt-3.5-turbo spider-sql --max_samples 25', 'initial_settings': {'visible': False}}, 'created_by': '', 'run_id': '240327024443FACXGMKA', 'created_at': '2024-03-27 02:44:43.626043'}</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaT</td>
    </tr>
    <tr>
      <th>1</th>
      <td>NaN</td>
      <td>{'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaN</td>
      <td>NaT</td>
    </tr>
    <tr>
      <th>2</th>
      <td>NaN</td>
      <td>NaN</td>
      <td>240327024443FACXGMKA</td>
      <td>0.0</td>
      <td>spider-sql.dev.88</td>
      <td>sampling</td>
      <td>{'prompt': [{'content': 'Answer the following question with syntactically correct SQLite SQL.


key: _node_content
string_value:
{"id_": "afd87d68-113c-4437-ab27-c6f502ec63a8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "417fb6de-d5a5-4a36-90b5-f47bf9fd58c8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "6653a273dcd5ea8f7c8b99df5e5dc9da02b8fa2281a5448872f66faccbb384eb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d3abbdfa-00b7-4db9-8c2c-57a613f96f35", "node_type": "1", "metadata": {}, "hash": "8fd03c3cad930b21286a3d08a17e04d228d35937cc112ef200190f164b83a650", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 37479, "end_char_idx": 40052, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Be creative but the SQL must be correct.
Use only the following tables and columns:
Table: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)
Table: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)
Table: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)

Question: Find the average rank of winners in all matches.
', 'role': 'system'}], 'sampled': ['SELECT AVG(winner_rank) AS average_rank_of_winners
FROM matches;']}</td>
      <td></td>
      <td>2024-03-27 02:44:44.821110+00:00</td>
    </tr>
    <tr>
      <th>3</th>
      <td>NaN</td>
      <td>NaN</td>
      <td>240327024443FACXGMKA</td>
      <td>1.0</td>
      <td>spider-sql.dev.82</td>
      <td>sampling</td>
      <td>{'prompt': [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.
Use only the following tables and columns:
Table: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)
Table: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)
Table: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)

Question: Find the total number of matches.


key: _node_content
string_value:
{"id_": "d3abbdfa-00b7-4db9-8c2c-57a613f96f35", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "afd87d68-113c-4437-ab27-c6f502ec63a8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8e0cc7c740d25642b408b58fe3bc7fd3ec8cfa7f75b0e496cc34483c8da52c94", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c3262d22-c2f9-4791-a866-e24d7e04f79b", "node_type": "1", "metadata": {}, "hash": "0ec710541b81819af80b8e7dbcb45936042d6b4ae46b4f832dc9a5bc72cec976", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 39005, "end_char_idx": 41789, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Be creative but the SQL must be correct.
Use only the following tables and columns:
Table: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)
Table: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)
Table: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)

Question: Find the total number of matches.
', 'role': 'system'}], 'sampled': ['SELECT COUNT(*) AS total_matches
FROM matches;']}</td>
      <td></td>
      <td>2024-03-27 02:44:44.831848+00:00</td>
    </tr>
    <tr>
      <th>4</th>
      <td>NaN</td>
      <td>NaN</td>
      <td>240327024443FACXGMKA</td>
      <td>2.0</td>
      <td>spider-sql.dev.25</td>
      <td>sampling</td>
      <td>{'prompt': [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.
Use only the following tables and columns:
Table: continents. Columns: ContId (number), Continent (text)
Table: countries. Columns: CountryId (number), CountryName (text), Continent (number)
Table: car_makers. Columns: Id (number), Maker (text), FullName (text), Country (text)
Table: model_list. Columns: ModelId (number), Maker (number), Model (text)
Table: car_names. Columns: MakeId (number), Model (text), Make (text)
Table: cars_data. Columns: Id (number), MPG (text), Cylinders (number), Edispl (number), Horsepower (text), Weight (number), Accelerate (number), Year (number)

Question: How many countries exist?
', 'role': 'system'}], 'sampled': ['SELECT COUNT(*) AS TotalCountries
FROM countries;']}</td>
      <td></td>
      <td>2024-03-27 02:44:44.996647+00:00</td>
    </tr>
  </tbody>
</table>
</div>



```python
# processing the log events generated by oaieval

with open(events, "r") as f:
    events_df = pd.read_json(f, lines=True)
```

This file will contain structured logs of the evaluation. The first entry provides a detailed specification of the evaluation, including the completion functions, evaluation name, run configuration, creator’s name, run ID, and creation timestamp.


```python
display(events_df.iloc[0].spec)
```


key: _node_content
string_value:
{"id_": "c3262d22-c2f9-4791-a866-e24d7e04f79b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d3abbdfa-00b7-4db9-8c2c-57a613f96f35", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "0aa72862556fd4d1087ed1e904559866eb7b1ed33010f83d36c963b3f7b2ba7a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d90657ad-654d-4f84-adaf-789d37b373a4", "node_type": "1", "metadata": {}, "hash": "2ec1f596b22aaf971bfc5a8f8ff999c198dc0be20e825f80bc68689950cad9b3", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 40498, "end_char_idx": 43047, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Be creative but the SQL must be correct.
Use only the following tables and columns:
Table: continents. Columns: ContId (number), Continent (text)
Table: countries. Columns: CountryId (number), CountryName (text), Continent (number)
Table: car_makers. Columns: Id (number), Maker (text), FullName (text), Country (text)
Table: model_list. Columns: ModelId (number), Maker (number), Model (text)
Table: car_names. Columns: MakeId (number), Model (text), Make (text)
Table: cars_data. Columns: Id (number), MPG (text), Cylinders (number), Edispl (number), Horsepower (text), Weight (number), Accelerate (number), Year (number)

Question: How many countries exist?
', 'role': 'system'}], 'sampled': ['SELECT COUNT(*) AS TotalCountries
FROM countries;']}</td>
      <td></td>
      <td>2024-03-27 02:44:44.996647+00:00</td>
    </tr>
  </tbody>
</table>
</div>



```python
# processing the log events generated by oaieval

with open(events, "r") as f:
    events_df = pd.read_json(f, lines=True)
```

This file will contain structured logs of the evaluation. The first entry provides a detailed specification of the evaluation, including the completion functions, evaluation name, run configuration, creator’s name, run ID, and creation timestamp.


```python
display(events_df.iloc[0].spec)
```


    {'completion_fns': ['gpt-3.5-turbo'],
     'eval_name': 'spider-sql.dev.v0',
     'base_eval': 'spider-sql',
     'split': 'dev',
     'run_config': {'completion_fns': ['gpt-3.5-turbo'],
      'eval_spec': {'cls': 'evals.elsuite.modelgraded.classify:ModelBasedClassify',
       'registry_path': '/Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry',
       'args': {'samples_jsonl': 'sql/spider_sql.jsonl',
        'eval_type': 'cot_classify',
        'modelgraded_spec': 'sql'},
       'key': 'spider-sql.dev.v0',
       'group': 'sql'},
      'seed': 20220722,
      'max_samples': 25,
      'command': '/Users/shyamal/.virtualenvs/openai/bin/oaieval gpt-3.5-turbo spider-sql --max_samples 25',
      'initial_settings': {'visible': False}},
     'created_by': '',
     'run_id': '240327024443FACXGMKA',
     'created_at': '2024-03-27 02:44:43.626043'}


Let's also look at the entry which provides the final report of the evaluation.


```python
display(events_df.dropna(subset=['final_report']).iloc[0]['final_report'])
```


    {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}


We can also review individual evaluation events that provide specific samples (`sample_id`), results, event types, and metadata.


key: _node_content
string_value:
{"id_": "d90657ad-654d-4f84-adaf-789d37b373a4", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c3262d22-c2f9-4791-a866-e24d7e04f79b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8bf7e16f58920fd038da862bae8d9519166af984e5674ee69520d94badf1102b", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "9d7cbc01-d5c7-449b-8e82-9ad10a83a90c", "node_type": "1", "metadata": {}, "hash": "8b13c7ab4b4dfd52de72c9adbe24ae0c4bcfb57d878aff96a2106c02a6abfea0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 41744, "end_char_idx": 43229, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
display(events_df.iloc[0].spec)
```


    {'completion_fns': ['gpt-3.5-turbo'],
     'eval_name': 'spider-sql.dev.v0',
     'base_eval': 'spider-sql',
     'split': 'dev',
     'run_config': {'completion_fns': ['gpt-3.5-turbo'],
      'eval_spec': {'cls': 'evals.elsuite.modelgraded.classify:ModelBasedClassify',
       'registry_path': '/Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry',
       'args': {'samples_jsonl': 'sql/spider_sql.jsonl',
        'eval_type': 'cot_classify',
        'modelgraded_spec': 'sql'},
       'key': 'spider-sql.dev.v0',
       'group': 'sql'},
      'seed': 20220722,
      'max_samples': 25,
      'command': '/Users/shyamal/.virtualenvs/openai/bin/oaieval gpt-3.5-turbo spider-sql --max_samples 25',
      'initial_settings': {'visible': False}},
     'created_by': '',
     'run_id': '240327024443FACXGMKA',
     'created_at': '2024-03-27 02:44:43.626043'}


Let's also look at the entry which provides the final report of the evaluation.


```python
display(events_df.dropna(subset=['final_report']).iloc[0]['final_report'])
```


    {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}


We can also review individual evaluation events that provide specific samples (`sample_id`), results, event types, and metadata.


```python
pd.set_option('display.max_colwidth', None)  # None means no truncation
display(events_df.iloc[2][['run_id', 'event_id', 'sample_id', 'type', 'data', 'created_at']])
```


key: _node_content
string_value:
{"id_": "9d7cbc01-d5c7-449b-8e82-9ad10a83a90c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d90657ad-654d-4f84-adaf-789d37b373a4", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c51f334d524e91795754f0f48a6969035394eb4997035234fc6564420e9b915c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "98a3e310-da47-47b5-8e62-14d853534075", "node_type": "1", "metadata": {}, "hash": "1c4d59968cf169d38bc2b3a2d85d03618c48f1467be680f6a4ebb8c74c7a023e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 42680, "end_char_idx": 51567, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Let's also look at the entry which provides the final report of the evaluation.


```python
display(events_df.dropna(subset=['final_report']).iloc[0]['final_report'])
```


    {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}


We can also review individual evaluation events that provide specific samples (`sample_id`), results, event types, and metadata.


```python
pd.set_option('display.max_colwidth', None)  # None means no truncation
display(events_df.iloc[2][['run_id', 'event_id', 'sample_id', 'type', 'data', 'created_at']])
```


    run_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                240327024443FACXGMKA
    event_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               0.0
    sample_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                spider-sql.dev.88
    type                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              sampling
    data          {'prompt': [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.
    Use only the following tables and columns:
    Table: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)
    Table: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)
    Table: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)
    
    Question: Find the average rank of winners in all matches.
    ', 'role': 'system'}], 'sampled': ['SELECT AVG(winner_rank) AS average_rank_of_winners
    FROM matches;']}
    created_at                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                2024-03-27 02:44:44.821110+00:00
    Name: 2, dtype: object



```python
# Inspect samples
for i, row in events_df[events_df['type'] == 'sampling'].head(5).iterrows():
    data = pd.json_normalize(row['data'])
    print(f"Prompt: {data['prompt'].iloc[0]}")
    print(f"Sampled: {data['sampled'].iloc[0]}")
    print("-" * 10)
```

    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\nTable: matches.


key: _node_content
string_value:
{"id_": "98a3e310-da47-47b5-8e62-14d853534075", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "9d7cbc01-d5c7-449b-8e82-9ad10a83a90c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "6f579e2bac6958c8aa7072094195da82d0df93353548b4f117052193c79f76d9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "e399e844-2589-4cb0-816f-5104331be6b0", "node_type": "1", "metadata": {}, "hash": "d448681c028549f6f2c1d6c21014c4d15631cfa5b7cf63b8cf6cca01c6042648", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 50968, "end_char_idx": 53972, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Inspect samples
for i, row in events_df[events_df['type'] == 'sampling'].head(5).iterrows():
    data = pd.json_normalize(row['data'])
    print(f"Prompt: {data['prompt'].iloc[0]}")
    print(f"Sampled: {data['sampled'].iloc[0]}")
    print("-" * 10)
```

    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\nTable: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)\nTable: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)\n\nQuestion: Find the average rank of winners in all matches.\n', 'role': 'system'}]
    Sampled: ['SELECT AVG(winner_rank) AS average_rank_of_winners\nFROM matches;']
    ----------
    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\nTable: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)\nTable: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)\n\nQuestion: Find the total number of matches.\n', 'role': 'system'}]
    Sampled: ['SELECT COUNT(*) AS total_matches\nFROM matches;']
    ----------
    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: continents.


key: _node_content
string_value:
{"id_": "e399e844-2589-4cb0-816f-5104331be6b0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "98a3e310-da47-47b5-8e62-14d853534075", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "efe845d6cd0b8d48286c17dbf185fb910817e6237a605a9429e948392dee6c8c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "56ee54a0-0b89-4021-a602-ba0b238a7d5f", "node_type": "1", "metadata": {}, "hash": "8081cfa1c0edb27d083aaa3a7baaaeaa758676d384ae77990b417c063565201a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 52616, "end_char_idx": 55242, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\nTable: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)\nTable: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)\n\nQuestion: Find the total number of matches.\n', 'role': 'system'}]
    Sampled: ['SELECT COUNT(*) AS total_matches\nFROM matches;']
    ----------
    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: continents. Columns: ContId (number), Continent (text)\nTable: countries. Columns: CountryId (number), CountryName (text), Continent (number)\nTable: car_makers. Columns: Id (number), Maker (text), FullName (text), Country (text)\nTable: model_list. Columns: ModelId (number), Maker (number), Model (text)\nTable: car_names. Columns: MakeId (number), Model (text), Make (text)\nTable: cars_data. Columns: Id (number), MPG (text), Cylinders (number), Edispl (number), Horsepower (text), Weight (number), Accelerate (number), Year (number)\n\nQuestion: How many countries exist?\n', 'role': 'system'}]
    Sampled: ['SELECT COUNT(*) AS TotalCountries\nFROM countries;']
    ----------
    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: TV_Channel. Columns: id (text), series_name (text), Country (text), Language (text), Content (text), Pixel_aspect_ratio_PAR (text), Hight_definition_TV (text), Pay_per_view_PPV (text), Package_Option (text)\nTable: TV_series. Columns: id (number), Episode (text), Air_Date (text), Rating (text), Share (number), 18_49_Rating_Share (text), Viewers_m (text), Weekly_Rank (number), Channel (text)\nTable: Cartoon.


key: _node_content
string_value:
{"id_": "56ee54a0-0b89-4021-a602-ba0b238a7d5f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "e399e844-2589-4cb0-816f-5104331be6b0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "2dac83b1bce5b1d9821abe9fdf90e512ba52cf62cf03ab3b89a73c7a7b827cb3", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "abac6e26-f2ec-46f8-9506-f802acae65cb", "node_type": "1", "metadata": {}, "hash": "dad931d4882b288500dbac7b74da622f9de656e29875e150bb5538b7803bf14a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 53868, "end_char_idx": 56584, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: continents. Columns: ContId (number), Continent (text)\nTable: countries. Columns: CountryId (number), CountryName (text), Continent (number)\nTable: car_makers. Columns: Id (number), Maker (text), FullName (text), Country (text)\nTable: model_list. Columns: ModelId (number), Maker (number), Model (text)\nTable: car_names. Columns: MakeId (number), Model (text), Make (text)\nTable: cars_data. Columns: Id (number), MPG (text), Cylinders (number), Edispl (number), Horsepower (text), Weight (number), Accelerate (number), Year (number)\n\nQuestion: How many countries exist?\n', 'role': 'system'}]
    Sampled: ['SELECT COUNT(*) AS TotalCountries\nFROM countries;']
    ----------
    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: TV_Channel. Columns: id (text), series_name (text), Country (text), Language (text), Content (text), Pixel_aspect_ratio_PAR (text), Hight_definition_TV (text), Pay_per_view_PPV (text), Package_Option (text)\nTable: TV_series. Columns: id (number), Episode (text), Air_Date (text), Rating (text), Share (number), 18_49_Rating_Share (text), Viewers_m (text), Weekly_Rank (number), Channel (text)\nTable: Cartoon. Columns: id (number), Title (text), Directed_by (text), Written_by (text), Original_air_date (text), Production_code (number), Channel (text)\n\nQuestion: What is the name and directors of all the cartoons that are ordered by air date?\n', 'role': 'system'}]
    Sampled: ['SELECT Title, Directed_by\nFROM Cartoon\nORDER BY Original_air_date;']
    ----------
    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: stadium. Columns: Stadium_ID (number), Location (text), Name (text), Capacity (number), Highest (number), Lowest (number), Average (number)\nTable: singer. Columns: Singer_ID (number), Name (text), Country (text), Song_Name (text), Song_release_year (text), Age (number), Is_male (others)\nTable: concert. Columns: concert_ID (number), concert_Name (text), Theme (text), Stadium_ID (text), Year (text)\nTable: singer_in_concert. Columns: concert_ID (number), Singer_ID (text)\n\nQuestion: Show the name and the release year of the song by the youngest singer.\n', 'role': 'system'}]
    Sampled: ['```sql\nSELECT s.Name, s.Song_release_year\nFROM singer s\nWHERE s.Age = (SELECT MIN(Age) FROM singer)\n```']
    ----------


Let's review our failures to understand which tests did not succeed.


key: _node_content
string_value:
{"id_": "abac6e26-f2ec-46f8-9506-f802acae65cb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "56ee54a0-0b89-4021-a602-ba0b238a7d5f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "b759c9da2bf2f2536d413c263443f979c9667c39c375ef49fa37fac02e2c86a9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "32de8531-d78f-4cae-869b-643bfc2439a7", "node_type": "1", "metadata": {}, "hash": "2b257916321e80c1f8c4b87ad84d80904cdb749c8f3349e8efb261dd3cf7dc6e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 55058, "end_char_idx": 58210, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Columns: id (number), Episode (text), Air_Date (text), Rating (text), Share (number), 18_49_Rating_Share (text), Viewers_m (text), Weekly_Rank (number), Channel (text)\nTable: Cartoon. Columns: id (number), Title (text), Directed_by (text), Written_by (text), Original_air_date (text), Production_code (number), Channel (text)\n\nQuestion: What is the name and directors of all the cartoons that are ordered by air date?\n', 'role': 'system'}]
    Sampled: ['SELECT Title, Directed_by\nFROM Cartoon\nORDER BY Original_air_date;']
    ----------
    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: stadium. Columns: Stadium_ID (number), Location (text), Name (text), Capacity (number), Highest (number), Lowest (number), Average (number)\nTable: singer. Columns: Singer_ID (number), Name (text), Country (text), Song_Name (text), Song_release_year (text), Age (number), Is_male (others)\nTable: concert. Columns: concert_ID (number), concert_Name (text), Theme (text), Stadium_ID (text), Year (text)\nTable: singer_in_concert. Columns: concert_ID (number), Singer_ID (text)\n\nQuestion: Show the name and the release year of the song by the youngest singer.\n', 'role': 'system'}]
    Sampled: ['```sql\nSELECT s.Name, s.Song_release_year\nFROM singer s\nWHERE s.Age = (SELECT MIN(Age) FROM singer)\n```']
    ----------


Let's review our failures to understand which tests did not succeed.


```python
def pretty_print_text(prompt):
    # Define markers for the sections
    markers = {
        "question": "[Question]:",
        "expert": "[Expert]:",
        "submission": "[Submission]:",
        "end": "[END DATA]"
    }
    
    # Function to extract text between markers
    def extract_text(start_marker, end_marker):
        start = prompt.find(start_marker) + len(start_marker)
        end = prompt.find(end_marker)
        text = prompt[start:end].strip()
        if start_marker == markers["question"]:
            text = text.split("\n\nQuestion:")[-1].strip() if "\n\nQuestion:" in text else text
        elif start_marker == markers["submission"]:
            text = text.replace("```sql", "").replace("```", "").strip()
        return text
    
    # Extracting text for each section
    question_text = extract_text(markers["question"], markers["expert"])
    expert_text = extract_text(markers["expert"], markers["submission"])
    submission_text = extract_text(markers["submission"], markers["end"])
    
    # HTML color codes and formatting
    colors = {
        "question": '<span style="color: #0000FF;">QUESTION:<br>', 
        "expert": '<span style="color: #008000;">EXPECTED:<br>',  
        "submission": '<span style="color: #FFA500;">SUBMISSION:<br>' 
    }
    color_end = '</span>'
    
    # Display each section with color
    from IPython.display import display, HTML
    display(HTML(f"{colors['question']}{question_text}{color_end}"))
    display(HTML(f"{colors['expert']}{expert_text}{color_end}"))
    display(HTML(f"{colors['submission']}{submission_text}{color_end}"))
```


key: _node_content
string_value:
{"id_": "32de8531-d78f-4cae-869b-643bfc2439a7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "abac6e26-f2ec-46f8-9506-f802acae65cb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "57266b63ea63aeababf4f2c80f873e1339a5029ffb143149a18054eb9f8dc622", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "7848bf6c-3848-4c35-9fd0-f75b3c9bac4b", "node_type": "1", "metadata": {}, "hash": "9e9f577507627bfa8df69233d7f66af92445e05915055a1bce054270fc75cce6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 56516, "end_char_idx": 59647, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
Let's review our failures to understand which tests did not succeed.


```python
def pretty_print_text(prompt):
    # Define markers for the sections
    markers = {
        "question": "[Question]:",
        "expert": "[Expert]:",
        "submission": "[Submission]:",
        "end": "[END DATA]"
    }
    
    # Function to extract text between markers
    def extract_text(start_marker, end_marker):
        start = prompt.find(start_marker) + len(start_marker)
        end = prompt.find(end_marker)
        text = prompt[start:end].strip()
        if start_marker == markers["question"]:
            text = text.split("\n\nQuestion:")[-1].strip() if "\n\nQuestion:" in text else text
        elif start_marker == markers["submission"]:
            text = text.replace("```sql", "").replace("```", "").strip()
        return text
    
    # Extracting text for each section
    question_text = extract_text(markers["question"], markers["expert"])
    expert_text = extract_text(markers["expert"], markers["submission"])
    submission_text = extract_text(markers["submission"], markers["end"])
    
    # HTML color codes and formatting
    colors = {
        "question": '<span style="color: #0000FF;">QUESTION:<br>', 
        "expert": '<span style="color: #008000;">EXPECTED:<br>',  
        "submission": '<span style="color: #FFA500;">SUBMISSION:<br>' 
    }
    color_end = '</span>'
    
    # Display each section with color
    from IPython.display import display, HTML
    display(HTML(f"{colors['question']}{question_text}{color_end}"))
    display(HTML(f"{colors['expert']}{expert_text}{color_end}"))
    display(HTML(f"{colors['submission']}{submission_text}{color_end}"))
```


```python
# Inspect metrics where choice is made and print only the prompt, result, and expected result if the choice is incorrect
for i, row in events_df[events_df['type'] == 'metrics'].iterrows():
    if row['data']['choice'] == 'Incorrect':
        # Get the previous row's data, which contains the prompt and the expected result
        prev_row = events_df.iloc[i-1]
        prompt = prev_row['data']['prompt'][0]['content'] if 'prompt' in prev_row['data'] and len(prev_row['data']['prompt']) > 0 else "Prompt not available"
        expected_result = prev_row['data'].get('ideal', 'Expected result not provided')
        
        # Current row's data will be the actual result
        result = row['data'].get('result', 'Actual result not provided')
        
        pretty_print_text(prompt)
        print("-" * 40)
```


<span style="color: #0000FF;">QUESTION:<br>How many countries have a republic as their form of government?

************</span>



<span style="color: #008000;">EXPECTED:<br>SELECT count(*) FROM country WHERE GovernmentForm  =  "Republic"
************</span>



<span style="color: #FFA500;">SUBMISSION:<br>SELECT COUNT(*) 
FROM country 
WHERE GovernmentForm LIKE '%Republic%'

************</span>


    ----------------------------------------



<span style="color: #0000FF;">QUESTION:<br>Return the document id, template id, and description for the document with the name Robbin CV.

************</span>


key: _node_content
string_value:
{"id_": "7848bf6c-3848-4c35-9fd0-f75b3c9bac4b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "32de8531-d78f-4cae-869b-643bfc2439a7", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "bac33913ada2e5360bf72540fb18a424adeee8410530d3cfbcad0f4b23840045", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0d21d848-2c0c-4065-945f-aa629eb21e1d", "node_type": "1", "metadata": {}, "hash": "42e338f1df2f76b100b9cbf663bcd88ce512a131678cd3692062fe336d720aea", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 58213, "end_char_idx": 61303, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Inspect metrics where choice is made and print only the prompt, result, and expected result if the choice is incorrect
for i, row in events_df[events_df['type'] == 'metrics'].iterrows():
    if row['data']['choice'] == 'Incorrect':
        # Get the previous row's data, which contains the prompt and the expected result
        prev_row = events_df.iloc[i-1]
        prompt = prev_row['data']['prompt'][0]['content'] if 'prompt' in prev_row['data'] and len(prev_row['data']['prompt']) > 0 else "Prompt not available"
        expected_result = prev_row['data'].get('ideal', 'Expected result not provided')
        
        # Current row's data will be the actual result
        result = row['data'].get('result', 'Actual result not provided')
        
        pretty_print_text(prompt)
        print("-" * 40)
```


<span style="color: #0000FF;">QUESTION:<br>How many countries have a republic as their form of government?

************</span>



<span style="color: #008000;">EXPECTED:<br>SELECT count(*) FROM country WHERE GovernmentForm  =  "Republic"
************</span>



<span style="color: #FFA500;">SUBMISSION:<br>SELECT COUNT(*) 
FROM country 
WHERE GovernmentForm LIKE '%Republic%'

************</span>


    ----------------------------------------



<span style="color: #0000FF;">QUESTION:<br>Return the document id, template id, and description for the document with the name Robbin CV.

************</span>



<span style="color: #008000;">EXPECTED:<br>SELECT document_id ,  template_id ,  Document_Description FROM Documents WHERE document_name  =  "Robbin CV"
************</span>



<span style="color: #FFA500;">SUBMISSION:<br>SELECT Documents.Document_ID, Documents.Template_ID, Documents.Document_Description
FROM Documents
JOIN Templates ON Documents.Template_ID = Templates.Template_ID
WHERE Documents.Document_Name = 'Robbin CV';

************</span>


    ----------------------------------------



<span style="color: #0000FF;">QUESTION:<br>Which professionals live in the state of Indiana or have done treatment on more than 2 treatments? List his or her id, last name and cell phone.

************</span>



<span style="color: #008000;">EXPECTED:<br>SELECT professional_id ,  last_name ,  cell_number FROM Professionals WHERE state  =  'Indiana' UNION SELECT T1.professional_id ,  T1.last_name ,  T1.cell_number FROM Professionals AS T1 JOIN Treatments AS T2 ON T1.professional_id  =  T2.professional_id GROUP BY T1.professional_id HAVING count(*)  >  2
************</span>



<span style="color: #FFA500;">SUBMISSION:<br>SELECT professional_id, last_name, cell_number
FROM Professionals
WHERE state = 'Indiana'
OR professional_id IN (
    SELECT professional_id
    FROM Treatments
    GROUP BY professional_id
    HAVING COUNT(*) > 2
);
************</span>


    ----------------------------------------



<span style="color: #0000FF;">QUESTION:<br>What is the continent name which Anguilla belongs to?

************</span>



<span style="color: #008000;">EXPECTED:<br>SELECT Continent FROM country WHERE Name  =  "Anguilla"
************</span>


key: _node_content
string_value:
{"id_": "0d21d848-2c0c-4065-945f-aa629eb21e1d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "7848bf6c-3848-4c35-9fd0-f75b3c9bac4b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "59cd05af2fba430ae2c191492e8cc7f531aefe1341d9053270aa4ce8dfd55449", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "0c727f9d-090f-4ca0-80a3-8d14dad639f6", "node_type": "1", "metadata": {}, "hash": "d93fb102df7799c5f49af1b446e5f57ad73eee8ef49a7ad1a8e6aaa7035ae59f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 59651, "end_char_idx": 61846, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<span style="color: #008000;">EXPECTED:<br>SELECT document_id ,  template_id ,  Document_Description FROM Documents WHERE document_name  =  "Robbin CV"
************</span>



<span style="color: #FFA500;">SUBMISSION:<br>SELECT Documents.Document_ID, Documents.Template_ID, Documents.Document_Description
FROM Documents
JOIN Templates ON Documents.Template_ID = Templates.Template_ID
WHERE Documents.Document_Name = 'Robbin CV';

************</span>


    ----------------------------------------



<span style="color: #0000FF;">QUESTION:<br>Which professionals live in the state of Indiana or have done treatment on more than 2 treatments? List his or her id, last name and cell phone.

************</span>



<span style="color: #008000;">EXPECTED:<br>SELECT professional_id ,  last_name ,  cell_number FROM Professionals WHERE state  =  'Indiana' UNION SELECT T1.professional_id ,  T1.last_name ,  T1.cell_number FROM Professionals AS T1 JOIN Treatments AS T2 ON T1.professional_id  =  T2.professional_id GROUP BY T1.professional_id HAVING count(*)  >  2
************</span>



<span style="color: #FFA500;">SUBMISSION:<br>SELECT professional_id, last_name, cell_number
FROM Professionals
WHERE state = 'Indiana'
OR professional_id IN (
    SELECT professional_id
    FROM Treatments
    GROUP BY professional_id
    HAVING COUNT(*) > 2
);
************</span>


    ----------------------------------------



<span style="color: #0000FF;">QUESTION:<br>What is the continent name which Anguilla belongs to?

************</span>



<span style="color: #008000;">EXPECTED:<br>SELECT Continent FROM country WHERE Name  =  "Anguilla"
************</span>



<span style="color: #FFA500;">SUBMISSION:<br>SELECT c.Continent
FROM country c
WHERE c.Code = 'AIA';

************</span>


    ----------------------------------------



<span style="color: #0000FF;">QUESTION:<br>How many airlines do we have?

************</span>



<span style="color: #008000;">EXPECTED:<br>SELECT count(*) FROM AIRLINES
************</span>



<span style="color: #FFA500;">SUBMISSION:<br>SELECT COUNT(DISTINCT Airline) AS TotalAirlines
FROM airlines;
************</span>


    ----------------------------------------


key: _node_content
string_value:
{"id_": "0c727f9d-090f-4ca0-80a3-8d14dad639f6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "a7ec993c-08fc-4643-844a-c524bc7c3f16", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "768291709437390dd549b3ed69ab84d94f65bcf3ff582852d98e4f92a5a40095", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "0d21d848-2c0c-4065-945f-aa629eb21e1d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "fff891177ef2fbb3dfbc338898f05a9ac776770b836b585d37c049a8cde026c7", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 60362, "end_char_idx": 63108, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
<span style="color: #008000;">EXPECTED:<br>SELECT professional_id ,  last_name ,  cell_number FROM Professionals WHERE state  =  'Indiana' UNION SELECT T1.professional_id ,  T1.last_name ,  T1.cell_number FROM Professionals AS T1 JOIN Treatments AS T2 ON T1.professional_id  =  T2.professional_id GROUP BY T1.professional_id HAVING count(*)  >  2
************</span>



<span style="color: #FFA500;">SUBMISSION:<br>SELECT professional_id, last_name, cell_number
FROM Professionals
WHERE state = 'Indiana'
OR professional_id IN (
    SELECT professional_id
    FROM Treatments
    GROUP BY professional_id
    HAVING COUNT(*) > 2
);
************</span>


    ----------------------------------------



<span style="color: #0000FF;">QUESTION:<br>What is the continent name which Anguilla belongs to?

************</span>



<span style="color: #008000;">EXPECTED:<br>SELECT Continent FROM country WHERE Name  =  "Anguilla"
************</span>



<span style="color: #FFA500;">SUBMISSION:<br>SELECT c.Continent
FROM country c
WHERE c.Code = 'AIA';

************</span>


    ----------------------------------------



<span style="color: #0000FF;">QUESTION:<br>How many airlines do we have?

************</span>



<span style="color: #008000;">EXPECTED:<br>SELECT count(*) FROM AIRLINES
************</span>



<span style="color: #FFA500;">SUBMISSION:<br>SELECT COUNT(DISTINCT Airline) AS TotalAirlines
FROM airlines;
************</span>


    ----------------------------------------


Reviewing some of the failures we see the following:
* The second incorrect answer had an unnecessary join with the 'Templates' table. Our eval was able to accurately identify this and flag this as incorrect. 
* Few other answers have minor syntax differences that caused the answers to get flagged.
  * In situations like this, it would be worthwhile exploring whether we should continue iterating on the prompt to ensure certain stylistic choices, or if we should modify the evaluation suite to capture this variation.
  * This type of failure hints at the potential need for model-graded evals as a way to ensure accuracy in grading the results

# Conclusion

Building out effective evals is a core part of the development cycle of LLM-based applications. The OpenAI Evals framework provides the core structure of building evals out of the box, and allows you to quickly spin up new tests for your various use cases. In this guide, we demonstrated step-by-step how to create an eval, run it, and analyze the results.

The example shown in this guide represent a straightfoward use case for evals. As you continue to explore this framework, we recommend you explore creating more complex model-graded evals for actual production use cases. Happy evaluating!


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb


key: _node_content
string_value:
{"id_": "c6a66e83-a6f1-4772-a174-1323d2df995c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99de0ab8-a81a-4792-80de-b1084e9bd657", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3d87226cf3e1ecbc42d0bed8b11ca11fa5c935369f6c9e45c85efd4d1eec2978", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "fbaa895d-5682-484b-97ff-699217657376", "node_type": "1", "metadata": {}, "hash": "4987a94b9964c66fd52540d6ed08d64f33ec1c0cc7725cb5e845dbc12a104f94", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3549, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Evaluate RAG with LlamaIndex

In this notebook we will look into building an RAG pipeline and evaluating it with LlamaIndex. It has following 3 sections.

1. Understanding Retrieval Augmented Generation (RAG).
2. Building RAG with LlamaIndex.
3. Evaluating RAG with LlamaIndex.

**Retrieval Augmented Generation (RAG)**

LLMs are trained on vast datasets, but these will not include your specific data. Retrieval-Augmented Generation (RAG) addresses this by dynamically incorporating your data during the generation process. This is done not by altering the training data of LLMs, but by allowing the model to access and utilize your data in real-time to provide more tailored and contextually relevant responses.

In RAG, your data is loaded and and prepared for queries or “indexed”. User queries act on the index, which filters your data down to the most relevant context. This context and your query then go to the LLM along with a prompt, and the LLM provides a response.

Even if what you’re building is a chatbot or an agent, you’ll want to know RAG techniques for getting data into your application.

![RAG Overview](../../images/llamaindex_rag_overview.png)

**Stages within RAG**

There are five key stages within RAG, which in turn will be a part of any larger application you build. These are:

**Loading:** this refers to getting your data from where it lives – whether it’s text files, PDFs, another website, a database, or an API – into your pipeline. LlamaHub provides hundreds of connectors to choose from.

**Indexing:** this means creating a data structure that allows for querying the data. For LLMs this nearly always means creating vector embeddings, numerical representations of the meaning of your data, as well as numerous other metadata strategies to make it easy to accurately find contextually relevant data.

**Storing:** Once your data is indexed, you will want to store your index, along with any other metadata, to avoid the need to re-index it.

**Querying:** for any given indexing strategy there are many ways you can utilize LLMs and LlamaIndex data structures to query, including sub-queries, multi-step queries and hybrid strategies.

**Evaluation:** a critical step in any pipeline is checking how effective it is relative to other strategies, or when you make changes. Evaluation provides objective measures of how accurate, faithful and fast your responses to queries are.

## Build RAG system.

Now that we have understood the significance of RAG system, let's build a simple RAG pipeline.


```python
!pip install llama-index
```


```python
# The nest_asyncio module enables the nesting of asynchronous functions within an already running async loop.
# This is necessary because Jupyter notebooks inherently operate in an asynchronous loop.
# By applying nest_asyncio, we can run additional async functions within this existing loop without conflicts.
import nest_asyncio

nest_asyncio.apply()

from llama_index.evaluation import generate_question_context_pairs
from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext
from llama_index.node_parser import SimpleNodeParser
from llama_index.evaluation import generate_question_context_pairs
from llama_index.evaluation import RetrieverEvaluator
from llama_index.llms import OpenAI

import os
import pandas as pd
```

Set Your OpenAI API Key


```python
os.environ['OPENAI_API_KEY'] = 'YOUR OPENAI API KEY'
```

Let's use [Paul Graham Essay text](https://www.paulgraham.com/worked.html) for building RAG pipeline.

#### Download Data


key: _node_content
string_value:
{"id_": "fbaa895d-5682-484b-97ff-699217657376", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99de0ab8-a81a-4792-80de-b1084e9bd657", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3d87226cf3e1ecbc42d0bed8b11ca11fa5c935369f6c9e45c85efd4d1eec2978", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c6a66e83-a6f1-4772-a174-1323d2df995c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0faf228e5bc8a80222528f722a0c78d455d584894cd1b845a2c73dc0ab710031", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2327a528-3ed7-48a0-ac0b-4c80e409d1bb", "node_type": "1", "metadata": {}, "hash": "61da286d96fb963babd58cdeeadd7f9bc332f4579fcbc66c3f1f1b17c7cb20ea", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 2536, "end_char_idx": 5052, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
!pip install llama-index
```


```python
# The nest_asyncio module enables the nesting of asynchronous functions within an already running async loop.
# This is necessary because Jupyter notebooks inherently operate in an asynchronous loop.
# By applying nest_asyncio, we can run additional async functions within this existing loop without conflicts.
import nest_asyncio

nest_asyncio.apply()

from llama_index.evaluation import generate_question_context_pairs
from llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext
from llama_index.node_parser import SimpleNodeParser
from llama_index.evaluation import generate_question_context_pairs
from llama_index.evaluation import RetrieverEvaluator
from llama_index.llms import OpenAI

import os
import pandas as pd
```

Set Your OpenAI API Key


```python
os.environ['OPENAI_API_KEY'] = 'YOUR OPENAI API KEY'
```

Let's use [Paul Graham Essay text](https://www.paulgraham.com/worked.html) for building RAG pipeline.

#### Download Data


```python
!mkdir -p 'data/paul_graham/'
!curl 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -o 'data/paul_graham/paul_graham_essay.txt'
```

      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                     Dload  Upload   Total   Spent    Left  Speed
    100 75042  100 75042    0     0   190k      0 --:--:-- --:--:-- --:--:--  190k--:--  0:00:03 24586


#### Load Data and Build Index.


```python
documents = SimpleDirectoryReader("./data/paul_graham/").load_data()

# Define an LLM
llm = OpenAI(model="gpt-4")

# Build index with a chunk_size of 512
node_parser = SimpleNodeParser.from_defaults(chunk_size=512)
nodes = node_parser.get_nodes_from_documents(documents)
vector_index = VectorStoreIndex(nodes)
```

Build a QueryEngine and start querying.


```python
query_engine = vector_index.as_query_engine()
```


```python
response_vector = query_engine.query("What did the author do growing up?")
```

Check response.


```python
response_vector.response
```




    'The author wrote short stories and worked on programming, specifically on an IBM 1401 computer using an early version of Fortran.'



By default it retrieves `two` similar nodes/ chunks. You can modify that in `vector_index.as_query_engine(similarity_top_k=k)`.

Let's check the text in each of these retrieved nodes.


```python
# First retrieved node
response_vector.source_nodes[0].get_text()
```


key: _node_content
string_value:
{"id_": "2327a528-3ed7-48a0-ac0b-4c80e409d1bb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99de0ab8-a81a-4792-80de-b1084e9bd657", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3d87226cf3e1ecbc42d0bed8b11ca11fa5c935369f6c9e45c85efd4d1eec2978", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "fbaa895d-5682-484b-97ff-699217657376", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "4776de8bd34aa62c6ac9e6f8020b6b188b3d52d21a6e2cd3601f82e94aebe17f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4ea5f7bf-0ddd-4b38-924a-96835c00f885", "node_type": "1", "metadata": {}, "hash": "81bd1a393b6cb99db4709af65021450a8d4448932336a3cef5fe84a52cdd8971", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4034, "end_char_idx": 7154, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
#### Load Data and Build Index.


```python
documents = SimpleDirectoryReader("./data/paul_graham/").load_data()

# Define an LLM
llm = OpenAI(model="gpt-4")

# Build index with a chunk_size of 512
node_parser = SimpleNodeParser.from_defaults(chunk_size=512)
nodes = node_parser.get_nodes_from_documents(documents)
vector_index = VectorStoreIndex(nodes)
```

Build a QueryEngine and start querying.


```python
query_engine = vector_index.as_query_engine()
```


```python
response_vector = query_engine.query("What did the author do growing up?")
```

Check response.


```python
response_vector.response
```




    'The author wrote short stories and worked on programming, specifically on an IBM 1401 computer using an early version of Fortran.'



By default it retrieves `two` similar nodes/ chunks. You can modify that in `vector_index.as_query_engine(similarity_top_k=k)`.

Let's check the text in each of these retrieved nodes.


```python
# First retrieved node
response_vector.source_nodes[0].get_text()
```




    'What I Worked On\n\nFebruary 2021\n\nBefore college the two main things I worked on, outside of school, were writing and programming. I didn\'t write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.\n\nThe first programs I tried writing were on the IBM 1401 that our school district used for what was then called "data processing." This was in 9th grade, so I was 13 or 14. The school district\'s 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain\'s lair down there, with all these alien-looking machines — CPU, disk drives, printer, card reader — sitting up on a raised floor under bright fluorescent lights.\n\nThe language we used was an early version of Fortran. You had to type programs on punch cards, then stack them in the card reader and press a button to load the program into memory and run it. The result would ordinarily be to print something on the spectacularly loud printer.\n\nI was puzzled by the 1401. I couldn\'t figure out what to do with it. And in retrospect there\'s not much I could have done with it. The only form of input to programs was data stored on punched cards, and I didn\'t have any data stored on punched cards. The only other option was to do things that didn\'t rely on any input, like calculate approximations of pi, but I didn\'t know enough math to do anything interesting of that type. So I\'m not surprised I can\'t remember any programs I wrote, because they can\'t have done much. My clearest memory is of the moment I learned it was possible for programs not to terminate, when one of mine didn\'t. On a machine without time-sharing, this was a social as well as a technical error, as the data center manager\'s expression made clear.\n\nWith microcomputers, everything changed.'




```python
# Second retrieved node
response_vector.source_nodes[1].get_text()
```


key: _node_content
string_value:
{"id_": "4ea5f7bf-0ddd-4b38-924a-96835c00f885", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99de0ab8-a81a-4792-80de-b1084e9bd657", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3d87226cf3e1ecbc42d0bed8b11ca11fa5c935369f6c9e45c85efd4d1eec2978", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2327a528-3ed7-48a0-ac0b-4c80e409d1bb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "41469b2004099464480bf9ddf983ad852d923e2c5b371f2a0807b4061e4c58e0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "588f2f3f-a86e-4639-8cf0-d322c5f2879c", "node_type": "1", "metadata": {}, "hash": "c04fa934ab1372695a643ad2181d320704d196e8cbcd866226719624c27f30ca", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7074, "end_char_idx": 9296, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Second retrieved node
response_vector.source_nodes[1].get_text()
```




    "It felt like I was doing life right. I remember that because I was slightly dismayed at how novel it felt. The good news is that I had more moments like this over the next few years.\n\nIn the summer of 2016 we moved to England. We wanted our kids to see what it was like living in another country, and since I was a British citizen by birth, that seemed the obvious choice. We only meant to stay for a year, but we liked it so much that we still live there. So most of Bel was written in England.\n\nIn the fall of 2019, Bel was finally finished. Like McCarthy's original Lisp, it's a spec rather than an implementation, although like McCarthy's Lisp it's a spec expressed as code.\n\nNow that I could write essays again, I wrote a bunch about topics I'd had stacked up. I kept writing essays through 2020, but I also started to think about other things I could work on. How should I choose what to do? Well, how had I chosen what to work on in the past? I wrote an essay for myself to answer that question, and I was surprised how long and messy the answer turned out to be. If this surprised me, who'd lived it, then I thought perhaps it would be interesting to other people, and encouraging to those with similarly messy lives. So I wrote a more detailed version for others to read, and this is the last sentence of it.\n\n\n\n\n\n\n\n\n\nNotes\n\n[1] My experience skipped a step in the evolution of computers: time-sharing machines with interactive OSes. I went straight from batch processing to microcomputers, which made microcomputers seem all the more exciting.\n\n[2] Italian words for abstract concepts can nearly always be predicted from their English cognates (except for occasional traps like polluzione). It's the everyday words that differ. So if you string together a lot of abstract concepts with a few simple verbs, you can make a little Italian go a long way.\n\n[3] I lived at Piazza San Felice 4, so my walk to the Accademia went straight down the spine of old Florence: past the Pitti, across the bridge, past Orsanmichele, between the Duomo and the Baptistery, and then up Via Ricasoli to Piazza San Marco."


key: _node_content
string_value:
{"id_": "588f2f3f-a86e-4639-8cf0-d322c5f2879c", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99de0ab8-a81a-4792-80de-b1084e9bd657", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3d87226cf3e1ecbc42d0bed8b11ca11fa5c935369f6c9e45c85efd4d1eec2978", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4ea5f7bf-0ddd-4b38-924a-96835c00f885", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "6f011ed0db493d5c4e165df089c4b780d3302efa11ac001c3de1418cedf7b846", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ecbcd0ba-b2a4-485c-bfb2-010bede41053", "node_type": "1", "metadata": {}, "hash": "b73f12652763c10651caedcb96949757445bf1aea680b1f95df7293ebc735e5f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9300, "end_char_idx": 12800, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
We have built a RAG pipeline and now need to evaluate its performance. We can assess our RAG system/query engine using LlamaIndex's core evaluation modules. Let's examine how to leverage these tools to quantify the quality of our retrieval-augmented generation system.

## Evaluation

Evaluation should serve as the primary metric for assessing your RAG application. It determines whether the pipeline will produce accurate responses based on the data sources and a range of queries.

While it's beneficial to examine individual queries and responses at the start, this approach may become impractical as the volume of edge cases and failures increases. Instead, it may be more effective to establish a suite of summary metrics or automated evaluations. These tools can provide insights into overall system performance and indicate specific areas that may require closer scrutiny.

In a RAG system, evaluation focuses on two critical aspects:

*   **Retrieval Evaluation:** This assesses the accuracy and relevance of the information retrieved by the system.
*   **Response Evaluation:** This measures the quality and appropriateness of the responses generated by the system based on the retrieved information.

#### Question-Context Pair Generation:

For the evaluation of a RAG system, it's essential to have queries that can fetch the correct context and subsequently generate an appropriate response. `LlamaIndex` offers a `generate_question_context_pairs` module specifically for crafting questions and context pairs which can be used in the assessment of the RAG system of both Retrieval and Response Evaluation. For more details on Question Generation, please refer to the [documentation](https://docs.llamaindex.ai/en/stable/examples/evaluation/QuestionGeneration.html).


```python
qa_dataset = generate_question_context_pairs(
    nodes,
    llm=llm,
    num_questions_per_chunk=2
)
```

    100%|██████████| 58/58 [06:26<00:00,  6.67s/it]


#### Retrieval Evaluation:

We are now prepared to conduct our retrieval evaluations. We will execute our `RetrieverEvaluator` using the evaluation dataset we have generated.

We first create the `Retriever` and then define two functions: `get_eval_results`, which operates our retriever on the dataset, and `display_results`, which presents the outcomes of the evaluation.

Let's create the retriever.


```python
retriever = vector_index.as_retriever(similarity_top_k=2)
```

Define `RetrieverEvaluator`. We use **Hit Rate** and **MRR** metrics to evaluate our Retriever.

**Hit Rate:**

Hit rate calculates the fraction of queries where the correct answer is found within the top-k retrieved documents. In simpler terms, it’s about how often our system gets it right within the top few guesses.

**Mean Reciprocal Rank (MRR):**

For each query, MRR evaluates the system’s accuracy by looking at the rank of the highest-placed relevant document. Specifically, it’s the average of the reciprocals of these ranks across all the queries. So, if the first relevant document is the top result, the reciprocal rank is 1; if it’s second, the reciprocal rank is 1/2, and so on.

Let's check these metrics to check the performance of out retriever.


```python
retriever_evaluator = RetrieverEvaluator.from_metric_names(
    ["mrr", "hit_rate"], retriever=retriever
)
```


```python
# Evaluate
eval_results = await retriever_evaluator.aevaluate_dataset(qa_dataset)
```

Let's define a function to display the Retrieval evaluation results in table format.


key: _node_content
string_value:
{"id_": "ecbcd0ba-b2a4-485c-bfb2-010bede41053", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99de0ab8-a81a-4792-80de-b1084e9bd657", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3d87226cf3e1ecbc42d0bed8b11ca11fa5c935369f6c9e45c85efd4d1eec2978", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "588f2f3f-a86e-4639-8cf0-d322c5f2879c", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7944b33bcec19cba256b20f74f99120037af9ad27b8eb2ca5160f4d21b219b29", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "41c48007-c539-48f0-979e-218f5498dc83", "node_type": "1", "metadata": {}, "hash": "f0c32c09851d0ec78ec39e7758ebb04c7f7983d69d4700d7061f846cfd345133", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 11252, "end_char_idx": 13969, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
#### Retrieval Evaluation:

We are now prepared to conduct our retrieval evaluations. We will execute our `RetrieverEvaluator` using the evaluation dataset we have generated.

We first create the `Retriever` and then define two functions: `get_eval_results`, which operates our retriever on the dataset, and `display_results`, which presents the outcomes of the evaluation.

Let's create the retriever.


```python
retriever = vector_index.as_retriever(similarity_top_k=2)
```

Define `RetrieverEvaluator`. We use **Hit Rate** and **MRR** metrics to evaluate our Retriever.

**Hit Rate:**

Hit rate calculates the fraction of queries where the correct answer is found within the top-k retrieved documents. In simpler terms, it’s about how often our system gets it right within the top few guesses.

**Mean Reciprocal Rank (MRR):**

For each query, MRR evaluates the system’s accuracy by looking at the rank of the highest-placed relevant document. Specifically, it’s the average of the reciprocals of these ranks across all the queries. So, if the first relevant document is the top result, the reciprocal rank is 1; if it’s second, the reciprocal rank is 1/2, and so on.

Let's check these metrics to check the performance of out retriever.


```python
retriever_evaluator = RetrieverEvaluator.from_metric_names(
    ["mrr", "hit_rate"], retriever=retriever
)
```


```python
# Evaluate
eval_results = await retriever_evaluator.aevaluate_dataset(qa_dataset)
```

Let's define a function to display the Retrieval evaluation results in table format.


```python
def display_results(name, eval_results):
    """Display results from evaluate."""

    metric_dicts = []
    for eval_result in eval_results:
        metric_dict = eval_result.metric_vals_dict
        metric_dicts.append(metric_dict)

    full_df = pd.DataFrame(metric_dicts)

    hit_rate = full_df["hit_rate"].mean()
    mrr = full_df["mrr"].mean()

    metric_df = pd.DataFrame(
        {"Retriever Name": [name], "Hit Rate": [hit_rate], "MRR": [mrr]}
    )

    return metric_df
```


```python
display_results("OpenAI Embedding Retriever", eval_results)
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Retriever Name</th>
      <th>Hit Rate</th>
      <th>MRR</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>OpenAI Embedding Retriever</td>
      <td>0.758621</td>
      <td>0.62069</td>
    </tr>
  </tbody>
</table>
</div>


key: _node_content
string_value:
{"id_": "41c48007-c539-48f0-979e-218f5498dc83", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99de0ab8-a81a-4792-80de-b1084e9bd657", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3d87226cf3e1ecbc42d0bed8b11ca11fa5c935369f6c9e45c85efd4d1eec2978", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ecbcd0ba-b2a4-485c-bfb2-010bede41053", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "51f9e6e0f79a9aa24393431eadcff2f8124fd66df028e16c46e2706035e69a48", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "4429f711-4749-4bc0-9d76-591fc7cd7a80", "node_type": "1", "metadata": {}, "hash": "549106f8cb76e64f06462e529b22bf76d9967224215f05a7db7b7fad0c367dee", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12619, "end_char_idx": 15282, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Evaluate
eval_results = await retriever_evaluator.aevaluate_dataset(qa_dataset)
```

Let's define a function to display the Retrieval evaluation results in table format.


```python
def display_results(name, eval_results):
    """Display results from evaluate."""

    metric_dicts = []
    for eval_result in eval_results:
        metric_dict = eval_result.metric_vals_dict
        metric_dicts.append(metric_dict)

    full_df = pd.DataFrame(metric_dicts)

    hit_rate = full_df["hit_rate"].mean()
    mrr = full_df["mrr"].mean()

    metric_df = pd.DataFrame(
        {"Retriever Name": [name], "Hit Rate": [hit_rate], "MRR": [mrr]}
    )

    return metric_df
```


```python
display_results("OpenAI Embedding Retriever", eval_results)
```




<div>
<style scoped>
    .dataframe tbody tr th:only-of-type {
        vertical-align: middle;
    }

    .dataframe tbody tr th {
        vertical-align: top;
    }

    .dataframe thead th {
        text-align: right;
    }
</style>
<table border="1" class="dataframe">
  <thead>
    <tr style="text-align: right;">
      <th></th>
      <th>Retriever Name</th>
      <th>Hit Rate</th>
      <th>MRR</th>
    </tr>
  </thead>
  <tbody>
    <tr>
      <th>0</th>
      <td>OpenAI Embedding Retriever</td>
      <td>0.758621</td>
      <td>0.62069</td>
    </tr>
  </tbody>
</table>
</div>



#### Observation:

The Retriever with OpenAI Embedding  demonstrates a performance with a hit rate of `0.7586`, while the MRR, at `0.6206`, suggests there's room for improvement in ensuring the most relevant results appear at the top. The observation that MRR is less than the hit rate indicates that the top-ranking results aren't always the most relevant. Enhancing MRR could involve the use of rerankers, which refine the order of retrieved documents. For a deeper understanding of how rerankers can optimize retrieval metrics, refer to the detailed discussion in our [blog post](https://blog.llamaindex.ai/boosting-rag-picking-the-best-embedding-reranker-models-42d079022e83).

#### Response Evaluation:

1. FaithfulnessEvaluator: Measures if the response from a query engine matches any source nodes which is useful for measuring if the response is hallucinated.
2. Relevancy Evaluator: Measures if the response + source nodes match the query.


```python
# Get the list of queries from the above created dataset

queries = list(qa_dataset.queries.values())
```

#### Faithfulness Evaluator

Let's start with FaithfulnessEvaluator.

We will use `gpt-3.5-turbo` for generating response for a given query and `gpt-4` for evaluation.

Let's create service_context seperately for `gpt-3.5-turbo` and `gpt-4`.


key: _node_content
string_value:
{"id_": "4429f711-4749-4bc0-9d76-591fc7cd7a80", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99de0ab8-a81a-4792-80de-b1084e9bd657", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3d87226cf3e1ecbc42d0bed8b11ca11fa5c935369f6c9e45c85efd4d1eec2978", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "41c48007-c539-48f0-979e-218f5498dc83", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "5eb97467bb193a103c8e6a850a974b9c4fcc3c3b1cfe2ee03fef586f40a5a743", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c11070c4-617a-4159-961b-59187d5109bb", "node_type": "1", "metadata": {}, "hash": "4c2c05e99bc0e5d6ed09c4eb42b8b5e8d04fb4adfcd0cbafe1145c8497b96755", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13973, "end_char_idx": 17109, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
#### Observation:

The Retriever with OpenAI Embedding  demonstrates a performance with a hit rate of `0.7586`, while the MRR, at `0.6206`, suggests there's room for improvement in ensuring the most relevant results appear at the top. The observation that MRR is less than the hit rate indicates that the top-ranking results aren't always the most relevant. Enhancing MRR could involve the use of rerankers, which refine the order of retrieved documents. For a deeper understanding of how rerankers can optimize retrieval metrics, refer to the detailed discussion in our [blog post](https://blog.llamaindex.ai/boosting-rag-picking-the-best-embedding-reranker-models-42d079022e83).

#### Response Evaluation:

1. FaithfulnessEvaluator: Measures if the response from a query engine matches any source nodes which is useful for measuring if the response is hallucinated.
2. Relevancy Evaluator: Measures if the response + source nodes match the query.


```python
# Get the list of queries from the above created dataset

queries = list(qa_dataset.queries.values())
```

#### Faithfulness Evaluator

Let's start with FaithfulnessEvaluator.

We will use `gpt-3.5-turbo` for generating response for a given query and `gpt-4` for evaluation.

Let's create service_context seperately for `gpt-3.5-turbo` and `gpt-4`.


```python
# gpt-3.5-turbo
gpt35 = OpenAI(temperature=0, model="gpt-3.5-turbo")
service_context_gpt35 = ServiceContext.from_defaults(llm=gpt35)

# gpt-4
gpt4 = OpenAI(temperature=0, model="gpt-4")
service_context_gpt4 = ServiceContext.from_defaults(llm=gpt4)
```

Create a `QueryEngine` with `gpt-3.5-turbo` service_context to generate response for the query.


```python
vector_index = VectorStoreIndex(nodes, service_context = service_context_gpt35)
query_engine = vector_index.as_query_engine()
```

Create a  FaithfulnessEvaluator.





```python
from llama_index.evaluation import FaithfulnessEvaluator
faithfulness_gpt4 = FaithfulnessEvaluator(service_context=service_context_gpt4)
```

Let's evaluate on one question.


```python
eval_query = queries[10]

eval_query
```




    "Based on the author's experience and observations, why did he consider the AI practices during his first year of grad school as a hoax? Provide specific examples from the text to support your answer."



Generate response first and use faithfull evaluator.


```python
response_vector = query_engine.query(eval_query)
```


```python
# Compute faithfulness evaluation

eval_result = faithfulness_gpt4.evaluate_response(response=response_vector)
```


```python
# You can check passing parameter in eval_result if it passed the evaluation.
eval_result.passing
```




    True



#### Relevancy Evaluator

RelevancyEvaluator is useful to measure if the response and source nodes (retrieved context) match the query. Useful to see if response actually answers the query.

Instantiate `RelevancyEvaluator` for relevancy evaluation with `gpt-4`


```python
from llama_index.evaluation import RelevancyEvaluator

relevancy_gpt4 = RelevancyEvaluator(service_context=service_context_gpt4)
```

Let's do relevancy evaluation for one of the query.


key: _node_content
string_value:
{"id_": "c11070c4-617a-4159-961b-59187d5109bb", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99de0ab8-a81a-4792-80de-b1084e9bd657", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3d87226cf3e1ecbc42d0bed8b11ca11fa5c935369f6c9e45c85efd4d1eec2978", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "4429f711-4749-4bc0-9d76-591fc7cd7a80", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "512b06e91036f3f8068628ddfbd58047555177c8911aa6f1414a643ec3489b9f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "6005e630-a0cf-4519-b795-fa4667ae78bd", "node_type": "1", "metadata": {}, "hash": "029bba2fa3cccf385dc9a81d85edb398af99c2381665be7e9461197e76b7cbc1", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15646, "end_char_idx": 18901, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
vector_index = VectorStoreIndex(nodes, service_context = service_context_gpt35)
query_engine = vector_index.as_query_engine()
```

Create a  FaithfulnessEvaluator.





```python
from llama_index.evaluation import FaithfulnessEvaluator
faithfulness_gpt4 = FaithfulnessEvaluator(service_context=service_context_gpt4)
```

Let's evaluate on one question.


```python
eval_query = queries[10]

eval_query
```




    "Based on the author's experience and observations, why did he consider the AI practices during his first year of grad school as a hoax? Provide specific examples from the text to support your answer."



Generate response first and use faithfull evaluator.


```python
response_vector = query_engine.query(eval_query)
```


```python
# Compute faithfulness evaluation

eval_result = faithfulness_gpt4.evaluate_response(response=response_vector)
```


```python
# You can check passing parameter in eval_result if it passed the evaluation.
eval_result.passing
```




    True



#### Relevancy Evaluator

RelevancyEvaluator is useful to measure if the response and source nodes (retrieved context) match the query. Useful to see if response actually answers the query.

Instantiate `RelevancyEvaluator` for relevancy evaluation with `gpt-4`


```python
from llama_index.evaluation import RelevancyEvaluator

relevancy_gpt4 = RelevancyEvaluator(service_context=service_context_gpt4)
```

Let's do relevancy evaluation for one of the query.


```python
# Pick a query
query = queries[10]

query
```




    "Based on the author's experience and observations, why did he consider the AI practices during his first year of grad school as a hoax? Provide specific examples from the text to support your answer."




```python
# Generate response.
# response_vector has response and source nodes (retrieved context)
response_vector = query_engine.query(query)

# Relevancy evaluation
eval_result = relevancy_gpt4.evaluate_response(
    query=query, response=response_vector
)
```


```python
# You can check passing parameter in eval_result if it passed the evaluation.
eval_result.passing
```




    True




```python
# You can get the feedback for the evaluation.
eval_result.feedback
```




    'YES'



#### Batch Evaluator:

Now that we have done FaithFulness and Relevancy Evaluation independently. LlamaIndex has `BatchEvalRunner` to compute multiple evaluations in batch wise manner.


```python
from llama_index.evaluation import BatchEvalRunner

# Let's pick top 10 queries to do evaluation
batch_eval_queries = queries[:10]

# Initiate BatchEvalRunner to compute FaithFulness and Relevancy Evaluation.
runner = BatchEvalRunner(
    {"faithfulness": faithfulness_gpt4, "relevancy": relevancy_gpt4},
    workers=8,
)

# Compute evaluation
eval_results = await runner.aevaluate_queries(
    query_engine, queries=batch_eval_queries
)
```


```python
# Let's get faithfulness score

faithfulness_score = sum(result.passing for result in eval_results['faithfulness']) / len(eval_results['faithfulness'])

faithfulness_score
```




    1.0




```python
# Let's get relevancy score

relevancy_score = sum(result.passing for result in eval_results['relevancy']) / len(eval_results['relevancy'])

relevancy_score

```




    1.0


key: _node_content
string_value:
{"id_": "6005e630-a0cf-4519-b795-fa4667ae78bd", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "99de0ab8-a81a-4792-80de-b1084e9bd657", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "3d87226cf3e1ecbc42d0bed8b11ca11fa5c935369f6c9e45c85efd4d1eec2978", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c11070c4-617a-4159-961b-59187d5109bb", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "ce531dd10450efe578a6e37a674d623bea6daa1ef24f646395d62a410b313f26", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17176, "end_char_idx": 19579, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
"Based on the author's experience and observations, why did he consider the AI practices during his first year of grad school as a hoax? Provide specific examples from the text to support your answer."




```python
# Generate response.
# response_vector has response and source nodes (retrieved context)
response_vector = query_engine.query(query)

# Relevancy evaluation
eval_result = relevancy_gpt4.evaluate_response(
    query=query, response=response_vector
)
```


```python
# You can check passing parameter in eval_result if it passed the evaluation.
eval_result.passing
```




    True




```python
# You can get the feedback for the evaluation.
eval_result.feedback
```




    'YES'



#### Batch Evaluator:

Now that we have done FaithFulness and Relevancy Evaluation independently. LlamaIndex has `BatchEvalRunner` to compute multiple evaluations in batch wise manner.


```python
from llama_index.evaluation import BatchEvalRunner

# Let's pick top 10 queries to do evaluation
batch_eval_queries = queries[:10]

# Initiate BatchEvalRunner to compute FaithFulness and Relevancy Evaluation.
runner = BatchEvalRunner(
    {"faithfulness": faithfulness_gpt4, "relevancy": relevancy_gpt4},
    workers=8,
)

# Compute evaluation
eval_results = await runner.aevaluate_queries(
    query_engine, queries=batch_eval_queries
)
```


```python
# Let's get faithfulness score

faithfulness_score = sum(result.passing for result in eval_results['faithfulness']) / len(eval_results['faithfulness'])

faithfulness_score
```




    1.0




```python
# Let's get relevancy score

relevancy_score = sum(result.passing for result in eval_results['relevancy']) / len(eval_results['relevancy'])

relevancy_score

```




    1.0



#### Observation:

Faithfulness score of `1.0` signifies that the generated answers contain no hallucinations and are entirely based on retrieved context.

Relevancy score of `1.0` suggests that the answers generated are consistently aligned with the retrieved context and the queries.

## Conclusion

In this notebook, we have explored how to build and evaluate a RAG pipeline using LlamaIndex, with a specific focus on evaluating the retrieval system and generated responses within the pipeline. 

LlamaIndex offers a variety of other evaluation modules as well, which you can explore further [here](https://docs.llamaindex.ai/en/stable/module_guides/evaluating/root.html)


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb


key: _node_content
string_value:
{"id_": "c977feff-b0e1-4f7a-bf3e-a74e92aebe27", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b0987757-cb7f-4604-90a1-17aede6900d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9ea262c5cfe1d5088303e0fa4d72d2d33cdf85e20df26faa3a3a20edfa01e0a7", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d83f4722-5cd5-48f1-a614-94e57bee184b", "node_type": "1", "metadata": {}, "hash": "55eed429aec6c27c9bdf5797d80594800ff9cf6a85932198ef18202f8ac12339", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 2064, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT

This notebook provides step by step instuctions on using Azure AI Search (f.k.a Azure Cognitive Search) as a vector database with OpenAI embeddings, then creating an Azure Function on top to plug into a Custom GPT in ChatGPT. 

This can be a solution for customers looking to set up RAG infrastructure contained within Azure, and exposing it as an endpoint to integrate that with other platforms such as ChatGPT.

Azure AI Search is a cloud search service that gives developers infrastructure, APIs, and tools for building a rich search experience over private, heterogeneous content in web, mobile, and enterprise applications. 

Azure Functions is a serverless compute service that runs event-driven code, automatically managing infrastructure, scaling, and integrating with other Azure services.

## Prerequisites:
For the purposes of this exercise you must have the following:
- Azure user with permission to create [Azure AI Search Service](https://learn.microsoft.com/azure/search/) and Azure Function Apps
- Azure subscription ID and a resource group.
- [OpenAI Key](https://platform.openai.com/account/api-keys) 

# Architecture
Below is a diagram of the architecture of this solution, which we'll walk through step-by-step.

![azure-rag-architecture.png](../../../../images/azure-rag-architecture.png)


> Note: This architecture pattern of vector data store + serverless functions can be extrapolated to other vector data stores. For example, if you would want to use something like Postgres within Azure, you'd change the [Configure Azure AI Search Settings](#configure-azure-ai-search-settings) step to set-up the requirements for Postgres, you'd modify the [Create Azure AI Vector Search](#create-azure-ai-vector-search) to create the database and table in Postgres instead, and you'd update the `function_app.py` code in this repository to query Postgres instead of Azure AI Search. The data preparation and creation of the Azure Function would stay consistent.


key: _node_content
string_value:
{"id_": "d83f4722-5cd5-48f1-a614-94e57bee184b", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b0987757-cb7f-4604-90a1-17aede6900d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9ea262c5cfe1d5088303e0fa4d72d2d33cdf85e20df26faa3a3a20edfa01e0a7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c977feff-b0e1-4f7a-bf3e-a74e92aebe27", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "5ba85cea8ed56930c58a15b0a691baf4b4558f36f0510b22fea47a413019b210", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "678698b6-6d92-48d7-96ba-07d8e06858d6", "node_type": "1", "metadata": {}, "hash": "d5e5ff2e1959591ad2c62a929fa8af3de32e3f5cc13cb74fbd3d787c163e437e", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1402, "end_char_idx": 4481, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
> Note: This architecture pattern of vector data store + serverless functions can be extrapolated to other vector data stores. For example, if you would want to use something like Postgres within Azure, you'd change the [Configure Azure AI Search Settings](#configure-azure-ai-search-settings) step to set-up the requirements for Postgres, you'd modify the [Create Azure AI Vector Search](#create-azure-ai-vector-search) to create the database and table in Postgres instead, and you'd update the `function_app.py` code in this repository to query Postgres instead of Azure AI Search. The data preparation and creation of the Azure Function would stay consistent. 


# Table of Contents:

1. **[Setup of Environment](#set-up-environment)**
    Setup environment by installing and importing the required libraries and configuring our Azure settings. Includes:
     - [Install and Import Required Libraries](#install-and-import-required-libraries)
     - [Configure OpenAI Settings](#configure-openai-settings)
     - [Configure Azure AI Search Settings](#configure-azure-ai-search-settings)
 

2. **[Prepare Data](#prepare-data)** Prepare the data for uploading by embedding the documents, as well as capturing additional metadata. We will use a subset of OpenAI's docs as example data for this.
 
3. **[Create Azure AI Vector Search](#create-azure-ai-vector-search)** Create an Azure AI Vector Search and upload the data we've prepared. Includes:
     - [Create Index](#create-index): Steps to create an index in Azure AI Search.
     - [Upload Data](#upload-data): Instructions to upload data to Azure AI Search.
     - [Test Search](#test-search): Steps to test the search functionality.
 
4. **[Create Azure Function](#create-azure-function)** Create an Azure Function to interact with the Azure AI Vector Search. Includes:
     - [Create Storage Account](#create-storage-account): Steps to create a storage account for the Azure Function.
     - [Create Function App](#create-function-app): Instructions to create a function app in Azure.
 
5. **[Input in a Custom GPT in ChatGPT](#input-in-a-custom-gpt-in-chatgpt)** Integrate the Azure Function with a Custom GPT in ChatGPT. Includes:
     - [Create OpenAPI Spec](#create-openapi-spec): Steps to create an OpenAPI specification for the Azure Function.
     - [Create GPT Instructions](#create-gpt-instructions): Instructions to create GPT-specific instructions for the integration.




# Set up environment
We'll set up our environment by importing the required libraries and configuring our Azure settings.

## Install and import required libraries
We categorize these libraries into standard Python libraries, third-party libraries, and Azure-related libraries for readability.


```python
! pip install -q wget
! pip install -q azure-search-documents 
! pip install -q azure-identity
! pip install -q openai
! pip install -q azure-mgmt-search
! pip install -q pandas
! pip install -q azure-mgmt-resource 
! pip install -q azure-mgmt-storage
! pip install -q pyperclip
! pip install -q PyPDF2
! pip install -q tiktoken
```


key: _node_content
string_value:
{"id_": "678698b6-6d92-48d7-96ba-07d8e06858d6", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b0987757-cb7f-4604-90a1-17aede6900d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9ea262c5cfe1d5088303e0fa4d72d2d33cdf85e20df26faa3a3a20edfa01e0a7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d83f4722-5cd5-48f1-a614-94e57bee184b", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c891640ca6e94ee5afb409917d4b535b88dd3ddd7374e8514432cccdb675375d", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "cf98f97a-2445-4efb-8e6b-a711a9fb2c85", "node_type": "1", "metadata": {}, "hash": "fac34c7bfe3727c21a1238c42e099bb83f0148da3f07d7da94308f4af2a37770", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3843, "end_char_idx": 6713, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# Set up environment
We'll set up our environment by importing the required libraries and configuring our Azure settings.

## Install and import required libraries
We categorize these libraries into standard Python libraries, third-party libraries, and Azure-related libraries for readability.


```python
! pip install -q wget
! pip install -q azure-search-documents 
! pip install -q azure-identity
! pip install -q openai
! pip install -q azure-mgmt-search
! pip install -q pandas
! pip install -q azure-mgmt-resource 
! pip install -q azure-mgmt-storage
! pip install -q pyperclip
! pip install -q PyPDF2
! pip install -q tiktoken
```


```python
# Standard Libraries
import json  
import os
import platform
import subprocess
import csv
from itertools import islice
import uuid
import shutil
import concurrent.futures

# Third-Party Libraries
import pandas as pd
from PyPDF2 import PdfReader
import tiktoken
from dotenv import load_dotenv
import pyperclip

# OpenAI Libraries (note we use OpenAI directly here, but you can replace with Azure OpenAI as needed)
from openai import OpenAI

# Azure Identity and Credentials
from azure.identity import DefaultAzureCredential, InteractiveBrowserCredential
from azure.core.credentials import AzureKeyCredential  
from azure.core.exceptions import HttpResponseError

# Azure Search Documents
from azure.search.documents import SearchClient, SearchIndexingBufferedSender  
from azure.search.documents.indexes import SearchIndexClient  
from azure.search.documents.models import (
    VectorizedQuery
)
from azure.search.documents.indexes.models import (
    HnswAlgorithmConfiguration,
    HnswParameters,
    SearchField,
    SearchableField,
    SearchFieldDataType,
    SearchIndex,
    SimpleField,
    VectorSearch,
    VectorSearchAlgorithmKind,
    VectorSearchAlgorithmMetric,
    VectorSearchProfile,
)

# Azure Management Clients
from azure.mgmt.search import SearchManagementClient
from azure.mgmt.resource import ResourceManagementClient, SubscriptionClient
from azure.mgmt.storage import StorageManagementClient
```

## Configure OpenAI settings

Before going through this section, make sure you have your OpenAI API key.



```python
openai_api_key = os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as an env var>") # Saving this as a variable to reference in function app in later step
openai_client = OpenAI(api_key=openai_api_key)
embeddings_model = "text-embedding-3-small" # We'll use this by default, but you can change to your text-embedding-3-large if desired
```

## Configure Azure AI Search Settings
You can locate your Azure AI Search service details in the Azure Portal or programmatically via the [Search Management SDK](https://learn.microsoft.com/rest/api/searchmanagement/).


#### Prerequisites:
- Subscription ID from Azure
- Resource Group name from Azure
- Region in Azure


key: _node_content
string_value:
{"id_": "cf98f97a-2445-4efb-8e6b-a711a9fb2c85", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b0987757-cb7f-4604-90a1-17aede6900d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9ea262c5cfe1d5088303e0fa4d72d2d33cdf85e20df26faa3a3a20edfa01e0a7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "678698b6-6d92-48d7-96ba-07d8e06858d6", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "53a4f03c09ddefb4bacf7c45a5411bb838ad586e33f33ef017eaddd095b4cef9", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "f3ae4ec7-d1f8-432c-bd92-e5d74f799774", "node_type": "1", "metadata": {}, "hash": "8b072004b85a152337416fbe4fe776eee7fd5b9e4dad835a6a57bf19c37fd374", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6026, "end_char_idx": 9267, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
openai_api_key = os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as an env var>") # Saving this as a variable to reference in function app in later step
openai_client = OpenAI(api_key=openai_api_key)
embeddings_model = "text-embedding-3-small" # We'll use this by default, but you can change to your text-embedding-3-large if desired
```

## Configure Azure AI Search Settings
You can locate your Azure AI Search service details in the Azure Portal or programmatically via the [Search Management SDK](https://learn.microsoft.com/rest/api/searchmanagement/).


#### Prerequisites:
- Subscription ID from Azure
- Resource Group name from Azure
- Region in Azure


```python
# Update the below with your values
subscription_id="<enter_your_subscription_id>"
resource_group="<enter_your_resource_group>"

## Make sure to choose a region that supports the proper products. We've defaulted to "eastus" below. https://azure.microsoft.com/en-us/explore/global-infrastructure/products-by-region/#products-by-region_tab5
region = "eastus"
credential = InteractiveBrowserCredential()
subscription_client = SubscriptionClient(credential)
subscription = next(subscription_client.subscriptions.list())
```

#### Create and Configure Azure AI Search Service
Below we'll generate a unique name for the search service, set up the service properties, and create the search service.


```python
# Initialize the SearchManagementClient with the provided credentials and subscription ID
search_management_client = SearchManagementClient(
    credential=credential,
    subscription_id=subscription_id,
)

# Generate a unique name for the search service using UUID, but you can change this if you'd like.
generated_uuid = str(uuid.uuid4())
search_service_name = "search-service-gpt-demo" + generated_uuid
## The below is the default endpoint structure that is created when you create a search service. This may differ based on your Azure settings.
search_service_endpoint = 'https://'+search_service_name+'.search.windows.net'

# Create or update the search service with the specified parameters
response = search_management_client.services.begin_create_or_update(
    resource_group_name=resource_group,
    search_service_name=search_service_name,
    service={
        "location": region,
        "properties": {"hostingMode": "default", "partitionCount": 1, "replicaCount": 1},
        # We are using the free pricing tier for this demo. You are only allowed one free search service per subscription.
        "sku": {"name": "free"},
        "tags": {"app-name": "Search service demo"},
    },
).result()

# Convert the response to a dictionary and then to a pretty-printed JSON string
response_dict = response.as_dict()
response_json = json.dumps(response_dict, indent=4)

print(response_json)
print("Search Service Name:" + search_service_name)
print("Search Service Endpoint:" + search_service_endpoint)

```

#### Get the Search Service API Key
Now that we have the search service up and running, we need the [Search Service API Key](https://learn.microsoft.com/en-us/azure/search/search-security-api-keys?tabs=rest-use,portal-find,portal-query), which we'll use to initiate the index creation, and later to execute the search.


key: _node_content
string_value:
{"id_": "f3ae4ec7-d1f8-432c-bd92-e5d74f799774", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b0987757-cb7f-4604-90a1-17aede6900d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9ea262c5cfe1d5088303e0fa4d72d2d33cdf85e20df26faa3a3a20edfa01e0a7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "cf98f97a-2445-4efb-8e6b-a711a9fb2c85", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "b745a46e58552127728ef862a0063e9b5ab79e2189c4d20bbf083c10189220a8", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "694d2eab-3742-496d-940f-d23a740a903e", "node_type": "1", "metadata": {}, "hash": "43ae46e9a96cffdd57e67dd4216650cd12021baa719055bc26b55c64691191a2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 7420, "end_char_idx": 10245, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Initialize the SearchManagementClient with the provided credentials and subscription ID
search_management_client = SearchManagementClient(
    credential=credential,
    subscription_id=subscription_id,
)

# Generate a unique name for the search service using UUID, but you can change this if you'd like.
generated_uuid = str(uuid.uuid4())
search_service_name = "search-service-gpt-demo" + generated_uuid
## The below is the default endpoint structure that is created when you create a search service. This may differ based on your Azure settings.
search_service_endpoint = 'https://'+search_service_name+'.search.windows.net'

# Create or update the search service with the specified parameters
response = search_management_client.services.begin_create_or_update(
    resource_group_name=resource_group,
    search_service_name=search_service_name,
    service={
        "location": region,
        "properties": {"hostingMode": "default", "partitionCount": 1, "replicaCount": 1},
        # We are using the free pricing tier for this demo. You are only allowed one free search service per subscription.
        "sku": {"name": "free"},
        "tags": {"app-name": "Search service demo"},
    },
).result()

# Convert the response to a dictionary and then to a pretty-printed JSON string
response_dict = response.as_dict()
response_json = json.dumps(response_dict, indent=4)

print(response_json)
print("Search Service Name:" + search_service_name)
print("Search Service Endpoint:" + search_service_endpoint)

```

#### Get the Search Service API Key
Now that we have the search service up and running, we need the [Search Service API Key](https://learn.microsoft.com/en-us/azure/search/search-security-api-keys?tabs=rest-use,portal-find,portal-query), which we'll use to initiate the index creation, and later to execute the search.


```python
# Retrieve the admin keys for the search service
try:
    response = search_management_client.admin_keys.get(
        resource_group_name=resource_group,
        search_service_name=search_service_name,
    )
    # Extract the primary API key from the response and save as a variable to be used later
    search_service_api_key = response.primary_key
    print("Successfully retrieved the API key.")
except Exception as e:
    print(f"Failed to retrieve the API key: {e}")
```

# Prepare data
We're going to embed and store a few pages of the OpenAI docs in the oai_docs folder. We'll first embed each, add it to a CSV, and then use that CSV to upload to the index.

In order to handle longer text files beyond the context of 8191 tokens, we can either use the chunk embeddings separately, or combine them in some way, such as averaging (weighted by the size of each chunk).

We will take a function from Python's own cookbook that breaks up a sequence into chunks.


key: _node_content
string_value:
{"id_": "694d2eab-3742-496d-940f-d23a740a903e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b0987757-cb7f-4604-90a1-17aede6900d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9ea262c5cfe1d5088303e0fa4d72d2d33cdf85e20df26faa3a3a20edfa01e0a7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "f3ae4ec7-d1f8-432c-bd92-e5d74f799774", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "e7c6b08311452a9d5c95438328c8c45604a40e793f5d3ca2f3cb0ef1d163ae4f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c96e6ae5-f365-44e5-9a2e-53c632a9fe38", "node_type": "1", "metadata": {}, "hash": "3d9749de9765f0c484cdf81bc66dc55f49e425a7732f978a7a12119851d90473", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9270, "end_char_idx": 12536, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Retrieve the admin keys for the search service
try:
    response = search_management_client.admin_keys.get(
        resource_group_name=resource_group,
        search_service_name=search_service_name,
    )
    # Extract the primary API key from the response and save as a variable to be used later
    search_service_api_key = response.primary_key
    print("Successfully retrieved the API key.")
except Exception as e:
    print(f"Failed to retrieve the API key: {e}")
```

# Prepare data
We're going to embed and store a few pages of the OpenAI docs in the oai_docs folder. We'll first embed each, add it to a CSV, and then use that CSV to upload to the index.

In order to handle longer text files beyond the context of 8191 tokens, we can either use the chunk embeddings separately, or combine them in some way, such as averaging (weighted by the size of each chunk).

We will take a function from Python's own cookbook that breaks up a sequence into chunks.


```python
def batched(iterable, n):
    """Batch data into tuples of length n. The last batch may be shorter."""
    # batched('ABCDEFG', 3) --> ABC DEF G
    if n < 1:
        raise ValueError('n must be at least one')
    it = iter(iterable)
    while (batch := tuple(islice(it, n))):
        yield batch

```

Now we define a function that encodes a string into tokens and then breaks it up into chunks. We'll use tiktoken, a fast open-source tokenizer by OpenAI.

To read more about counting tokens with Tiktoken, check out [this cookbook](https://cookbook.openai.com/examples/how_to_count_tokens_with_tiktoken). 




```python
def chunked_tokens(text, chunk_length, encoding_name='cl100k_base'):
    # Get the encoding object for the specified encoding name. OpenAI's tiktoken library, which is used in this notebook, currently supports two encodings: 'bpe' and 'cl100k_base'. The 'bpe' encoding is used for GPT-3 and earlier models, while 'cl100k_base' is used for newer models like GPT-4.
    encoding = tiktoken.get_encoding(encoding_name)
    # Encode the input text into tokens
    tokens = encoding.encode(text)
    # Create an iterator that yields chunks of tokens of the specified length
    chunks_iterator = batched(tokens, chunk_length)
    # Yield each chunk from the iterator
    yield from chunks_iterator
```

Finally, we can write a function that safely handles embedding requests, even when the input text is longer than the maximum context length, by chunking the input tokens and embedding each chunk individually. The average flag can be set to True to return the weighted average of the chunk embeddings, or False to simply return the unmodified list of chunk embeddings.

> Note: there are other, more sophisticated techniques you can take here, including:
> - using GPT-4o to capture images/chart descriptions for embedding.
> - keeping text overlap between the chunks to minimize cutting off important context.
> - chunking based on paragraphs or sections.
> - adding more descriptive metadata about each article.


```python
## Change the below based on model. The below is for the latest embeddings models from OpenAI, so you can leave as is unless you are using a different embedding model..
EMBEDDING_CTX_LENGTH = 8191
EMBEDDING_ENCODING='cl100k_base'
```


key: _node_content
string_value:
{"id_": "c96e6ae5-f365-44e5-9a2e-53c632a9fe38", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b0987757-cb7f-4604-90a1-17aede6900d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9ea262c5cfe1d5088303e0fa4d72d2d33cdf85e20df26faa3a3a20edfa01e0a7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "694d2eab-3742-496d-940f-d23a740a903e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "8c6e87519f24b94726cf128870bddb44bd48b8cc221c2ddb23bd982ee4f8f9bb", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "dbace265-84df-48c7-8e60-49ab714fa678", "node_type": "1", "metadata": {}, "hash": "3fad5a39935ab46eecd10c3b745833bc1065f43e8c6abf8c64398510f00eede6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 10870, "end_char_idx": 13962, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def chunked_tokens(text, chunk_length, encoding_name='cl100k_base'):
    # Get the encoding object for the specified encoding name. OpenAI's tiktoken library, which is used in this notebook, currently supports two encodings: 'bpe' and 'cl100k_base'. The 'bpe' encoding is used for GPT-3 and earlier models, while 'cl100k_base' is used for newer models like GPT-4.
    encoding = tiktoken.get_encoding(encoding_name)
    # Encode the input text into tokens
    tokens = encoding.encode(text)
    # Create an iterator that yields chunks of tokens of the specified length
    chunks_iterator = batched(tokens, chunk_length)
    # Yield each chunk from the iterator
    yield from chunks_iterator
```

Finally, we can write a function that safely handles embedding requests, even when the input text is longer than the maximum context length, by chunking the input tokens and embedding each chunk individually. The average flag can be set to True to return the weighted average of the chunk embeddings, or False to simply return the unmodified list of chunk embeddings.

> Note: there are other, more sophisticated techniques you can take here, including:
> - using GPT-4o to capture images/chart descriptions for embedding.
> - keeping text overlap between the chunks to minimize cutting off important context.
> - chunking based on paragraphs or sections.
> - adding more descriptive metadata about each article.


```python
## Change the below based on model. The below is for the latest embeddings models from OpenAI, so you can leave as is unless you are using a different embedding model..
EMBEDDING_CTX_LENGTH = 8191
EMBEDDING_ENCODING='cl100k_base'
```


```python
def generate_embeddings(text, model):
    # Generate embeddings for the provided text using the specified model
    embeddings_response = openai_client.embeddings.create(model=model, input=text)
    # Extract the embedding data from the response
    embedding = embeddings_response.data[0].embedding
    return embedding

def len_safe_get_embedding(text, model=embeddings_model, max_tokens=EMBEDDING_CTX_LENGTH, encoding_name=EMBEDDING_ENCODING):
    # Initialize lists to store embeddings and corresponding text chunks
    chunk_embeddings = []
    chunk_texts = []
    # Iterate over chunks of tokens from the input text
    for chunk in chunked_tokens(text, chunk_length=max_tokens, encoding_name=encoding_name):
        # Generate embeddings for each chunk and append to the list
        chunk_embeddings.append(generate_embeddings(chunk, model=model))
        # Decode the chunk back to text and append to the list
        chunk_texts.append(tiktoken.get_encoding(encoding_name).decode(chunk))
    # Return the list of chunk embeddings and the corresponding text chunks
    return chunk_embeddings, chunk_texts
```

Next, we can define a helper function that will capture additional metadata about the documents. This is useful to use as a metadata filter for search queries, and capturing richer data for search. 

In this example, I'll choose from a list of categories to use later on in a metadata filter.


key: _node_content
string_value:
{"id_": "dbace265-84df-48c7-8e60-49ab714fa678", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b0987757-cb7f-4604-90a1-17aede6900d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9ea262c5cfe1d5088303e0fa4d72d2d33cdf85e20df26faa3a3a20edfa01e0a7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c96e6ae5-f365-44e5-9a2e-53c632a9fe38", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4c26def1a147344a116bae0584e005976bdcad661f0217404a88662a578060f0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d30d367d-84c4-434c-bbc4-18a174e4f2ae", "node_type": "1", "metadata": {}, "hash": "b31bb4a6fc5a48e1591deb96ac6db9d49beb87daf6417a8ea1bb02b6065157ab", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12293, "end_char_idx": 15261, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
## Change the below based on model. The below is for the latest embeddings models from OpenAI, so you can leave as is unless you are using a different embedding model..
EMBEDDING_CTX_LENGTH = 8191
EMBEDDING_ENCODING='cl100k_base'
```


```python
def generate_embeddings(text, model):
    # Generate embeddings for the provided text using the specified model
    embeddings_response = openai_client.embeddings.create(model=model, input=text)
    # Extract the embedding data from the response
    embedding = embeddings_response.data[0].embedding
    return embedding

def len_safe_get_embedding(text, model=embeddings_model, max_tokens=EMBEDDING_CTX_LENGTH, encoding_name=EMBEDDING_ENCODING):
    # Initialize lists to store embeddings and corresponding text chunks
    chunk_embeddings = []
    chunk_texts = []
    # Iterate over chunks of tokens from the input text
    for chunk in chunked_tokens(text, chunk_length=max_tokens, encoding_name=encoding_name):
        # Generate embeddings for each chunk and append to the list
        chunk_embeddings.append(generate_embeddings(chunk, model=model))
        # Decode the chunk back to text and append to the list
        chunk_texts.append(tiktoken.get_encoding(encoding_name).decode(chunk))
    # Return the list of chunk embeddings and the corresponding text chunks
    return chunk_embeddings, chunk_texts
```

Next, we can define a helper function that will capture additional metadata about the documents. This is useful to use as a metadata filter for search queries, and capturing richer data for search. 

In this example, I'll choose from a list of categories to use later on in a metadata filter.


```python
## These are the categories I will be using for the categorization task. You can change these as needed based on your use case.
categories = ['authentication','models','techniques','tools','setup','billing_limits','other']

def categorize_text(text, categories):
    # Create a prompt for categorization
    messages = [
        {"role": "system", "content": f"""You are an expert in LLMs, and you will be given text that corresponds to an article in OpenAI's documentation.
         Categorize the document into one of these categories: {', '.join(categories)}. Only respond with the category name and nothing else."""},
        {"role": "user", "content": text}
    ]
    try:
        # Call the OpenAI API to categorize the text
        response = openai_client.chat.completions.create(
            model="gpt-4o",
            messages=messages
        )
        # Extract the category from the response
        category = response.choices[0].message.content
        return category
    except Exception as e:
        print(f"Error categorizing text: {str(e)}")
        return None
```

Now, we can define some helper functions to process the .txt files in the oai_docs folder within the data folder. You can use this with your own data as well and supports both .txt and .pdf files.


key: _node_content
string_value:
{"id_": "d30d367d-84c4-434c-bbc4-18a174e4f2ae", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b0987757-cb7f-4604-90a1-17aede6900d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9ea262c5cfe1d5088303e0fa4d72d2d33cdf85e20df26faa3a3a20edfa01e0a7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "dbace265-84df-48c7-8e60-49ab714fa678", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "4674f112dbc0736065b247c4a3ed9be1100d60aac532b80770941de18f27eead", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "29efb7d2-e435-40ca-86b2-dd29a9f4964f", "node_type": "1", "metadata": {}, "hash": "5ab9ebcfaaf610da64bfd282c681175b718da1a6aab53fad94b53ae79559cad4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 13965, "end_char_idx": 17109, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
## These are the categories I will be using for the categorization task. You can change these as needed based on your use case.
categories = ['authentication','models','techniques','tools','setup','billing_limits','other']

def categorize_text(text, categories):
    # Create a prompt for categorization
    messages = [
        {"role": "system", "content": f"""You are an expert in LLMs, and you will be given text that corresponds to an article in OpenAI's documentation.
         Categorize the document into one of these categories: {', '.join(categories)}. Only respond with the category name and nothing else."""},
        {"role": "user", "content": text}
    ]
    try:
        # Call the OpenAI API to categorize the text
        response = openai_client.chat.completions.create(
            model="gpt-4o",
            messages=messages
        )
        # Extract the category from the response
        category = response.choices[0].message.content
        return category
    except Exception as e:
        print(f"Error categorizing text: {str(e)}")
        return None
```

Now, we can define some helper functions to process the .txt files in the oai_docs folder within the data folder. You can use this with your own data as well and supports both .txt and .pdf files.


```python
def extract_text_from_pdf(pdf_path):
    # Initialize the PDF reader
    reader = PdfReader(pdf_path)
    text = ""
    # Iterate through each page in the PDF and extract text
    for page in reader.pages:
        text += page.extract_text()
    return text

def process_file(file_path, idx, categories, embeddings_model):
    file_name = os.path.basename(file_path)
    print(f"Processing file {idx + 1}: {file_name}")
    
    # Read text content from .txt files
    if file_name.endswith('.txt'):
        with open(file_path, 'r', encoding='utf-8') as file:
            text = file.read()
    # Extract text content from .pdf files
    elif file_name.endswith('.pdf'):
        text = extract_text_from_pdf(file_path)
    
    title = file_name
    # Generate embeddings for the title
    title_vectors, title_text = len_safe_get_embedding(title, embeddings_model)
    print(f"Generated title embeddings for {file_name}")
    
    # Generate embeddings for the content
    content_vectors, content_text = len_safe_get_embedding(text, embeddings_model)
    print(f"Generated content embeddings for {file_name}")
    
    category = categorize_text(' '.join(content_text), categories)
    print(f"Categorized {file_name} as {category}")
    
    # Prepare the data to be appended
    data = []
    for i, content_vector in enumerate(content_vectors):
        data.append({
            "id": f"{idx}_{i}",
            "vector_id": f"{idx}_{i}",
            "title": title_text[0],
            "text": content_text[i],
            "title_vector": json.dumps(title_vectors[0]),  # Assuming title is short and has only one chunk
            "content_vector": json.dumps(content_vector),
            "category": category
        })
        print(f"Appended data for chunk {i + 1}/{len(content_vectors)} of {file_name}")
    
    return data


key: _node_content
string_value:
{"id_": "29efb7d2-e435-40ca-86b2-dd29a9f4964f", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b0987757-cb7f-4604-90a1-17aede6900d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9ea262c5cfe1d5088303e0fa4d72d2d33cdf85e20df26faa3a3a20edfa01e0a7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d30d367d-84c4-434c-bbc4-18a174e4f2ae", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "bed8e7562514244dcd97355e8100012f30c569f1abb05848d34f3367570760c5", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b684d357-584e-456e-9423-96425f6711ba", "node_type": "1", "metadata": {}, "hash": "e7146340321845649d3b045caf59e98d38150892b0d9f6f0f95d8b51b08d50a2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 17112, "end_char_idx": 20019, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```

We'll now use this helper function to process our OpenAI documentation. Feel free to update this to use your own data by changing the folder in `process_files` below.

Note that this will process the documents in chosen folder concurrently, so this should take <30 seconds if using txt files, and slightly longer if using PDFs.


```python
## Customize the location below if you are using different data besides the OpenAI documentation. Note that if you are using a different dataset, you will need to update the categories list as well.
folder_name = "../../../data/oai_docs"

files = [os.path.join(folder_name, f) for f in os.listdir(folder_name) if f.endswith('.txt') or f.endswith('.pdf')]
data = []

# Process each file concurrently
with concurrent.futures.ThreadPoolExecutor() as executor:
    futures = {executor.submit(process_file, file_path, idx, categories, embeddings_model): idx for idx, file_path in enumerate(files)}
    for future in concurrent.futures.as_completed(futures):
        try:
            result = future.result()
            data.extend(result)
        except Exception as e:
            print(f"Error processing file: {str(e)}")

# Write the data to a CSV file
csv_file = os.path.join("..", "embedded_data.csv")
with open(csv_file, 'w', newline='', encoding='utf-8') as csvfile:
    fieldnames = ["id", "vector_id", "title", "text", "title_vector", "content_vector","category"]
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    writer.writeheader()
    for row in data:
        writer.writerow(row)
        print(f"Wrote row with id {row['id']} to CSV")

# Convert the CSV file to a Dataframe
article_df = pd.read_csv("../embedded_data.csv")
# Read vectors from strings back into a list using json.loads
article_df["title_vector"] = article_df.title_vector.apply(json.loads)
article_df["content_vector"] = article_df.content_vector.apply(json.loads)
article_df["vector_id"] = article_df["vector_id"].apply(str)
article_df["category"] = article_df["category"].apply(str)
article_df.head()

```

We now have an `embedded_data.csv` file with six columns that we can upload to our vector database! 

# Create Azure AI Vector Search

##  Create index
We'll define and create a search index using the `SearchIndexClient` from the Azure AI Search Python SDK. The index incorporates both vector search and hybrid search capabilities. For more details, visit Microsoft's documentation on how to [Create a Vector Index](https://learn.microsoft.com/azure/search/vector-search-how-to-create-index?.tabs=config-2023-11-01%2Crest-2023-11-01%2Cpush%2Cportal-check-index)


```python
index_name = "azure-ai-search-openai-cookbook-demo"
# index_name = "<insert_name_for_index>"

index_client = SearchIndexClient(
    endpoint=search_service_endpoint, credential=AzureKeyCredential(search_service_api_key)
)
# Define the fields for the index. Update these based on your data.


key: _node_content
string_value:
{"id_": "b684d357-584e-456e-9423-96425f6711ba", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b0987757-cb7f-4604-90a1-17aede6900d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9ea262c5cfe1d5088303e0fa4d72d2d33cdf85e20df26faa3a3a20edfa01e0a7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "29efb7d2-e435-40ca-86b2-dd29a9f4964f", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "3032ea66e4af260e1262a0e1b1a616ff310ac6644fa2177f863f1debe5b34949", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "563cc870-4bda-4f9f-9ebd-dfa9f1606ad0", "node_type": "1", "metadata": {}, "hash": "09095bd7aa3b65d85450acf1978e92b20e1e3294808891f48d59b508d7cf2d89", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19720, "end_char_idx": 23108, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
index_name = "azure-ai-search-openai-cookbook-demo"
# index_name = "<insert_name_for_index>"

index_client = SearchIndexClient(
    endpoint=search_service_endpoint, credential=AzureKeyCredential(search_service_api_key)
)
# Define the fields for the index. Update these based on your data.
# Each field represents a column in the search index
fields = [
    SimpleField(name="id", type=SearchFieldDataType.String),  # Simple string field for document ID
    SimpleField(name="vector_id", type=SearchFieldDataType.String, key=True),  # Key field for the index
    # SimpleField(name="url", type=SearchFieldDataType.String),  # URL field (commented out)
    SearchableField(name="title", type=SearchFieldDataType.String),  # Searchable field for document title
    SearchableField(name="text", type=SearchFieldDataType.String),  # Searchable field for document text
    SearchField(
        name="title_vector",
        type=SearchFieldDataType.Collection(SearchFieldDataType.Single),  # Collection of single values for title vector
        vector_search_dimensions=1536,  # Number of dimensions in the vector
        vector_search_profile_name="my-vector-config",  # Profile name for vector search configuration
    ),
    SearchField(
        name="content_vector",
        type=SearchFieldDataType.Collection(SearchFieldDataType.Single),  # Collection of single values for content vector
        vector_search_dimensions=1536,  # Number of dimensions in the vector
        vector_search_profile_name="my-vector-config",  # Profile name for vector search configuration
    ),
    SearchableField(name="category", type=SearchFieldDataType.String, filterable=True),  # Searchable field for document category
]

# This configuration defines the algorithm and parameters for vector search
vector_search = VectorSearch(
    algorithms=[
        HnswAlgorithmConfiguration(
            name="my-hnsw",  # Name of the HNSW algorithm configuration
            kind=VectorSearchAlgorithmKind.HNSW,  # Type of algorithm
            parameters=HnswParameters(
                m=4,  # Number of bi-directional links created for every new element
                ef_construction=400,  # Size of the dynamic list for the nearest neighbors during construction
                ef_search=500,  # Size of the dynamic list for the nearest neighbors during search
                metric=VectorSearchAlgorithmMetric.COSINE,  # Distance metric used for the search
            ),
        )
    ],
    profiles=[
        VectorSearchProfile(
            name="my-vector-config",  # Name of the vector search profile
            algorithm_configuration_name="my-hnsw",  # Reference to the algorithm configuration
        )
    ],
)

# Create the search index with the vector search configuration
# This combines all the configurations into a single search index
index = SearchIndex(
    name=index_name,  # Name of the index
    fields=fields,  # Fields defined for the index
    vector_search=vector_search  # Vector search configuration

)

# Create or update the index
# This sends the index definition to the Azure Search service
result = index_client.create_index(index)
print(f"{result.name} created")  # Output the name of the created index
```

## Upload Data

Now we'll upload the articles from above that we've stored in `embedded_data.csv` from a pandas DataFrame to an Azure AI Search index.


key: _node_content
string_value:
{"id_": "563cc870-4bda-4f9f-9ebd-dfa9f1606ad0", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b0987757-cb7f-4604-90a1-17aede6900d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9ea262c5cfe1d5088303e0fa4d72d2d33cdf85e20df26faa3a3a20edfa01e0a7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b684d357-584e-456e-9423-96425f6711ba", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "d0959856d8e889ffea7877ca07c26b36dba43a2cb0cf0fe5b24953bc45f21b0c", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5d8a9e2d-819e-4499-ab62-50a6d12ba09e", "node_type": "1", "metadata": {}, "hash": "e139fca35c98549565441e2770a6762598fb370601f11d77955b961d92712860", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 23109, "end_char_idx": 25723, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
For a detailed guide on data import strategies and best practices, refer to  [Data Import in Azure AI Search](https://learn.microsoft.com/azure/search/search-what-is-data-import).



```python
# Convert the 'id' and 'vector_id' columns to string so one of them can serve as our key field
article_df["id"] = article_df["id"].astype(str)
article_df["vector_id"] = article_df["vector_id"].astype(str)

# Convert the DataFrame to a list of dictionaries
documents = article_df.to_dict(orient="records")

# Log the number of documents to be uploaded
print(f"Number of documents to upload: {len(documents)}")

# Create a SearchIndexingBufferedSender
batch_client = SearchIndexingBufferedSender(
    search_service_endpoint, index_name, AzureKeyCredential(search_service_api_key)
)
# Get the first document to check its schema
first_document = documents[0]

# Get the index schema
index_schema = index_client.get_index(index_name)

# Get the field names from the index schema
index_fields = {field.name: field.type for field in index_schema.fields}

# Check each field in the first document
for field, value in first_document.items():
    if field not in index_fields:
        print(f"Field '{field}' is not in the index schema.")

# Check for any fields in the index schema that are not in the documents
for field in index_fields:
    if field not in first_document:
        print(f"Field '{field}' is in the index schema but not in the documents.")

try:
    if documents:
        # Add upload actions for all documents in a single call
        upload_result = batch_client.upload_documents(documents=documents)

        # Check if the upload was successful
        # Manually flush to send any remaining documents in the buffer
        batch_client.flush()
        
        print(f"Uploaded {len(documents)} documents in total")
    else:
        print("No documents to upload.")
except HttpResponseError as e:
    print(f"An error occurred: {e}")
    raise  # Re-raise the exception to ensure it errors out
finally:
    # Clean up resources
    batch_client.close()
```

## Test search
Now that the data is uploaded, we'll test both vector similarity search and hybrid search locally below to make sure it is working as expected.

You can test both a pure vector search and hybrid search. Pure vector search passes in `None` to the `search_text` below and will only search on vector similarity. Hybrid search will combines the capabilities of traditional keyword-based search by passing in the query text `query` to the `search_text` with vector-based similarity search to provide more relevant and contextual results.


key: _node_content
string_value:
{"id_": "5d8a9e2d-819e-4499-ab62-50a6d12ba09e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b0987757-cb7f-4604-90a1-17aede6900d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9ea262c5cfe1d5088303e0fa4d72d2d33cdf85e20df26faa3a3a20edfa01e0a7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "563cc870-4bda-4f9f-9ebd-dfa9f1606ad0", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "70f6a25494fed03c0bbe225a2111886e8113816649c32418fc925a6da3941cbc", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "86c8eaf4-1f53-46ef-bdad-075a2c4f25b2", "node_type": "1", "metadata": {}, "hash": "bb4beb9b584c369cccb62e2d23a84ebf72500450ed5c571d1a03c4f1ad43c0aa", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 25727, "end_char_idx": 27870, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
query = "What model should I use to embed?"
# Note: we'll have the GPT choose the category automatically once we put it in ChatGPT
category ="models"

search_client = SearchClient(search_service_endpoint, index_name, AzureKeyCredential(search_service_api_key))
vector_query = VectorizedQuery(vector=generate_embeddings(query, embeddings_model), k_nearest_neighbors=3, fields="content_vector")
  
results = search_client.search(  
    search_text=None, # Pass in None if you want to use pure vector search, and `query` if you want to use hybrid search
    vector_queries= [vector_query], 
    select=["title", "text"],
    filter=f"category eq '{category}'" 
)

for result in results:  
    print(result)

```

## Create Azure Function

Azure Functions are an easy way to build an API on top of our new AI search. Our code (see the `function_app.py` file in this folder, or linked [here](https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/function_app.py)) does the following:

1. Takes in an input of the user's query, search index endpoint, the index name, the k_nearest_neighbors*, the search column to use (either content_vector or title_vector), and whether it should use a hybrid query
2. Takes the user's query and embeds it.
3. Conducts a vector search and retrieves relevant text chunks.
4. Returns those relevant text chunks as the response body. 

*In the context of vector search, k_nearest_neighbors specifies the number of "closest" vectors (in terms of cosine similarity) that the search should return. For example, if k_nearest_neighbors is set to 3, the search will return the 3 vectors in the index that are most similar to the query vector.

> Note that this Azure Function _does not have any authentication_. However, you can set authentication on it following docs [here](https://learn.microsoft.com/en-us/azure/azure-functions/security-concepts?tabs=v4)

### Create storage account

We can create a new storage account using the code below, but feel free to skip that block and modify the subsequent steps to use an existing storage account. This may take up to 30 seconds.


key: _node_content
string_value:
{"id_": "86c8eaf4-1f53-46ef-bdad-075a2c4f25b2", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b0987757-cb7f-4604-90a1-17aede6900d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9ea262c5cfe1d5088303e0fa4d72d2d33cdf85e20df26faa3a3a20edfa01e0a7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5d8a9e2d-819e-4499-ab62-50a6d12ba09e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "d86410606f7e1d5391dc53d600bf49fcf18adccd138880158388d080885773b1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "5fea35b9-32e2-40fc-bb21-3fc8d7e0b453", "node_type": "1", "metadata": {}, "hash": "d39ef8b1882dbf951f971da693677e563d83c5a644cd2f797b5123549bf259f9", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27873, "end_char_idx": 29766, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
## Update below with a different name
storage_account_name = "<enter-storage-account-name>"

## Use below SKU or any other SKU as per your requirement
sku = "Standard_LRS"
resource_client = ResourceManagementClient(credential, subscription_id)
storage_client = StorageManagementClient(credential, subscription_id)

# Create resource group if it doesn't exist
rg_result = resource_client.resource_groups.create_or_update(resource_group, {"location": region})

# Create storage account
storage_async_operation = storage_client.storage_accounts.begin_create(
    resource_group,
    storage_account_name,
    {
        "sku": {"name": sku},
        "kind": "StorageV2",
        "location": region,
    },
)
storage_account = storage_async_operation.result()

print(f"Storage account {storage_account.name} created")

```

### Create Function App
This Function App is where the python code will execute once it is triggered via a GPT Action. To read more about Function Apps, see the docs [here](https://learn.microsoft.com/en-us/azure/azure-functions/functions-overview?pivots=programming-language-csharp). 

To deploy Function Apps, we'll need to use the Azure CLI and Azure Functions Core Tools. 

> The below will attempt to install it and run it based on your platform type in your virtual environment, but if that does not work, read the Azure documentation to figure out how to install [Azure Function Core Tools](https://learn.microsoft.com/en-us/azure/azure-functions/create-first-function-cli-python?tabs=linux,bash,azure-cli,browser) and [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). After doing that, run the below `subprocess.run` commands in your terminal after navigating to this folder.

First we'll make sure we have the relevant tools in the environment in order to run the Azure commands necessary. This may take a few minutes to install.


key: _node_content
string_value:
{"id_": "5fea35b9-32e2-40fc-bb21-3fc8d7e0b453", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b0987757-cb7f-4604-90a1-17aede6900d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9ea262c5cfe1d5088303e0fa4d72d2d33cdf85e20df26faa3a3a20edfa01e0a7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "86c8eaf4-1f53-46ef-bdad-075a2c4f25b2", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "2a7551b5bb2b30a73d32f32c5c6e3e95e50a59fda34880384bb2873607203643", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "ba3ae13b-caea-4268-a761-e863f7bc923e", "node_type": "1", "metadata": {}, "hash": "cce1f9e13ec7f459778f6b35924847e19ac9e435703b7c3a2f1992ce004177f4", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 27873, "end_char_idx": 32120, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
## Update below with a different name
storage_account_name = "<enter-storage-account-name>"

## Use below SKU or any other SKU as per your requirement
sku = "Standard_LRS"
resource_client = ResourceManagementClient(credential, subscription_id)
storage_client = StorageManagementClient(credential, subscription_id)

# Create resource group if it doesn't exist
rg_result = resource_client.resource_groups.create_or_update(resource_group, {"location": region})

# Create storage account
storage_async_operation = storage_client.storage_accounts.begin_create(
    resource_group,
    storage_account_name,
    {
        "sku": {"name": sku},
        "kind": "StorageV2",
        "location": region,
    },
)
storage_account = storage_async_operation.result()

print(f"Storage account {storage_account.name} created")

```

### Create Function App
This Function App is where the python code will execute once it is triggered via a GPT Action. To read more about Function Apps, see the docs [here](https://learn.microsoft.com/en-us/azure/azure-functions/functions-overview?pivots=programming-language-csharp). 

To deploy Function Apps, we'll need to use the Azure CLI and Azure Functions Core Tools. 

> The below will attempt to install it and run it based on your platform type in your virtual environment, but if that does not work, read the Azure documentation to figure out how to install [Azure Function Core Tools](https://learn.microsoft.com/en-us/azure/azure-functions/create-first-function-cli-python?tabs=linux,bash,azure-cli,browser) and [Azure CLI](https://learn.microsoft.com/en-us/cli/azure/install-azure-cli). After doing that, run the below `subprocess.run` commands in your terminal after navigating to this folder.

First we'll make sure we have the relevant tools in the environment in order to run the Azure commands necessary. This may take a few minutes to install.


```python
os_type = platform.system()

if os_type == "Windows":
    # Install Azure Functions Core Tools on Windows
    subprocess.run(["npm", "install", "-g", "azure-functions-core-tools@3", "--unsafe-perm", "true"], check=True)
    # Install Azure CLI on Windows
    subprocess.run(["powershell", "-Command", "Invoke-WebRequest -Uri https://aka.ms/installazurecliwindows -OutFile .\\AzureCLI.msi; Start-Process msiexec.exe -ArgumentList '/I AzureCLI.msi /quiet' -Wait"], check=True)
elif os_type == "Darwin":  # MacOS
    # Install Azure Functions Core Tools on MacOS
    if platform.machine() == 'arm64':
        # For M1 Macs
        subprocess.run(["arch", "-arm64", "brew", "install", "azure-functions-core-tools@3"], check=True)
    else:
        # For Intel Macs
        subprocess.run(["brew", "install", "azure-functions-core-tools@3"], check=True)
    # Install Azure CLI on MacOS
    subprocess.run(["brew", "update"], check=True)
    subprocess.run(["brew", "install", "azure-cli"], check=True)
elif os_type == "Linux":
    # Install Azure Functions Core Tools on Linux
    subprocess.run(["curl", "https://packages.microsoft.com/keys/microsoft.asc", "|", "gpg", "--dearmor", ">", "microsoft.gpg"], check=True, shell=True)
    subprocess.run(["sudo", "mv", "microsoft.gpg", "/etc/apt/trusted.gpg.d/microsoft.gpg"], check=True)
    subprocess.run(["sudo", "sh", "-c", "'echo \"deb [arch=amd64] https://packages.microsoft.com/repos/microsoft-ubuntu-$(lsb_release -cs)-prod $(lsb_release -cs) main\" > /etc/apt/sources.list.d/dotnetdev.list'"], check=True, shell=True)
    subprocess.run(["sudo", "apt-get", "update"], check=True)
    subprocess.run(["sudo", "apt-get", "install", "azure-functions-core-tools-3"], check=True)
    # Install Azure CLI on Linux
    subprocess.run(["curl", "-sL", "https://aka.ms/InstallAzureCLIDeb", "|", "sudo", "bash"], check=True, shell=True)
else:
    # Raise an error if the operating system is not supported
    raise OSError("Unsupported operating system")

# Verify the installation of Azure Functions Core Tools
subprocess.run(["func", "--version"], check=True)
# Verify the installation of Azure CLI
subprocess.run(["az", "--version"], check=True)

subprocess.run([
    "az", "login"
], check=True)
```

Now, we need to create a `local.settings.json` file with our key environment variables for Azure


key: _node_content
string_value:
{"id_": "ba3ae13b-caea-4268-a761-e863f7bc923e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b0987757-cb7f-4604-90a1-17aede6900d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9ea262c5cfe1d5088303e0fa4d72d2d33cdf85e20df26faa3a3a20edfa01e0a7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "5fea35b9-32e2-40fc-bb21-3fc8d7e0b453", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "12aa7973ee8daa149e24b7add66fe72a0cf5e20987326182ee43cddf2239c85f", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "24409061-b68d-4c16-b3d6-e00b093c011a", "node_type": "1", "metadata": {}, "hash": "f8e113bb04626d5e684065345efc6974796b1f58d61f345249e97a92ff125c70", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 32123, "end_char_idx": 34773, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
local_settings_content = f"""
{{
  "IsEncrypted": false,
  "Values": {{
    "AzureWebJobsStorage": "UseDevelopmentStorage=true",
    "FUNCTIONS_WORKER_RUNTIME": "python",
    "OPENAI_API_KEY": "{openai_api_key}",
    "EMBEDDINGS_MODEL": "{embeddings_model}",
    "SEARCH_SERVICE_API_KEY": "{search_service_api_key}",
  }}
}}
"""

with open("local.settings.json", "w") as file:
    file.write(local_settings_content)
```

Check the `local.settings.json` file and make sure that the environment variables match what you expect. 

Now, give your app a name below, and you are ready to create your Function App and then publish your function. 


```python
# Replace this with your own values. This name will appear in the URL of the API call https://<app_name>.azurewebsites.net
app_name = "<app-name>"

subprocess.run([
    "az", "functionapp", "create",
    "--resource-group", resource_group,
    "--consumption-plan-location", region,
    "--runtime", "python",
    "--name", app_name,
    "--storage-account", storage_account_name,
    "--os-type", "Linux",
], check=True)
```

Once we've created the Function App, we now want to add the configuration variables to the function app to use in the function. Specifically, we need the `OPENAI_API_KEY`, the `SEARCH_SERVICE_API_KEY`, and the `EMBEDDINGS_MODEL` as these are all used in the `function_app.py` code.


```python
# Collect the relevant environment variables 
env_vars = {
    "OPENAI_API_KEY": openai_api_key,
    "SEARCH_SERVICE_API_KEY": search_service_api_key,
    "EMBEDDINGS_MODEL": embeddings_model
}

# Create the settings argument for the az functionapp create command
settings_args = []
for key, value in env_vars.items():
    settings_args.append(f"{key}={value}")

subprocess.run([
    "az", "functionapp", "config", "appsettings", "set",
    "--name", app_name,
    "--resource-group", resource_group,
    "--settings", *settings_args
], check=True)
```

We are now ready to publish your function code `function_app.py` to the Azure Function. This may take up to 10 minutes to deploy. Once this is finished, we now have an API endpoint using an Azure Function on top of Azure AI Search.


```python
subprocess.run([
    "func", "azure", "functionapp", "publish", app_name
], check=True)
```

## Input in a Custom GPT in ChatGPT
Now that we have an Azure Function that queries this Vector Search Index, let's put it as a GPT Action!

See documentation [here](https://openai.com/index/introducing-gpts/) on GPTs and [here](https://platform.openai.com/docs/actions) on GPT Actions. Use the below as the instructions for the GPT and as the OpenAPI spec for the GPT Action.


key: _node_content
string_value:
{"id_": "24409061-b68d-4c16-b3d6-e00b093c011a", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b0987757-cb7f-4604-90a1-17aede6900d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9ea262c5cfe1d5088303e0fa4d72d2d33cdf85e20df26faa3a3a20edfa01e0a7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "ba3ae13b-caea-4268-a761-e863f7bc923e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "5d97d29d4cbddc89f4cb9ad10573cac28991b67d652be83ac6e7b83276c16f49", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "c1e36bba-b819-457a-9f6b-78097156f93e", "node_type": "1", "metadata": {}, "hash": "db47f4fc4659cd63107b42684fcbba220fb24903d26e9ef7289139908389dbc0", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 33496, "end_char_idx": 35367, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Collect the relevant environment variables 
env_vars = {
    "OPENAI_API_KEY": openai_api_key,
    "SEARCH_SERVICE_API_KEY": search_service_api_key,
    "EMBEDDINGS_MODEL": embeddings_model
}

# Create the settings argument for the az functionapp create command
settings_args = []
for key, value in env_vars.items():
    settings_args.append(f"{key}={value}")

subprocess.run([
    "az", "functionapp", "config", "appsettings", "set",
    "--name", app_name,
    "--resource-group", resource_group,
    "--settings", *settings_args
], check=True)
```

We are now ready to publish your function code `function_app.py` to the Azure Function. This may take up to 10 minutes to deploy. Once this is finished, we now have an API endpoint using an Azure Function on top of Azure AI Search.


```python
subprocess.run([
    "func", "azure", "functionapp", "publish", app_name
], check=True)
```

## Input in a Custom GPT in ChatGPT
Now that we have an Azure Function that queries this Vector Search Index, let's put it as a GPT Action!

See documentation [here](https://openai.com/index/introducing-gpts/) on GPTs and [here](https://platform.openai.com/docs/actions) on GPT Actions. Use the below as the instructions for the GPT and as the OpenAPI spec for the GPT Action.


### Create OpenAPI Spec
Below is a sample OpenAPI spec. When we run the block below, a functional spec should be copied to the clipboard to paste in the GPT Action.

Note that this does not have any authentication by default, but you can set up Azure Functions with OAuth by following the pattern in [this cookbook](https://cookbook.openai.com/examples/chatgpt/gpt_actions_library/gpt_middleware_azure_function#part-2-set-up-auth) in the Authentication section or looking at the documentation [here](https://learn.microsoft.com/en-us/azure/app-service/overview-authentication-authorization).


key: _node_content
string_value:
{"id_": "c1e36bba-b819-457a-9f6b-78097156f93e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b0987757-cb7f-4604-90a1-17aede6900d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9ea262c5cfe1d5088303e0fa4d72d2d33cdf85e20df26faa3a3a20edfa01e0a7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "24409061-b68d-4c16-b3d6-e00b093c011a", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "c93a11e6315b8a63a3eec04d827520891fb3cd3dc2fd235cc377f753922519a6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "071cd804-ffd2-4e98-9191-e0ab4584a5e7", "node_type": "1", "metadata": {}, "hash": "656140e72a3e6217c4006e5160f1f47f0387be5d092554f2077a7aeeb402c574", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 34294, "end_char_idx": 38270, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
subprocess.run([
    "func", "azure", "functionapp", "publish", app_name
], check=True)
```

## Input in a Custom GPT in ChatGPT
Now that we have an Azure Function that queries this Vector Search Index, let's put it as a GPT Action!

See documentation [here](https://openai.com/index/introducing-gpts/) on GPTs and [here](https://platform.openai.com/docs/actions) on GPT Actions. Use the below as the instructions for the GPT and as the OpenAPI spec for the GPT Action.


### Create OpenAPI Spec
Below is a sample OpenAPI spec. When we run the block below, a functional spec should be copied to the clipboard to paste in the GPT Action.

Note that this does not have any authentication by default, but you can set up Azure Functions with OAuth by following the pattern in [this cookbook](https://cookbook.openai.com/examples/chatgpt/gpt_actions_library/gpt_middleware_azure_function#part-2-set-up-auth) in the Authentication section or looking at the documentation [here](https://learn.microsoft.com/en-us/azure/app-service/overview-authentication-authorization).


```python

spec = f"""
openapi: 3.1.0
info:
  title: Vector Similarity Search API
  description: API for performing vector similarity search.
  version: 1.0.0
servers:
  - url: https://{app_name}.azurewebsites.net/api
    description: Main (production) server
paths:
  /vector_similarity_search:
    post:
      operationId: vectorSimilaritySearch
      summary: Perform a vector similarity search.
      requestBody:
        required: true
        content:
          application/json:
            schema:
              type: object
              properties:
                search_service_endpoint:
                  type: string
                  description: The endpoint of the search service.
                index_name:
                  type: string
                  description: The name of the search index.
                query:
                  type: string
                  description: The search query.
                k_nearest_neighbors:
                  type: integer
                  description: The number of nearest neighbors to return.
                search_column:
                  type: string
                  description: The name of the search column.
                use_hybrid_query:
                  type: boolean
                  description: Whether to use a hybrid query.
                category:
                  type: string
                  description: category to filter.
              required:
                - search_service_endpoint
                - index_name
                - query
                - k_nearest_neighbors
                - search_column
                - use_hybrid_query
      responses:
        '200':
          description: A successful response with the search results.
          content:
            application/json:
              schema:
                type: object
                properties:
                  results:
                    type: array
                    items:
                      type: object
                      properties:
                        id:
                          type: string
                          description: The identifier of the result item.
                        score:
                          type: number
                          description: The similarity score of the result item.
                        content:
                          type: object
                          description: The content of the result item.
        '400':
          description: Bad request due to missing or invalid parameters.
        '500':
          description: Internal server error.
"""
pyperclip.copy(spec)
print("OpenAPI spec copied to clipboard")
print(spec)
```

### Create GPT Instructions

Feel free to modify instructions as you see fit. Check out our docs [here](https://platform.openai.com/docs/guides/prompt-engineering) for some tips on prompt engineering.


key: _node_content
string_value:
{"id_": "071cd804-ffd2-4e98-9191-e0ab4584a5e7", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "b0987757-cb7f-4604-90a1-17aede6900d7", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "9ea262c5cfe1d5088303e0fa4d72d2d33cdf85e20df26faa3a3a20edfa01e0a7", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "c1e36bba-b819-457a-9f6b-78097156f93e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "openai_cookbooks"}, "hash": "33116bfd0af99e5de1443a758a47a8bfb2542d8dd545598eb9e9842b190608a2", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 38273, "end_char_idx": 39941, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
instructions = f'''
You are an OAI docs assistant. You have an action in your knowledge base where you can make a POST request to search for information. The POST request should always include: {{
    "search_service_endpoint": "{search_service_endpoint}",
    "index_name": {index_name},
    "query": "<user_query>",
    "k_nearest_neighbors": 1,
    "search_column": "content_vector",
    "use_hybrid_query": true,
    "category": "<category>"
}}. Only the query and category change based on the user's request. Your goal is to assist users by performing searches using this POST request and providing them with relevant information based on the query.

You must only include knowledge you get from your action in your response.
The category must be from the following list: {categories}, which you should determine based on the user's query. If you cannot determine, then do not include the category in the POST request.
'''
pyperclip.copy(instructions)
print("GPT Instructions copied to clipboard")
print(instructions)
```

We now have a GPT that queries a vector database! 

# Recap
We've now successfully integrated Azure AI Search with GPT Actions in ChatGPT by doing the following:
1. embedded them using OpenAI's embeddings, while adding some additional metadata using gpt-4o.
2. uploaded that data to Azure AI Search.
3. created an endpoint to query it using Azure Functions.
4. incorporated it into a Custom GPT. 

Our GPT can now retrieve information to help answer user queries, making it much more accurate and customized to our data. Here's the GPT in action:

# ![azure-rag-quickstart-gpt.png](../../../../images/azure-rag-quickstart-gpt.png)


key: url
string_value:
https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb


key: _node_content
string_value:
{"id_": "d9d35db3-0fa0-4167-b92e-c040a878ac59", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6d8bb99b-2027-4815-90bd-14a51cfd3979", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f33c7ece243df97d2c659b7677c9ac7a517784feee18f15772f79b17a9198eda", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "2e8300b6-1321-41f3-a981-1ab111468643", "node_type": "1", "metadata": {}, "hash": "c5bafd3bc8794b4db019664a796c0d8b29846910e4b65edfbdcb2924a6a95fdd", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 0, "end_char_idx": 3391, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
# GCP Bigquery with GCP Functions and GPT actions in ChatGPT

This notebook provides step-by-step instructions on using Google Cloud BigQuery as a database with vector search capabilities, with OpenAI embeddings, then creating a Google Cloud Function on top to plug into a Custom GPT in ChatGPT.

This can be a solution for customers looking to set up RAG infrastructure contained within Google Cloud Platform (GCP), and exposing it as an endpoint to integrate that with other platforms such as ChatGPT.

Google Cloud BigQuery is a fully-managed, serverless data warehouse that enables super-fast SQL queries using the processing power of Google's infrastructure. It allows developers to store and analyze massive datasets with ease.

Google Cloud Functions is a lightweight, event-based, asynchronous compute solution that allows you to create small, single-purpose functions that respond to cloud events without managing servers or runtime environments.

## Pre-requisites:

To run this cookbook, you must have:
- A GCP project you have access to
- GCP user with permission to create a BigQuery dataset and Google Cloud Function
- [GCP CLI](https://cloud.google.com/sdk/docs/downloads-interactive) installed and connected
- OpenAI API key
- ChatGPT Plus, Teams or Enterprise subscription


## Architecture

Below is a diagram of the architecture of this solution, which we'll walk through step-by-step:

![bigquery-rag-architecture.png](../../../../images/bigquery_rag_architecture.png)

## Table of Contents

1. **[Setup of Environment](#set-up-environment)** Setup environment by installing and importing the required libraries and configuring our GCP settings. Includes:
    - [Install and Import Required Libraries](#install-and-import-required-libraries)
    - [Configure GCP project](#configure-gcp-project)
    - [Configure OpenAI Settings](#configure-openai-settings)


2. **[Prepare Data](#prepare-data)** Prepare the data for uploading by embedding the documents, as well as capturing additional metadata. We will use a subset of OpenAI's docs as example data for this.

3. **[Create BigQuery Table with Vector search](#create-bigquery-table-with-vector-search)**  
Create a BigQuery table and upload the data we've prepared. Includes:

    - [Create Dataset](#create-bigquery-dataset): Steps to create a dataset in BigQuery.
    - [Create Table and upload data](#creating-table-and-upload-data): Instructions to create a table in BigQuery.

4. **[Create GCP Function](#create-gcp-function)** using gcloud CLI and environment variables computed previously

5. **[Input in a Custom GPT in ChatGPT](#input-in-a-custom-gpt-in-chatgpt)** Perform searches on the embedded data in BigQuery:

    - [Vector Search](#test-search): Steps to perform vector-based search queries.
    - [Metadata filtering Search](#perform-search-with-metadata-filtering): Instructions for performing metadata filtering.

# Set up environment

## Install and import required libraries
The below libraries can be categorized as standard Python libraries, third-party libraries, and GCP-related libraries.


```python
! pip install -q google-auth
! pip install -q openai
! pip install -q pandas
! pip install -q google-cloud-functions
! pip install -q python-dotenv
! pip install -q pyperclip
! pip install -q PyPDF2
! pip install -q tiktoken
! pip install -q google-cloud-bigquery
! pip install -q pyyaml
```


key: _node_content
string_value:
{"id_": "2e8300b6-1321-41f3-a981-1ab111468643", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6d8bb99b-2027-4815-90bd-14a51cfd3979", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f33c7ece243df97d2c659b7677c9ac7a517784feee18f15772f79b17a9198eda", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d9d35db3-0fa0-4167-b92e-c040a878ac59", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "d23a421ed82039aafafd0373b97df9bc1aef28acc7eced23d89f50612decb183", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3cbb42a3-e59d-4462-963b-b41181817c9d", "node_type": "1", "metadata": {}, "hash": "4cd030e92db96cce29e305a931a4f97ad05cb06f9e0b0a11e61a52f736143000", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 1880, "end_char_idx": 4717, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
2. **[Prepare Data](#prepare-data)** Prepare the data for uploading by embedding the documents, as well as capturing additional metadata. We will use a subset of OpenAI's docs as example data for this.

3. **[Create BigQuery Table with Vector search](#create-bigquery-table-with-vector-search)**  
Create a BigQuery table and upload the data we've prepared. Includes:

    - [Create Dataset](#create-bigquery-dataset): Steps to create a dataset in BigQuery.
    - [Create Table and upload data](#creating-table-and-upload-data): Instructions to create a table in BigQuery.

4. **[Create GCP Function](#create-gcp-function)** using gcloud CLI and environment variables computed previously

5. **[Input in a Custom GPT in ChatGPT](#input-in-a-custom-gpt-in-chatgpt)** Perform searches on the embedded data in BigQuery:

    - [Vector Search](#test-search): Steps to perform vector-based search queries.
    - [Metadata filtering Search](#perform-search-with-metadata-filtering): Instructions for performing metadata filtering.

# Set up environment

## Install and import required libraries
The below libraries can be categorized as standard Python libraries, third-party libraries, and GCP-related libraries.


```python
! pip install -q google-auth
! pip install -q openai
! pip install -q pandas
! pip install -q google-cloud-functions
! pip install -q python-dotenv
! pip install -q pyperclip
! pip install -q PyPDF2
! pip install -q tiktoken
! pip install -q google-cloud-bigquery
! pip install -q pyyaml
```


```python
# Standard Libraries
import json  
import os
import csv
import shutil
from itertools import islice
import concurrent.futures
import yaml

# Third-Party Libraries
import pandas as pd
import numpy as np
from PyPDF2 import PdfReader
import tiktoken
from dotenv import load_dotenv
import pyperclip

# OpenAI Libraries
from openai import OpenAI

# Google Cloud Identity and Credentials
from google.auth import default
from google.cloud import bigquery
from google.cloud import functions_v1

```

## Configure GCP project

If not already set-up, we'll install GCP CLI's, authenticate to GCP and set your default project.


```python
# Add gcloud to PATH
os.environ['PATH'] += os.pathsep + os.path.expanduser('~/google-cloud-sdk/bin')

# Verify gcloud is in PATH
! gcloud --version
```


```python
! gcloud auth application-default login
```


```python
project_id = "<insert_project_id>"  # Replace with your actual project ID
! gcloud config set project {project_id}
```


```python
! gcloud services enable cloudfunctions.googleapis.com
! gcloud services enable cloudbuild.googleapis.com
! gcloud services enable bigquery.googleapis.com
```

## Configure OpenAI settings

This section guides you through setting up authentication for  OpenAI. Before going through this section, make sure you have your OpenAI API key.


key: _node_content
string_value:
{"id_": "3cbb42a3-e59d-4462-963b-b41181817c9d", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6d8bb99b-2027-4815-90bd-14a51cfd3979", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f33c7ece243df97d2c659b7677c9ac7a517784feee18f15772f79b17a9198eda", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "2e8300b6-1321-41f3-a981-1ab111468643", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "b4cb7839ff84c4a9a8b54b49afeed7a27f917d7b0559e7dc362da34d52363b85", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "11925185-6512-45e5-9dac-e22fe24920ce", "node_type": "1", "metadata": {}, "hash": "6b9295ddbfda570c14f3a67cebd57f417a21591783af65ddafdb6eae88c0066a", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 3394, "end_char_idx": 6320, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Standard Libraries
import json  
import os
import csv
import shutil
from itertools import islice
import concurrent.futures
import yaml

# Third-Party Libraries
import pandas as pd
import numpy as np
from PyPDF2 import PdfReader
import tiktoken
from dotenv import load_dotenv
import pyperclip

# OpenAI Libraries
from openai import OpenAI

# Google Cloud Identity and Credentials
from google.auth import default
from google.cloud import bigquery
from google.cloud import functions_v1

```

## Configure GCP project

If not already set-up, we'll install GCP CLI's, authenticate to GCP and set your default project.


```python
# Add gcloud to PATH
os.environ['PATH'] += os.pathsep + os.path.expanduser('~/google-cloud-sdk/bin')

# Verify gcloud is in PATH
! gcloud --version
```


```python
! gcloud auth application-default login
```


```python
project_id = "<insert_project_id>"  # Replace with your actual project ID
! gcloud config set project {project_id}
```


```python
! gcloud services enable cloudfunctions.googleapis.com
! gcloud services enable cloudbuild.googleapis.com
! gcloud services enable bigquery.googleapis.com
```

## Configure OpenAI settings

This section guides you through setting up authentication for  OpenAI. Before going through this section, make sure you have your OpenAI API key.


```python
openai_api_key = os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as an env var>") # Saving this as a variable to reference in function app in later step
openai_client = OpenAI(api_key=openai_api_key)
embeddings_model = "text-embedding-3-small" # We'll use this by default, but you can change to your text-embedding-3-large if desired
```

## Configure GCP BigQuery with Vector Search capabilities

This section explains how to create a dataset in BigQuery and store vectors of float, used for embeddings & vector search.


```python
from google.auth import default

# Use default credentials
credentials, project_id = default()
region = "us-central1" # e.g: "us-central1"
print("Default Project ID:", project_id)
```

# Prepare data
We're going to embed and store a few pages of the OpenAI docs in the oai_docs folder. We'll first embed each, add it to a CSV, and then use that CSV to upload to the index.

We are going to use some techniques highlighted in [this cookbook](khttps://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb). This is a quick way to embed text, without taking into account variables like sections, using our vision model to describe images/graphs/diagrams, overlapping text between chunks for longer documents, etc. 

In order to handle longer text files beyond the context of 8191 tokens, we can either use the chunk embeddings separately, or combine them in some way, such as averaging (weighted by the size of each chunk).

We will take a function from Python's own cookbook that breaks up a sequence into chunks.


key: _node_content
string_value:
{"id_": "11925185-6512-45e5-9dac-e22fe24920ce", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6d8bb99b-2027-4815-90bd-14a51cfd3979", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f33c7ece243df97d2c659b7677c9ac7a517784feee18f15772f79b17a9198eda", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "3cbb42a3-e59d-4462-963b-b41181817c9d", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "c47eb9144a08411c4cf8c2003542c555afaabf2120b7beb09f7279b05f9c5d7a", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "37526d93-f835-42f6-a51b-ee82291e8711", "node_type": "1", "metadata": {}, "hash": "f3fb08707aa80906a3f571bca14128050c2a2bf69974d9ff55d91f6a77283445", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 4720, "end_char_idx": 6939, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
openai_api_key = os.environ.get("OPENAI_API_KEY", "<your OpenAI API key if not set as an env var>") # Saving this as a variable to reference in function app in later step
openai_client = OpenAI(api_key=openai_api_key)
embeddings_model = "text-embedding-3-small" # We'll use this by default, but you can change to your text-embedding-3-large if desired
```

## Configure GCP BigQuery with Vector Search capabilities

This section explains how to create a dataset in BigQuery and store vectors of float, used for embeddings & vector search.


```python
from google.auth import default

# Use default credentials
credentials, project_id = default()
region = "us-central1" # e.g: "us-central1"
print("Default Project ID:", project_id)
```

# Prepare data
We're going to embed and store a few pages of the OpenAI docs in the oai_docs folder. We'll first embed each, add it to a CSV, and then use that CSV to upload to the index.

We are going to use some techniques highlighted in [this cookbook](khttps://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb). This is a quick way to embed text, without taking into account variables like sections, using our vision model to describe images/graphs/diagrams, overlapping text between chunks for longer documents, etc. 

In order to handle longer text files beyond the context of 8191 tokens, we can either use the chunk embeddings separately, or combine them in some way, such as averaging (weighted by the size of each chunk).

We will take a function from Python's own cookbook that breaks up a sequence into chunks.


```python
def batched(iterable, n):
    """Batch data into tuples of length n. The last batch may be shorter."""
    # batched('ABCDEFG', 3) --> ABC DEF G
    if n < 1:
        raise ValueError('n must be at least one')
    it = iter(iterable)
    while (batch := tuple(islice(it, n))):
        yield batch

```

Now we define a function that encodes a string into tokens and then breaks it up into chunks. We'll use tiktoken, a fast open-source tokenizer by OpenAI.

To read more about counting tokens with Tiktoken, check out [this cookbook](https://cookbook.openai.com/examples/how_to_count_tokens_with_tiktoken).


key: _node_content
string_value:
{"id_": "37526d93-f835-42f6-a51b-ee82291e8711", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6d8bb99b-2027-4815-90bd-14a51cfd3979", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f33c7ece243df97d2c659b7677c9ac7a517784feee18f15772f79b17a9198eda", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "11925185-6512-45e5-9dac-e22fe24920ce", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a0cdcda9e01b5cabb9a7c97a7bebfa59f7004400b1d922351dd04c33fa346b64", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "d7490cf5-1ea1-4762-b2b9-0f0f47d886f4", "node_type": "1", "metadata": {}, "hash": "0bbaedbdefd83c5e44bc6134ff43612eb435216189896806d8ff5d396e83ad2b", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 5271, "end_char_idx": 8333, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
from google.auth import default

# Use default credentials
credentials, project_id = default()
region = "us-central1" # e.g: "us-central1"
print("Default Project ID:", project_id)
```

# Prepare data
We're going to embed and store a few pages of the OpenAI docs in the oai_docs folder. We'll first embed each, add it to a CSV, and then use that CSV to upload to the index.

We are going to use some techniques highlighted in [this cookbook](khttps://github.com/openai/openai-cookbook/blob/main/examples/Embedding_long_inputs.ipynb). This is a quick way to embed text, without taking into account variables like sections, using our vision model to describe images/graphs/diagrams, overlapping text between chunks for longer documents, etc. 

In order to handle longer text files beyond the context of 8191 tokens, we can either use the chunk embeddings separately, or combine them in some way, such as averaging (weighted by the size of each chunk).

We will take a function from Python's own cookbook that breaks up a sequence into chunks.


```python
def batched(iterable, n):
    """Batch data into tuples of length n. The last batch may be shorter."""
    # batched('ABCDEFG', 3) --> ABC DEF G
    if n < 1:
        raise ValueError('n must be at least one')
    it = iter(iterable)
    while (batch := tuple(islice(it, n))):
        yield batch

```

Now we define a function that encodes a string into tokens and then breaks it up into chunks. We'll use tiktoken, a fast open-source tokenizer by OpenAI.

To read more about counting tokens with Tiktoken, check out [this cookbook](https://cookbook.openai.com/examples/how_to_count_tokens_with_tiktoken). 




```python
def chunked_tokens(text, chunk_length, encoding_name='cl100k_base'):
    # Get the encoding object for the specified encoding name. OpenAI's tiktoken library, which is used in this notebook, currently supports two encodings: 'bpe' and 'cl100k_base'. The 'bpe' encoding is used for GPT-3 and earlier models, while 'cl100k_base' is used for newer models like GPT-4.
    encoding = tiktoken.get_encoding(encoding_name)
    # Encode the input text into tokens
    tokens = encoding.encode(text)
    # Create an iterator that yields chunks of tokens of the specified length
    chunks_iterator = batched(tokens, chunk_length)
    # Yield each chunk from the iterator
    yield from chunks_iterator
```

Finally, we can write a function that safely handles embedding requests, even when the input text is longer than the maximum context length, by chunking the input tokens and embedding each chunk individually. The average flag can be set to True to return the weighted average of the chunk embeddings, or False to simply return the unmodified list of chunk embeddings.

> Note: there are other techniques you can take here, including:
> - using GPT-4o to capture images/chart descriptions for embedding
> - chunking based on paragraphs or sections
> - adding more descriptive metadata about each article.


```python
EMBEDDING_CTX_LENGTH = 8191
EMBEDDING_ENCODING='cl100k_base'
```


key: _node_content
string_value:
{"id_": "d7490cf5-1ea1-4762-b2b9-0f0f47d886f4", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6d8bb99b-2027-4815-90bd-14a51cfd3979", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f33c7ece243df97d2c659b7677c9ac7a517784feee18f15772f79b17a9198eda", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "37526d93-f835-42f6-a51b-ee82291e8711", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "7e7631cb37ba540368952374d3eea8afd542cc3e6fbe167e8b979bc8546a90f0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "193f08b4-1ce8-4e61-9a4d-312ee78e6778", "node_type": "1", "metadata": {}, "hash": "2b5e1b9f7608212a2ec302776456580097b7d74222eff2d0240c7430e00fa9e6", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 6945, "end_char_idx": 9655, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
def chunked_tokens(text, chunk_length, encoding_name='cl100k_base'):
    # Get the encoding object for the specified encoding name. OpenAI's tiktoken library, which is used in this notebook, currently supports two encodings: 'bpe' and 'cl100k_base'. The 'bpe' encoding is used for GPT-3 and earlier models, while 'cl100k_base' is used for newer models like GPT-4.
    encoding = tiktoken.get_encoding(encoding_name)
    # Encode the input text into tokens
    tokens = encoding.encode(text)
    # Create an iterator that yields chunks of tokens of the specified length
    chunks_iterator = batched(tokens, chunk_length)
    # Yield each chunk from the iterator
    yield from chunks_iterator
```

Finally, we can write a function that safely handles embedding requests, even when the input text is longer than the maximum context length, by chunking the input tokens and embedding each chunk individually. The average flag can be set to True to return the weighted average of the chunk embeddings, or False to simply return the unmodified list of chunk embeddings.

> Note: there are other techniques you can take here, including:
> - using GPT-4o to capture images/chart descriptions for embedding
> - chunking based on paragraphs or sections
> - adding more descriptive metadata about each article.


```python
EMBEDDING_CTX_LENGTH = 8191
EMBEDDING_ENCODING='cl100k_base'
```


```python
def generate_embeddings(text, model):
    # Generate embeddings for the provided text using the specified model
    embeddings_response = openai_client.embeddings.create(model=model, input=text)
    # Extract the embedding data from the response
    embedding = embeddings_response.data[0].embedding
    return embedding

def len_safe_get_embedding(text, model=embeddings_model, max_tokens=EMBEDDING_CTX_LENGTH, encoding_name=EMBEDDING_ENCODING):
    # Initialize lists to store embeddings and corresponding text chunks
    chunk_embeddings = []
    chunk_texts = []
    # Iterate over chunks of tokens from the input text
    for chunk in chunked_tokens(text, chunk_length=max_tokens, encoding_name=encoding_name):
        # Generate embeddings for each chunk and append to the list
        chunk_embeddings.append(generate_embeddings(chunk, model=model))
        # Decode the chunk back to text and append to the list
        chunk_texts.append(tiktoken.get_encoding(encoding_name).decode(chunk))
    # Return the list of chunk embeddings and the corresponding text chunks
    return chunk_embeddings, chunk_texts
```

Next, we can define a helper function that will capture additional metadata about the documents. In this example, I'll choose from a list of categories to use later on in a metadata filter


key: _node_content
string_value:
{"id_": "193f08b4-1ce8-4e61-9a4d-312ee78e6778", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6d8bb99b-2027-4815-90bd-14a51cfd3979", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f33c7ece243df97d2c659b7677c9ac7a517784feee18f15772f79b17a9198eda", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "d7490cf5-1ea1-4762-b2b9-0f0f47d886f4", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "9bb3442c9bc7bba34887ed6f0fd954454314daae4093255cce1e9d91ee0a2e9e", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "1668009c-2d50-48aa-a186-17bac7fe754e", "node_type": "1", "metadata": {}, "hash": "623f1fecfebe40b62e2a23264c8595063edee949efa1dc755eef4c9bce04154c", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 8259, "end_char_idx": 10819, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
EMBEDDING_CTX_LENGTH = 8191
EMBEDDING_ENCODING='cl100k_base'
```


```python
def generate_embeddings(text, model):
    # Generate embeddings for the provided text using the specified model
    embeddings_response = openai_client.embeddings.create(model=model, input=text)
    # Extract the embedding data from the response
    embedding = embeddings_response.data[0].embedding
    return embedding

def len_safe_get_embedding(text, model=embeddings_model, max_tokens=EMBEDDING_CTX_LENGTH, encoding_name=EMBEDDING_ENCODING):
    # Initialize lists to store embeddings and corresponding text chunks
    chunk_embeddings = []
    chunk_texts = []
    # Iterate over chunks of tokens from the input text
    for chunk in chunked_tokens(text, chunk_length=max_tokens, encoding_name=encoding_name):
        # Generate embeddings for each chunk and append to the list
        chunk_embeddings.append(generate_embeddings(chunk, model=model))
        # Decode the chunk back to text and append to the list
        chunk_texts.append(tiktoken.get_encoding(encoding_name).decode(chunk))
    # Return the list of chunk embeddings and the corresponding text chunks
    return chunk_embeddings, chunk_texts
```

Next, we can define a helper function that will capture additional metadata about the documents. In this example, I'll choose from a list of categories to use later on in a metadata filter


```python
categories = ['authentication','models','techniques','tools','setup','billing_limits','other']

def categorize_text(text, categories):

    # Create a prompt for categorization
    messages = [
        {"role": "system", "content": f"""You are an expert in LLMs, and you will be given text that corresponds to an article in OpenAI's documentation.
         Categorize the document into one of these categories: {', '.join(categories)}. Only respond with the category name and nothing else."""},
        {"role": "user", "content": text}
    ]
    try:
        # Call the OpenAI API to categorize the text
        response = openai_client.chat.completions.create(
            model="gpt-4o",
            messages=messages
        )

        # Extract the category from the response
        category = response.choices[0].message.content
        return category
    except Exception as e:
        print(f"Error categorizing text: {str(e)}")
        return None

# Example usage
```

Now, we can define some helper functions to process the .txt files in the oai_docs folder. Feel free to use this on your own data, this supports both .txt and .pdf files.


key: _node_content
string_value:
{"id_": "1668009c-2d50-48aa-a186-17bac7fe754e", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6d8bb99b-2027-4815-90bd-14a51cfd3979", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f33c7ece243df97d2c659b7677c9ac7a517784feee18f15772f79b17a9198eda", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "193f08b4-1ce8-4e61-9a4d-312ee78e6778", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "cd7400dbeb1b4a5831e2744614f17a057d9f6591e16bce76219aabb908c64123", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "b2f0ff1c-2de3-41e8-81e4-a90f0e02abe5", "node_type": "1", "metadata": {}, "hash": "e08e2c5fd0170b7c8cc8e6e3df85e67796414bb52220b59e469ff6663b4dee41", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 9658, "end_char_idx": 13000, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
categories = ['authentication','models','techniques','tools','setup','billing_limits','other']

def categorize_text(text, categories):

    # Create a prompt for categorization
    messages = [
        {"role": "system", "content": f"""You are an expert in LLMs, and you will be given text that corresponds to an article in OpenAI's documentation.
         Categorize the document into one of these categories: {', '.join(categories)}. Only respond with the category name and nothing else."""},
        {"role": "user", "content": text}
    ]
    try:
        # Call the OpenAI API to categorize the text
        response = openai_client.chat.completions.create(
            model="gpt-4o",
            messages=messages
        )

        # Extract the category from the response
        category = response.choices[0].message.content
        return category
    except Exception as e:
        print(f"Error categorizing text: {str(e)}")
        return None

# Example usage
```

Now, we can define some helper functions to process the .txt files in the oai_docs folder. Feel free to use this on your own data, this supports both .txt and .pdf files.


```python
def extract_text_from_pdf(pdf_path):
    # Initialize the PDF reader
    reader = PdfReader(pdf_path)
    text = ""
    # Iterate through each page in the PDF and extract text
    for page in reader.pages:
        text += page.extract_text()
    return text

def process_file(file_path, idx, categories, embeddings_model):
    file_name = os.path.basename(file_path)
    print(f"Processing file {idx + 1}: {file_name}")
    
    # Read text content from .txt files
    if file_name.endswith('.txt'):
        with open(file_path, 'r', encoding='utf-8') as file:
            text = file.read()
    # Extract text content from .pdf files
    elif file_name.endswith('.pdf'):
        text = extract_text_from_pdf(file_path)
    
    title = file_name
    # Generate embeddings for the title
    title_vectors, title_text = len_safe_get_embedding(title, embeddings_model)
    print(f"Generated title embeddings for {file_name}")
    
    # Generate embeddings for the content
    content_vectors, content_text = len_safe_get_embedding(text, embeddings_model)
    print(f"Generated content embeddings for {file_name}")
    
    category = categorize_text(' '.join(content_text), categories)
    print(f"Categorized {file_name} as {category}")
    
    # Prepare the data to be appended
    data = []
    for i, content_vector in enumerate(content_vectors):
        data.append({
            "id": f"{idx}_{i}",
            "vector_id": f"{idx}_{i}",
            "title": title_text[0],
            "text": content_text[i],
            "title_vector": json.dumps(title_vectors[0]),  # Assuming title is short and has only one chunk
            "content_vector": json.dumps(content_vector),
            "category": category
        })
        print(f"Appended data for chunk {i + 1}/{len(content_vectors)} of {file_name}")
    
    return data


```

We'll now use this helper function to process our OpenAI documentation. Feel free to update this to use your own data by changing the folder in process_files below.

Note that this will process the documents in chosen folder concurrently, so this should take <30 seconds if using txt files, and slightly longer if using PDFs.


key: _node_content
string_value:
{"id_": "b2f0ff1c-2de3-41e8-81e4-a90f0e02abe5", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6d8bb99b-2027-4815-90bd-14a51cfd3979", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f33c7ece243df97d2c659b7677c9ac7a517784feee18f15772f79b17a9198eda", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "1668009c-2d50-48aa-a186-17bac7fe754e", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "082174960000d20a34b09973c3fbd7bfad5086696b28786c7e0b45f45ca374f1", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "8c1c4ec3-21cc-4778-b373-cdb713bf1b53", "node_type": "1", "metadata": {}, "hash": "f2085f28758fdb0c73ed95052e135a043109b2ea71298089988ad8cbdcd40127", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 12670, "end_char_idx": 16063, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```

We'll now use this helper function to process our OpenAI documentation. Feel free to update this to use your own data by changing the folder in process_files below.

Note that this will process the documents in chosen folder concurrently, so this should take <30 seconds if using txt files, and slightly longer if using PDFs.


```python
## Customize the location below if you are using different data besides the OpenAI documentation. Note that if you are using a different dataset, you will need to update the categories list as well.
folder_name = "../../../data/oai_docs"

files = [os.path.join(folder_name, f) for f in os.listdir(folder_name) if f.endswith('.txt') or f.endswith('.pdf')]
data = []

# Process each file concurrently
with concurrent.futures.ThreadPoolExecutor() as executor:
    futures = {executor.submit(process_file, file_path, idx, categories, embeddings_model): idx for idx, file_path in enumerate(files)}
    for future in concurrent.futures.as_completed(futures):
        try:
            result = future.result()
            data.extend(result)
        except Exception as e:
            print(f"Error processing file: {str(e)}")

# Write the data to a CSV file
csv_file = os.path.join("..", "embedded_data.csv")
with open(csv_file, 'w', newline='', encoding='utf-8') as csvfile:
    fieldnames = ["id", "vector_id", "title", "text", "title_vector", "content_vector","category"]
    writer = csv.DictWriter(csvfile, fieldnames=fieldnames)
    writer.writeheader()
    for row in data:
        writer.writerow(row)
        print(f"Wrote row with id {row['id']} to CSV")

# Convert the CSV file to a Dataframe
article_df = pd.read_csv("../embedded_data.csv")
# Read vectors from strings back into a list using json.loads
article_df["title_vector"] = article_df.title_vector.apply(json.loads)
article_df["content_vector"] = article_df.content_vector.apply(json.loads)
article_df["vector_id"] = article_df["vector_id"].apply(str)
article_df["category"] = article_df["category"].apply(str)
article_df.head()
```

We now have an `embedded_data.csv` file with six columns that we can upload to our vector database! 

# Create BigQuery table with Vector Search

## Create BigQuery dataset

We'll leverage Google SDK and create a dataset named "oai_docs" with a table name of "embedded_data", but feel free to change those variables (you can also change regions).

*PS: We won't create a BigQuery index, that could improve the performance of the vector search, because such index requires more than 1k rows in our dataset which we don't have in our example, but feel free to leverage that for your own use-case.*


```python
# Create bigquery table

from google.cloud import bigquery
from google.api_core.exceptions import Conflict

# Define the dataset ID (project_id.dataset_id)
raw_dataset_id = 'oai_docs'
dataset_id = project_id + '.' + raw_dataset_id

client = bigquery.Client(credentials=credentials, project=project_id)

# Construct a full Dataset object to send to the API
dataset = bigquery.Dataset(dataset_id)

# Specify the geographic location where the dataset should reside
dataset.location = "US"

# Send the dataset to the API for creation
try:
    dataset = client.create_dataset(dataset, timeout=30)
    print(f"Created dataset {client.project}.{dataset.dataset_id}")
except Conflict:
    print(f"dataset {dataset.dataset_id } already exists")
    
```


key: _node_content
string_value:
{"id_": "8c1c4ec3-21cc-4778-b373-cdb713bf1b53", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6d8bb99b-2027-4815-90bd-14a51cfd3979", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f33c7ece243df97d2c659b7677c9ac7a517784feee18f15772f79b17a9198eda", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "b2f0ff1c-2de3-41e8-81e4-a90f0e02abe5", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "5afaaf11ac9e333d7247c43a3561e3fba47488faf4cc0ba6ce67cbffde4fc700", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "42a16ca5-c2b4-4287-a43b-cf6936acc404", "node_type": "1", "metadata": {}, "hash": "003341fcda74a53d30a7b84fa016d50c6e9a114908a9843fa4043648939c0572", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15309, "end_char_idx": 16819, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Create bigquery table

from google.cloud import bigquery
from google.api_core.exceptions import Conflict

# Define the dataset ID (project_id.dataset_id)
raw_dataset_id = 'oai_docs'
dataset_id = project_id + '.' + raw_dataset_id

client = bigquery.Client(credentials=credentials, project=project_id)

# Construct a full Dataset object to send to the API
dataset = bigquery.Dataset(dataset_id)

# Specify the geographic location where the dataset should reside
dataset.location = "US"

# Send the dataset to the API for creation
try:
    dataset = client.create_dataset(dataset, timeout=30)
    print(f"Created dataset {client.project}.{dataset.dataset_id}")
except Conflict:
    print(f"dataset {dataset.dataset_id } already exists")
    
```


```python
# Read the CSV file, properly handling multiline fields
csv_file_path = "../embedded_data.csv"
df = pd.read_csv(csv_file_path, engine='python', quotechar='"', quoting=1)

# Display the first few rows of the dataframe
df.head()

```

## Creating table and upload data

We'll create the table with the attribute name and types. Note the 'content_vector' attribute that allows to store a vector of float for a single row, which we'll use for our vector search.

This code will then loop on our CSVs previously created to insert the rows into Bigquery. 
If you run this code multiple time, multiple identical rows will be inserted which will give less accurate results when doing search (you could put uniqueness on IDs or clean the DB each time).


key: _node_content
string_value:
{"id_": "42a16ca5-c2b4-4287-a43b-cf6936acc404", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6d8bb99b-2027-4815-90bd-14a51cfd3979", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f33c7ece243df97d2c659b7677c9ac7a517784feee18f15772f79b17a9198eda", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "8c1c4ec3-21cc-4778-b373-cdb713bf1b53", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "52fc3546cedfc595efeb9e5d64dfe227913aedf908368c4c8f6e42f93c67e4b6", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "459d5a93-d77d-41b5-bb19-3e7866abffb8", "node_type": "1", "metadata": {}, "hash": "59f81da7ede3c10aee2cbbeffb071bd1445387e05f33083c260beb115b90706f", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 15309, "end_char_idx": 19086, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
# Create bigquery table

from google.cloud import bigquery
from google.api_core.exceptions import Conflict

# Define the dataset ID (project_id.dataset_id)
raw_dataset_id = 'oai_docs'
dataset_id = project_id + '.' + raw_dataset_id

client = bigquery.Client(credentials=credentials, project=project_id)

# Construct a full Dataset object to send to the API
dataset = bigquery.Dataset(dataset_id)

# Specify the geographic location where the dataset should reside
dataset.location = "US"

# Send the dataset to the API for creation
try:
    dataset = client.create_dataset(dataset, timeout=30)
    print(f"Created dataset {client.project}.{dataset.dataset_id}")
except Conflict:
    print(f"dataset {dataset.dataset_id } already exists")
    
```


```python
# Read the CSV file, properly handling multiline fields
csv_file_path = "../embedded_data.csv"
df = pd.read_csv(csv_file_path, engine='python', quotechar='"', quoting=1)

# Display the first few rows of the dataframe
df.head()

```

## Creating table and upload data

We'll create the table with the attribute name and types. Note the 'content_vector' attribute that allows to store a vector of float for a single row, which we'll use for our vector search.

This code will then loop on our CSVs previously created to insert the rows into Bigquery. 
If you run this code multiple time, multiple identical rows will be inserted which will give less accurate results when doing search (you could put uniqueness on IDs or clean the DB each time).


```python
# Read the CSV file, properly handling multiline fields
dataset_id = project_id + '.' + raw_dataset_id
client = bigquery.Client(credentials=credentials, project=project_id)
csv_file_path = "../embedded_data.csv"
df = pd.read_csv(csv_file_path, engine='python', quotechar='"', quoting=1)

# Preprocess the data to ensure content_vector is correctly formatted
# removing last and first character which are brackets [], comma splitting and converting to float
def preprocess_content_vector(row):
    row['content_vector'] = [float(x) for x in row['content_vector'][1:-1].split(',')]
    return row

# Apply preprocessing to the dataframe
df = df.apply(preprocess_content_vector, axis=1)

# Define the schema of the final table
final_schema = [
    bigquery.SchemaField("id", "STRING"),
    bigquery.SchemaField("vector_id", "STRING"),
    bigquery.SchemaField("title", "STRING"),
    bigquery.SchemaField("text", "STRING"),
    bigquery.SchemaField("title_vector", "STRING"),
    bigquery.SchemaField("content_vector", "FLOAT64", mode="REPEATED"),
    bigquery.SchemaField("category", "STRING"),
]

# Define the final table ID
raw_table_id = 'embedded_data'
final_table_id = f'{dataset_id}.' + raw_table_id

# Create the final table object
final_table = bigquery.Table(final_table_id, schema=final_schema)

# Send the table to the API for creation
final_table = client.create_table(final_table, exists_ok=True)  # API request
print(f"Created final table {project_id}.{final_table.dataset_id}.{final_table.table_id}")

# Convert DataFrame to list of dictionaries for BigQuery insertion
rows_to_insert = df.to_dict(orient='records')

# Upload data to the final table
errors = client.insert_rows_json(f"{final_table.dataset_id}.{final_table.table_id}", rows_to_insert)  # API request

if errors:
    print(f"Encountered errors while inserting rows: {errors}")
else:
    print(f"Successfully loaded data into {dataset_id}:{final_table_id}")
```

# Test search
Now that the data is uploaded, we'll test both pure vector similarity search and with metadata filtering locally below to make sure it is working as expected.

You can test both a pure vector search and metadata filtering.

The query below is pure vector search, where we don't filter out on category.


key: _node_content
string_value:
{"id_": "459d5a93-d77d-41b5-bb19-3e7866abffb8", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6d8bb99b-2027-4815-90bd-14a51cfd3979", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f33c7ece243df97d2c659b7677c9ac7a517784feee18f15772f79b17a9198eda", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "42a16ca5-c2b4-4287-a43b-cf6936acc404", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "a959916ed8d8c1c39ff77a87fdc6f6b24f0d7d5e39433b8e670d1bec64c1ac15", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "91a942a1-247a-46ad-9fd3-603558a03223", "node_type": "1", "metadata": {}, "hash": "ab2130edfcb8bbe698d982ed63d3d8ffccc2d2c34625df1ee31ee6243f932e96", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 19089, "end_char_idx": 21729, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


key: chroma:document
string_value:
```python
query = "What model should I use to embed?"
category = "models"

embedding_query = generate_embeddings(query, embeddings_model)
embedding_query_list = ', '.join(map(str, embedding_query))

query = f"""
WITH search_results AS (
  SELECT query.id AS query_id, base.id AS base_id, distance
  FROM VECTOR_SEARCH(
    TABLE oai_docs.embedded_data, 'content_vector',
    (SELECT ARRAY[{embedding_query_list}] AS content_vector, 'query_vector' AS id),
    top_k => 2, distance_type => 'COSINE', options => '{{"use_brute_force": true}}')
)
SELECT sr.query_id, sr.base_id, sr.distance, ed.text, ed.title
FROM search_results sr
JOIN oai_docs.embedded_data ed ON sr.base_id = ed.id
ORDER BY sr.distance ASC
"""

query_job = client.query(query)
results = query_job.result()  # Wait for the job to complete

for row in results:
    print(f"query_id: {row['query_id']}, base_id: {row['base_id']}, distance: {row['distance']}, text_truncated: {row['text'][0:100]}")

```

## Perform search with metadata filtering
Metadata filtering allows to restrict findings that have certain attributes on top of having the closest semantic findings of vector search.

The provided code snippet demonstrates how to execute a query with metadata filtering:


```python

query = "What model should I use to embed?"
category = "models"

embedding_query = generate_embeddings(query, embeddings_model)
embedding_query_list = ', '.join(map(str, embedding_query))


query = f"""
WITH search_results AS (
  SELECT query.id AS query_id, base.id AS base_id, distance
  FROM VECTOR_SEARCH(
    (SELECT * FROM oai_docs.embedded_data WHERE category = '{category}'), 
    'content_vector',
    (SELECT ARRAY[{embedding_query_list}] AS content_vector, 'query_vector' AS id),
    top_k => 4, distance_type => 'COSINE', options => '{{"use_brute_force": true}}')
)
SELECT sr.query_id, sr.base_id, sr.distance, ed.text, ed.title, ed.category
FROM search_results sr
JOIN oai_docs.embedded_data ed ON sr.base_id = ed.id
ORDER BY sr.distance ASC
"""


query_job = client.query(query)
results = query_job.result()  # Wait for the job to complete

for row in results:
    print(f"category: {row['category']}, title: {row['title']}, base_id: {row['base_id']}, distance: {row['distance']}, text_truncated: {row['text'][0:100]}")

```

# Create GCP function

## Exporting variables

We'll deploy the function in `main.py` in this folder (also available [here](https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/main.py)).

In a first step, we'll export the variables to target our table/dataset as well as to generate Embeddings using OpenAI's API.


key: _node_content
string_value:
{"id_": "91a942a1-247a-46ad-9fd3-603558a03223", "embedding": null, "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "excluded_embed_metadata_keys": ["url", "tokens", "retrieve_doc", "source"], "excluded_llm_metadata_keys": ["title", "tokens", "retrieve_doc", "source"], "relationships": {"1": {"node_id": "6d8bb99b-2027-4815-90bd-14a51cfd3979", "node_type": "4", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "f33c7ece243df97d2c659b7677c9ac7a517784feee18f15772f79b17a9198eda", "class_name": "RelatedNodeInfo"}, "2": {"node_id": "459d5a93-d77d-41b5-bb19-3e7866abffb8", "node_type": "1", "metadata": {"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/gcp/Getting_started_with_bigquery_vector_search_and_openai.ipynb", "title": "GCP Bigquery with GCP Functions and GPT actions in ChatGPT", "tokens": 6067, "retrieve_doc": true, "source": "openai_cookbooks"}, "hash": "0b1b81765eba46fbc2873fb7f328eb60e7e0fc58be5d2541019025b43e3bb6b0", "class_name": "RelatedNodeInfo"}, "3": {"node_id": "3fa737df-7e98-4621-b267-d847aad08cab", "node_type": "1", "metadata": {}, "hash": "b97fd9c62afe7a50f9f0171b909e00403e4bdda0468e1ca5916711ef0486cc7d", "class_name": "RelatedNodeInfo"}}, "text": "", "mimetype": "text/plain", "start_char_idx": 20329, "end_char_idx": 22620, "text_template": "{metadata_str}\n\n{content}", "metadata_template": "{key}: {value}", "metadata_seperator": "\n", "class_name": "TextNode"}


