# embeddings_queue

created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 386a0d42-a514-41d0-a952-928faf037642
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"386a0d42-a514-41d0-a952-928faf037642\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4773235a-eb59-4363-857d-783a66776844\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"355c443da9ae5d37fdfaec0607e185767d650838712ba271587c11ed2370e57a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"e69d24f9-7f00-4e1e-a446-20ad42547b3d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"95c5cdd8a888a4a50454d7672f9c198c1edabd1ce5a706351e5f15ba5e51e17c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2450, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4773235a-eb59-4363-857d-783a66776844", "doc_id": "4773235a-eb59-4363-857d-783a66776844", "ref_doc_id": "4773235a-eb59-4363-857d-783a66776844", "chroma:document": "# How to evaluate a summarization task\n\nIn this notebook we delve into the evaluation techniques for abstractive summarization tasks using a simple example. We explore traditional evaluation methods like [ROUGE](https://aclanthology.org/W04-1013/) and [BERTScore](https://arxiv.org/abs/1904.09675), in addition to showcasing a more novel approach using LLMs as evaluators.\n\nEvaluating the quality of summaries is a time-consuming process, as it involves different quality metrics such as coherence, conciseness, readability and content. Traditional automatic evaluation metrics such as `ROUGE` and `BERTScore` and others are concrete and reliable, but they may not correlate well with the actual quality of summaries. They show relatively low correlation with human judgments, especially for open-ended generation tasks ([Liu et al., 2023](https://arxiv.org/pdf/2303.16634.pdf)). There's a growing need to lean on human evaluations, user feedback, or model-based metrics while being vigilant about potential biases. While human judgment provides invaluable insights, it is often not scalable and can be cost-prohibitive.\n\nIn addition to these traditional metrics, we showcase a method ([G-Eval](https://arxiv.org/pdf/2303.16634.pdf)) that leverages Large Language Models (LLMs) as a novel, reference-free metric for assessing abstractive summaries. In this case, we use `gpt-4` to score candidate outputs. `gpt-4` has effectively learned an internal model of language quality that allows it to differentiate between fluent, coherent text and low-quality text. Harnessing this internal scoring mechanism allows auto-evaluation of new candidate outputs generated by an LLM.\n\n\n## Setup\n\n\n\n```python\n# Installing necessary packages for the evaluation\n# rouge: For evaluating with ROUGE metric\n# bert_score: For evaluating with BERTScore\n# openai: To interact with OpenAI's API\n!pip install rouge --quiet\n!pip install bert_score --quiet\n!pip install openai --quiet\n```\n\n\n```python\nfrom openai import OpenAI\nimport os\nimport re\nimport pandas as pd\n\n# Python Implementation of the ROUGE Metric\nfrom rouge import Rouge\n\n# BERTScore leverages the pre-trained contextual embeddings from BERT and matches words in candidate and reference sentences by cosine similarity.\nfrom bert_score import BERTScorer\n\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n\n```\n\n\n    <IPython.core.display.Javascript object>"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: e69d24f9-7f00-4e1e-a446-20ad42547b3d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"e69d24f9-7f00-4e1e-a446-20ad42547b3d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4773235a-eb59-4363-857d-783a66776844\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"355c443da9ae5d37fdfaec0607e185767d650838712ba271587c11ed2370e57a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"386a0d42-a514-41d0-a952-928faf037642\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"5b0fd405fe45abb01f9ca4ce3184eacdd666bc46b6b5770b9db85f5ac732b3ab\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8809a5d1-bc8a-48ed-b9d5-c579a00d603b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c8f266d2cac970d57cd8bde60d8d84155be9ea038460f0370ee9be4541f6362e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1674, \"end_char_idx\": 6892, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4773235a-eb59-4363-857d-783a66776844", "doc_id": "4773235a-eb59-4363-857d-783a66776844", "ref_doc_id": "4773235a-eb59-4363-857d-783a66776844", "chroma:document": "## Setup\n\n\n\n```python\n# Installing necessary packages for the evaluation\n# rouge: For evaluating with ROUGE metric\n# bert_score: For evaluating with BERTScore\n# openai: To interact with OpenAI's API\n!pip install rouge --quiet\n!pip install bert_score --quiet\n!pip install openai --quiet\n```\n\n\n```python\nfrom openai import OpenAI\nimport os\nimport re\nimport pandas as pd\n\n# Python Implementation of the ROUGE Metric\nfrom rouge import Rouge\n\n# BERTScore leverages the pre-trained contextual embeddings from BERT and matches words in candidate and reference sentences by cosine similarity.\nfrom bert_score import BERTScorer\n\nclient = OpenAI(api_key=os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as env var>\"))\n\n```\n\n\n    <IPython.core.display.Javascript object>\n\n\n## Example task\n\nFor the purposes of this notebook we'll use the example summarization below. Notice that we provide two generated summaries to compare, and a reference human-written summary, which evaluation metrics like `ROUGE` and `BERTScore` require.\n\nExcerpt (`excerpt`):\n\n> OpenAI's mission is to ensure that artificial general intelligence (AGI) benefits all of humanity. OpenAI will build safe and beneficial AGI directly, but will also consider its mission fulfilled if its work aids others to achieve this outcome. OpenAI follows several key principles for this purpose. First, broadly distributed benefits - any influence over AGI's deployment will be used for the benefit of all, and to avoid harmful uses or undue concentration of power. Second, long-term safety - OpenAI is committed to doing the research to make AGI safe, and to promote the adoption of such research across the AI community. Third, technical leadership - OpenAI aims to be at the forefront of AI capabilities. Fourth, a cooperative orientation - OpenAI actively cooperates with other research and policy institutions, and seeks to create a global community working together to address AGI's global challenges.\n\nSummaries:\n\n| Reference Summary /`ref_summary` (human generated)                                                                                                                                                                                                                                                                                                                         | Eval Summary 1 / `eval_summary_1` (system generated)                                                                                                                                                                                                                                                                                                                               | Eval Summary 2 / `eval_summary_2` (system generated)                                                                                                                                                                                                                                                   |\n| -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------ |\n| OpenAI aims to ensure artificial general intelligence (AGI) is used for everyone's benefit, avoiding harmful uses or undue power concentration. It is committed to researching AGI safety, promoting such studies among the AI community. OpenAI seeks to lead in AI capabilities and cooperates with global research and policy institutions to address AGI's challenges. | OpenAI aims to AGI benefits all humanity, avoiding harmful uses and power concentration. It pioneers research into safe and beneficial AGI and promotes adoption globally. OpenAI maintains technical leadership in AI while cooperating with global institutions to address AGI challenges. It seeks to lead a collaborative worldwide effort developing AGI for collective good. | OpenAI aims to ensure AGI is for everyone's use, totally avoiding harmful stuff or big power concentration. Committed to researching AGI's safe side, promoting these studies in AI folks. OpenAI wants to be top in AI things and works with worldwide research, policy groups to figure AGI's stuff. |\n\nTake a moment to figure out which summary you'd personally prefer and the one that captures OpenAI's mission really well."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 8809a5d1-bc8a-48ed-b9d5-c579a00d603b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"8809a5d1-bc8a-48ed-b9d5-c579a00d603b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4773235a-eb59-4363-857d-783a66776844\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"355c443da9ae5d37fdfaec0607e185767d650838712ba271587c11ed2370e57a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"e69d24f9-7f00-4e1e-a446-20ad42547b3d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b06cfe88479f728a46dcbe777aa10fc7c0e855cb3135f2d0a5132196cf50ec24\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"581b2610-dd66-4754-93be-8f921e84242c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b4521af41da3037c5cbad4fccbc8f93dac9bc5779bbf896d17fe26f9b4101f53\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6896, \"end_char_idx\": 10220, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4773235a-eb59-4363-857d-783a66776844", "doc_id": "4773235a-eb59-4363-857d-783a66776844", "ref_doc_id": "4773235a-eb59-4363-857d-783a66776844", "chroma:document": "```python\nexcerpt = \"OpenAI's mission is to ensure that artificial general intelligence (AGI) benefits all of humanity. OpenAI will build safe and beneficial AGI directly, but will also consider its mission fulfilled if its work aids others to achieve this outcome. OpenAI follows several key principles for this purpose. First, broadly distributed benefits - any influence over AGI's deployment will be used for the benefit of all, and to avoid harmful uses or undue concentration of power. Second, long-term safety - OpenAI is committed to doing the research to make AGI safe, and to promote the adoption of such research across the AI community. Third, technical leadership - OpenAI aims to be at the forefront of AI capabilities. Fourth, a cooperative orientation - OpenAI actively cooperates with other research and policy institutions, and seeks to create a global community working together to address AGI's global challenges.\"\nref_summary = \"OpenAI aims to ensure artificial general intelligence (AGI) is used for everyone's benefit, avoiding harmful uses or undue power concentration. It is committed to researching AGI safety, promoting such studies among the AI community. OpenAI seeks to lead in AI capabilities and cooperates with global research and policy institutions to address AGI's challenges.\"\neval_summary_1 = \"OpenAI aims to AGI benefits all humanity, avoiding harmful uses and power concentration. It pioneers research into safe and beneficial AGI and promotes adoption globally. OpenAI maintains technical leadership in AI while cooperating with global institutions to address AGI challenges. It seeks to lead a collaborative worldwide effort developing AGI for collective good.\"\neval_summary_2 = \"OpenAI aims to ensure AGI is for everyone's use, totally avoiding harmful stuff or big power concentration. Committed to researching AGI's safe side, promoting these studies in AI folks. OpenAI wants to be top in AI things and works with worldwide research, policy groups to figure AGI's stuff.\"\n```\n\n\n    <IPython.core.display.Javascript object>\n\n\n## Evaluating using ROUGE\n\n[ROUGE](https://aclanthology.org/W04-1013/), which stands for Recall-Oriented Understudy for Gisting Evaluation, primarily gauges the overlap of words between a generated output and a reference text. It's a prevalent metric for evaluating automatic summarization tasks. Among its variants, `ROUGE-L` offers insights into the longest contiguous match between system-generated and reference summaries, gauging how well the system retains the original summary's essence.\n\n\n\n```python\n# function to calculate the Rouge score\ndef get_rouge_scores(text1, text2):\n    rouge = Rouge()\n    return rouge.get_scores(text1, text2)\n\n\nrouge_scores_out = []\n\n# Calculate the ROUGE scores for both summaries using reference\neval_1_rouge = get_rouge_scores(eval_summary_1, ref_summary)\neval_2_rouge = get_rouge_scores(eval_summary_2, ref_summary)\n\nfor metric in [\"rouge-1\", \"rouge-2\", \"rouge-l\"]:\n    for label in [\"F-Score\"]:\n        eval_1_score = eval_1_rouge[0][metric][label[0].lower()]\n        eval_2_score = eval_2_rouge[0][metric][label[0].lower()]\n\n        row = {\n            \"Metric\": f\"{metric} ({label})\",\n            \"Summary 1\": eval_1_score,\n            \"Summary 2\": eval_2_score,\n        }\n        rouge_scores_out.append(row)"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 581b2610-dd66-4754-93be-8f921e84242c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"581b2610-dd66-4754-93be-8f921e84242c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4773235a-eb59-4363-857d-783a66776844\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"355c443da9ae5d37fdfaec0607e185767d650838712ba271587c11ed2370e57a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8809a5d1-bc8a-48ed-b9d5-c579a00d603b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"64330718e9e3593a942bcc9cad8904d810122eb698ce7b4bd18c51440b41f724\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"685c1e9e-0d31-4a11-af70-87f1312bedc2\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"81476445bc57001c5fb5142b26eee5186b5a76ce47defb6e9d76e4d65e7bb780\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8924, \"end_char_idx\": 10535, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4773235a-eb59-4363-857d-783a66776844", "doc_id": "4773235a-eb59-4363-857d-783a66776844", "ref_doc_id": "4773235a-eb59-4363-857d-783a66776844", "chroma:document": "<IPython.core.display.Javascript object>\n\n\n## Evaluating using ROUGE\n\n[ROUGE](https://aclanthology.org/W04-1013/), which stands for Recall-Oriented Understudy for Gisting Evaluation, primarily gauges the overlap of words between a generated output and a reference text. It's a prevalent metric for evaluating automatic summarization tasks. Among its variants, `ROUGE-L` offers insights into the longest contiguous match between system-generated and reference summaries, gauging how well the system retains the original summary's essence.\n\n\n\n```python\n# function to calculate the Rouge score\ndef get_rouge_scores(text1, text2):\n    rouge = Rouge()\n    return rouge.get_scores(text1, text2)\n\n\nrouge_scores_out = []\n\n# Calculate the ROUGE scores for both summaries using reference\neval_1_rouge = get_rouge_scores(eval_summary_1, ref_summary)\neval_2_rouge = get_rouge_scores(eval_summary_2, ref_summary)\n\nfor metric in [\"rouge-1\", \"rouge-2\", \"rouge-l\"]:\n    for label in [\"F-Score\"]:\n        eval_1_score = eval_1_rouge[0][metric][label[0].lower()]\n        eval_2_score = eval_2_rouge[0][metric][label[0].lower()]\n\n        row = {\n            \"Metric\": f\"{metric} ({label})\",\n            \"Summary 1\": eval_1_score,\n            \"Summary 2\": eval_2_score,\n        }\n        rouge_scores_out.append(row)\n\n\ndef highlight_max(s):\n    is_max = s == s.max()\n    return [\n        \"background-color: lightgreen\" if v else \"background-color: white\"\n        for v in is_max\n    ]\n\n\nrouge_scores_out = (\n    pd.DataFrame(rouge_scores_out)\n    .set_index(\"Metric\")\n    .style.apply(highlight_max, axis=1)\n)\n\nrouge_scores_out\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 685c1e9e-0d31-4a11-af70-87f1312bedc2
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"685c1e9e-0d31-4a11-af70-87f1312bedc2\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4773235a-eb59-4363-857d-783a66776844\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"355c443da9ae5d37fdfaec0607e185767d650838712ba271587c11ed2370e57a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"581b2610-dd66-4754-93be-8f921e84242c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1bb7665301b34b255c896bd529ab4a94ba6220a660e67426fbee031b71860990\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4e63e7c3-ad58-405c-88b5-5eb5bc4ec769\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"eee3558fa20b8944ebcd928b79ba588daac76dc4eed49cb5bec8a0f8df83185d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9465, \"end_char_idx\": 11966, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4773235a-eb59-4363-857d-783a66776844", "doc_id": "4773235a-eb59-4363-857d-783a66776844", "ref_doc_id": "4773235a-eb59-4363-857d-783a66776844", "chroma:document": "```python\n# function to calculate the Rouge score\ndef get_rouge_scores(text1, text2):\n    rouge = Rouge()\n    return rouge.get_scores(text1, text2)\n\n\nrouge_scores_out = []\n\n# Calculate the ROUGE scores for both summaries using reference\neval_1_rouge = get_rouge_scores(eval_summary_1, ref_summary)\neval_2_rouge = get_rouge_scores(eval_summary_2, ref_summary)\n\nfor metric in [\"rouge-1\", \"rouge-2\", \"rouge-l\"]:\n    for label in [\"F-Score\"]:\n        eval_1_score = eval_1_rouge[0][metric][label[0].lower()]\n        eval_2_score = eval_2_rouge[0][metric][label[0].lower()]\n\n        row = {\n            \"Metric\": f\"{metric} ({label})\",\n            \"Summary 1\": eval_1_score,\n            \"Summary 2\": eval_2_score,\n        }\n        rouge_scores_out.append(row)\n\n\ndef highlight_max(s):\n    is_max = s == s.max()\n    return [\n        \"background-color: lightgreen\" if v else \"background-color: white\"\n        for v in is_max\n    ]\n\n\nrouge_scores_out = (\n    pd.DataFrame(rouge_scores_out)\n    .set_index(\"Metric\")\n    .style.apply(highlight_max, axis=1)\n)\n\nrouge_scores_out\n```\n\n\n\n\n<style type=\"text/css\">\n#T_7e6ac_row0_col0, #T_7e6ac_row1_col1, #T_7e6ac_row2_col0 {\n  background-color: white;\n}\n#T_7e6ac_row0_col1, #T_7e6ac_row1_col0, #T_7e6ac_row2_col1 {\n  background-color: lightgreen;\n}\n</style>\n<table id=\"T_7e6ac\">\n  <thead>\n    <tr>\n      <th class=\"blank level0\" >&nbsp;</th>\n      <th id=\"T_7e6ac_level0_col0\" class=\"col_heading level0 col0\" >Summary 1</th>\n      <th id=\"T_7e6ac_level0_col1\" class=\"col_heading level0 col1\" >Summary 2</th>\n    </tr>\n    <tr>\n      <th class=\"index_name level0\" >Metric</th>\n      <th class=\"blank col0\" >&nbsp;</th>\n      <th class=\"blank col1\" >&nbsp;</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th id=\"T_7e6ac_level0_row0\" class=\"row_heading level0 row0\" >rouge-1 (F-Score)</th>\n      <td id=\"T_7e6ac_row0_col0\" class=\"data row0 col0\" >0.488889</td>\n      <td id=\"T_7e6ac_row0_col1\" class=\"data row0 col1\" >0.511628</td>\n    </tr>\n    <tr>\n      <th id=\"T_7e6ac_level0_row1\" class=\"row_heading level0 row1\" >rouge-2 (F-Score)</th>\n      <td id=\"T_7e6ac_row1_col0\" class=\"data row1 col0\" >0.230769</td>\n      <td id=\"T_7e6ac_row1_col1\" class=\"data row1 col1\" >0.163265</td>\n    </tr>\n    <tr>\n      <th id=\"T_7e6ac_level0_row2\" class=\"row_heading level0 row2\" >rouge-l (F-Score)</th>\n      <td id=\"T_7e6ac_row2_col0\" class=\"data row2 col0\" >0.488889</td>\n      <td id=\"T_7e6ac_row2_col1\" class=\"data row2 col1\" >0.511628</td>\n    </tr>\n  </tbody>\n</table>"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 4e63e7c3-ad58-405c-88b5-5eb5bc4ec769
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"4e63e7c3-ad58-405c-88b5-5eb5bc4ec769\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4773235a-eb59-4363-857d-783a66776844\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"355c443da9ae5d37fdfaec0607e185767d650838712ba271587c11ed2370e57a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"685c1e9e-0d31-4a11-af70-87f1312bedc2\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"5e115bea4dec1d0f6a7816d4cb2cf8434baaddd66e28d470b591828002c5d018\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"99daf5b3-874d-4e8b-ad9d-d87d36ade12e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d01e0636fc392f1e46e62a08904eaaf217b146c4946a8d5c24e556b92a550980\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11976, \"end_char_idx\": 15206, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4773235a-eb59-4363-857d-783a66776844", "doc_id": "4773235a-eb59-4363-857d-783a66776844", "ref_doc_id": "4773235a-eb59-4363-857d-783a66776844", "chroma:document": "<IPython.core.display.Javascript object>\n\n\nThe table shows the `ROUGE` scores for evaluating two different summaries against a reference text. In the case of `rouge-1`, Summary 2 outperforms Summary 1, indicating a better overlap of individual words and for `rouge-l`, Summary 2 has a higher score, implying a closer match in the longest common subsequences, and thus a potentially better overall summarization in capturing the main content and order of the original text. Since Summary 2 has many words and short phrases directly lifted from the excerpt, its overlap with the reference summary would likely be higher, leading to higher `ROUGE` scores.\n\nWhile `ROUGE` and similar metrics, such as [BLEU](https://aclanthology.org/P02-1040.pdf) and [METEOR](https://www.cs.cmu.edu/~alavie/METEOR/), offer quantitative measures, they often fail to capture the true essence of a well-generated summary. They also correlate worse with human scores. Given the advancements in LLMs, which are adept at producing fluent and coherent summaries, traditional metrics like `ROUGE` may inadvertently penalize these models. This is especially true if the summaries are articulated differently but still encapsulate the core information accurately.\n\n\n## Evaluating using BERTScore\n\nROUGE relies on the exact presence of words in both the predicted and reference texts, failing to interpret the underlying semantics. This is where [BERTScore](https://arxiv.org/abs/1904.09675) comes in and leverages the contextual embeddings from the BERT model, aiming to evaluate the similarity between a predicted and a reference sentence in the context of machine-generated text. By comparing embeddings from both sentences, `BERTScore` captures semantic similarities that might be missed by traditional n-gram based metrics.\n\n\n\n```python\n# Instantiate the BERTScorer object for English language\nscorer = BERTScorer(lang=\"en\")\n\n# Calculate BERTScore for the summary 1 against the excerpt\n# P1, R1, F1_1 represent Precision, Recall, and F1 Score respectively\nP1, R1, F1_1 = scorer.score([eval_summary_1], [ref_summary])\n\n# Calculate BERTScore for summary 2 against the excerpt\n# P2, R2, F2_2 represent Precision, Recall, and F1 Score respectively\nP2, R2, F2_2 = scorer.score([eval_summary_2], [ref_summary])\n\nprint(\"Summary 1 F1 Score:\", F1_1.tolist()[0])\nprint(\"Summary 2 F1 Score:\", F2_2.tolist()[0])\n```\n\n    Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight', 'lm_head.decoder.weight', 'lm_head.bias']\n    - This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n    - This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n\n\n    Summary 1 F1 Score: 0.9227314591407776\n    Summary 2 F1 Score: 0.9189572930335999\n\n\n\n    <IPython.core.display.Javascript object>"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 99daf5b3-874d-4e8b-ad9d-d87d36ade12e
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"99daf5b3-874d-4e8b-ad9d-d87d36ade12e\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4773235a-eb59-4363-857d-783a66776844\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"355c443da9ae5d37fdfaec0607e185767d650838712ba271587c11ed2370e57a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4e63e7c3-ad58-405c-88b5-5eb5bc4ec769\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"28bd46c34406acba0446c38a6b0c14aa786c51302e55e773b720066d15ed6dab\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f477feab-edcf-4036-a9a0-3f314527d570\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1439f8bb4fc7a2a79da913830b7604d9dd81f85ace5d5ffb1ac5bb2d0d6db099\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13777, \"end_char_idx\": 15737, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4773235a-eb59-4363-857d-783a66776844", "doc_id": "4773235a-eb59-4363-857d-783a66776844", "ref_doc_id": "4773235a-eb59-4363-857d-783a66776844", "chroma:document": "```python\n# Instantiate the BERTScorer object for English language\nscorer = BERTScorer(lang=\"en\")\n\n# Calculate BERTScore for the summary 1 against the excerpt\n# P1, R1, F1_1 represent Precision, Recall, and F1 Score respectively\nP1, R1, F1_1 = scorer.score([eval_summary_1], [ref_summary])\n\n# Calculate BERTScore for summary 2 against the excerpt\n# P2, R2, F2_2 represent Precision, Recall, and F1 Score respectively\nP2, R2, F2_2 = scorer.score([eval_summary_2], [ref_summary])\n\nprint(\"Summary 1 F1 Score:\", F1_1.tolist()[0])\nprint(\"Summary 2 F1 Score:\", F2_2.tolist()[0])\n```\n\n    Some weights of the model checkpoint at roberta-large were not used when initializing RobertaModel: ['lm_head.dense.weight', 'lm_head.dense.bias', 'lm_head.layer_norm.bias', 'lm_head.layer_norm.weight', 'lm_head.decoder.weight', 'lm_head.bias']\n    - This IS expected if you are initializing RobertaModel from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).\n    - This IS NOT expected if you are initializing RobertaModel from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).\n\n\n    Summary 1 F1 Score: 0.9227314591407776\n    Summary 2 F1 Score: 0.9189572930335999\n\n\n\n    <IPython.core.display.Javascript object>\n\n\nThe close F1 Scores between the summaries indicate that they may perform similarly in capturing the key information. However, this small difference should be interpreted with caution. Since `BERTScore` may not fully grasp subtleties and high-level concepts that a human evaluator might understand, reliance solely on this metric could lead to misinterpreting the actual quality and nuances of the summary. An integrated approach combining `BERTScore` with human judgment and other metrics could offer a more reliable evaluation."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: f477feab-edcf-4036-a9a0-3f314527d570
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"f477feab-edcf-4036-a9a0-3f314527d570\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4773235a-eb59-4363-857d-783a66776844\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"355c443da9ae5d37fdfaec0607e185767d650838712ba271587c11ed2370e57a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"99daf5b3-874d-4e8b-ad9d-d87d36ade12e\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9ba5124d5d53521790c3ebacb6920411b882963b258984c23ee2f192a46f0028\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"0e244d68-0118-4502-bd2b-50c993451cb3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c7abc59b4b1407c97964e334311db93595655397b1fa1b4054e0aa57e13bb992\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15077, \"end_char_idx\": 17368, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4773235a-eb59-4363-857d-783a66776844", "doc_id": "4773235a-eb59-4363-857d-783a66776844", "ref_doc_id": "4773235a-eb59-4363-857d-783a66776844", "chroma:document": "Summary 1 F1 Score: 0.9227314591407776\n    Summary 2 F1 Score: 0.9189572930335999\n\n\n\n    <IPython.core.display.Javascript object>\n\n\nThe close F1 Scores between the summaries indicate that they may perform similarly in capturing the key information. However, this small difference should be interpreted with caution. Since `BERTScore` may not fully grasp subtleties and high-level concepts that a human evaluator might understand, reliance solely on this metric could lead to misinterpreting the actual quality and nuances of the summary. An integrated approach combining `BERTScore` with human judgment and other metrics could offer a more reliable evaluation.\n\n\n## Evaluating using GPT-4\n\nHere we implement an example **reference-free** text evaluator using `gpt-4`, inspired by the [G-Eval]((https://arxiv.org/pdf/2303.16634.pdf)) framework which evaluates the quality of generated text using large language models. Unlike metrics like `ROUGE` or `BERTScore` that rely on comparison to reference summaries, the `gpt-4` based evaluator assesses the quality of generated content based solely on the input prompt and text, without any ground truth references. This makes it applicable to new datasets and tasks where human references are sparse or unavailable. \n\nHere's an overview of this method:\n\n1. We define four distinct criteria:\n    1. **Relevance**: Evaluates if the summary includes only important information and excludes redundancies.\n    2. **Coherence**: Assesses the logical flow and organization of the summary.\n    3. **Consistency**: Checks if the summary aligns with the facts in the source document.\n    4. **Fluency**: Rates the grammar and readability of the summary.\n2. We craft prompts for each of these criteria, taking the original document and the summary as inputs, and leveraging chain-of-thought generation and guiding the model to output a numeric score from 1-5 for each criteria. \n3. We generate scores from `gpt-4` with the defined prompts, comparing them across summaries.\n\nIn this demonstration, we're using a direct scoring function where `gpt-4` generates a discrete score (1-5) for each metric. Normalizing the scores and taking a weighted sum could result in more robust, continuous scores that better reflect the quality and diversity of the summaries."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 0e244d68-0118-4502-bd2b-50c993451cb3
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"0e244d68-0118-4502-bd2b-50c993451cb3\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4773235a-eb59-4363-857d-783a66776844\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"355c443da9ae5d37fdfaec0607e185767d650838712ba271587c11ed2370e57a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f477feab-edcf-4036-a9a0-3f314527d570\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"699c9f1a3149ae7edbb9c337c0da2e4d8c4a1abc5af9319228c368cadc3b0152\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f1c36a00-f333-457f-a5cd-03d87f544afd\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"23a177c88aa82909084813256b18dd5831fa17f2ed7924b26ca8c4ea6db46ade\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15740, \"end_char_idx\": 20652, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4773235a-eb59-4363-857d-783a66776844", "doc_id": "4773235a-eb59-4363-857d-783a66776844", "ref_doc_id": "4773235a-eb59-4363-857d-783a66776844", "chroma:document": "## Evaluating using GPT-4\n\nHere we implement an example **reference-free** text evaluator using `gpt-4`, inspired by the [G-Eval]((https://arxiv.org/pdf/2303.16634.pdf)) framework which evaluates the quality of generated text using large language models. Unlike metrics like `ROUGE` or `BERTScore` that rely on comparison to reference summaries, the `gpt-4` based evaluator assesses the quality of generated content based solely on the input prompt and text, without any ground truth references. This makes it applicable to new datasets and tasks where human references are sparse or unavailable. \n\nHere's an overview of this method:\n\n1. We define four distinct criteria:\n    1. **Relevance**: Evaluates if the summary includes only important information and excludes redundancies.\n    2. **Coherence**: Assesses the logical flow and organization of the summary.\n    3. **Consistency**: Checks if the summary aligns with the facts in the source document.\n    4. **Fluency**: Rates the grammar and readability of the summary.\n2. We craft prompts for each of these criteria, taking the original document and the summary as inputs, and leveraging chain-of-thought generation and guiding the model to output a numeric score from 1-5 for each criteria. \n3. We generate scores from `gpt-4` with the defined prompts, comparing them across summaries.\n\nIn this demonstration, we're using a direct scoring function where `gpt-4` generates a discrete score (1-5) for each metric. Normalizing the scores and taking a weighted sum could result in more robust, continuous scores that better reflect the quality and diversity of the summaries.\n\n\n```python\n# Evaluation prompt template based on G-Eval\nEVALUATION_PROMPT_TEMPLATE = \"\"\"\nYou will be given one summary written for an article. Your task is to rate the summary on one metric.\nPlease make sure you read and understand these instructions very carefully. \nPlease keep this document open while reviewing, and refer to it as needed.\n\nEvaluation Criteria:\n\n{criteria}\n\nEvaluation Steps:\n\n{steps}\n\nExample:\n\nSource Text:\n\n{document}\n\nSummary:\n\n{summary}\n\nEvaluation Form (scores ONLY):\n\n- {metric_name}\n\"\"\"\n\n# Metric 1: Relevance\n\nRELEVANCY_SCORE_CRITERIA = \"\"\"\nRelevance(1-5) - selection of important content from the source. \\\nThe summary should include only important information from the source document. \\\nAnnotators were instructed to penalize summaries which contained redundancies and excess information.\n\"\"\"\n\nRELEVANCY_SCORE_STEPS = \"\"\"\n1. Read the summary and the source document carefully.\n2. Compare the summary to the source document and identify the main points of the article.\n3. Assess how well the summary covers the main points of the article, and how much irrelevant or redundant information it contains.\n4. Assign a relevance score from 1 to 5.\n\"\"\"\n\n# Metric 2: Coherence\n\nCOHERENCE_SCORE_CRITERIA = \"\"\"\nCoherence(1-5) - the collective quality of all sentences. \\\nWe align this dimension with the DUC quality question of structure and coherence \\\nwhereby \"the summary should be well-structured and well-organized. \\\nThe summary should not just be a heap of related information, but should build from sentence to a\\\ncoherent body of information about a topic.\"\n\"\"\"\n\nCOHERENCE_SCORE_STEPS = \"\"\"\n1. Read the article carefully and identify the main topic and key points.\n2. Read the summary and compare it to the article. Check if the summary covers the main topic and key points of the article,\nand if it presents them in a clear and logical order.\n3. Assign a score for coherence on a scale of 1 to 5, where 1 is the lowest and 5 is the highest based on the Evaluation Criteria.\n\"\"\"\n\n# Metric 3: Consistency\n\nCONSISTENCY_SCORE_CRITERIA = \"\"\"\nConsistency(1-5) - the factual alignment between the summary and the summarized source. \\\nA factually consistent summary contains only statements that are entailed by the source document. \\\nAnnotators were also asked to penalize summaries that contained hallucinated facts.\n\"\"\"\n\nCONSISTENCY_SCORE_STEPS = \"\"\"\n1. Read the article carefully and identify the main facts and details it presents.\n2. Read the summary and compare it to the article. Check if the summary contains any factual errors that are not supported by the article.\n3. Assign a score for consistency based on the Evaluation Criteria.\n\"\"\"\n\n# Metric 4: Fluency\n\nFLUENCY_SCORE_CRITERIA = \"\"\"\nFluency(1-3): the quality of the summary in terms of grammar, spelling, punctuation, word choice, and sentence structure.\n1: Poor. The summary has many errors that make it hard to understand or sound unnatural.\n2: Fair. The summary has some errors that affect the clarity or smoothness of the text, but the main points are still comprehensible.\n3: Good. The summary has few or no errors and is easy to read and follow.\n\"\"\"\n\nFLUENCY_SCORE_STEPS = \"\"\"\nRead the summary and evaluate its fluency based on the given criteria. Assign a fluency score from 1 to 3.\n\"\"\""}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: f1c36a00-f333-457f-a5cd-03d87f544afd
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"f1c36a00-f333-457f-a5cd-03d87f544afd\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4773235a-eb59-4363-857d-783a66776844\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"355c443da9ae5d37fdfaec0607e185767d650838712ba271587c11ed2370e57a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0e244d68-0118-4502-bd2b-50c993451cb3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"e368c556f5729724a2dd2cec7f7bd750f43a869a0da767ef7b38e4a322d62e95\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"9b1cf4a2-27cd-4330-8d11-395d3104ef04\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3ea6e35c127f1e8581148d8689708c8b5e08709865a2b0e091e648c8e0b35793\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 20655, \"end_char_idx\": 22257, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4773235a-eb59-4363-857d-783a66776844", "doc_id": "4773235a-eb59-4363-857d-783a66776844", "ref_doc_id": "4773235a-eb59-4363-857d-783a66776844", "chroma:document": "def get_geval_score(\n    criteria: str, steps: str, document: str, summary: str, metric_name: str\n):\n    prompt = EVALUATION_PROMPT_TEMPLATE.format(\n        criteria=criteria,\n        steps=steps,\n        metric_name=metric_name,\n        document=document,\n        summary=summary,\n    )\n    response = client.chat.completions.create(\n        model=\"gpt-4\",\n        messages=[{\"role\": \"user\", \"content\": prompt}],\n        temperature=0,\n        max_tokens=5,\n        top_p=1,\n        frequency_penalty=0,\n        presence_penalty=0,\n    )\n    return response.choices[0].message.content\n\n\nevaluation_metrics = {\n    \"Relevance\": (RELEVANCY_SCORE_CRITERIA, RELEVANCY_SCORE_STEPS),\n    \"Coherence\": (COHERENCE_SCORE_CRITERIA, COHERENCE_SCORE_STEPS),\n    \"Consistency\": (CONSISTENCY_SCORE_CRITERIA, CONSISTENCY_SCORE_STEPS),\n    \"Fluency\": (FLUENCY_SCORE_CRITERIA, FLUENCY_SCORE_STEPS),\n}\n\nsummaries = {\"Summary 1\": eval_summary_1, \"Summary 2\": eval_summary_2}\n\ndata = {\"Evaluation Type\": [], \"Summary Type\": [], \"Score\": []}\n\nfor eval_type, (criteria, steps) in evaluation_metrics.items():\n    for summ_type, summary in summaries.items():\n        data[\"Evaluation Type\"].append(eval_type)\n        data[\"Summary Type\"].append(summ_type)\n        result = get_geval_score(criteria, steps, excerpt, summary, eval_type)\n        score_num = int(result.strip())\n        data[\"Score\"].append(score_num)\n\npivot_df = pd.DataFrame(data, index=None).pivot(\n    index=\"Evaluation Type\", columns=\"Summary Type\", values=\"Score\"\n)\nstyled_pivot_df = pivot_df.style.apply(highlight_max, axis=1)\ndisplay(styled_pivot_df)\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 9b1cf4a2-27cd-4330-8d11-395d3104ef04
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"9b1cf4a2-27cd-4330-8d11-395d3104ef04\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4773235a-eb59-4363-857d-783a66776844\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"355c443da9ae5d37fdfaec0607e185767d650838712ba271587c11ed2370e57a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f1c36a00-f333-457f-a5cd-03d87f544afd\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"2794403f1839eadbf308bf5987cd42e1d68d20d8c3cce2b2d8743cb0a1972971\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"2054816d-4161-40f4-a18b-9994822fafdf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"88d39cc6f73eb420872c3104aa885fca7521be5c7395c9bcdd5bf0637f81b1e2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 21243, \"end_char_idx\": 23910, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4773235a-eb59-4363-857d-783a66776844", "doc_id": "4773235a-eb59-4363-857d-783a66776844", "ref_doc_id": "4773235a-eb59-4363-857d-783a66776844", "chroma:document": "evaluation_metrics = {\n    \"Relevance\": (RELEVANCY_SCORE_CRITERIA, RELEVANCY_SCORE_STEPS),\n    \"Coherence\": (COHERENCE_SCORE_CRITERIA, COHERENCE_SCORE_STEPS),\n    \"Consistency\": (CONSISTENCY_SCORE_CRITERIA, CONSISTENCY_SCORE_STEPS),\n    \"Fluency\": (FLUENCY_SCORE_CRITERIA, FLUENCY_SCORE_STEPS),\n}\n\nsummaries = {\"Summary 1\": eval_summary_1, \"Summary 2\": eval_summary_2}\n\ndata = {\"Evaluation Type\": [], \"Summary Type\": [], \"Score\": []}\n\nfor eval_type, (criteria, steps) in evaluation_metrics.items():\n    for summ_type, summary in summaries.items():\n        data[\"Evaluation Type\"].append(eval_type)\n        data[\"Summary Type\"].append(summ_type)\n        result = get_geval_score(criteria, steps, excerpt, summary, eval_type)\n        score_num = int(result.strip())\n        data[\"Score\"].append(score_num)\n\npivot_df = pd.DataFrame(data, index=None).pivot(\n    index=\"Evaluation Type\", columns=\"Summary Type\", values=\"Score\"\n)\nstyled_pivot_df = pivot_df.style.apply(highlight_max, axis=1)\ndisplay(styled_pivot_df)\n```\n\n\n<style type=\"text/css\">\n#T_94fab_row0_col0, #T_94fab_row1_col0, #T_94fab_row1_col1, #T_94fab_row2_col0, #T_94fab_row3_col0 {\n  background-color: lightgreen;\n}\n#T_94fab_row0_col1, #T_94fab_row2_col1, #T_94fab_row3_col1 {\n  background-color: white;\n}\n</style>\n<table id=\"T_94fab\">\n  <thead>\n    <tr>\n      <th class=\"index_name level0\" >Summary Type</th>\n      <th id=\"T_94fab_level0_col0\" class=\"col_heading level0 col0\" >Summary 1</th>\n      <th id=\"T_94fab_level0_col1\" class=\"col_heading level0 col1\" >Summary 2</th>\n    </tr>\n    <tr>\n      <th class=\"index_name level0\" >Evaluation Type</th>\n      <th class=\"blank col0\" >&nbsp;</th>\n      <th class=\"blank col1\" >&nbsp;</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th id=\"T_94fab_level0_row0\" class=\"row_heading level0 row0\" >Coherence</th>\n      <td id=\"T_94fab_row0_col0\" class=\"data row0 col0\" >5</td>\n      <td id=\"T_94fab_row0_col1\" class=\"data row0 col1\" >3</td>\n    </tr>\n    <tr>\n      <th id=\"T_94fab_level0_row1\" class=\"row_heading level0 row1\" >Consistency</th>\n      <td id=\"T_94fab_row1_col0\" class=\"data row1 col0\" >5</td>\n      <td id=\"T_94fab_row1_col1\" class=\"data row1 col1\" >5</td>\n    </tr>\n    <tr>\n      <th id=\"T_94fab_level0_row2\" class=\"row_heading level0 row2\" >Fluency</th>\n      <td id=\"T_94fab_row2_col0\" class=\"data row2 col0\" >3</td>\n      <td id=\"T_94fab_row2_col1\" class=\"data row2 col1\" >2</td>\n    </tr>\n    <tr>\n      <th id=\"T_94fab_level0_row3\" class=\"row_heading level0 row3\" >Relevance</th>\n      <td id=\"T_94fab_row3_col0\" class=\"data row3 col0\" >5</td>\n      <td id=\"T_94fab_row3_col1\" class=\"data row3 col1\" >4</td>\n    </tr>\n  </tbody>\n</table>"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 2054816d-4161-40f4-a18b-9994822fafdf
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb", "title": "How to evaluate a summarization task", "tokens": 5999, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"2054816d-4161-40f4-a18b-9994822fafdf\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"4773235a-eb59-4363-857d-783a66776844\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"355c443da9ae5d37fdfaec0607e185767d650838712ba271587c11ed2370e57a\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"9b1cf4a2-27cd-4330-8d11-395d3104ef04\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/How_to_eval_abstractive_summarization.ipynb\", \"title\": \"How to evaluate a summarization task\", \"tokens\": 5999, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"9c501db8e60b49887c2fd2ed80433d6459e084c28099e005f48e3926a5d4b64d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 23919, \"end_char_idx\": 26471, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "4773235a-eb59-4363-857d-783a66776844", "doc_id": "4773235a-eb59-4363-857d-783a66776844", "ref_doc_id": "4773235a-eb59-4363-857d-783a66776844", "chroma:document": "<IPython.core.display.Javascript object>\n\n\n\nOverall, the Summary 1 appears to outperform Summary 2 in three of the four categories (Coherence, Relevance and Fluency). Both summaries are found to be consistent with each other. The result might suggest that Summary 1 is generally preferable based on the given evaluation criteria.\n\n### Limitations\n\nNote that LLM-based metrics could have a bias towards preferring LLM-generated texts over human-written texts. Additionally LLM based metrics are sensitive to system messages/prompts. We recommend experimenting with other techniques that can help improve performance and/or get consistent scores, striking the right balance between high-quality expensive evaluation and automated evaluations. It is also worth noting that this scoring methodology is currently limited by `gpt-4`'s context window.\n\n## Conclusion\n\nEvaluating abstractive summarization remains an open area for further improvement. Traditional metrics like `ROUGE`, `BLEU`, and `BERTScore` provide useful automatic evaluation but have limitations in capturing semantic similarity and nuanced aspects of summarization quality. Moreover, they require reference outputs which can be expensive to collect/label. LLM-based metrics offer promise as a reference-free method of evaluating coherence, fluency, and relevance. However, they too have potential biases favoring text generated by LLMs. Ultimately, a combination of automatic metrics and human evaluation is ideal for reliably assessing abstractive summarization systems. While human evaluation is indispensable for gaining a comprehensive understanding of summary quality, it should be complemented with automated evaluation to enable efficient, large-scale testing. The field will continue to evolve more robust evaluation techniques, balancing quality, scalability, and fairness. Advancing evaluation methods is crucial for driving progress in production applications.\n\n## References\n\n- [G-EVAL: NLG Evaluation Using GPT-4 with Better Human Alignment](https://arxiv.org/pdf/2303.16634.pdf) - Liu Y, Iter D, Xu Y, Wang S, Xu R, Zhu C. Published May, 2023.\n- [BERTScore: Evaluating Text Generation with BERT](https://arxiv.org/abs/1904.09675) - Zhang T, Kishore V, Wu F, Weinberger KQ, Artzi Y. Published online February, 2020.\n- [ROUGE: A Package for Automatic Evaluation of Summaries](https://aclanthology.org/W04-1013/) - Lin CY. Published July, 2004.\n- [SummEval: Re-evaluating Summarization Evaluation](https://aclanthology.org/2021.tacl-1.24) - Fabbri et al. Published April, 2021."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 0fe6fb09-2d07-48ba-bfd3-761e92cd4fbb
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"0fe6fb09-2d07-48ba-bfd3-761e92cd4fbb\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d18906e8-9fec-49e5-96bc-f73d8280b8ed\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ab4c3631d8abc55b383c576a8b460c6c41062f442df7d0e09a6ead8c1240672b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3413, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "# Getting Started with OpenAI Evals\n\nThe [OpenAI Evals](https://github.com/openai/evals/tree/main) framework consists of\n1. A framework to evaluate an LLM (large language model) or a system built on top of an LLM.\n2. An open-source registry of challenging evals\n\nThis notebook will cover:\n* Introduction to Evaluation and the [OpenAI Evals](https://github.com/openai/evals/tree/main) library\n* Building an Eval\n* Running an Eval\n\n#### What are evaluations/ `evals`?\n\nEvaluation is the process of validating and testing the outputs that your LLM applications are producing. Having strong evaluations (\"evals\") will mean a more stable, reliable application that is resilient to code and model changes. An eval is a task used to measure the quality of the output of an LLM or LLM system. Given an input prompt, an output is generated. We evaluate this output with a set of ideal answers and find the quality of the LLM system.\n\n#### Importance of Evaluations\n\nIf you are building with foundational models like `GPT-4`, creating high quality evals is one of the most impactful things you can do. Developing AI solutions involves an iterative design process. [Without evals, it can be very difficult and time intensive to understand](https://youtu.be/XGJNo8TpuVA?feature=shared&t=1089) how different model versions and prompts might affect your use case.\n\nWith OpenAI\u2019s [continuous model upgrades](https://platform.openai.com/docs/models/continuous-model-upgrades), evals allow you to efficiently test model performance for your use cases in a standardized way. Developing a suite of evals customized to your objectives will help you quickly and effectively understand how new models may perform for your use cases. You can also make evals a part of your CI/CD pipeline to make sure you achieve the desired accuracy before deploying.\n\n#### Types of evals\n\nThere are two main ways we can evaluate/grade completions: writing some validation logic in code\nor using the model itself to inspect the answer. We\u2019ll introduce each with some examples.\n\n**Writing logic for answer checking**\n\nThe simplest and most common type of eval has an input and an ideal response or answer. For example,\nwe can have an eval sample where the input is \"What year was Obama elected president for the first\ntime?\" and the ideal answer is \"2008\". We feed the input to a model and get the completion. If the model\nsays \"2008\", it is then graded as correct. We can write a string match to check if the completion includes the phrase \"2008\". If it does, we consider it correct.\n\nConsider another eval where the input is to generate valid JSON: We can write some code that\nattempts to parse the completion as JSON and then considers the completion correct if it is\nparsable.\n\n**Model grading: A two stage process where the model first answers the question, then we ask a\nmodel to look at the response to check if it\u2019s correct.**\n\nConsider an input that asks the model to write a funny joke. The model then generates a\ncompletion. We then create a new input to the model to answer the question: \"Is this following\njoke funny? First reason step by step, then answer yes or no\" that includes the completion.\" We\nfinally consider the original completion correct if the new model completion ends with \"yes\".\n\nModel grading works best with the latest, most powerful models like `GPT-4` and if we give them the ability\nto reason before making a judgment."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: d18906e8-9fec-49e5-96bc-f73d8280b8ed
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"d18906e8-9fec-49e5-96bc-f73d8280b8ed\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"0fe6fb09-2d07-48ba-bfd3-761e92cd4fbb\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"537ca8c81f09315f96e383d0c0153e578d0cc82744ee9d51f3d5dbed2300c0a7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"632506ad-5e74-4f99-a094-8f43502f54a1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0fc158e077f087629de75301c3a8ad9ef30edf09d88ff0e6a6816311e60c4f1a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1711, \"end_char_idx\": 4725, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "You can also make evals a part of your CI/CD pipeline to make sure you achieve the desired accuracy before deploying.\n\n#### Types of evals\n\nThere are two main ways we can evaluate/grade completions: writing some validation logic in code\nor using the model itself to inspect the answer. We\u2019ll introduce each with some examples.\n\n**Writing logic for answer checking**\n\nThe simplest and most common type of eval has an input and an ideal response or answer. For example,\nwe can have an eval sample where the input is \"What year was Obama elected president for the first\ntime?\" and the ideal answer is \"2008\". We feed the input to a model and get the completion. If the model\nsays \"2008\", it is then graded as correct. We can write a string match to check if the completion includes the phrase \"2008\". If it does, we consider it correct.\n\nConsider another eval where the input is to generate valid JSON: We can write some code that\nattempts to parse the completion as JSON and then considers the completion correct if it is\nparsable.\n\n**Model grading: A two stage process where the model first answers the question, then we ask a\nmodel to look at the response to check if it\u2019s correct.**\n\nConsider an input that asks the model to write a funny joke. The model then generates a\ncompletion. We then create a new input to the model to answer the question: \"Is this following\njoke funny? First reason step by step, then answer yes or no\" that includes the completion.\" We\nfinally consider the original completion correct if the new model completion ends with \"yes\".\n\nModel grading works best with the latest, most powerful models like `GPT-4` and if we give them the ability\nto reason before making a judgment. Model grading will have an error rate, so it is important to validate\nthe performance with human evaluation before running the evals at scale. For best results, it makes\nsense to use a different model to do grading from the one that did the completion, like using `GPT-4` to\ngrade `GPT-3.5` answers.\n\n#### OpenAI Eval Templates\n\nIn using evals, we have discovered several \"templates\" that accommodate many different benchmarks. We have implemented these templates in the OpenAI Evals library to simplify the development of new evals. For example, we have defined 2 types of eval templates that can be used out of the box:\n\n* **Basic Eval Templates**: These contain deterministic functions to compare the output to the ideal_answers. In cases where the desired model response has very little variation, such as answering multiple choice questions or simple questions with a straightforward answer, we have found this following templates to be useful.\n\n* **Model-Graded Templates**: These contain functions where an LLM compares the output to the ideal_answers and attempts to judge the accuracy. In cases where the desired model response can contain significant variation, such as answering an open-ended question, we have found that using the model to grade itself is a viable strategy for automated evaluation."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 632506ad-5e74-4f99-a094-8f43502f54a1
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"632506ad-5e74-4f99-a094-8f43502f54a1\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d18906e8-9fec-49e5-96bc-f73d8280b8ed\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"905d57394360a6a570f9440e9aa059fe1ad7fc1453d3b6ba148e6b6aff7d505e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f01ed2a1-b547-436e-9e5b-733a7364292c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"ef15426fa16a5af3c64c0c7b38afeb6eb2d83277228ce31476ef3c04a0a2e8e7\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2892, \"end_char_idx\": 6042, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "**\n\nConsider an input that asks the model to write a funny joke. The model then generates a\ncompletion. We then create a new input to the model to answer the question: \"Is this following\njoke funny? First reason step by step, then answer yes or no\" that includes the completion.\" We\nfinally consider the original completion correct if the new model completion ends with \"yes\".\n\nModel grading works best with the latest, most powerful models like `GPT-4` and if we give them the ability\nto reason before making a judgment. Model grading will have an error rate, so it is important to validate\nthe performance with human evaluation before running the evals at scale. For best results, it makes\nsense to use a different model to do grading from the one that did the completion, like using `GPT-4` to\ngrade `GPT-3.5` answers.\n\n#### OpenAI Eval Templates\n\nIn using evals, we have discovered several \"templates\" that accommodate many different benchmarks. We have implemented these templates in the OpenAI Evals library to simplify the development of new evals. For example, we have defined 2 types of eval templates that can be used out of the box:\n\n* **Basic Eval Templates**: These contain deterministic functions to compare the output to the ideal_answers. In cases where the desired model response has very little variation, such as answering multiple choice questions or simple questions with a straightforward answer, we have found this following templates to be useful.\n\n* **Model-Graded Templates**: These contain functions where an LLM compares the output to the ideal_answers and attempts to judge the accuracy. In cases where the desired model response can contain significant variation, such as answering an open-ended question, we have found that using the model to grade itself is a viable strategy for automated evaluation.\n\n\n### Getting Setup\n\nFirst, go to [github.com/openai/evals](https://github.com/openai/evals), clone the repository with `git clone git@github.com:openai/evals.git` and go through the [setup instructions](https://github.com/openai/evals). \n\nTo run evals later in this notebook, you will need to set up and specify your OpenAI API key. After you obtain an API key, specify it using the `OPENAI_API_KEY` environment variable. \n\nPlease be aware of the costs associated with using the API when running evals.\n\n\n```python\nfrom openai import OpenAI\nimport pandas as pd\n\nclient = OpenAI()\n```\n\n## Building an evaluation for OpenAI Evals framework\n\nAt its core, an eval is a dataset and an eval class that is defined in a YAML file. To start creating an eval, we need\n\n1. The test dataset in the `jsonl` format.\n2. The eval template to be used\n\n### Creating the eval dataset\nLets create a dataset for a use case where we are evaluating the model's ability to generate syntactically correct SQL. In this use case, we have a series of tables that are related to car manufacturing\n\nFirst we will need to create a system prompt that we would like to evaluate. We will pass in instructions for the model as well as an overview of the table structure:\n```\n\"TASK: Answer the following question with syntactically correct SQLite SQL."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: f01ed2a1-b547-436e-9e5b-733a7364292c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"f01ed2a1-b547-436e-9e5b-733a7364292c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"632506ad-5e74-4f99-a094-8f43502f54a1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"5c7e39ab26308c37d9e227f3a766d76bb758b803e4d1de2cf910ebcdd690defb\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3486cb1d-0a76-4f05-a660-3af31c19541f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"098fa12002a403201c2dc3ce999656ad9eb8764397035ddb0aa8c39930dd9500\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4365, \"end_char_idx\": 7321, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "* **Model-Graded Templates**: These contain functions where an LLM compares the output to the ideal_answers and attempts to judge the accuracy. In cases where the desired model response can contain significant variation, such as answering an open-ended question, we have found that using the model to grade itself is a viable strategy for automated evaluation.\n\n\n### Getting Setup\n\nFirst, go to [github.com/openai/evals](https://github.com/openai/evals), clone the repository with `git clone git@github.com:openai/evals.git` and go through the [setup instructions](https://github.com/openai/evals). \n\nTo run evals later in this notebook, you will need to set up and specify your OpenAI API key. After you obtain an API key, specify it using the `OPENAI_API_KEY` environment variable. \n\nPlease be aware of the costs associated with using the API when running evals.\n\n\n```python\nfrom openai import OpenAI\nimport pandas as pd\n\nclient = OpenAI()\n```\n\n## Building an evaluation for OpenAI Evals framework\n\nAt its core, an eval is a dataset and an eval class that is defined in a YAML file. To start creating an eval, we need\n\n1. The test dataset in the `jsonl` format.\n2. The eval template to be used\n\n### Creating the eval dataset\nLets create a dataset for a use case where we are evaluating the model's ability to generate syntactically correct SQL. In this use case, we have a series of tables that are related to car manufacturing\n\nFirst we will need to create a system prompt that we would like to evaluate. We will pass in instructions for the model as well as an overview of the table structure:\n```\n\"TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]\"\n```\n\nFor this prompt, we can ask a specific question:\n```\n\"Q: how many car makers are their in germany?\"\n```\n\nAnd we have an expected answer:\n```\n\"A: SELECT count ( * )  FROM CAR_MAKERS AS T1 JOIN COUNTRIES AS T2 ON T1.Country   =   T2.CountryId WHERE T2.CountryName   =   'germany'\"\n```\n\nThe dataset needs to be in the following format:\n```\n\"input\": [{\"role\": \"system\", \"content\": \"<input prompt>\"}, {\"role\": \"user\", \"content\": <user input>}, \"ideal\": \"correct answer\"]\n```\n\nPutting it all together, we get:\n```\n{\"input\": [{\"role\": \"system\", \"content\": \"TASK: Answer the following question with syntactically correct SQLite SQL."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 3486cb1d-0a76-4f05-a660-3af31c19541f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"3486cb1d-0a76-4f05-a660-3af31c19541f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f01ed2a1-b547-436e-9e5b-733a7364292c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a22aa479214276bff88bc0b01a5c28f8f5a115506ec614d5bba522a4103f9bc3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"beef8862-fca9-49c1-893c-283236f2088b\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"a7179d32f86a295a13789a06d4f3696e44fedf34b12da732b4b2e2b6d6a01d54\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 5873, \"end_char_idx\": 8791, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "We will pass in instructions for the model as well as an overview of the table structure:\n```\n\"TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]\"\n```\n\nFor this prompt, we can ask a specific question:\n```\n\"Q: how many car makers are their in germany?\"\n```\n\nAnd we have an expected answer:\n```\n\"A: SELECT count ( * )  FROM CAR_MAKERS AS T1 JOIN COUNTRIES AS T2 ON T1.Country   =   T2.CountryId WHERE T2.CountryName   =   'germany'\"\n```\n\nThe dataset needs to be in the following format:\n```\n\"input\": [{\"role\": \"system\", \"content\": \"<input prompt>\"}, {\"role\": \"user\", \"content\": <user input>}, \"ideal\": \"correct answer\"]\n```\n\nPutting it all together, we get:\n```\n{\"input\": [{\"role\": \"system\", \"content\": \"TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]\\n\"}, {\"role\": \"system\", \"content\": \"Q: how many car makers are their in germany\"}, \"ideal\": [\"A: SELECT count ( * )  FROM CAR_MAKERS AS T1 JOIN COUNTRIES AS T2 ON T1.Country   =   T2.CountryId WHERE T2.CountryName   =   'germany'\"]}\n```\n\n\nOne way to speed up the process of building eval datasets, is to use `GPT-4` to generate synthetic data\n\n\n```python\n## Use GPT-4 to generate synthetic data\n# Define the system prompt and user input (these should be filled as per the specific use case)\nsystem_prompt = \"\"\"You are a helpful assistant that can ask questions about a database table and write SQL queries to answer the question.\n    A user will pass in a table schema and your job is to return a question answer pairing. The question should relevant to the schema of the table,\n    and you can speculate on its contents."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: beef8862-fca9-49c1-893c-283236f2088b
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"beef8862-fca9-49c1-893c-283236f2088b\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3486cb1d-0a76-4f05-a660-3af31c19541f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"5b894b0bb3e6168ed715d4849c691025380b27770c2aa1f857ca23f7f0f8ad0d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bcfeed51-e06b-4d30-b607-3696dd211b03\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"944dd52fafe091ffa8cacc3ed42efca11af9cbbaae0e63d1dd6f8567fc92b130\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7322, \"end_char_idx\": 9957, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]\\n\"}, {\"role\": \"system\", \"content\": \"Q: how many car makers are their in germany\"}, \"ideal\": [\"A: SELECT count ( * )  FROM CAR_MAKERS AS T1 JOIN COUNTRIES AS T2 ON T1.Country   =   T2.CountryId WHERE T2.CountryName   =   'germany'\"]}\n```\n\n\nOne way to speed up the process of building eval datasets, is to use `GPT-4` to generate synthetic data\n\n\n```python\n## Use GPT-4 to generate synthetic data\n# Define the system prompt and user input (these should be filled as per the specific use case)\nsystem_prompt = \"\"\"You are a helpful assistant that can ask questions about a database table and write SQL queries to answer the question.\n    A user will pass in a table schema and your job is to return a question answer pairing. The question should relevant to the schema of the table,\n    and you can speculate on its contents. You will then have to generate a SQL query to answer the question. Below are some examples of what this should look like.\n\n    Example 1\n    ```````````\n    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\\n\n    Assistant Response:\n    Q: How many visitors have visited the museum with the most staff?\n    A: SELECT count ( * )  FROM VISIT AS T1 JOIN MUSEUM AS T2 ON T1.Museum_ID   =   T2.Museum_ID WHERE T2.Num_of_Staff   =   ( SELECT max ( Num_of_Staff )  FROM MUSEUM ) \n    ```````````\n\n    Example 2\n    ```````````\n    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\\n\n    Assistant Response:\n    Q: What are the names who have a membership level higher than 4?"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: bcfeed51-e06b-4d30-b607-3696dd211b03
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"bcfeed51-e06b-4d30-b607-3696dd211b03\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"beef8862-fca9-49c1-893c-283236f2088b\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1156f85a72660cefb47305108b9aad9a9558605ef193a6928b8e583571df6c04\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a6eb36b6-0d07-4a6c-b609-5b5c4f72dff0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b20c3a7abd7d0dd5b2d836298bfb85360eeb3498b6d85f8a6af5dc2f9a7883e0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 8604, \"end_char_idx\": 10452, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "A user will pass in a table schema and your job is to return a question answer pairing. The question should relevant to the schema of the table,\n    and you can speculate on its contents. You will then have to generate a SQL query to answer the question. Below are some examples of what this should look like.\n\n    Example 1\n    ```````````\n    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\\n\n    Assistant Response:\n    Q: How many visitors have visited the museum with the most staff?\n    A: SELECT count ( * )  FROM VISIT AS T1 JOIN MUSEUM AS T2 ON T1.Museum_ID   =   T2.Museum_ID WHERE T2.Num_of_Staff   =   ( SELECT max ( Num_of_Staff )  FROM MUSEUM ) \n    ```````````\n\n    Example 2\n    ```````````\n    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\\n\n    Assistant Response:\n    Q: What are the names who have a membership level higher than 4?\n    A: SELECT Name   FROM VISITOR AS T1 WHERE T1.Level_of_membership   >   4 \n    ```````````\n\n    Example 3\n    ```````````\n    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\\n\n    Assistant Response:\n    Q: How many tickets of customer id 5?"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: a6eb36b6-0d07-4a6c-b609-5b5c4f72dff0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"a6eb36b6-0d07-4a6c-b609-5b5c4f72dff0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bcfeed51-e06b-4d30-b607-3696dd211b03\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f1ba12eb65fed75490f28773671a6399a87a2ceafa9461fb8774209b9d5b7788\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a80e68ff-bc8a-45f2-8990-5d2c09f664a3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"36b266f7fb045846f1558880bb533a39b93c3c4c5d4c17b4d9c8482a210e7bf2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9347, \"end_char_idx\": 12032, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "A: SELECT count ( * )  FROM VISIT AS T1 JOIN MUSEUM AS T2 ON T1.Museum_ID   =   T2.Museum_ID WHERE T2.Num_of_Staff   =   ( SELECT max ( Num_of_Staff )  FROM MUSEUM ) \n    ```````````\n\n    Example 2\n    ```````````\n    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\\n\n    Assistant Response:\n    Q: What are the names who have a membership level higher than 4?\n    A: SELECT Name   FROM VISITOR AS T1 WHERE T1.Level_of_membership   >   4 \n    ```````````\n\n    Example 3\n    ```````````\n    User input: Table museum, columns = [*,Museum_ID,Name,Num_of_Staff,Open_Year]\\nTable visit, columns = [*,Museum_ID,visitor_ID,Num_of_Ticket,Total_spent]\\nTable visitor, columns = [*,ID,Name,Level_of_membership,Age]\\nForeign_keys = [visit.visitor_ID = visitor.ID,visit.Museum_ID = museum.Museum_ID]\\n\n    Assistant Response:\n    Q: How many tickets of customer id 5?\n    A: SELECT count ( * )  FROM VISIT AS T1 JOIN VISITOR AS T2 ON T1.visitor_ID   =   T2.ID WHERE T2.ID   =   5 \n    ```````````\n    \"\"\"\n\nuser_input = \"Table car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]\"\n\nmessages = [{\n        \"role\": \"system\",\n        \"content\": system_prompt\n    },\n    {\n        \"role\": \"user\",\n        \"content\": user_input\n    }\n]\n\ncompletion = client.chat.completions.create(\n    model=\"gpt-4-turbo-preview\",\n    messages=messages,\n    temperature=0.7,\n    n=5\n)\n\nfor choice in completion.choices:\n    print(choice.message.content + \"\\n\")\n\n```\n\n    Q: What is the average horsepower for cars made in Europe?\n    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'\n    \n    Q: What is the average horsepower for cars made in the USA?"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: a80e68ff-bc8a-45f2-8990-5d2c09f664a3
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"a80e68ff-bc8a-45f2-8990-5d2c09f664a3\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a6eb36b6-0d07-4a6c-b609-5b5c4f72dff0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7aa6fbddd74b995dcbbc36cd7cacbe36480663dbd0d769891335d684c60601ea\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"bcc55f42-c6d4-4502-9044-c450610475f1\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b11368cbd5e605f39b7c48292cbf7cce0831e71d58e51a566ce46279fe839d9b\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11601, \"end_char_idx\": 13747, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'\n    \n    Q: What is the average horsepower for cars made in the USA?\n    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'\n    \n    Q: What is the average horsepower for cars produced in countries from the continent with the id '3'?\n    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.ContId = '3'\n    \n    Q: What is the average horsepower for cars made by makers from Europe?\n    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'\n    \n    Q: What is the average horsepower for cars made in the USA?\n    \n    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'\n    \n\n\nOnce we have the synthetic data, we need to convert it to match the format of the eval dataset.\n\n\n```python\neval_data = []\ninput_prompt = \"TASK: Answer the following question with syntactically correct SQLite SQL."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: bcc55f42-c6d4-4502-9044-c450610475f1
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"bcc55f42-c6d4-4502-9044-c450610475f1\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a80e68ff-bc8a-45f2-8990-5d2c09f664a3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"0298f1e39b55315088adfb4f3f6d6329f29d0dbda7697fa293871cb6fe5cc099\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"08761e6e-9ccb-4c6f-9995-8164f9275aa7\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2b450c9ac0bbdae406b9891bad4de9e27a58db3b165a5b0f405ba1c9eb765058\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12037, \"end_char_idx\": 14979, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'\n    \n    Q: What is the average horsepower for cars produced in countries from the continent with the id '3'?\n    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.ContId = '3'\n    \n    Q: What is the average horsepower for cars made by makers from Europe?\n    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'\n    \n    Q: What is the average horsepower for cars made in the USA?\n    \n    A: SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'\n    \n\n\nOnce we have the synthetic data, we need to convert it to match the format of the eval dataset.\n\n\n```python\neval_data = []\ninput_prompt = \"TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]\"\n\nfor choice in completion.choices:\n    question = choice.message.content.split(\"Q: \")[1].split(\"\\n\")[0]  # Extracting the question\n    answer = choice.message.content.split(\"\\nA: \")[1].split(\"\\n\")[0]  # Extracting the answer\n    eval_data.append({\n        \"input\": [\n            {\"role\": \"system\", \"content\": input_prompt},\n            {\"role\": \"user\", \"content\": question},\n        ],\n        \"ideal\": answer\n    })\n\nfor item in eval_data:\n    print(item)\n```\n\n    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 08761e6e-9ccb-4c6f-9995-8164f9275aa7
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"08761e6e-9ccb-4c6f-9995-8164f9275aa7\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"bcc55f42-c6d4-4502-9044-c450610475f1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"a992beddff463cf1ec106174fd3950b17f72b3e214ebf5571cf0fc9e314b34f7\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"63138cbb-ffbe-4e11-aa33-c0fcd1b706b9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5ac2d4cdb10c785db2011cd46194ded7b48ac4d8a3a3804c2a043b8e3ba77aea\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13534, \"end_char_idx\": 16212, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "Once we have the synthetic data, we need to convert it to match the format of the eval dataset.\n\n\n```python\neval_data = []\ninput_prompt = \"TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]\"\n\nfor choice in completion.choices:\n    question = choice.message.content.split(\"Q: \")[1].split(\"\\n\")[0]  # Extracting the question\n    answer = choice.message.content.split(\"\\nA: \")[1].split(\"\\n\")[0]  # Extracting the answer\n    eval_data.append({\n        \"input\": [\n            {\"role\": \"system\", \"content\": input_prompt},\n            {\"role\": \"user\", \"content\": question},\n        ],\n        \"ideal\": answer\n    })\n\nfor item in eval_data:\n    print(item)\n```\n\n    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in Europe?'}], 'ideal': \"SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'\"}\n    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 63138cbb-ffbe-4e11-aa33-c0fcd1b706b9
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"63138cbb-ffbe-4e11-aa33-c0fcd1b706b9\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"08761e6e-9ccb-4c6f-9995-8164f9275aa7\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"af22064f88d6189a1de45fd281b841f8eded122770294bb2bf062c3ffc04f0d5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"73c69b84-a235-487c-8ccd-2d04978ac16a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0759faa0866775bc7fd049a62d7ebf20e2e96bcb9f6eb5e0036a653002f4b05d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 14980, \"end_char_idx\": 17331, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in Europe?'}], 'ideal': \"SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'\"}\n    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in the USA?'}], 'ideal': \"SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'\"}\n    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 73c69b84-a235-487c-8ccd-2d04978ac16a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"73c69b84-a235-487c-8ccd-2d04978ac16a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"63138cbb-ffbe-4e11-aa33-c0fcd1b706b9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"91a49e13d2e436a9de636889f342c4297e73e988aa96f21915aeeb46fcfad4cd\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4b0b31c9-ff84-4ee4-bfa5-49be265b2ee9\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"feedee288015d6864fb34882ead95610e365e753a7191adbaffdf52914fa3c2a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 16213, \"end_char_idx\": 18598, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in the USA?'}], 'ideal': \"SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'\"}\n    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': \"What is the average horsepower for cars produced in countries from the continent with the id '3'?\"}], 'ideal': \"SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.ContId = '3'\"}\n    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 4b0b31c9-ff84-4ee4-bfa5-49be265b2ee9
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"4b0b31c9-ff84-4ee4-bfa5-49be265b2ee9\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"73c69b84-a235-487c-8ccd-2d04978ac16a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7a7db36fcbabf40e3c9ab0a48dd7e32b80b69e6f0c0dc12d8e142399edf2a48e\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"7d882348-db22-4dba-a023-0421f1a80f00\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2eeb961643986f96eaa89dfd6bcce8bc5c156ce5ebd6299b46993006b5d6c333\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17332, \"end_char_idx\": 19843, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': \"What is the average horsepower for cars produced in countries from the continent with the id '3'?\"}], 'ideal': \"SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.ContId = '3'\"}\n    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made by makers from Europe?'}], 'ideal': \"SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'\"}\n    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 7d882348-db22-4dba-a023-0421f1a80f00
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"7d882348-db22-4dba-a023-0421f1a80f00\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4b0b31c9-ff84-4ee4-bfa5-49be265b2ee9\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"57b20abc23badf1e99109c6c04a4e868b1c76d6b1798ba8385e8c72d925af632\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"35b4733b-4b04-4057-9263-0e50bd98e591\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f4101e8d4f54af493be4f6b69555c21d9f985f5a3be8fbfb485ad596ca6c88e0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 18599, \"end_char_idx\": 21288, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made by makers from Europe?'}], 'ideal': \"SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN model_list ON car_names.Model = model_list.Model JOIN car_makers ON model_list.Maker = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId JOIN continents ON countries.Continent = continents.ContId WHERE continents.Continent = 'Europe'\"}\n    {'input': [{'role': 'system', 'content': 'TASK: Answer the following question with syntactically correct SQLite SQL. The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in the USA?'}], 'ideal': \"SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'\"}\n\n\nNext we need to create the eval registry to run it in the framework.\n\nThe evals framework requires a `.yaml` file structured with the following properties:\n* `id` - An identifier for your eval\n* `description` - A short description of your eval\n* `disclaimer` - An additional notes about your eval\n* `metrics` - There are three types of eval metrics we can choose from: match, includes, fuzzyMatch\n\nFor our eval, we will configure the following:"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 35b4733b-4b04-4057-9263-0e50bd98e591
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"35b4733b-4b04-4057-9263-0e50bd98e591\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"7d882348-db22-4dba-a023-0421f1a80f00\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"7fc099c184db62dbf6840adac5cdcfb61fb39c5fa35664ab2f29c2f8255cdf4a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"acf289f3-836f-40ab-a6e9-8bb84f06e4f4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"056e66da09a52a90c9b9f0370fe26e54477be31ae1e67d339493a333e7063741\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 19844, \"end_char_idx\": 22364, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "The SQL should be correct and be in context of the previous question-answer pairs.\\nTable car_makers, columns = [*,Id,Maker,FullName,Country]\\nTable car_names, columns = [*,MakeId,Model,Make]\\nTable cars_data, columns = [*,Id,MPG,Cylinders,Edispl,Horsepower,Weight,Accelerate,Year]\\nTable continents, columns = [*,ContId,Continent]\\nTable countries, columns = [*,CountryId,CountryName,Continent]\\nTable model_list, columns = [*,ModelId,Maker,Model]\\nForeign_keys = [countries.Continent = continents.ContId,car_makers.Country = countries.CountryId,model_list.Maker = car_makers.Id,car_names.Model = model_list.Model,cars_data.Id = car_names.MakeId]'}, {'role': 'user', 'content': 'What is the average horsepower for cars made in the USA?'}], 'ideal': \"SELECT AVG(cars_data.Horsepower) FROM cars_data JOIN car_names ON cars_data.Id = car_names.MakeId JOIN car_makers ON car_names.MakeId = car_makers.Id JOIN countries ON car_makers.Country = countries.CountryId WHERE countries.CountryName = 'USA'\"}\n\n\nNext we need to create the eval registry to run it in the framework.\n\nThe evals framework requires a `.yaml` file structured with the following properties:\n* `id` - An identifier for your eval\n* `description` - A short description of your eval\n* `disclaimer` - An additional notes about your eval\n* `metrics` - There are three types of eval metrics we can choose from: match, includes, fuzzyMatch\n\nFor our eval, we will configure the following:\n\n\n```python\n\"\"\"\nspider-sql:\n  id: spider-sql.dev.v0\n  metrics: [accuracy]\n  description: Eval that scores SQL code from 194 examples in the Spider Text-to-SQL test dataset. The problems are selected by taking the first 10 problems for each database that appears in the test set.\n    Yu, Tao, et al. \\\"Spider; A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task.\\\" Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, https://doi.org/10.18653/v1/d18-1425.\n  disclaimer: Problems are solved zero-shot with no prompting other than the schema; performance may improve with training examples, fine tuning, or a different schema format. Evaluation is currently done through model-grading, where SQL code is not actually executed; the model may judge correct SQL to be incorrect, or vice-versa.\nspider-sql.dev.v0:\n  class: evals.elsuite.modelgraded.classify:ModelBasedClassify\n  args:\n    samples_jsonl: sql/spider_sql.jsonl\n    eval_type: cot_classify\n    modelgraded_spec: sql\n  \"\"\"\"\"\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: acf289f3-836f-40ab-a6e9-8bb84f06e4f4
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"acf289f3-836f-40ab-a6e9-8bb84f06e4f4\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"35b4733b-4b04-4057-9263-0e50bd98e591\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bdcca48abdd3168bf78075c4f0aa65ff101e80cc6e218b5fc4bc45da8212ecd9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a42427ce-f8ab-47a5-bed2-61088829b0cc\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"916cae0985b0e2eb687e7d89eed355ea3537abfedd50a5ef0e6545f213fbf667\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 20844, \"end_char_idx\": 23437, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "Next we need to create the eval registry to run it in the framework.\n\nThe evals framework requires a `.yaml` file structured with the following properties:\n* `id` - An identifier for your eval\n* `description` - A short description of your eval\n* `disclaimer` - An additional notes about your eval\n* `metrics` - There are three types of eval metrics we can choose from: match, includes, fuzzyMatch\n\nFor our eval, we will configure the following:\n\n\n```python\n\"\"\"\nspider-sql:\n  id: spider-sql.dev.v0\n  metrics: [accuracy]\n  description: Eval that scores SQL code from 194 examples in the Spider Text-to-SQL test dataset. The problems are selected by taking the first 10 problems for each database that appears in the test set.\n    Yu, Tao, et al. \\\"Spider; A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task.\\\" Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, https://doi.org/10.18653/v1/d18-1425.\n  disclaimer: Problems are solved zero-shot with no prompting other than the schema; performance may improve with training examples, fine tuning, or a different schema format. Evaluation is currently done through model-grading, where SQL code is not actually executed; the model may judge correct SQL to be incorrect, or vice-versa.\nspider-sql.dev.v0:\n  class: evals.elsuite.modelgraded.classify:ModelBasedClassify\n  args:\n    samples_jsonl: sql/spider_sql.jsonl\n    eval_type: cot_classify\n    modelgraded_spec: sql\n  \"\"\"\"\"\n```\n\n\n\n\n    '\\nspider-sql:\\n  id: spider-sql.dev.v0\\n  metrics: [accuracy]\\n  description: Eval that scores SQL code from 194 examples in the Spider Text-to-SQL test dataset. The problems are selected by taking the first 10 problems for each database that appears in the test set.\\n    Yu, Tao, et al. \"Spider; A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task.\" Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, https://doi.org/10.18653/v1/d18-1425.\\n  disclaimer: Problems are solved zero-shot with no prompting other than the schema; performance may improve with training examples, fine tuning, or a different schema format. Evaluation is currently done through model-grading, where SQL code is not actually executed; the model may judge correct SQL to be incorrect, or vice-versa.\\nspider-sql.dev.v0:\\n  class: evals.elsuite.modelgraded.classify:ModelBasedClassify\\n  args:\\n    samples_jsonl: sql/spider_sql.jsonl\\n    eval_type: cot_classify\\n    modelgraded_spec: sql\\n  '"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: a42427ce-f8ab-47a5-bed2-61088829b0cc
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"a42427ce-f8ab-47a5-bed2-61088829b0cc\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"acf289f3-836f-40ab-a6e9-8bb84f06e4f4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"4eabd8563c4fb11538fd4ae9f48eb84488cccb189104fd0bbdcd62add9937b8d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d7adf874-e18b-4ba0-9e45-c07214616e38\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c88731d5079dd1904859263f84017b8c722a22fd922d538a3b3b88189dd88dbc\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 22373, \"end_char_idx\": 25130, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "'\\nspider-sql:\\n  id: spider-sql.dev.v0\\n  metrics: [accuracy]\\n  description: Eval that scores SQL code from 194 examples in the Spider Text-to-SQL test dataset. The problems are selected by taking the first 10 problems for each database that appears in the test set.\\n    Yu, Tao, et al. \"Spider; A Large-Scale Human-Labeled Dataset for Complex and Cross-Domain Semantic Parsing and Text-to-SQL Task.\" Proceedings of the 2018 Conference on Empirical Methods in Natural Language Processing, 2018, https://doi.org/10.18653/v1/d18-1425.\\n  disclaimer: Problems are solved zero-shot with no prompting other than the schema; performance may improve with training examples, fine tuning, or a different schema format. Evaluation is currently done through model-grading, where SQL code is not actually executed; the model may judge correct SQL to be incorrect, or vice-versa.\\nspider-sql.dev.v0:\\n  class: evals.elsuite.modelgraded.classify:ModelBasedClassify\\n  args:\\n    samples_jsonl: sql/spider_sql.jsonl\\n    eval_type: cot_classify\\n    modelgraded_spec: sql\\n  '\n\n\n\n## Running an evaluation\n\nWe can run this eval using the `oaieval` CLI. To get setup, install the library: `pip install .` (if you are running the [OpenAI Evals library](github.com/openai/evals) locally) or `pip install oaieval` if you are running an existing eval.\n\nThen, run the eval using the CLI: `oaieval gpt-3.5-turbo spider-sql`\n\nThis command expects a model name and an eval set name. Note that we provide two command line interfaces (CLIs): `oaieval` for running a single eval and `oaievalset` for running a set of evals. The valid eval names are specified in the YAML files under `evals/registry/evals` and their corresponding implementations can be found in `evals/elsuite`.\n\n\n```python\n!pip install evals --quiet\n```\n\nThe `oaieval` CLI can accept various flags to modify the default behavior. You can run `oaieval --help` to see a full list of CLI options. \n\nAfter running that command, you\u2019ll see the final report of accuracy printed to the console, as well as a file path to a temporary file that contains the full report.\n\nThese CLIs can accept various flags to modify their default behavior. You can run `oaieval --help` to see a full list of CLI options. \n\n`oaieval` will search for the `spider-sql` eval YAML file in the `evals/registry/evals` directory, following the format specified in cell 4 above. The path to the eval dataset is specified in the eval YAML file under the args: parameter as `samples_jsonl: sql/spider_sql.jsonl`, with the file content in JSONL format (as generated in step 3 above).\n\nAfter running that command, you\u2019ll see the final report of accuracy printed to the console, as well as a file path to a temporary file that contains the full report."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: d7adf874-e18b-4ba0-9e45-c07214616e38
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"d7adf874-e18b-4ba0-9e45-c07214616e38\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a42427ce-f8ab-47a5-bed2-61088829b0cc\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"3abe7e75fce9b3ba125b30dc33002e5e725a066d56accd80e5cdf2b94636d971\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d7af657f-1df9-494f-a5fd-19c61b3705da\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0cafe37e6073396b41848498fbb85c6ebd20393fd8494a86f449b6b5ace42e5c\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 24129, \"end_char_idx\": 26532, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "```python\n!pip install evals --quiet\n```\n\nThe `oaieval` CLI can accept various flags to modify the default behavior. You can run `oaieval --help` to see a full list of CLI options. \n\nAfter running that command, you\u2019ll see the final report of accuracy printed to the console, as well as a file path to a temporary file that contains the full report.\n\nThese CLIs can accept various flags to modify their default behavior. You can run `oaieval --help` to see a full list of CLI options. \n\n`oaieval` will search for the `spider-sql` eval YAML file in the `evals/registry/evals` directory, following the format specified in cell 4 above. The path to the eval dataset is specified in the eval YAML file under the args: parameter as `samples_jsonl: sql/spider_sql.jsonl`, with the file content in JSONL format (as generated in step 3 above).\n\nAfter running that command, you\u2019ll see the final report of accuracy printed to the console, as well as a file path to a temporary file that contains the full report.\n\n\n```python\n!oaieval gpt-3.5-turbo spider-sql --max_samples 25\n```\n\n    [2024-03-26 19:44:39,836] [registry.py:257] Loading registry from /Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry/evals\n    [2024-03-26 19:44:43,623] [registry.py:257] Loading registry from /Users/shyamal/.evals/evals\n    [2024-03-26 19:44:43,635] [oaieval.py:189] \u001b[1;35mRun started: 240327024443FACXGMKA\u001b[0m\n    [2024-03-26 19:44:43,663] [registry.py:257] Loading registry from /Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry/modelgraded\n    [2024-03-26 19:44:43,851] [registry.py:257] Loading registry from /Users/shyamal/.evals/modelgraded\n    [2024-03-26 19:44:43,853] [data.py:90] Fetching /Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry/data/sql/spider_sql.jsonl\n    [2024-03-26 19:44:43,878] [eval.py:36] Evaluating 25 samples\n    [2024-03-26 19:44:43,952] [eval.py:144] Running in threaded mode with 10 threads!\n      0%|                                                    | 0/25 [00:00<?, ?it/s][2024-03-26 19:44:44,810] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:44,829] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:44,991] [_client.py:1026] HTTP Request: POST https://api."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: d7af657f-1df9-494f-a5fd-19c61b3705da
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"d7af657f-1df9-494f-a5fd-19c61b3705da\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d7adf874-e18b-4ba0-9e45-c07214616e38\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"6a0c91f89744370804d5b7369cf0dece63cb854b4cf20e5e3c5e5d523819695c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"3a95efac-4e14-4638-9f68-7cc26b22212c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"081235968d0e797ca990c2399a521982c7e437b5f730de75a4af4662fb794ee5\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 26123, \"end_char_idx\": 28011, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "0%|                                                    | 0/25 [00:00<?, ?it/s][2024-03-26 19:44:44,810] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:44,829] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:44,991] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:45,090] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:45,145] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:45,971] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:46,040] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:46,069] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:46,378] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:46,587] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:47,412] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n      4%|\u2588\u258a                                          | 1/25 [00:03<01:23,  3.46s/it][2024-03-26 19:44:47,714] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n      8%|\u2588\u2588\u2588\u258c                                        | 2/25 [00:03<00:36,  1.60s/it][2024-03-26 19:44:47,947] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 3a95efac-4e14-4638-9f68-7cc26b22212c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"3a95efac-4e14-4638-9f68-7cc26b22212c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d7af657f-1df9-494f-a5fd-19c61b3705da\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"4e201e4aeb421259ff970045e9646593dde3dc36740b618b3f59b05230c062c2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"c46feab2-62cc-4562-adee-73cfd12aba59\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"db366f425162c8b26059e702eeec766dcdd8d5f223f0f4222a143afb335fa1f8\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 27044, \"end_char_idx\": 28926, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:46,069] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:46,378] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:46,587] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:47,412] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n      4%|\u2588\u258a                                          | 1/25 [00:03<01:23,  3.46s/it][2024-03-26 19:44:47,714] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n      8%|\u2588\u2588\u2588\u258c                                        | 2/25 [00:03<00:36,  1.60s/it][2024-03-26 19:44:47,947] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     12%|\u2588\u2588\u2588\u2588\u2588\u258e                                      | 3/25 [00:03<00:21,  1.02it/s][2024-03-26 19:44:48,413] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:48,643] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     16%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588                                     | 4/25 [00:04<00:18,  1.15it/s][2024-03-26 19:44:48,909] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     20%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258a                                   | 5/25 [00:04<00:12,  1.54it/s][2024-03-26 19:44:49,131] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:49,500] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:49,"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: c46feab2-62cc-4562-adee-73cfd12aba59
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"c46feab2-62cc-4562-adee-73cfd12aba59\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"3a95efac-4e14-4638-9f68-7cc26b22212c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"11bc74c74de8a60bc51a735c26fc2d87a2c2a0cbf654a1238a09fe582840156d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"25c61180-24dc-425e-b952-3086abc3773d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"57a75db204086cf2fa4fe1c71627c2c33d8915580d7201ceb300d6824ea63b00\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 27972, \"end_char_idx\": 29854, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     12%|\u2588\u2588\u2588\u2588\u2588\u258e                                      | 3/25 [00:03<00:21,  1.02it/s][2024-03-26 19:44:48,413] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:48,643] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     16%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588                                     | 4/25 [00:04<00:18,  1.15it/s][2024-03-26 19:44:48,909] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     20%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258a                                   | 5/25 [00:04<00:12,  1.54it/s][2024-03-26 19:44:49,131] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:49,500] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:49,530] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     24%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258c                                 | 6/25 [00:05<00:12,  1.56it/s][2024-03-26 19:44:49,962] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:49,964] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:49,967] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     28%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258e                               | 7/25 [00:06<00:10,  1.73it/s][2024-03-26 19:44:50,577] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:50,602] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:50,"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 25c61180-24dc-425e-b952-3086abc3773d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"25c61180-24dc-425e-b952-3086abc3773d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c46feab2-62cc-4562-adee-73cfd12aba59\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"762005b84d436367f27d4e21fb212eb0369f33395775e2883ec20685a8e39bea\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"efee0efc-c3ca-4e1f-af3d-a7ff5699c1da\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"51de4367ae40c6512905d30f1b3b99adce02a0a291ab512d027c14ca99febed4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 28891, \"end_char_idx\": 30747, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "1 200 OK\"\n    [2024-03-26 19:44:49,530] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     24%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258c                                 | 6/25 [00:05<00:12,  1.56it/s][2024-03-26 19:44:49,962] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:49,964] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:49,967] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     28%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258e                               | 7/25 [00:06<00:10,  1.73it/s][2024-03-26 19:44:50,577] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:50,602] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:50,634] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:50,862] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:51,503] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:51,608] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     40%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258f                         | 10/25 [00:07<00:08,  1.79it/s][2024-03-26 19:44:51,801] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     44%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2589                        | 11/25 [00:07<00:06,  2.09it/s][2024-03-26 19:44:51,856] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: efee0efc-c3ca-4e1f-af3d-a7ff5699c1da
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"efee0efc-c3ca-4e1f-af3d-a7ff5699c1da\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"25c61180-24dc-425e-b952-3086abc3773d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1e0ad2110282d18f1ef19a1b658c3b17fc93e865758d982f410d76bfaa1abf86\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"b3538639-b016-443a-bf70-a03905cd697f\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f8306eec8985f16e0a9a4ee47cd9a2ccd16d9aff13d9cbd82a093be64dd1e450\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 29780, \"end_char_idx\": 31636, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:50,634] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:50,862] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:51,503] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:51,608] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     40%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258f                         | 10/25 [00:07<00:08,  1.79it/s][2024-03-26 19:44:51,801] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     44%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2589                        | 11/25 [00:07<00:06,  2.09it/s][2024-03-26 19:44:51,856] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:51,969] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:52,227] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     52%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258e                    | 13/25 [00:08<00:04,  2.65it/s][2024-03-26 19:44:52,450] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:52,526] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:52,615] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     56%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588                   | 14/25 [00:08<00:04,  2.64it/s][2024-03-26 19:44:52,625] [_client.py:1026] HTTP Request: POST https://api."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: b3538639-b016-443a-bf70-a03905cd697f
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"b3538639-b016-443a-bf70-a03905cd697f\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"efee0efc-c3ca-4e1f-af3d-a7ff5699c1da\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f213c1990915818ee1b7846887fb2acc14111db315fade72c90d31e269139104\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cdb21433-ede9-4ffc-baff-10326ec39121\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"7f901a8ee5ed0a6397597abe944a5b433ddd0c7cc86e71208e4597435b045557\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 30668, \"end_char_idx\": 32510, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:51,969] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:52,227] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     52%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258e                    | 13/25 [00:08<00:04,  2.65it/s][2024-03-26 19:44:52,450] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:52,526] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:52,615] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     56%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588                   | 14/25 [00:08<00:04,  2.64it/s][2024-03-26 19:44:52,625] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:52,777] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:53,653] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     60%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258a                 | 15/25 [00:09<00:05,  1.87it/s][2024-03-26 19:44:53,670] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:54,028] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     68%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258f             | 17/25 [00:10<00:03,  2.54it/s][2024-03-26 19:44:54,388] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:54,"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: cdb21433-ede9-4ffc-baff-10326ec39121
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"cdb21433-ede9-4ffc-baff-10326ec39121\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"b3538639-b016-443a-bf70-a03905cd697f\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"4e970975626189dbad51c4bfe7cfe8c17a8f8368d8456861aa9d422f8ef4b7d9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"97d7022d-a08a-4380-8ffd-58a06ea641f4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"202c11a22b026984c25fa6e1b160ff245da11a8bbfe0356ecfc1b605cd17d31d\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 31582, \"end_char_idx\": 33355, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "625] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:52,777] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:53,653] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     60%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258a                 | 15/25 [00:09<00:05,  1.87it/s][2024-03-26 19:44:53,670] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:54,028] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     68%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258f             | 17/25 [00:10<00:03,  2.54it/s][2024-03-26 19:44:54,388] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:54,396] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     72%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2589            | 18/25 [00:10<00:02,  2.58it/s][2024-03-26 19:44:54,529] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:54,585] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     76%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258b          | 19/25 [00:10<00:02,  2.94it/s][2024-03-26 19:44:54,980] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     80%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258d        | 20/25 [00:11<00:01,  2.82it/s][2024-03-26 19:44:55,152] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 97d7022d-a08a-4380-8ffd-58a06ea641f4
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"97d7022d-a08a-4380-8ffd-58a06ea641f4\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cdb21433-ede9-4ffc-baff-10326ec39121\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"eadca77773b6b4184b5aa4a89f3ce3b899a423f3d121a5d1f3223800a3d3d1f0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"db91f878-68de-4e2d-8da1-70b40df09eed\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b5e494782afe4346ea938d8e0449b6bfb661e6650f10622756d2e34a12b0ebc0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 32436, \"end_char_idx\": 34187, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:54,396] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     72%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2589            | 18/25 [00:10<00:02,  2.58it/s][2024-03-26 19:44:54,529] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    [2024-03-26 19:44:54,585] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     76%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258b          | 19/25 [00:10<00:02,  2.94it/s][2024-03-26 19:44:54,980] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     80%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258d        | 20/25 [00:11<00:01,  2.82it/s][2024-03-26 19:44:55,152] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     84%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588       | 21/25 [00:11<00:01,  3.27it/s][2024-03-26 19:44:56,420] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     88%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258a     | 22/25 [00:12<00:01,  1.75it/s][2024-03-26 19:44:56,984] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     92%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258c   | 23/25 [00:13<00:01,  1.76it/s][2024-03-26 19:44:57,370] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     96%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258e | 24/25 [00:13<00:00,  1.94it/s][2024-03-26 19:44:59,589] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: db91f878-68de-4e2d-8da1-70b40df09eed
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"db91f878-68de-4e2d-8da1-70b40df09eed\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"97d7022d-a08a-4380-8ffd-58a06ea641f4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"26955da41e73b6e743c193312851cde36ff65e60a40d5b4bdc8a1b91ea442ef3\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"46bcd003-beff-4de8-a3cf-bebf5851a848\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d68329986a0dde70d712bd8b52a4af48e59c7b1f51214cff0084da94d46b8da1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 33262, \"end_char_idx\": 34892, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "152] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     84%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588       | 21/25 [00:11<00:01,  3.27it/s][2024-03-26 19:44:56,420] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     88%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258a     | 22/25 [00:12<00:01,  1.75it/s][2024-03-26 19:44:56,984] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     92%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258c   | 23/25 [00:13<00:01,  1.76it/s][2024-03-26 19:44:57,370] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n     96%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u258e | 24/25 [00:13<00:00,  1.94it/s][2024-03-26 19:44:59,589] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 25/25 [00:15<00:00,  1.60it/s]\n    [2024-03-26 19:44:59,607] [record.py:360] Final report: {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}. Logged to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl\n    [2024-03-26 19:44:59,608] [oaieval.py:229] Final report:\n    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Correct: 20\n    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Incorrect: 5\n    [2024-03-26 19:44:59,608] [oaieval.py:231] score: 0.8\n    [2024-03-26 19:44:59,640] [record.py:349] Logged 75 rows of events to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl: insert_time=27.915ms"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 46bcd003-beff-4de8-a3cf-bebf5851a848
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"46bcd003-beff-4de8-a3cf-bebf5851a848\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"db91f878-68de-4e2d-8da1-70b40df09eed\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"5a8096b3fa80a96a039062b2f6bde8190acabef115f512d5974b9ebaf6c1b341\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d81cc2bf-ebe8-4130-b7a3-f3f1075c6bb4\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c76d41267ae2a9651a44e6c00718bfcfba021096ff7b65c123e3641062e87d15\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 34064, \"end_char_idx\": 35381, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "1.94it/s][2024-03-26 19:44:59,589] [_client.py:1026] HTTP Request: POST https://api.openai.com/v1/chat/completions \"HTTP/1.1 200 OK\"\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 25/25 [00:15<00:00,  1.60it/s]\n    [2024-03-26 19:44:59,607] [record.py:360] Final report: {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}. Logged to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl\n    [2024-03-26 19:44:59,608] [oaieval.py:229] Final report:\n    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Correct: 20\n    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Incorrect: 5\n    [2024-03-26 19:44:59,608] [oaieval.py:231] score: 0.8\n    [2024-03-26 19:44:59,640] [record.py:349] Logged 75 rows of events to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl: insert_time=27.915ms\n\n\n`oaievalset` expects a model name and an eval set name, for which the valid options are specified in the YAML files under `evals/registry/eval_sets`.\n\n### Going through eval logs\n\nThe eval logs are located at `/tmp/evallogs` and different log files are created for each evaluation run. \n\n\n```python\nlog_name = '240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl' # \"EDIT THIS\" - copy from above\nevents = f\"/tmp/evallogs/{log_name}\"\ndisplay(pd.read_json(events, lines=True).head(5))\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: d81cc2bf-ebe8-4130-b7a3-f3f1075c6bb4
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"d81cc2bf-ebe8-4130-b7a3-f3f1075c6bb4\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"46bcd003-beff-4de8-a3cf-bebf5851a848\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"6d6868ed6d5f4dce2a10ed612e9518d2db62e11f1ed1a951080b9e43892bde08\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"eb2c416b-c8b8-4577-b793-ad493068c9d0\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"df8d0a1d4d691c842a849196abc154b427afc611a470efec6e5c4ec9d7ad1557\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 34307, \"end_char_idx\": 37478, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "607] [record.py:360] Final report: {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}. Logged to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl\n    [2024-03-26 19:44:59,608] [oaieval.py:229] Final report:\n    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Correct: 20\n    [2024-03-26 19:44:59,608] [oaieval.py:231] counts/Incorrect: 5\n    [2024-03-26 19:44:59,608] [oaieval.py:231] score: 0.8\n    [2024-03-26 19:44:59,640] [record.py:349] Logged 75 rows of events to /tmp/evallogs/240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl: insert_time=27.915ms\n\n\n`oaievalset` expects a model name and an eval set name, for which the valid options are specified in the YAML files under `evals/registry/eval_sets`.\n\n### Going through eval logs\n\nThe eval logs are located at `/tmp/evallogs` and different log files are created for each evaluation run. \n\n\n```python\nlog_name = '240327024443FACXGMKA_gpt-3.5-turbo_spider-sql.jsonl' # \"EDIT THIS\" - copy from above\nevents = f\"/tmp/evallogs/{log_name}\"\ndisplay(pd.read_json(events, lines=True).head(5))\n```\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>spec</th>\n      <th>final_report</th>\n      <th>run_id</th>\n      <th>event_id</th>\n      <th>sample_id</th>\n      <th>type</th>\n      <th>data</th>\n      <th>created_by</th>\n      <th>created_at</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>{'completion_fns': ['gpt-3.5-turbo'], 'eval_name': 'spider-sql.dev.v0', 'base_eval': 'spider-sql', 'split': 'dev', 'run_config': {'completion_fns': ['gpt-3.5-turbo'], 'eval_spec': {'cls': 'evals.elsuite.modelgraded.classify:ModelBasedClassify', 'registry_path': '/Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry', 'args': {'samples_jsonl': 'sql/spider_sql.jsonl', 'eval_type': 'cot_classify', 'modelgraded_spec': 'sql'}, 'key': 'spider-sql.dev.v0', 'group': 'sql'}, 'seed': 20220722, 'max_samples': 25, 'command': '/Users/shyamal/.virtualenvs/openai/bin/oaieval gpt-3.5-turbo spider-sql --max_samples 25', 'initial_settings': {'visible': False}}, 'created_by': '', 'run_id': '240327024443FACXGMKA', 'created_at': '2024-03-27 02:44:43.626043'}</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaT</td>\n    </tr>\n    <tr>\n      <th>1</th>\n      <td>NaN</td>\n      <td>{'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>NaT</td>\n    </tr>\n    <tr>\n      <th>2</th>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>240327024443FACXGMKA</td>\n      <td>0.0</td>\n      <td>spider-sql.dev.88</td>\n      <td>sampling</td>\n      <td>{'prompt': [{'content': 'Answer the following question with syntactically correct SQLite SQL."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: eb2c416b-c8b8-4577-b793-ad493068c9d0
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"eb2c416b-c8b8-4577-b793-ad493068c9d0\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d81cc2bf-ebe8-4130-b7a3-f3f1075c6bb4\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"e377850c57244f1082dc511a930d95783028dd3979b026f1fd8c563b167c8f68\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"44258223-6eb2-4aeb-97fe-699d7b00d29c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8fd03c3cad930b21286a3d08a17e04d228d35937cc112ef200190f164b83a650\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 37479, \"end_char_idx\": 40052, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\nTable: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)\nTable: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)\n\nQuestion: Find the average rank of winners in all matches.\n', 'role': 'system'}], 'sampled': ['SELECT AVG(winner_rank) AS average_rank_of_winners\nFROM matches;']}</td>\n      <td></td>\n      <td>2024-03-27 02:44:44.821110+00:00</td>\n    </tr>\n    <tr>\n      <th>3</th>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>240327024443FACXGMKA</td>\n      <td>1.0</td>\n      <td>spider-sql.dev.82</td>\n      <td>sampling</td>\n      <td>{'prompt': [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\nTable: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)\nTable: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)\n\nQuestion: Find the total number of matches."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 44258223-6eb2-4aeb-97fe-699d7b00d29c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"44258223-6eb2-4aeb-97fe-699d7b00d29c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"eb2c416b-c8b8-4577-b793-ad493068c9d0\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"1221640b813198a828839ff7b00d3e1e2e53184463941c37f7648c94ad1f1e42\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ad50b65a-aeaf-4dc8-b898-212aeb8a4d2a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"0ec710541b81819af80b8e7dbcb45936042d6b4ae46b4f832dc9a5bc72cec976\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 39005, \"end_char_idx\": 41789, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\nTable: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)\nTable: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)\n\nQuestion: Find the total number of matches.\n', 'role': 'system'}], 'sampled': ['SELECT COUNT(*) AS total_matches\nFROM matches;']}</td>\n      <td></td>\n      <td>2024-03-27 02:44:44.831848+00:00</td>\n    </tr>\n    <tr>\n      <th>4</th>\n      <td>NaN</td>\n      <td>NaN</td>\n      <td>240327024443FACXGMKA</td>\n      <td>2.0</td>\n      <td>spider-sql.dev.25</td>\n      <td>sampling</td>\n      <td>{'prompt': [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: continents. Columns: ContId (number), Continent (text)\nTable: countries. Columns: CountryId (number), CountryName (text), Continent (number)\nTable: car_makers. Columns: Id (number), Maker (text), FullName (text), Country (text)\nTable: model_list. Columns: ModelId (number), Maker (number), Model (text)\nTable: car_names. Columns: MakeId (number), Model (text), Make (text)\nTable: cars_data. Columns: Id (number), MPG (text), Cylinders (number), Edispl (number), Horsepower (text), Weight (number), Accelerate (number), Year (number)\n\nQuestion: How many countries exist?\n', 'role': 'system'}], 'sampled': ['SELECT COUNT(*) AS TotalCountries\nFROM countries;']}</td>\n      <td></td>\n      <td>2024-03-27 02:44:44.996647+00:00</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n```python\n# processing the log events generated by oaieval\n\nwith open(events, \"r\") as f:\n    events_df = pd.read_json(f, lines=True)\n```\n\nThis file will contain structured logs of the evaluation. The first entry provides a detailed specification of the evaluation, including the completion functions, evaluation name, run configuration, creator\u2019s name, run ID, and creation timestamp.\n\n\n```python\ndisplay(events_df.iloc[0].spec)\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: ad50b65a-aeaf-4dc8-b898-212aeb8a4d2a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"ad50b65a-aeaf-4dc8-b898-212aeb8a4d2a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"44258223-6eb2-4aeb-97fe-699d7b00d29c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"373e3cd0da767d40d7e448029a7c867ea785cf333482471108c74f8911b20d04\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d3f1a823-3ccb-4148-8319-3f0212f5287d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2ec1f596b22aaf971bfc5a8f8ff999c198dc0be20e825f80bc68689950cad9b3\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 40498, \"end_char_idx\": 43047, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "Be creative but the SQL must be correct.\nUse only the following tables and columns:\nTable: continents. Columns: ContId (number), Continent (text)\nTable: countries. Columns: CountryId (number), CountryName (text), Continent (number)\nTable: car_makers. Columns: Id (number), Maker (text), FullName (text), Country (text)\nTable: model_list. Columns: ModelId (number), Maker (number), Model (text)\nTable: car_names. Columns: MakeId (number), Model (text), Make (text)\nTable: cars_data. Columns: Id (number), MPG (text), Cylinders (number), Edispl (number), Horsepower (text), Weight (number), Accelerate (number), Year (number)\n\nQuestion: How many countries exist?\n', 'role': 'system'}], 'sampled': ['SELECT COUNT(*) AS TotalCountries\nFROM countries;']}</td>\n      <td></td>\n      <td>2024-03-27 02:44:44.996647+00:00</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n```python\n# processing the log events generated by oaieval\n\nwith open(events, \"r\") as f:\n    events_df = pd.read_json(f, lines=True)\n```\n\nThis file will contain structured logs of the evaluation. The first entry provides a detailed specification of the evaluation, including the completion functions, evaluation name, run configuration, creator\u2019s name, run ID, and creation timestamp.\n\n\n```python\ndisplay(events_df.iloc[0].spec)\n```\n\n\n    {'completion_fns': ['gpt-3.5-turbo'],\n     'eval_name': 'spider-sql.dev.v0',\n     'base_eval': 'spider-sql',\n     'split': 'dev',\n     'run_config': {'completion_fns': ['gpt-3.5-turbo'],\n      'eval_spec': {'cls': 'evals.elsuite.modelgraded.classify:ModelBasedClassify',\n       'registry_path': '/Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry',\n       'args': {'samples_jsonl': 'sql/spider_sql.jsonl',\n        'eval_type': 'cot_classify',\n        'modelgraded_spec': 'sql'},\n       'key': 'spider-sql.dev.v0',\n       'group': 'sql'},\n      'seed': 20220722,\n      'max_samples': 25,\n      'command': '/Users/shyamal/.virtualenvs/openai/bin/oaieval gpt-3.5-turbo spider-sql --max_samples 25',\n      'initial_settings': {'visible': False}},\n     'created_by': '',\n     'run_id': '240327024443FACXGMKA',\n     'created_at': '2024-03-27 02:44:43.626043'}\n\n\nLet's also look at the entry which provides the final report of the evaluation.\n\n\n```python\ndisplay(events_df.dropna(subset=['final_report']).iloc[0]['final_report'])\n```\n\n\n    {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}\n\n\nWe can also review individual evaluation events that provide specific samples (`sample_id`), results, event types, and metadata."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: d3f1a823-3ccb-4148-8319-3f0212f5287d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"d3f1a823-3ccb-4148-8319-3f0212f5287d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ad50b65a-aeaf-4dc8-b898-212aeb8a4d2a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"ce8aebb9fc08a2a910f6628d9f96f2afac1e005eece2bf9149c4e58a24bfad64\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"64c9c70a-d118-43f3-b639-289942d15c55\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8b13c7ab4b4dfd52de72c9adbe24ae0c4bcfb57d878aff96a2106c02a6abfea0\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 41744, \"end_char_idx\": 43229, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "```python\ndisplay(events_df.iloc[0].spec)\n```\n\n\n    {'completion_fns': ['gpt-3.5-turbo'],\n     'eval_name': 'spider-sql.dev.v0',\n     'base_eval': 'spider-sql',\n     'split': 'dev',\n     'run_config': {'completion_fns': ['gpt-3.5-turbo'],\n      'eval_spec': {'cls': 'evals.elsuite.modelgraded.classify:ModelBasedClassify',\n       'registry_path': '/Users/shyamal/.virtualenvs/openai/lib/python3.11/site-packages/evals/registry',\n       'args': {'samples_jsonl': 'sql/spider_sql.jsonl',\n        'eval_type': 'cot_classify',\n        'modelgraded_spec': 'sql'},\n       'key': 'spider-sql.dev.v0',\n       'group': 'sql'},\n      'seed': 20220722,\n      'max_samples': 25,\n      'command': '/Users/shyamal/.virtualenvs/openai/bin/oaieval gpt-3.5-turbo spider-sql --max_samples 25',\n      'initial_settings': {'visible': False}},\n     'created_by': '',\n     'run_id': '240327024443FACXGMKA',\n     'created_at': '2024-03-27 02:44:43.626043'}\n\n\nLet's also look at the entry which provides the final report of the evaluation.\n\n\n```python\ndisplay(events_df.dropna(subset=['final_report']).iloc[0]['final_report'])\n```\n\n\n    {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}\n\n\nWe can also review individual evaluation events that provide specific samples (`sample_id`), results, event types, and metadata.\n\n\n```python\npd.set_option('display.max_colwidth', None)  # None means no truncation\ndisplay(events_df.iloc[2][['run_id', 'event_id', 'sample_id', 'type', 'data', 'created_at']])\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 64c9c70a-d118-43f3-b639-289942d15c55
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"64c9c70a-d118-43f3-b639-289942d15c55\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d3f1a823-3ccb-4148-8319-3f0212f5287d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"36aa2685b41ddfe9a8933fae51761e9d2f3c087e1701a66c0bd44a9109f5020d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"88a18639-d146-4485-9a3b-a152b9dded86\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"1c4d59968cf169d38bc2b3a2d85d03618c48f1467be680f6a4ebb8c74c7a023e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 42680, \"end_char_idx\": 51567, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "Let's also look at the entry which provides the final report of the evaluation.\n\n\n```python\ndisplay(events_df.dropna(subset=['final_report']).iloc[0]['final_report'])\n```\n\n\n    {'counts/Correct': 20, 'counts/Incorrect': 5, 'score': 0.8}\n\n\nWe can also review individual evaluation events that provide specific samples (`sample_id`), results, event types, and metadata.\n\n\n```python\npd.set_option('display.max_colwidth', None)  # None means no truncation\ndisplay(events_df.iloc[2][['run_id', 'event_id', 'sample_id', 'type', 'data', 'created_at']])\n```\n\n\n    run_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                240327024443FACXGMKA\n    event_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                               0.0\n    sample_id                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                spider-sql.dev.88\n    type                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                              sampling\n    data          {'prompt': [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\n    Use only the following tables and columns:\n    Table: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\n    Table: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)\n    Table: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)\n    \n    Question: Find the average rank of winners in all matches.\n    ', 'role': 'system'}], 'sampled': ['SELECT AVG(winner_rank) AS average_rank_of_winners\n    FROM matches;']}\n    created_at                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                                2024-03-27 02:44:44.821110+00:00\n    Name: 2, dtype: object\n\n\n\n```python\n# Inspect samples\nfor i, row in events_df[events_df['type'] == 'sampling'].head(5).iterrows():\n    data = pd.json_normalize(row['data'])\n    print(f\"Prompt: {data['prompt'].iloc[0]}\")\n    print(f\"Sampled: {data['sampled'].iloc[0]}\")\n    print(\"-\" * 10)\n```\n\n    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\\nUse only the following tables and columns:\\nTable: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\\nTable: matches."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 88a18639-d146-4485-9a3b-a152b9dded86
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"88a18639-d146-4485-9a3b-a152b9dded86\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"64c9c70a-d118-43f3-b639-289942d15c55\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"16568141afe5baf87b239f4d73cb866703f067d5740bf5255b2b85cf54a291c5\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"17804800-3945-401e-a082-a8544fb775d8\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d448681c028549f6f2c1d6c21014c4d15631cfa5b7cf63b8cf6cca01c6042648\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 50968, \"end_char_idx\": 53972, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "```python\n# Inspect samples\nfor i, row in events_df[events_df['type'] == 'sampling'].head(5).iterrows():\n    data = pd.json_normalize(row['data'])\n    print(f\"Prompt: {data['prompt'].iloc[0]}\")\n    print(f\"Sampled: {data['sampled'].iloc[0]}\")\n    print(\"-\" * 10)\n```\n\n    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\\nUse only the following tables and columns:\\nTable: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\\nTable: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)\\nTable: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)\\n\\nQuestion: Find the average rank of winners in all matches.\\n', 'role': 'system'}]\n    Sampled: ['SELECT AVG(winner_rank) AS average_rank_of_winners\\nFROM matches;']\n    ----------\n    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\\nUse only the following tables and columns:\\nTable: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\\nTable: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)\\nTable: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)\\n\\nQuestion: Find the total number of matches.\\n', 'role': 'system'}]\n    Sampled: ['SELECT COUNT(*) AS total_matches\\nFROM matches;']\n    ----------\n    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\\nUse only the following tables and columns:\\nTable: continents."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 17804800-3945-401e-a082-a8544fb775d8
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"17804800-3945-401e-a082-a8544fb775d8\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"88a18639-d146-4485-9a3b-a152b9dded86\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"648dc861250ba51d15c02facfb56f8e72ac3a5854e2d89133299d206cbb1ba52\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"81af1b42-c562-4fa8-acec-afa61888970d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8081cfa1c0edb27d083aaa3a7baaaeaa758676d384ae77990b417c063565201a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 52616, \"end_char_idx\": 55242, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "Be creative but the SQL must be correct.\\nUse only the following tables and columns:\\nTable: players. Columns: player_id (number), first_name (text), last_name (text), hand (text), birth_date (time), country_code (text)\\nTable: matches. Columns: best_of (number), draw_size (number), loser_age (number), loser_entry (text), loser_hand (text), loser_ht (number), loser_id (number), loser_ioc (text), loser_name (text), loser_rank (number), loser_rank_points (number), loser_seed (number), match_num (number), minutes (number), round (text), score (text), surface (text), tourney_date (time), tourney_id (text), tourney_level (text), tourney_name (text), winner_age (number), winner_entry (text), winner_hand (text), winner_ht (number), winner_id (number), winner_ioc (text), winner_name (text), winner_rank (number), winner_rank_points (number), winner_seed (number), year (number)\\nTable: rankings. Columns: ranking_date (time), ranking (number), player_id (number), ranking_points (number), tours (number)\\n\\nQuestion: Find the total number of matches.\\n', 'role': 'system'}]\n    Sampled: ['SELECT COUNT(*) AS total_matches\\nFROM matches;']\n    ----------\n    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\\nUse only the following tables and columns:\\nTable: continents. Columns: ContId (number), Continent (text)\\nTable: countries. Columns: CountryId (number), CountryName (text), Continent (number)\\nTable: car_makers. Columns: Id (number), Maker (text), FullName (text), Country (text)\\nTable: model_list. Columns: ModelId (number), Maker (number), Model (text)\\nTable: car_names. Columns: MakeId (number), Model (text), Make (text)\\nTable: cars_data. Columns: Id (number), MPG (text), Cylinders (number), Edispl (number), Horsepower (text), Weight (number), Accelerate (number), Year (number)\\n\\nQuestion: How many countries exist?\\n', 'role': 'system'}]\n    Sampled: ['SELECT COUNT(*) AS TotalCountries\\nFROM countries;']\n    ----------\n    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\\nUse only the following tables and columns:\\nTable: TV_Channel. Columns: id (text), series_name (text), Country (text), Language (text), Content (text), Pixel_aspect_ratio_PAR (text), Hight_definition_TV (text), Pay_per_view_PPV (text), Package_Option (text)\\nTable: TV_series. Columns: id (number), Episode (text), Air_Date (text), Rating (text), Share (number), 18_49_Rating_Share (text), Viewers_m (text), Weekly_Rank (number), Channel (text)\\nTable: Cartoon."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 81af1b42-c562-4fa8-acec-afa61888970d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"81af1b42-c562-4fa8-acec-afa61888970d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"17804800-3945-401e-a082-a8544fb775d8\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"ea4d7b4628e0b426c053f8f76e75ba29650ed84b4d85c3cff44526f5c29cf314\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"d5d2ac9b-1a2f-4267-9cc8-005680f0d7db\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"dad931d4882b288500dbac7b74da622f9de656e29875e150bb5538b7803bf14a\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 53868, \"end_char_idx\": 56584, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "Be creative but the SQL must be correct.\\nUse only the following tables and columns:\\nTable: continents. Columns: ContId (number), Continent (text)\\nTable: countries. Columns: CountryId (number), CountryName (text), Continent (number)\\nTable: car_makers. Columns: Id (number), Maker (text), FullName (text), Country (text)\\nTable: model_list. Columns: ModelId (number), Maker (number), Model (text)\\nTable: car_names. Columns: MakeId (number), Model (text), Make (text)\\nTable: cars_data. Columns: Id (number), MPG (text), Cylinders (number), Edispl (number), Horsepower (text), Weight (number), Accelerate (number), Year (number)\\n\\nQuestion: How many countries exist?\\n', 'role': 'system'}]\n    Sampled: ['SELECT COUNT(*) AS TotalCountries\\nFROM countries;']\n    ----------\n    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\\nUse only the following tables and columns:\\nTable: TV_Channel. Columns: id (text), series_name (text), Country (text), Language (text), Content (text), Pixel_aspect_ratio_PAR (text), Hight_definition_TV (text), Pay_per_view_PPV (text), Package_Option (text)\\nTable: TV_series. Columns: id (number), Episode (text), Air_Date (text), Rating (text), Share (number), 18_49_Rating_Share (text), Viewers_m (text), Weekly_Rank (number), Channel (text)\\nTable: Cartoon. Columns: id (number), Title (text), Directed_by (text), Written_by (text), Original_air_date (text), Production_code (number), Channel (text)\\n\\nQuestion: What is the name and directors of all the cartoons that are ordered by air date?\\n', 'role': 'system'}]\n    Sampled: ['SELECT Title, Directed_by\\nFROM Cartoon\\nORDER BY Original_air_date;']\n    ----------\n    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\\nUse only the following tables and columns:\\nTable: stadium. Columns: Stadium_ID (number), Location (text), Name (text), Capacity (number), Highest (number), Lowest (number), Average (number)\\nTable: singer. Columns: Singer_ID (number), Name (text), Country (text), Song_Name (text), Song_release_year (text), Age (number), Is_male (others)\\nTable: concert. Columns: concert_ID (number), concert_Name (text), Theme (text), Stadium_ID (text), Year (text)\\nTable: singer_in_concert. Columns: concert_ID (number), Singer_ID (text)\\n\\nQuestion: Show the name and the release year of the song by the youngest singer.\\n', 'role': 'system'}]\n    Sampled: ['```sql\\nSELECT s.Name, s.Song_release_year\\nFROM singer s\\nWHERE s.Age = (SELECT MIN(Age) FROM singer)\\n```']\n    ----------\n\n\nLet's review our failures to understand which tests did not succeed."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: d5d2ac9b-1a2f-4267-9cc8-005680f0d7db
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"d5d2ac9b-1a2f-4267-9cc8-005680f0d7db\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"81af1b42-c562-4fa8-acec-afa61888970d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"528c593ff142a4ba623e4f5e1cfb1fa95614203a9db55c1bea4d7ebb95e5017f\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"4ad58cd7-a41c-4c9f-ab4e-d71789ff2361\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"2b257916321e80c1f8c4b87ad84d80904cdb749c8f3349e8efb261dd3cf7dc6e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 55058, \"end_char_idx\": 58210, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "Columns: id (number), Episode (text), Air_Date (text), Rating (text), Share (number), 18_49_Rating_Share (text), Viewers_m (text), Weekly_Rank (number), Channel (text)\\nTable: Cartoon. Columns: id (number), Title (text), Directed_by (text), Written_by (text), Original_air_date (text), Production_code (number), Channel (text)\\n\\nQuestion: What is the name and directors of all the cartoons that are ordered by air date?\\n', 'role': 'system'}]\n    Sampled: ['SELECT Title, Directed_by\\nFROM Cartoon\\nORDER BY Original_air_date;']\n    ----------\n    Prompt: [{'content': 'Answer the following question with syntactically correct SQLite SQL. Be creative but the SQL must be correct.\\nUse only the following tables and columns:\\nTable: stadium. Columns: Stadium_ID (number), Location (text), Name (text), Capacity (number), Highest (number), Lowest (number), Average (number)\\nTable: singer. Columns: Singer_ID (number), Name (text), Country (text), Song_Name (text), Song_release_year (text), Age (number), Is_male (others)\\nTable: concert. Columns: concert_ID (number), concert_Name (text), Theme (text), Stadium_ID (text), Year (text)\\nTable: singer_in_concert. Columns: concert_ID (number), Singer_ID (text)\\n\\nQuestion: Show the name and the release year of the song by the youngest singer.\\n', 'role': 'system'}]\n    Sampled: ['```sql\\nSELECT s.Name, s.Song_release_year\\nFROM singer s\\nWHERE s.Age = (SELECT MIN(Age) FROM singer)\\n```']\n    ----------\n\n\nLet's review our failures to understand which tests did not succeed.\n\n\n```python\ndef pretty_print_text(prompt):\n    # Define markers for the sections\n    markers = {\n        \"question\": \"[Question]:\",\n        \"expert\": \"[Expert]:\",\n        \"submission\": \"[Submission]:\",\n        \"end\": \"[END DATA]\"\n    }\n    \n    # Function to extract text between markers\n    def extract_text(start_marker, end_marker):\n        start = prompt.find(start_marker) + len(start_marker)\n        end = prompt.find(end_marker)\n        text = prompt[start:end].strip()\n        if start_marker == markers[\"question\"]:\n            text = text.split(\"\\n\\nQuestion:\")[-1].strip() if \"\\n\\nQuestion:\" in text else text\n        elif start_marker == markers[\"submission\"]:\n            text = text.replace(\"```sql\", \"\").replace(\"```\", \"\").strip()\n        return text\n    \n    # Extracting text for each section\n    question_text = extract_text(markers[\"question\"], markers[\"expert\"])\n    expert_text = extract_text(markers[\"expert\"], markers[\"submission\"])\n    submission_text = extract_text(markers[\"submission\"], markers[\"end\"])\n    \n    # HTML color codes and formatting\n    colors = {\n        \"question\": '<span style=\"color: #0000FF;\">QUESTION:<br>', \n        \"expert\": '<span style=\"color: #008000;\">EXPECTED:<br>',  \n        \"submission\": '<span style=\"color: #FFA500;\">SUBMISSION:<br>' \n    }\n    color_end = '</span>'\n    \n    # Display each section with color\n    from IPython.display import display, HTML\n    display(HTML(f\"{colors['question']}{question_text}{color_end}\"))\n    display(HTML(f\"{colors['expert']}{expert_text}{color_end}\"))\n    display(HTML(f\"{colors['submission']}{submission_text}{color_end}\"))\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 4ad58cd7-a41c-4c9f-ab4e-d71789ff2361
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"4ad58cd7-a41c-4c9f-ab4e-d71789ff2361\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"d5d2ac9b-1a2f-4267-9cc8-005680f0d7db\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"09d93a689f6cce18573bd01690aa510c7ba4bc43dadbc50756312a0250c24852\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"f6624619-55c0-4b19-bdaa-6b819568e6a5\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"9e9f577507627bfa8df69233d7f66af92445e05915055a1bce054270fc75cce6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 56516, \"end_char_idx\": 59647, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "Let's review our failures to understand which tests did not succeed.\n\n\n```python\ndef pretty_print_text(prompt):\n    # Define markers for the sections\n    markers = {\n        \"question\": \"[Question]:\",\n        \"expert\": \"[Expert]:\",\n        \"submission\": \"[Submission]:\",\n        \"end\": \"[END DATA]\"\n    }\n    \n    # Function to extract text between markers\n    def extract_text(start_marker, end_marker):\n        start = prompt.find(start_marker) + len(start_marker)\n        end = prompt.find(end_marker)\n        text = prompt[start:end].strip()\n        if start_marker == markers[\"question\"]:\n            text = text.split(\"\\n\\nQuestion:\")[-1].strip() if \"\\n\\nQuestion:\" in text else text\n        elif start_marker == markers[\"submission\"]:\n            text = text.replace(\"```sql\", \"\").replace(\"```\", \"\").strip()\n        return text\n    \n    # Extracting text for each section\n    question_text = extract_text(markers[\"question\"], markers[\"expert\"])\n    expert_text = extract_text(markers[\"expert\"], markers[\"submission\"])\n    submission_text = extract_text(markers[\"submission\"], markers[\"end\"])\n    \n    # HTML color codes and formatting\n    colors = {\n        \"question\": '<span style=\"color: #0000FF;\">QUESTION:<br>', \n        \"expert\": '<span style=\"color: #008000;\">EXPECTED:<br>',  \n        \"submission\": '<span style=\"color: #FFA500;\">SUBMISSION:<br>' \n    }\n    color_end = '</span>'\n    \n    # Display each section with color\n    from IPython.display import display, HTML\n    display(HTML(f\"{colors['question']}{question_text}{color_end}\"))\n    display(HTML(f\"{colors['expert']}{expert_text}{color_end}\"))\n    display(HTML(f\"{colors['submission']}{submission_text}{color_end}\"))\n```\n\n\n```python\n# Inspect metrics where choice is made and print only the prompt, result, and expected result if the choice is incorrect\nfor i, row in events_df[events_df['type'] == 'metrics'].iterrows():\n    if row['data']['choice'] == 'Incorrect':\n        # Get the previous row's data, which contains the prompt and the expected result\n        prev_row = events_df.iloc[i-1]\n        prompt = prev_row['data']['prompt'][0]['content'] if 'prompt' in prev_row['data'] and len(prev_row['data']['prompt']) > 0 else \"Prompt not available\"\n        expected_result = prev_row['data'].get('ideal', 'Expected result not provided')\n        \n        # Current row's data will be the actual result\n        result = row['data'].get('result', 'Actual result not provided')\n        \n        pretty_print_text(prompt)\n        print(\"-\" * 40)\n```\n\n\n<span style=\"color: #0000FF;\">QUESTION:<br>How many countries have a republic as their form of government?\n\n************</span>\n\n\n\n<span style=\"color: #008000;\">EXPECTED:<br>SELECT count(*) FROM country WHERE GovernmentForm  =  \"Republic\"\n************</span>\n\n\n\n<span style=\"color: #FFA500;\">SUBMISSION:<br>SELECT COUNT(*) \nFROM country \nWHERE GovernmentForm LIKE '%Republic%'\n\n************</span>\n\n\n    ----------------------------------------\n\n\n\n<span style=\"color: #0000FF;\">QUESTION:<br>Return the document id, template id, and description for the document with the name Robbin CV.\n\n************</span>"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: f6624619-55c0-4b19-bdaa-6b819568e6a5
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"f6624619-55c0-4b19-bdaa-6b819568e6a5\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"4ad58cd7-a41c-4c9f-ab4e-d71789ff2361\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"63a9df17e0e4a87afeb853e4b81e89f0323f69453b31cc8e8c4eda8586329941\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8c75e9fc-f82f-4a56-8410-916acc148b4d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"42e338f1df2f76b100b9cbf663bcd88ce512a131678cd3692062fe336d720aea\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 58213, \"end_char_idx\": 61303, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "```python\n# Inspect metrics where choice is made and print only the prompt, result, and expected result if the choice is incorrect\nfor i, row in events_df[events_df['type'] == 'metrics'].iterrows():\n    if row['data']['choice'] == 'Incorrect':\n        # Get the previous row's data, which contains the prompt and the expected result\n        prev_row = events_df.iloc[i-1]\n        prompt = prev_row['data']['prompt'][0]['content'] if 'prompt' in prev_row['data'] and len(prev_row['data']['prompt']) > 0 else \"Prompt not available\"\n        expected_result = prev_row['data'].get('ideal', 'Expected result not provided')\n        \n        # Current row's data will be the actual result\n        result = row['data'].get('result', 'Actual result not provided')\n        \n        pretty_print_text(prompt)\n        print(\"-\" * 40)\n```\n\n\n<span style=\"color: #0000FF;\">QUESTION:<br>How many countries have a republic as their form of government?\n\n************</span>\n\n\n\n<span style=\"color: #008000;\">EXPECTED:<br>SELECT count(*) FROM country WHERE GovernmentForm  =  \"Republic\"\n************</span>\n\n\n\n<span style=\"color: #FFA500;\">SUBMISSION:<br>SELECT COUNT(*) \nFROM country \nWHERE GovernmentForm LIKE '%Republic%'\n\n************</span>\n\n\n    ----------------------------------------\n\n\n\n<span style=\"color: #0000FF;\">QUESTION:<br>Return the document id, template id, and description for the document with the name Robbin CV.\n\n************</span>\n\n\n\n<span style=\"color: #008000;\">EXPECTED:<br>SELECT document_id ,  template_id ,  Document_Description FROM Documents WHERE document_name  =  \"Robbin CV\"\n************</span>\n\n\n\n<span style=\"color: #FFA500;\">SUBMISSION:<br>SELECT Documents.Document_ID, Documents.Template_ID, Documents.Document_Description\nFROM Documents\nJOIN Templates ON Documents.Template_ID = Templates.Template_ID\nWHERE Documents.Document_Name = 'Robbin CV';\n\n************</span>\n\n\n    ----------------------------------------\n\n\n\n<span style=\"color: #0000FF;\">QUESTION:<br>Which professionals live in the state of Indiana or have done treatment on more than 2 treatments? List his or her id, last name and cell phone.\n\n************</span>\n\n\n\n<span style=\"color: #008000;\">EXPECTED:<br>SELECT professional_id ,  last_name ,  cell_number FROM Professionals WHERE state  =  'Indiana' UNION SELECT T1.professional_id ,  T1.last_name ,  T1.cell_number FROM Professionals AS T1 JOIN Treatments AS T2 ON T1.professional_id  =  T2.professional_id GROUP BY T1.professional_id HAVING count(*)  >  2\n************</span>\n\n\n\n<span style=\"color: #FFA500;\">SUBMISSION:<br>SELECT professional_id, last_name, cell_number\nFROM Professionals\nWHERE state = 'Indiana'\nOR professional_id IN (\n    SELECT professional_id\n    FROM Treatments\n    GROUP BY professional_id\n    HAVING COUNT(*) > 2\n);\n************</span>\n\n\n    ----------------------------------------\n\n\n\n<span style=\"color: #0000FF;\">QUESTION:<br>What is the continent name which Anguilla belongs to?\n\n************</span>\n\n\n\n<span style=\"color: #008000;\">EXPECTED:<br>SELECT Continent FROM country WHERE Name  =  \"Anguilla\"\n************</span>"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 8c75e9fc-f82f-4a56-8410-916acc148b4d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"8c75e9fc-f82f-4a56-8410-916acc148b4d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"f6624619-55c0-4b19-bdaa-6b819568e6a5\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"c47414adf8c82131e174e98002c786ecd2d67344a42ad6638b7c6b803d430b26\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"897484b4-dfdf-44e8-8d62-a07470517d4a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d93fb102df7799c5f49af1b446e5f57ad73eee8ef49a7ad1a8e6aaa7035ae59f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 59651, \"end_char_idx\": 61846, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "<span style=\"color: #008000;\">EXPECTED:<br>SELECT document_id ,  template_id ,  Document_Description FROM Documents WHERE document_name  =  \"Robbin CV\"\n************</span>\n\n\n\n<span style=\"color: #FFA500;\">SUBMISSION:<br>SELECT Documents.Document_ID, Documents.Template_ID, Documents.Document_Description\nFROM Documents\nJOIN Templates ON Documents.Template_ID = Templates.Template_ID\nWHERE Documents.Document_Name = 'Robbin CV';\n\n************</span>\n\n\n    ----------------------------------------\n\n\n\n<span style=\"color: #0000FF;\">QUESTION:<br>Which professionals live in the state of Indiana or have done treatment on more than 2 treatments? List his or her id, last name and cell phone.\n\n************</span>\n\n\n\n<span style=\"color: #008000;\">EXPECTED:<br>SELECT professional_id ,  last_name ,  cell_number FROM Professionals WHERE state  =  'Indiana' UNION SELECT T1.professional_id ,  T1.last_name ,  T1.cell_number FROM Professionals AS T1 JOIN Treatments AS T2 ON T1.professional_id  =  T2.professional_id GROUP BY T1.professional_id HAVING count(*)  >  2\n************</span>\n\n\n\n<span style=\"color: #FFA500;\">SUBMISSION:<br>SELECT professional_id, last_name, cell_number\nFROM Professionals\nWHERE state = 'Indiana'\nOR professional_id IN (\n    SELECT professional_id\n    FROM Treatments\n    GROUP BY professional_id\n    HAVING COUNT(*) > 2\n);\n************</span>\n\n\n    ----------------------------------------\n\n\n\n<span style=\"color: #0000FF;\">QUESTION:<br>What is the continent name which Anguilla belongs to?\n\n************</span>\n\n\n\n<span style=\"color: #008000;\">EXPECTED:<br>SELECT Continent FROM country WHERE Name  =  \"Anguilla\"\n************</span>\n\n\n\n<span style=\"color: #FFA500;\">SUBMISSION:<br>SELECT c.Continent\nFROM country c\nWHERE c.Code = 'AIA';\n\n************</span>\n\n\n    ----------------------------------------\n\n\n\n<span style=\"color: #0000FF;\">QUESTION:<br>How many airlines do we have?\n\n************</span>\n\n\n\n<span style=\"color: #008000;\">EXPECTED:<br>SELECT count(*) FROM AIRLINES\n************</span>\n\n\n\n<span style=\"color: #FFA500;\">SUBMISSION:<br>SELECT COUNT(DISTINCT Airline) AS TotalAirlines\nFROM airlines;\n************</span>\n\n\n    ----------------------------------------"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 897484b4-dfdf-44e8-8d62-a07470517d4a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb", "title": "Getting Started with OpenAI Evals", "tokens": 15953, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"897484b4-dfdf-44e8-8d62-a07470517d4a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"fb582c37-7251-4ebe-9352-7f5f2ea8cfc8\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"be7a39573e7c50cfb902d34957d7fd28ac908247da186b3e1c39135bf3586882\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8c75e9fc-f82f-4a56-8410-916acc148b4d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Getting_Started_with_OpenAI_Evals.ipynb\", \"title\": \"Getting Started with OpenAI Evals\", \"tokens\": 15953, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"8b57e6d940fe61350115a8390d949663a2e418726803cac45930aec61ac79c4e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 60362, \"end_char_idx\": 63108, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "ref_doc_id": "fb582c37-7251-4ebe-9352-7f5f2ea8cfc8", "chroma:document": "<span style=\"color: #008000;\">EXPECTED:<br>SELECT professional_id ,  last_name ,  cell_number FROM Professionals WHERE state  =  'Indiana' UNION SELECT T1.professional_id ,  T1.last_name ,  T1.cell_number FROM Professionals AS T1 JOIN Treatments AS T2 ON T1.professional_id  =  T2.professional_id GROUP BY T1.professional_id HAVING count(*)  >  2\n************</span>\n\n\n\n<span style=\"color: #FFA500;\">SUBMISSION:<br>SELECT professional_id, last_name, cell_number\nFROM Professionals\nWHERE state = 'Indiana'\nOR professional_id IN (\n    SELECT professional_id\n    FROM Treatments\n    GROUP BY professional_id\n    HAVING COUNT(*) > 2\n);\n************</span>\n\n\n    ----------------------------------------\n\n\n\n<span style=\"color: #0000FF;\">QUESTION:<br>What is the continent name which Anguilla belongs to?\n\n************</span>\n\n\n\n<span style=\"color: #008000;\">EXPECTED:<br>SELECT Continent FROM country WHERE Name  =  \"Anguilla\"\n************</span>\n\n\n\n<span style=\"color: #FFA500;\">SUBMISSION:<br>SELECT c.Continent\nFROM country c\nWHERE c.Code = 'AIA';\n\n************</span>\n\n\n    ----------------------------------------\n\n\n\n<span style=\"color: #0000FF;\">QUESTION:<br>How many airlines do we have?\n\n************</span>\n\n\n\n<span style=\"color: #008000;\">EXPECTED:<br>SELECT count(*) FROM AIRLINES\n************</span>\n\n\n\n<span style=\"color: #FFA500;\">SUBMISSION:<br>SELECT COUNT(DISTINCT Airline) AS TotalAirlines\nFROM airlines;\n************</span>\n\n\n    ----------------------------------------\n\n\nReviewing some of the failures we see the following:\n* The second incorrect answer had an unnecessary join with the 'Templates' table. Our eval was able to accurately identify this and flag this as incorrect. \n* Few other answers have minor syntax differences that caused the answers to get flagged.\n  * In situations like this, it would be worthwhile exploring whether we should continue iterating on the prompt to ensure certain stylistic choices, or if we should modify the evaluation suite to capture this variation.\n  * This type of failure hints at the potential need for model-graded evals as a way to ensure accuracy in grading the results\n\n# Conclusion\n\nBuilding out effective evals is a core part of the development cycle of LLM-based applications. The OpenAI Evals framework provides the core structure of building evals out of the box, and allows you to quickly spin up new tests for your various use cases. In this guide, we demonstrated step-by-step how to create an eval, run it, and analyze the results.\n\nThe example shown in this guide represent a straightfoward use case for evals. As you continue to explore this framework, we recommend you explore creating more complex model-graded evals for actual production use cases. Happy evaluating!"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: c20af387-7fe4-4025-95eb-071c7da081e1
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"c20af387-7fe4-4025-95eb-071c7da081e1\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e90b4ace-2a01-47fb-b16e-61b6d6e24a42\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b38a3d1ace59358d62d1d822324929594e81f81511747f16addff899e9e565e2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"a95dbc00-1bdc-44f2-a33d-92bd4a20883d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4987a94b9964c66fd52540d6ed08d64f33ec1c0cc7725cb5e845dbc12a104f94\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 3549, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "doc_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "ref_doc_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "chroma:document": "# Evaluate RAG with LlamaIndex\n\nIn this notebook we will look into building an RAG pipeline and evaluating it with LlamaIndex. It has following 3 sections.\n\n1. Understanding Retrieval Augmented Generation (RAG).\n2. Building RAG with LlamaIndex.\n3. Evaluating RAG with LlamaIndex.\n\n**Retrieval Augmented Generation (RAG)**\n\nLLMs are trained on vast datasets, but these will not include your specific data. Retrieval-Augmented Generation (RAG) addresses this by dynamically incorporating your data during the generation process. This is done not by altering the training data of LLMs, but by allowing the model to access and utilize your data in real-time to provide more tailored and contextually relevant responses.\n\nIn RAG, your data is loaded and and prepared for queries or \u201cindexed\u201d. User queries act on the index, which filters your data down to the most relevant context. This context and your query then go to the LLM along with a prompt, and the LLM provides a response.\n\nEven if what you\u2019re building is a chatbot or an agent, you\u2019ll want to know RAG techniques for getting data into your application.\n\n![RAG Overview](../../images/llamaindex_rag_overview.png)\n\n**Stages within RAG**\n\nThere are five key stages within RAG, which in turn will be a part of any larger application you build. These are:\n\n**Loading:** this refers to getting your data from where it lives \u2013 whether it\u2019s text files, PDFs, another website, a database, or an API \u2013 into your pipeline. LlamaHub provides hundreds of connectors to choose from.\n\n**Indexing:** this means creating a data structure that allows for querying the data. For LLMs this nearly always means creating vector embeddings, numerical representations of the meaning of your data, as well as numerous other metadata strategies to make it easy to accurately find contextually relevant data.\n\n**Storing:** Once your data is indexed, you will want to store your index, along with any other metadata, to avoid the need to re-index it.\n\n**Querying:** for any given indexing strategy there are many ways you can utilize LLMs and LlamaIndex data structures to query, including sub-queries, multi-step queries and hybrid strategies.\n\n**Evaluation:** a critical step in any pipeline is checking how effective it is relative to other strategies, or when you make changes. Evaluation provides objective measures of how accurate, faithful and fast your responses to queries are.\n\n## Build RAG system.\n\nNow that we have understood the significance of RAG system, let's build a simple RAG pipeline.\n\n\n```python\n!pip install llama-index\n```\n\n\n```python\n# The nest_asyncio module enables the nesting of asynchronous functions within an already running async loop.\n# This is necessary because Jupyter notebooks inherently operate in an asynchronous loop.\n# By applying nest_asyncio, we can run additional async functions within this existing loop without conflicts.\nimport nest_asyncio\n\nnest_asyncio.apply()\n\nfrom llama_index.evaluation import generate_question_context_pairs\nfrom llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext\nfrom llama_index.node_parser import SimpleNodeParser\nfrom llama_index.evaluation import generate_question_context_pairs\nfrom llama_index.evaluation import RetrieverEvaluator\nfrom llama_index.llms import OpenAI\n\nimport os\nimport pandas as pd\n```\n\nSet Your OpenAI API Key\n\n\n```python\nos.environ['OPENAI_API_KEY'] = 'YOUR OPENAI API KEY'\n```\n\nLet's use [Paul Graham Essay text](https://www.paulgraham.com/worked.html) for building RAG pipeline.\n\n#### Download Data"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: a95dbc00-1bdc-44f2-a33d-92bd4a20883d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"a95dbc00-1bdc-44f2-a33d-92bd4a20883d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e90b4ace-2a01-47fb-b16e-61b6d6e24a42\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b38a3d1ace59358d62d1d822324929594e81f81511747f16addff899e9e565e2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"c20af387-7fe4-4025-95eb-071c7da081e1\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"26b6d928f6a4c668716b90d0408473bd80f7ece1b184ccfca1f467dc8bfa2ee0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"226546d4-6117-4a3b-82fc-26e94c1100ad\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"61da286d96fb963babd58cdeeadd7f9bc332f4579fcbc66c3f1f1b17c7cb20ea\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 2536, \"end_char_idx\": 5052, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "doc_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "ref_doc_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "chroma:document": "```python\n!pip install llama-index\n```\n\n\n```python\n# The nest_asyncio module enables the nesting of asynchronous functions within an already running async loop.\n# This is necessary because Jupyter notebooks inherently operate in an asynchronous loop.\n# By applying nest_asyncio, we can run additional async functions within this existing loop without conflicts.\nimport nest_asyncio\n\nnest_asyncio.apply()\n\nfrom llama_index.evaluation import generate_question_context_pairs\nfrom llama_index import VectorStoreIndex, SimpleDirectoryReader, ServiceContext\nfrom llama_index.node_parser import SimpleNodeParser\nfrom llama_index.evaluation import generate_question_context_pairs\nfrom llama_index.evaluation import RetrieverEvaluator\nfrom llama_index.llms import OpenAI\n\nimport os\nimport pandas as pd\n```\n\nSet Your OpenAI API Key\n\n\n```python\nos.environ['OPENAI_API_KEY'] = 'YOUR OPENAI API KEY'\n```\n\nLet's use [Paul Graham Essay text](https://www.paulgraham.com/worked.html) for building RAG pipeline.\n\n#### Download Data\n\n\n```python\n!mkdir -p 'data/paul_graham/'\n!curl 'https://raw.githubusercontent.com/run-llama/llama_index/main/docs/examples/data/paul_graham/paul_graham_essay.txt' -o 'data/paul_graham/paul_graham_essay.txt'\n```\n\n      % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current\n                                     Dload  Upload   Total   Spent    Left  Speed\n    100 75042  100 75042    0     0   190k      0 --:--:-- --:--:-- --:--:--  190k--:--  0:00:03 24586\n\n\n#### Load Data and Build Index.\n\n\n```python\ndocuments = SimpleDirectoryReader(\"./data/paul_graham/\").load_data()\n\n# Define an LLM\nllm = OpenAI(model=\"gpt-4\")\n\n# Build index with a chunk_size of 512\nnode_parser = SimpleNodeParser.from_defaults(chunk_size=512)\nnodes = node_parser.get_nodes_from_documents(documents)\nvector_index = VectorStoreIndex(nodes)\n```\n\nBuild a QueryEngine and start querying.\n\n\n```python\nquery_engine = vector_index.as_query_engine()\n```\n\n\n```python\nresponse_vector = query_engine.query(\"What did the author do growing up?\")\n```\n\nCheck response.\n\n\n```python\nresponse_vector.response\n```\n\n\n\n\n    'The author wrote short stories and worked on programming, specifically on an IBM 1401 computer using an early version of Fortran.'\n\n\n\nBy default it retrieves `two` similar nodes/ chunks. You can modify that in `vector_index.as_query_engine(similarity_top_k=k)`.\n\nLet's check the text in each of these retrieved nodes.\n\n\n```python\n# First retrieved node\nresponse_vector.source_nodes[0].get_text()\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 226546d4-6117-4a3b-82fc-26e94c1100ad
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"226546d4-6117-4a3b-82fc-26e94c1100ad\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e90b4ace-2a01-47fb-b16e-61b6d6e24a42\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b38a3d1ace59358d62d1d822324929594e81f81511747f16addff899e9e565e2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"a95dbc00-1bdc-44f2-a33d-92bd4a20883d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bd6f7eb2febb306d15c0d48f6ed3403dd29d1291782ec60f3d62bc862608885a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"ee2d687b-bbd5-49b2-b942-97336299b5be\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"81bd1a393b6cb99db4709af65021450a8d4448932336a3cef5fe84a52cdd8971\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 4034, \"end_char_idx\": 7154, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "doc_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "ref_doc_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "chroma:document": "#### Load Data and Build Index.\n\n\n```python\ndocuments = SimpleDirectoryReader(\"./data/paul_graham/\").load_data()\n\n# Define an LLM\nllm = OpenAI(model=\"gpt-4\")\n\n# Build index with a chunk_size of 512\nnode_parser = SimpleNodeParser.from_defaults(chunk_size=512)\nnodes = node_parser.get_nodes_from_documents(documents)\nvector_index = VectorStoreIndex(nodes)\n```\n\nBuild a QueryEngine and start querying.\n\n\n```python\nquery_engine = vector_index.as_query_engine()\n```\n\n\n```python\nresponse_vector = query_engine.query(\"What did the author do growing up?\")\n```\n\nCheck response.\n\n\n```python\nresponse_vector.response\n```\n\n\n\n\n    'The author wrote short stories and worked on programming, specifically on an IBM 1401 computer using an early version of Fortran.'\n\n\n\nBy default it retrieves `two` similar nodes/ chunks. You can modify that in `vector_index.as_query_engine(similarity_top_k=k)`.\n\nLet's check the text in each of these retrieved nodes.\n\n\n```python\n# First retrieved node\nresponse_vector.source_nodes[0].get_text()\n```\n\n\n\n\n    'What I Worked On\\n\\nFebruary 2021\\n\\nBefore college the two main things I worked on, outside of school, were writing and programming. I didn\\'t write essays. I wrote what beginning writers were supposed to write then, and probably still are: short stories. My stories were awful. They had hardly any plot, just characters with strong feelings, which I imagined made them deep.\\n\\nThe first programs I tried writing were on the IBM 1401 that our school district used for what was then called \"data processing.\" This was in 9th grade, so I was 13 or 14. The school district\\'s 1401 happened to be in the basement of our junior high school, and my friend Rich Draves and I got permission to use it. It was like a mini Bond villain\\'s lair down there, with all these alien-looking machines \u2014 CPU, disk drives, printer, card reader \u2014 sitting up on a raised floor under bright fluorescent lights.\\n\\nThe language we used was an early version of Fortran. You had to type programs on punch cards, then stack them in the card reader and press a button to load the program into memory and run it. The result would ordinarily be to print something on the spectacularly loud printer.\\n\\nI was puzzled by the 1401. I couldn\\'t figure out what to do with it. And in retrospect there\\'s not much I could have done with it. The only form of input to programs was data stored on punched cards, and I didn\\'t have any data stored on punched cards. The only other option was to do things that didn\\'t rely on any input, like calculate approximations of pi, but I didn\\'t know enough math to do anything interesting of that type. So I\\'m not surprised I can\\'t remember any programs I wrote, because they can\\'t have done much. My clearest memory is of the moment I learned it was possible for programs not to terminate, when one of mine didn\\'t. On a machine without time-sharing, this was a social as well as a technical error, as the data center manager\\'s expression made clear.\\n\\nWith microcomputers, everything changed.'\n\n\n\n\n```python\n# Second retrieved node\nresponse_vector.source_nodes[1].get_text()\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: ee2d687b-bbd5-49b2-b942-97336299b5be
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"ee2d687b-bbd5-49b2-b942-97336299b5be\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e90b4ace-2a01-47fb-b16e-61b6d6e24a42\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b38a3d1ace59358d62d1d822324929594e81f81511747f16addff899e9e565e2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"226546d4-6117-4a3b-82fc-26e94c1100ad\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f7241d41b5d21073ca5f7e71fe357b9337dde8ee99728302181191ae3738b522\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"fa37b546-43f2-44a9-8600-160e3f0efe0d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"c04fa934ab1372695a643ad2181d320704d196e8cbcd866226719624c27f30ca\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7074, \"end_char_idx\": 9296, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "doc_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "ref_doc_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "chroma:document": "```python\n# Second retrieved node\nresponse_vector.source_nodes[1].get_text()\n```\n\n\n\n\n    \"It felt like I was doing life right. I remember that because I was slightly dismayed at how novel it felt. The good news is that I had more moments like this over the next few years.\\n\\nIn the summer of 2016 we moved to England. We wanted our kids to see what it was like living in another country, and since I was a British citizen by birth, that seemed the obvious choice. We only meant to stay for a year, but we liked it so much that we still live there. So most of Bel was written in England.\\n\\nIn the fall of 2019, Bel was finally finished. Like McCarthy's original Lisp, it's a spec rather than an implementation, although like McCarthy's Lisp it's a spec expressed as code.\\n\\nNow that I could write essays again, I wrote a bunch about topics I'd had stacked up. I kept writing essays through 2020, but I also started to think about other things I could work on. How should I choose what to do? Well, how had I chosen what to work on in the past? I wrote an essay for myself to answer that question, and I was surprised how long and messy the answer turned out to be. If this surprised me, who'd lived it, then I thought perhaps it would be interesting to other people, and encouraging to those with similarly messy lives. So I wrote a more detailed version for others to read, and this is the last sentence of it.\\n\\n\\n\\n\\n\\n\\n\\n\\n\\nNotes\\n\\n[1] My experience skipped a step in the evolution of computers: time-sharing machines with interactive OSes. I went straight from batch processing to microcomputers, which made microcomputers seem all the more exciting.\\n\\n[2] Italian words for abstract concepts can nearly always be predicted from their English cognates (except for occasional traps like polluzione). It's the everyday words that differ. So if you string together a lot of abstract concepts with a few simple verbs, you can make a little Italian go a long way.\\n\\n[3] I lived at Piazza San Felice 4, so my walk to the Accademia went straight down the spine of old Florence: past the Pitti, across the bridge, past Orsanmichele, between the Duomo and the Baptistery, and then up Via Ricasoli to Piazza San Marco.\""}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: fa37b546-43f2-44a9-8600-160e3f0efe0d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"fa37b546-43f2-44a9-8600-160e3f0efe0d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e90b4ace-2a01-47fb-b16e-61b6d6e24a42\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b38a3d1ace59358d62d1d822324929594e81f81511747f16addff899e9e565e2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"ee2d687b-bbd5-49b2-b942-97336299b5be\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b581d58e34e242155afae629a949a342fbaa8652abd44be85f621b68a62a9336\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"dfc131c8-d27d-470b-a074-c2fd99bc14fe\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b73f12652763c10651caedcb96949757445bf1aea680b1f95df7293ebc735e5f\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9300, \"end_char_idx\": 12800, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "doc_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "ref_doc_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "chroma:document": "We have built a RAG pipeline and now need to evaluate its performance. We can assess our RAG system/query engine using LlamaIndex's core evaluation modules. Let's examine how to leverage these tools to quantify the quality of our retrieval-augmented generation system.\n\n## Evaluation\n\nEvaluation should serve as the primary metric for assessing your RAG application. It determines whether the pipeline will produce accurate responses based on the data sources and a range of queries.\n\nWhile it's beneficial to examine individual queries and responses at the start, this approach may become impractical as the volume of edge cases and failures increases. Instead, it may be more effective to establish a suite of summary metrics or automated evaluations. These tools can provide insights into overall system performance and indicate specific areas that may require closer scrutiny.\n\nIn a RAG system, evaluation focuses on two critical aspects:\n\n*   **Retrieval Evaluation:** This assesses the accuracy and relevance of the information retrieved by the system.\n*   **Response Evaluation:** This measures the quality and appropriateness of the responses generated by the system based on the retrieved information.\n\n#### Question-Context Pair Generation:\n\nFor the evaluation of a RAG system, it's essential to have queries that can fetch the correct context and subsequently generate an appropriate response. `LlamaIndex` offers a `generate_question_context_pairs` module specifically for crafting questions and context pairs which can be used in the assessment of the RAG system of both Retrieval and Response Evaluation. For more details on Question Generation, please refer to the [documentation](https://docs.llamaindex.ai/en/stable/examples/evaluation/QuestionGeneration.html).\n\n\n```python\nqa_dataset = generate_question_context_pairs(\n    nodes,\n    llm=llm,\n    num_questions_per_chunk=2\n)\n```\n\n    100%|\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588\u2588| 58/58 [06:26<00:00,  6.67s/it]\n\n\n#### Retrieval Evaluation:\n\nWe are now prepared to conduct our retrieval evaluations. We will execute our `RetrieverEvaluator` using the evaluation dataset we have generated.\n\nWe first create the `Retriever` and then define two functions: `get_eval_results`, which operates our retriever on the dataset, and `display_results`, which presents the outcomes of the evaluation.\n\nLet's create the retriever.\n\n\n```python\nretriever = vector_index.as_retriever(similarity_top_k=2)\n```\n\nDefine `RetrieverEvaluator`. We use **Hit Rate** and **MRR** metrics to evaluate our Retriever.\n\n**Hit Rate:**\n\nHit rate calculates the fraction of queries where the correct answer is found within the top-k retrieved documents. In simpler terms, it\u2019s about how often our system gets it right within the top few guesses.\n\n**Mean Reciprocal Rank (MRR):**\n\nFor each query, MRR evaluates the system\u2019s accuracy by looking at the rank of the highest-placed relevant document. Specifically, it\u2019s the average of the reciprocals of these ranks across all the queries. So, if the first relevant document is the top result, the reciprocal rank is 1; if it\u2019s second, the reciprocal rank is 1/2, and so on.\n\nLet's check these metrics to check the performance of out retriever.\n\n\n```python\nretriever_evaluator = RetrieverEvaluator.from_metric_names(\n    [\"mrr\", \"hit_rate\"], retriever=retriever\n)\n```\n\n\n```python\n# Evaluate\neval_results = await retriever_evaluator.aevaluate_dataset(qa_dataset)\n```\n\nLet's define a function to display the Retrieval evaluation results in table format."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: dfc131c8-d27d-470b-a074-c2fd99bc14fe
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"dfc131c8-d27d-470b-a074-c2fd99bc14fe\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e90b4ace-2a01-47fb-b16e-61b6d6e24a42\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b38a3d1ace59358d62d1d822324929594e81f81511747f16addff899e9e565e2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"fa37b546-43f2-44a9-8600-160e3f0efe0d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"de885a81ee2ce481634f514b6f53a520d611e03ee0e98efa088b90a41d32af3d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"76bfa6ea-1be2-4371-bab9-f1090d05996a\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"f0c32c09851d0ec78ec39e7758ebb04c7f7983d69d4700d7061f846cfd345133\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 11252, \"end_char_idx\": 13969, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "doc_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "ref_doc_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "chroma:document": "#### Retrieval Evaluation:\n\nWe are now prepared to conduct our retrieval evaluations. We will execute our `RetrieverEvaluator` using the evaluation dataset we have generated.\n\nWe first create the `Retriever` and then define two functions: `get_eval_results`, which operates our retriever on the dataset, and `display_results`, which presents the outcomes of the evaluation.\n\nLet's create the retriever.\n\n\n```python\nretriever = vector_index.as_retriever(similarity_top_k=2)\n```\n\nDefine `RetrieverEvaluator`. We use **Hit Rate** and **MRR** metrics to evaluate our Retriever.\n\n**Hit Rate:**\n\nHit rate calculates the fraction of queries where the correct answer is found within the top-k retrieved documents. In simpler terms, it\u2019s about how often our system gets it right within the top few guesses.\n\n**Mean Reciprocal Rank (MRR):**\n\nFor each query, MRR evaluates the system\u2019s accuracy by looking at the rank of the highest-placed relevant document. Specifically, it\u2019s the average of the reciprocals of these ranks across all the queries. So, if the first relevant document is the top result, the reciprocal rank is 1; if it\u2019s second, the reciprocal rank is 1/2, and so on.\n\nLet's check these metrics to check the performance of out retriever.\n\n\n```python\nretriever_evaluator = RetrieverEvaluator.from_metric_names(\n    [\"mrr\", \"hit_rate\"], retriever=retriever\n)\n```\n\n\n```python\n# Evaluate\neval_results = await retriever_evaluator.aevaluate_dataset(qa_dataset)\n```\n\nLet's define a function to display the Retrieval evaluation results in table format.\n\n\n```python\ndef display_results(name, eval_results):\n    \"\"\"Display results from evaluate.\"\"\"\n\n    metric_dicts = []\n    for eval_result in eval_results:\n        metric_dict = eval_result.metric_vals_dict\n        metric_dicts.append(metric_dict)\n\n    full_df = pd.DataFrame(metric_dicts)\n\n    hit_rate = full_df[\"hit_rate\"].mean()\n    mrr = full_df[\"mrr\"].mean()\n\n    metric_df = pd.DataFrame(\n        {\"Retriever Name\": [name], \"Hit Rate\": [hit_rate], \"MRR\": [mrr]}\n    )\n\n    return metric_df\n```\n\n\n```python\ndisplay_results(\"OpenAI Embedding Retriever\", eval_results)\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Retriever Name</th>\n      <th>Hit Rate</th>\n      <th>MRR</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>OpenAI Embedding Retriever</td>\n      <td>0.758621</td>\n      <td>0.62069</td>\n    </tr>\n  </tbody>\n</table>\n</div>"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 76bfa6ea-1be2-4371-bab9-f1090d05996a
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"76bfa6ea-1be2-4371-bab9-f1090d05996a\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e90b4ace-2a01-47fb-b16e-61b6d6e24a42\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b38a3d1ace59358d62d1d822324929594e81f81511747f16addff899e9e565e2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"dfc131c8-d27d-470b-a074-c2fd99bc14fe\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"f282f79be69cc3aec08ac21252e066feaeaec8bf4899f4f3ff0eb6cf106382e9\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"05691b43-3b22-4883-aedb-01ca04e7fedf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"549106f8cb76e64f06462e529b22bf76d9967224215f05a7db7b7fad0c367dee\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12619, \"end_char_idx\": 15282, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "doc_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "ref_doc_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "chroma:document": "```python\n# Evaluate\neval_results = await retriever_evaluator.aevaluate_dataset(qa_dataset)\n```\n\nLet's define a function to display the Retrieval evaluation results in table format.\n\n\n```python\ndef display_results(name, eval_results):\n    \"\"\"Display results from evaluate.\"\"\"\n\n    metric_dicts = []\n    for eval_result in eval_results:\n        metric_dict = eval_result.metric_vals_dict\n        metric_dicts.append(metric_dict)\n\n    full_df = pd.DataFrame(metric_dicts)\n\n    hit_rate = full_df[\"hit_rate\"].mean()\n    mrr = full_df[\"mrr\"].mean()\n\n    metric_df = pd.DataFrame(\n        {\"Retriever Name\": [name], \"Hit Rate\": [hit_rate], \"MRR\": [mrr]}\n    )\n\n    return metric_df\n```\n\n\n```python\ndisplay_results(\"OpenAI Embedding Retriever\", eval_results)\n```\n\n\n\n\n<div>\n<style scoped>\n    .dataframe tbody tr th:only-of-type {\n        vertical-align: middle;\n    }\n\n    .dataframe tbody tr th {\n        vertical-align: top;\n    }\n\n    .dataframe thead th {\n        text-align: right;\n    }\n</style>\n<table border=\"1\" class=\"dataframe\">\n  <thead>\n    <tr style=\"text-align: right;\">\n      <th></th>\n      <th>Retriever Name</th>\n      <th>Hit Rate</th>\n      <th>MRR</th>\n    </tr>\n  </thead>\n  <tbody>\n    <tr>\n      <th>0</th>\n      <td>OpenAI Embedding Retriever</td>\n      <td>0.758621</td>\n      <td>0.62069</td>\n    </tr>\n  </tbody>\n</table>\n</div>\n\n\n\n#### Observation:\n\nThe Retriever with OpenAI Embedding  demonstrates a performance with a hit rate of `0.7586`, while the MRR, at `0.6206`, suggests there's room for improvement in ensuring the most relevant results appear at the top. The observation that MRR is less than the hit rate indicates that the top-ranking results aren't always the most relevant. Enhancing MRR could involve the use of rerankers, which refine the order of retrieved documents. For a deeper understanding of how rerankers can optimize retrieval metrics, refer to the detailed discussion in our [blog post](https://blog.llamaindex.ai/boosting-rag-picking-the-best-embedding-reranker-models-42d079022e83).\n\n#### Response Evaluation:\n\n1. FaithfulnessEvaluator: Measures if the response from a query engine matches any source nodes which is useful for measuring if the response is hallucinated.\n2. Relevancy Evaluator: Measures if the response + source nodes match the query.\n\n\n```python\n# Get the list of queries from the above created dataset\n\nqueries = list(qa_dataset.queries.values())\n```\n\n#### Faithfulness Evaluator\n\nLet's start with FaithfulnessEvaluator.\n\nWe will use `gpt-3.5-turbo` for generating response for a given query and `gpt-4` for evaluation.\n\nLet's create service_context seperately for `gpt-3.5-turbo` and `gpt-4`."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 05691b43-3b22-4883-aedb-01ca04e7fedf
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"05691b43-3b22-4883-aedb-01ca04e7fedf\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e90b4ace-2a01-47fb-b16e-61b6d6e24a42\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b38a3d1ace59358d62d1d822324929594e81f81511747f16addff899e9e565e2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"76bfa6ea-1be2-4371-bab9-f1090d05996a\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"8aba12e49e80298de7671f3db595bcb7e4563ccc3dda0d9d88958f1d5554c8a2\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"cc8a4d15-69c0-4108-8efe-3effb3c3c600\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"4c2c05e99bc0e5d6ed09c4eb42b8b5e8d04fb4adfcd0cbafe1145c8497b96755\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13973, \"end_char_idx\": 17109, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "doc_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "ref_doc_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "chroma:document": "#### Observation:\n\nThe Retriever with OpenAI Embedding  demonstrates a performance with a hit rate of `0.7586`, while the MRR, at `0.6206`, suggests there's room for improvement in ensuring the most relevant results appear at the top. The observation that MRR is less than the hit rate indicates that the top-ranking results aren't always the most relevant. Enhancing MRR could involve the use of rerankers, which refine the order of retrieved documents. For a deeper understanding of how rerankers can optimize retrieval metrics, refer to the detailed discussion in our [blog post](https://blog.llamaindex.ai/boosting-rag-picking-the-best-embedding-reranker-models-42d079022e83).\n\n#### Response Evaluation:\n\n1. FaithfulnessEvaluator: Measures if the response from a query engine matches any source nodes which is useful for measuring if the response is hallucinated.\n2. Relevancy Evaluator: Measures if the response + source nodes match the query.\n\n\n```python\n# Get the list of queries from the above created dataset\n\nqueries = list(qa_dataset.queries.values())\n```\n\n#### Faithfulness Evaluator\n\nLet's start with FaithfulnessEvaluator.\n\nWe will use `gpt-3.5-turbo` for generating response for a given query and `gpt-4` for evaluation.\n\nLet's create service_context seperately for `gpt-3.5-turbo` and `gpt-4`.\n\n\n```python\n# gpt-3.5-turbo\ngpt35 = OpenAI(temperature=0, model=\"gpt-3.5-turbo\")\nservice_context_gpt35 = ServiceContext.from_defaults(llm=gpt35)\n\n# gpt-4\ngpt4 = OpenAI(temperature=0, model=\"gpt-4\")\nservice_context_gpt4 = ServiceContext.from_defaults(llm=gpt4)\n```\n\nCreate a `QueryEngine` with `gpt-3.5-turbo` service_context to generate response for the query.\n\n\n```python\nvector_index = VectorStoreIndex(nodes, service_context = service_context_gpt35)\nquery_engine = vector_index.as_query_engine()\n```\n\nCreate a  FaithfulnessEvaluator.\n\n\n\n\n\n```python\nfrom llama_index.evaluation import FaithfulnessEvaluator\nfaithfulness_gpt4 = FaithfulnessEvaluator(service_context=service_context_gpt4)\n```\n\nLet's evaluate on one question.\n\n\n```python\neval_query = queries[10]\n\neval_query\n```\n\n\n\n\n    \"Based on the author's experience and observations, why did he consider the AI practices during his first year of grad school as a hoax? Provide specific examples from the text to support your answer.\"\n\n\n\nGenerate response first and use faithfull evaluator.\n\n\n```python\nresponse_vector = query_engine.query(eval_query)\n```\n\n\n```python\n# Compute faithfulness evaluation\n\neval_result = faithfulness_gpt4.evaluate_response(response=response_vector)\n```\n\n\n```python\n# You can check passing parameter in eval_result if it passed the evaluation.\neval_result.passing\n```\n\n\n\n\n    True\n\n\n\n#### Relevancy Evaluator\n\nRelevancyEvaluator is useful to measure if the response and source nodes (retrieved context) match the query. Useful to see if response actually answers the query.\n\nInstantiate `RelevancyEvaluator` for relevancy evaluation with `gpt-4`\n\n\n```python\nfrom llama_index.evaluation import RelevancyEvaluator\n\nrelevancy_gpt4 = RelevancyEvaluator(service_context=service_context_gpt4)\n```\n\nLet's do relevancy evaluation for one of the query."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: cc8a4d15-69c0-4108-8efe-3effb3c3c600
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"cc8a4d15-69c0-4108-8efe-3effb3c3c600\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e90b4ace-2a01-47fb-b16e-61b6d6e24a42\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b38a3d1ace59358d62d1d822324929594e81f81511747f16addff899e9e565e2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"05691b43-3b22-4883-aedb-01ca04e7fedf\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"439b6ed2e091a8ecdd1c2e2682515b8b15ff3ebce21a65b07e1e00842f77400c\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"309e1a45-bf44-417b-95d1-a3a0a5ece65c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"029bba2fa3cccf385dc9a81d85edb398af99c2381665be7e9461197e76b7cbc1\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 15646, \"end_char_idx\": 18901, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "doc_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "ref_doc_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "chroma:document": "```python\nvector_index = VectorStoreIndex(nodes, service_context = service_context_gpt35)\nquery_engine = vector_index.as_query_engine()\n```\n\nCreate a  FaithfulnessEvaluator.\n\n\n\n\n\n```python\nfrom llama_index.evaluation import FaithfulnessEvaluator\nfaithfulness_gpt4 = FaithfulnessEvaluator(service_context=service_context_gpt4)\n```\n\nLet's evaluate on one question.\n\n\n```python\neval_query = queries[10]\n\neval_query\n```\n\n\n\n\n    \"Based on the author's experience and observations, why did he consider the AI practices during his first year of grad school as a hoax? Provide specific examples from the text to support your answer.\"\n\n\n\nGenerate response first and use faithfull evaluator.\n\n\n```python\nresponse_vector = query_engine.query(eval_query)\n```\n\n\n```python\n# Compute faithfulness evaluation\n\neval_result = faithfulness_gpt4.evaluate_response(response=response_vector)\n```\n\n\n```python\n# You can check passing parameter in eval_result if it passed the evaluation.\neval_result.passing\n```\n\n\n\n\n    True\n\n\n\n#### Relevancy Evaluator\n\nRelevancyEvaluator is useful to measure if the response and source nodes (retrieved context) match the query. Useful to see if response actually answers the query.\n\nInstantiate `RelevancyEvaluator` for relevancy evaluation with `gpt-4`\n\n\n```python\nfrom llama_index.evaluation import RelevancyEvaluator\n\nrelevancy_gpt4 = RelevancyEvaluator(service_context=service_context_gpt4)\n```\n\nLet's do relevancy evaluation for one of the query.\n\n\n```python\n# Pick a query\nquery = queries[10]\n\nquery\n```\n\n\n\n\n    \"Based on the author's experience and observations, why did he consider the AI practices during his first year of grad school as a hoax? Provide specific examples from the text to support your answer.\"\n\n\n\n\n```python\n# Generate response.\n# response_vector has response and source nodes (retrieved context)\nresponse_vector = query_engine.query(query)\n\n# Relevancy evaluation\neval_result = relevancy_gpt4.evaluate_response(\n    query=query, response=response_vector\n)\n```\n\n\n```python\n# You can check passing parameter in eval_result if it passed the evaluation.\neval_result.passing\n```\n\n\n\n\n    True\n\n\n\n\n```python\n# You can get the feedback for the evaluation.\neval_result.feedback\n```\n\n\n\n\n    'YES'\n\n\n\n#### Batch Evaluator:\n\nNow that we have done FaithFulness and Relevancy Evaluation independently. LlamaIndex has `BatchEvalRunner` to compute multiple evaluations in batch wise manner.\n\n\n```python\nfrom llama_index.evaluation import BatchEvalRunner\n\n# Let's pick top 10 queries to do evaluation\nbatch_eval_queries = queries[:10]\n\n# Initiate BatchEvalRunner to compute FaithFulness and Relevancy Evaluation.\nrunner = BatchEvalRunner(\n    {\"faithfulness\": faithfulness_gpt4, \"relevancy\": relevancy_gpt4},\n    workers=8,\n)\n\n# Compute evaluation\neval_results = await runner.aevaluate_queries(\n    query_engine, queries=batch_eval_queries\n)\n```\n\n\n```python\n# Let's get faithfulness score\n\nfaithfulness_score = sum(result.passing for result in eval_results['faithfulness']) / len(eval_results['faithfulness'])\n\nfaithfulness_score\n```\n\n\n\n\n    1.0\n\n\n\n\n```python\n# Let's get relevancy score\n\nrelevancy_score = sum(result.passing for result in eval_results['relevancy']) / len(eval_results['relevancy'])\n\nrelevancy_score\n\n```\n\n\n\n\n    1.0"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 309e1a45-bf44-417b-95d1-a3a0a5ece65c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb", "title": "Evaluate RAG with LlamaIndex", "tokens": 4522, "retrieve_doc": true, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"309e1a45-bf44-417b-95d1-a3a0a5ece65c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"e90b4ace-2a01-47fb-b16e-61b6d6e24a42\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b38a3d1ace59358d62d1d822324929594e81f81511747f16addff899e9e565e2\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"cc8a4d15-69c0-4108-8efe-3effb3c3c600\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/evaluation/Evaluate_RAG_with_LlamaIndex.ipynb\", \"title\": \"Evaluate RAG with LlamaIndex\", \"tokens\": 4522, \"retrieve_doc\": true, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"bc5e4ed17e14a8e41ec26a70b4f9392f3058853b18c691eb89efbfe4fa560133\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 17176, \"end_char_idx\": 19579, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "doc_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "ref_doc_id": "e90b4ace-2a01-47fb-b16e-61b6d6e24a42", "chroma:document": "\"Based on the author's experience and observations, why did he consider the AI practices during his first year of grad school as a hoax? Provide specific examples from the text to support your answer.\"\n\n\n\n\n```python\n# Generate response.\n# response_vector has response and source nodes (retrieved context)\nresponse_vector = query_engine.query(query)\n\n# Relevancy evaluation\neval_result = relevancy_gpt4.evaluate_response(\n    query=query, response=response_vector\n)\n```\n\n\n```python\n# You can check passing parameter in eval_result if it passed the evaluation.\neval_result.passing\n```\n\n\n\n\n    True\n\n\n\n\n```python\n# You can get the feedback for the evaluation.\neval_result.feedback\n```\n\n\n\n\n    'YES'\n\n\n\n#### Batch Evaluator:\n\nNow that we have done FaithFulness and Relevancy Evaluation independently. LlamaIndex has `BatchEvalRunner` to compute multiple evaluations in batch wise manner.\n\n\n```python\nfrom llama_index.evaluation import BatchEvalRunner\n\n# Let's pick top 10 queries to do evaluation\nbatch_eval_queries = queries[:10]\n\n# Initiate BatchEvalRunner to compute FaithFulness and Relevancy Evaluation.\nrunner = BatchEvalRunner(\n    {\"faithfulness\": faithfulness_gpt4, \"relevancy\": relevancy_gpt4},\n    workers=8,\n)\n\n# Compute evaluation\neval_results = await runner.aevaluate_queries(\n    query_engine, queries=batch_eval_queries\n)\n```\n\n\n```python\n# Let's get faithfulness score\n\nfaithfulness_score = sum(result.passing for result in eval_results['faithfulness']) / len(eval_results['faithfulness'])\n\nfaithfulness_score\n```\n\n\n\n\n    1.0\n\n\n\n\n```python\n# Let's get relevancy score\n\nrelevancy_score = sum(result.passing for result in eval_results['relevancy']) / len(eval_results['relevancy'])\n\nrelevancy_score\n\n```\n\n\n\n\n    1.0\n\n\n\n#### Observation:\n\nFaithfulness score of `1.0` signifies that the generated answers contain no hallucinations and are entirely based on retrieved context.\n\nRelevancy score of `1.0` suggests that the answers generated are consistently aligned with the retrieved context and the queries.\n\n## Conclusion\n\nIn this notebook, we have explored how to build and evaluate a RAG pipeline using LlamaIndex, with a specific focus on evaluating the retrieval system and generated responses within the pipeline. \n\nLlamaIndex offers a variety of other evaluation modules as well, which you can explore further [here](https://docs.llamaindex.ai/en/stable/module_guides/evaluating/root.html)"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 15c4a4dc-2200-41e5-b41d-4c60cbdb0bff
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"15c4a4dc-2200-41e5-b41d-4c60cbdb0bff\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"379642c4-6259-407d-8bc7-20486f37e723\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"98759ab3f29100bf093ebd1d9dd7e4e2a5592f3db8faba9515e6b412b3ff62f6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"870bcff0-4d93-45eb-9978-4e36e947c310\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"55eed429aec6c27c9bdf5797d80594800ff9cf6a85932198ef18202f8ac12339\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 0, \"end_char_idx\": 2064, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "379642c4-6259-407d-8bc7-20486f37e723", "doc_id": "379642c4-6259-407d-8bc7-20486f37e723", "ref_doc_id": "379642c4-6259-407d-8bc7-20486f37e723", "chroma:document": "# Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\n\nThis notebook provides step by step instuctions on using Azure AI Search (f.k.a Azure Cognitive Search) as a vector database with OpenAI embeddings, then creating an Azure Function on top to plug into a Custom GPT in ChatGPT. \n\nThis can be a solution for customers looking to set up RAG infrastructure contained within Azure, and exposing it as an endpoint to integrate that with other platforms such as ChatGPT.\n\nAzure AI Search is a cloud search service that gives developers infrastructure, APIs, and tools for building a rich search experience over private, heterogeneous content in web, mobile, and enterprise applications. \n\nAzure Functions is a serverless compute service that runs event-driven code, automatically managing infrastructure, scaling, and integrating with other Azure services.\n\n## Prerequisites:\nFor the purposes of this exercise you must have the following:\n- Azure user with permission to create [Azure AI Search Service](https://learn.microsoft.com/azure/search/) and Azure Function Apps\n- Azure subscription ID and a resource group.\n- [OpenAI Key](https://platform.openai.com/account/api-keys) \n\n# Architecture\nBelow is a diagram of the architecture of this solution, which we'll walk through step-by-step.\n\n![azure-rag-architecture.png](../../../../images/azure-rag-architecture.png)\n\n\n> Note: This architecture pattern of vector data store + serverless functions can be extrapolated to other vector data stores. For example, if you would want to use something like Postgres within Azure, you'd change the [Configure Azure AI Search Settings](#configure-azure-ai-search-settings) step to set-up the requirements for Postgres, you'd modify the [Create Azure AI Vector Search](#create-azure-ai-vector-search) to create the database and table in Postgres instead, and you'd update the `function_app.py` code in this repository to query Postgres instead of Azure AI Search. The data preparation and creation of the Azure Function would stay consistent."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 870bcff0-4d93-45eb-9978-4e36e947c310
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"870bcff0-4d93-45eb-9978-4e36e947c310\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"379642c4-6259-407d-8bc7-20486f37e723\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"98759ab3f29100bf093ebd1d9dd7e4e2a5592f3db8faba9515e6b412b3ff62f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"15c4a4dc-2200-41e5-b41d-4c60cbdb0bff\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"079bfe227786117efb760d5f9f0f5fd8b38dd423e858494f2b3641237134388a\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"275a6b5b-9ad5-41ee-a533-ff68763eb0af\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"d5e5ff2e1959591ad2c62a929fa8af3de32e3f5cc13cb74fbd3d787c163e437e\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 1402, \"end_char_idx\": 4481, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "379642c4-6259-407d-8bc7-20486f37e723", "doc_id": "379642c4-6259-407d-8bc7-20486f37e723", "ref_doc_id": "379642c4-6259-407d-8bc7-20486f37e723", "chroma:document": "> Note: This architecture pattern of vector data store + serverless functions can be extrapolated to other vector data stores. For example, if you would want to use something like Postgres within Azure, you'd change the [Configure Azure AI Search Settings](#configure-azure-ai-search-settings) step to set-up the requirements for Postgres, you'd modify the [Create Azure AI Vector Search](#create-azure-ai-vector-search) to create the database and table in Postgres instead, and you'd update the `function_app.py` code in this repository to query Postgres instead of Azure AI Search. The data preparation and creation of the Azure Function would stay consistent. \n\n\n# Table of Contents:\n\n1. **[Setup of Environment](#set-up-environment)**\n    Setup environment by installing and importing the required libraries and configuring our Azure settings. Includes:\n     - [Install and Import Required Libraries](#install-and-import-required-libraries)\n     - [Configure OpenAI Settings](#configure-openai-settings)\n     - [Configure Azure AI Search Settings](#configure-azure-ai-search-settings)\n \n\n2. **[Prepare Data](#prepare-data)** Prepare the data for uploading by embedding the documents, as well as capturing additional metadata. We will use a subset of OpenAI's docs as example data for this.\n \n3. **[Create Azure AI Vector Search](#create-azure-ai-vector-search)** Create an Azure AI Vector Search and upload the data we've prepared. Includes:\n     - [Create Index](#create-index): Steps to create an index in Azure AI Search.\n     - [Upload Data](#upload-data): Instructions to upload data to Azure AI Search.\n     - [Test Search](#test-search): Steps to test the search functionality.\n \n4. **[Create Azure Function](#create-azure-function)** Create an Azure Function to interact with the Azure AI Vector Search. Includes:\n     - [Create Storage Account](#create-storage-account): Steps to create a storage account for the Azure Function.\n     - [Create Function App](#create-function-app): Instructions to create a function app in Azure.\n \n5. **[Input in a Custom GPT in ChatGPT](#input-in-a-custom-gpt-in-chatgpt)** Integrate the Azure Function with a Custom GPT in ChatGPT. Includes:\n     - [Create OpenAPI Spec](#create-openapi-spec): Steps to create an OpenAPI specification for the Azure Function.\n     - [Create GPT Instructions](#create-gpt-instructions): Instructions to create GPT-specific instructions for the integration.\n\n\n\n\n# Set up environment\nWe'll set up our environment by importing the required libraries and configuring our Azure settings.\n\n## Install and import required libraries\nWe categorize these libraries into standard Python libraries, third-party libraries, and Azure-related libraries for readability.\n\n\n```python\n! pip install -q wget\n! pip install -q azure-search-documents \n! pip install -q azure-identity\n! pip install -q openai\n! pip install -q azure-mgmt-search\n! pip install -q pandas\n! pip install -q azure-mgmt-resource \n! pip install -q azure-mgmt-storage\n! pip install -q pyperclip\n! pip install -q PyPDF2\n! pip install -q tiktoken\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 275a6b5b-9ad5-41ee-a533-ff68763eb0af
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"275a6b5b-9ad5-41ee-a533-ff68763eb0af\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"379642c4-6259-407d-8bc7-20486f37e723\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"98759ab3f29100bf093ebd1d9dd7e4e2a5592f3db8faba9515e6b412b3ff62f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"870bcff0-4d93-45eb-9978-4e36e947c310\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"89c3d858de809eca6242ff97533fad2e6da7d83fd1e27312f0cb5937f73c4bda\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"09ad5027-5f67-41b2-b20d-ae5632f29acf\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"fac34c7bfe3727c21a1238c42e099bb83f0148da3f07d7da94308f4af2a37770\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 3843, \"end_char_idx\": 6713, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "379642c4-6259-407d-8bc7-20486f37e723", "doc_id": "379642c4-6259-407d-8bc7-20486f37e723", "ref_doc_id": "379642c4-6259-407d-8bc7-20486f37e723", "chroma:document": "# Set up environment\nWe'll set up our environment by importing the required libraries and configuring our Azure settings.\n\n## Install and import required libraries\nWe categorize these libraries into standard Python libraries, third-party libraries, and Azure-related libraries for readability.\n\n\n```python\n! pip install -q wget\n! pip install -q azure-search-documents \n! pip install -q azure-identity\n! pip install -q openai\n! pip install -q azure-mgmt-search\n! pip install -q pandas\n! pip install -q azure-mgmt-resource \n! pip install -q azure-mgmt-storage\n! pip install -q pyperclip\n! pip install -q PyPDF2\n! pip install -q tiktoken\n```\n\n\n```python\n# Standard Libraries\nimport json  \nimport os\nimport platform\nimport subprocess\nimport csv\nfrom itertools import islice\nimport uuid\nimport shutil\nimport concurrent.futures\n\n# Third-Party Libraries\nimport pandas as pd\nfrom PyPDF2 import PdfReader\nimport tiktoken\nfrom dotenv import load_dotenv\nimport pyperclip\n\n# OpenAI Libraries (note we use OpenAI directly here, but you can replace with Azure OpenAI as needed)\nfrom openai import OpenAI\n\n# Azure Identity and Credentials\nfrom azure.identity import DefaultAzureCredential, InteractiveBrowserCredential\nfrom azure.core.credentials import AzureKeyCredential  \nfrom azure.core.exceptions import HttpResponseError\n\n# Azure Search Documents\nfrom azure.search.documents import SearchClient, SearchIndexingBufferedSender  \nfrom azure.search.documents.indexes import SearchIndexClient  \nfrom azure.search.documents.models import (\n    VectorizedQuery\n)\nfrom azure.search.documents.indexes.models import (\n    HnswAlgorithmConfiguration,\n    HnswParameters,\n    SearchField,\n    SearchableField,\n    SearchFieldDataType,\n    SearchIndex,\n    SimpleField,\n    VectorSearch,\n    VectorSearchAlgorithmKind,\n    VectorSearchAlgorithmMetric,\n    VectorSearchProfile,\n)\n\n# Azure Management Clients\nfrom azure.mgmt.search import SearchManagementClient\nfrom azure.mgmt.resource import ResourceManagementClient, SubscriptionClient\nfrom azure.mgmt.storage import StorageManagementClient\n```\n\n## Configure OpenAI settings\n\nBefore going through this section, make sure you have your OpenAI API key.\n\n\n\n```python\nopenai_api_key = os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as an env var>\") # Saving this as a variable to reference in function app in later step\nopenai_client = OpenAI(api_key=openai_api_key)\nembeddings_model = \"text-embedding-3-small\" # We'll use this by default, but you can change to your text-embedding-3-large if desired\n```\n\n## Configure Azure AI Search Settings\nYou can locate your Azure AI Search service details in the Azure Portal or programmatically via the [Search Management SDK](https://learn.microsoft.com/rest/api/searchmanagement/).\n\n\n#### Prerequisites:\n- Subscription ID from Azure\n- Resource Group name from Azure\n- Region in Azure"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 09ad5027-5f67-41b2-b20d-ae5632f29acf
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"09ad5027-5f67-41b2-b20d-ae5632f29acf\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"379642c4-6259-407d-8bc7-20486f37e723\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"98759ab3f29100bf093ebd1d9dd7e4e2a5592f3db8faba9515e6b412b3ff62f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"275a6b5b-9ad5-41ee-a533-ff68763eb0af\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"b0e2713b1f94251ba7cf0434415b038350f8af109ab58a12147b009bec514ef8\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"89d75a97-1d9b-4b28-ad49-5e88f414a6fa\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"8b072004b85a152337416fbe4fe776eee7fd5b9e4dad835a6a57bf19c37fd374\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 6026, \"end_char_idx\": 9267, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "379642c4-6259-407d-8bc7-20486f37e723", "doc_id": "379642c4-6259-407d-8bc7-20486f37e723", "ref_doc_id": "379642c4-6259-407d-8bc7-20486f37e723", "chroma:document": "```python\nopenai_api_key = os.environ.get(\"OPENAI_API_KEY\", \"<your OpenAI API key if not set as an env var>\") # Saving this as a variable to reference in function app in later step\nopenai_client = OpenAI(api_key=openai_api_key)\nembeddings_model = \"text-embedding-3-small\" # We'll use this by default, but you can change to your text-embedding-3-large if desired\n```\n\n## Configure Azure AI Search Settings\nYou can locate your Azure AI Search service details in the Azure Portal or programmatically via the [Search Management SDK](https://learn.microsoft.com/rest/api/searchmanagement/).\n\n\n#### Prerequisites:\n- Subscription ID from Azure\n- Resource Group name from Azure\n- Region in Azure\n\n\n```python\n# Update the below with your values\nsubscription_id=\"<enter_your_subscription_id>\"\nresource_group=\"<enter_your_resource_group>\"\n\n## Make sure to choose a region that supports the proper products. We've defaulted to \"eastus\" below. https://azure.microsoft.com/en-us/explore/global-infrastructure/products-by-region/#products-by-region_tab5\nregion = \"eastus\"\ncredential = InteractiveBrowserCredential()\nsubscription_client = SubscriptionClient(credential)\nsubscription = next(subscription_client.subscriptions.list())\n```\n\n#### Create and Configure Azure AI Search Service\nBelow we'll generate a unique name for the search service, set up the service properties, and create the search service.\n\n\n```python\n# Initialize the SearchManagementClient with the provided credentials and subscription ID\nsearch_management_client = SearchManagementClient(\n    credential=credential,\n    subscription_id=subscription_id,\n)\n\n# Generate a unique name for the search service using UUID, but you can change this if you'd like.\ngenerated_uuid = str(uuid.uuid4())\nsearch_service_name = \"search-service-gpt-demo\" + generated_uuid\n## The below is the default endpoint structure that is created when you create a search service. This may differ based on your Azure settings.\nsearch_service_endpoint = 'https://'+search_service_name+'.search.windows.net'\n\n# Create or update the search service with the specified parameters\nresponse = search_management_client.services.begin_create_or_update(\n    resource_group_name=resource_group,\n    search_service_name=search_service_name,\n    service={\n        \"location\": region,\n        \"properties\": {\"hostingMode\": \"default\", \"partitionCount\": 1, \"replicaCount\": 1},\n        # We are using the free pricing tier for this demo. You are only allowed one free search service per subscription.\n        \"sku\": {\"name\": \"free\"},\n        \"tags\": {\"app-name\": \"Search service demo\"},\n    },\n).result()\n\n# Convert the response to a dictionary and then to a pretty-printed JSON string\nresponse_dict = response.as_dict()\nresponse_json = json.dumps(response_dict, indent=4)\n\nprint(response_json)\nprint(\"Search Service Name:\" + search_service_name)\nprint(\"Search Service Endpoint:\" + search_service_endpoint)\n\n```\n\n#### Get the Search Service API Key\nNow that we have the search service up and running, we need the [Search Service API Key](https://learn.microsoft.com/en-us/azure/search/search-security-api-keys?tabs=rest-use,portal-find,portal-query), which we'll use to initiate the index creation, and later to execute the search."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 89d75a97-1d9b-4b28-ad49-5e88f414a6fa
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"89d75a97-1d9b-4b28-ad49-5e88f414a6fa\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"379642c4-6259-407d-8bc7-20486f37e723\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"98759ab3f29100bf093ebd1d9dd7e4e2a5592f3db8faba9515e6b412b3ff62f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"09ad5027-5f67-41b2-b20d-ae5632f29acf\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"31cd38309586f4776877492777214d8c31a1d9c8fa6f3bb03ae24e1dbe012776\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"8e60ec8c-ec17-4ff1-a6e0-ef9a6680e4b3\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"43ae46e9a96cffdd57e67dd4216650cd12021baa719055bc26b55c64691191a2\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 7420, \"end_char_idx\": 10245, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "379642c4-6259-407d-8bc7-20486f37e723", "doc_id": "379642c4-6259-407d-8bc7-20486f37e723", "ref_doc_id": "379642c4-6259-407d-8bc7-20486f37e723", "chroma:document": "```python\n# Initialize the SearchManagementClient with the provided credentials and subscription ID\nsearch_management_client = SearchManagementClient(\n    credential=credential,\n    subscription_id=subscription_id,\n)\n\n# Generate a unique name for the search service using UUID, but you can change this if you'd like.\ngenerated_uuid = str(uuid.uuid4())\nsearch_service_name = \"search-service-gpt-demo\" + generated_uuid\n## The below is the default endpoint structure that is created when you create a search service. This may differ based on your Azure settings.\nsearch_service_endpoint = 'https://'+search_service_name+'.search.windows.net'\n\n# Create or update the search service with the specified parameters\nresponse = search_management_client.services.begin_create_or_update(\n    resource_group_name=resource_group,\n    search_service_name=search_service_name,\n    service={\n        \"location\": region,\n        \"properties\": {\"hostingMode\": \"default\", \"partitionCount\": 1, \"replicaCount\": 1},\n        # We are using the free pricing tier for this demo. You are only allowed one free search service per subscription.\n        \"sku\": {\"name\": \"free\"},\n        \"tags\": {\"app-name\": \"Search service demo\"},\n    },\n).result()\n\n# Convert the response to a dictionary and then to a pretty-printed JSON string\nresponse_dict = response.as_dict()\nresponse_json = json.dumps(response_dict, indent=4)\n\nprint(response_json)\nprint(\"Search Service Name:\" + search_service_name)\nprint(\"Search Service Endpoint:\" + search_service_endpoint)\n\n```\n\n#### Get the Search Service API Key\nNow that we have the search service up and running, we need the [Search Service API Key](https://learn.microsoft.com/en-us/azure/search/search-security-api-keys?tabs=rest-use,portal-find,portal-query), which we'll use to initiate the index creation, and later to execute the search.\n\n\n```python\n# Retrieve the admin keys for the search service\ntry:\n    response = search_management_client.admin_keys.get(\n        resource_group_name=resource_group,\n        search_service_name=search_service_name,\n    )\n    # Extract the primary API key from the response and save as a variable to be used later\n    search_service_api_key = response.primary_key\n    print(\"Successfully retrieved the API key.\")\nexcept Exception as e:\n    print(f\"Failed to retrieve the API key: {e}\")\n```\n\n# Prepare data\nWe're going to embed and store a few pages of the OpenAI docs in the oai_docs folder. We'll first embed each, add it to a CSV, and then use that CSV to upload to the index.\n\nIn order to handle longer text files beyond the context of 8191 tokens, we can either use the chunk embeddings separately, or combine them in some way, such as averaging (weighted by the size of each chunk).\n\nWe will take a function from Python's own cookbook that breaks up a sequence into chunks."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 8e60ec8c-ec17-4ff1-a6e0-ef9a6680e4b3
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"8e60ec8c-ec17-4ff1-a6e0-ef9a6680e4b3\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"379642c4-6259-407d-8bc7-20486f37e723\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"98759ab3f29100bf093ebd1d9dd7e4e2a5592f3db8faba9515e6b412b3ff62f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"89d75a97-1d9b-4b28-ad49-5e88f414a6fa\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"2054fa8d68e8dc447dd8401b8d84ff4db67fafddbfaa52cc6a156fd808acfc6d\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"edebc805-a2d9-41ce-8443-9b47ef3a9e2d\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3d9749de9765f0c484cdf81bc66dc55f49e425a7732f978a7a12119851d90473\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 9270, \"end_char_idx\": 12536, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "379642c4-6259-407d-8bc7-20486f37e723", "doc_id": "379642c4-6259-407d-8bc7-20486f37e723", "ref_doc_id": "379642c4-6259-407d-8bc7-20486f37e723", "chroma:document": "```python\n# Retrieve the admin keys for the search service\ntry:\n    response = search_management_client.admin_keys.get(\n        resource_group_name=resource_group,\n        search_service_name=search_service_name,\n    )\n    # Extract the primary API key from the response and save as a variable to be used later\n    search_service_api_key = response.primary_key\n    print(\"Successfully retrieved the API key.\")\nexcept Exception as e:\n    print(f\"Failed to retrieve the API key: {e}\")\n```\n\n# Prepare data\nWe're going to embed and store a few pages of the OpenAI docs in the oai_docs folder. We'll first embed each, add it to a CSV, and then use that CSV to upload to the index.\n\nIn order to handle longer text files beyond the context of 8191 tokens, we can either use the chunk embeddings separately, or combine them in some way, such as averaging (weighted by the size of each chunk).\n\nWe will take a function from Python's own cookbook that breaks up a sequence into chunks.\n\n\n```python\ndef batched(iterable, n):\n    \"\"\"Batch data into tuples of length n. The last batch may be shorter.\"\"\"\n    # batched('ABCDEFG', 3) --> ABC DEF G\n    if n < 1:\n        raise ValueError('n must be at least one')\n    it = iter(iterable)\n    while (batch := tuple(islice(it, n))):\n        yield batch\n\n```\n\nNow we define a function that encodes a string into tokens and then breaks it up into chunks. We'll use tiktoken, a fast open-source tokenizer by OpenAI.\n\nTo read more about counting tokens with Tiktoken, check out [this cookbook](https://cookbook.openai.com/examples/how_to_count_tokens_with_tiktoken). \n\n\n\n\n```python\ndef chunked_tokens(text, chunk_length, encoding_name='cl100k_base'):\n    # Get the encoding object for the specified encoding name. OpenAI's tiktoken library, which is used in this notebook, currently supports two encodings: 'bpe' and 'cl100k_base'. The 'bpe' encoding is used for GPT-3 and earlier models, while 'cl100k_base' is used for newer models like GPT-4.\n    encoding = tiktoken.get_encoding(encoding_name)\n    # Encode the input text into tokens\n    tokens = encoding.encode(text)\n    # Create an iterator that yields chunks of tokens of the specified length\n    chunks_iterator = batched(tokens, chunk_length)\n    # Yield each chunk from the iterator\n    yield from chunks_iterator\n```\n\nFinally, we can write a function that safely handles embedding requests, even when the input text is longer than the maximum context length, by chunking the input tokens and embedding each chunk individually. The average flag can be set to True to return the weighted average of the chunk embeddings, or False to simply return the unmodified list of chunk embeddings.\n\n> Note: there are other, more sophisticated techniques you can take here, including:\n> - using GPT-4o to capture images/chart descriptions for embedding.\n> - keeping text overlap between the chunks to minimize cutting off important context.\n> - chunking based on paragraphs or sections.\n> - adding more descriptive metadata about each article.\n\n\n```python\n## Change the below based on model. The below is for the latest embeddings models from OpenAI, so you can leave as is unless you are using a different embedding model..\nEMBEDDING_CTX_LENGTH = 8191\nEMBEDDING_ENCODING='cl100k_base'\n```"}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: edebc805-a2d9-41ce-8443-9b47ef3a9e2d
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"edebc805-a2d9-41ce-8443-9b47ef3a9e2d\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"379642c4-6259-407d-8bc7-20486f37e723\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"98759ab3f29100bf093ebd1d9dd7e4e2a5592f3db8faba9515e6b412b3ff62f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"8e60ec8c-ec17-4ff1-a6e0-ef9a6680e4b3\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"79ca571432b6928aef471f406095e201a84ce6fb0eb2399b6756d2fd855215a0\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"419e1a72-9ef2-4e7a-81a3-50b356be494c\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"3fad5a39935ab46eecd10c3b745833bc1065f43e8c6abf8c64398510f00eede6\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 10870, \"end_char_idx\": 13962, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "379642c4-6259-407d-8bc7-20486f37e723", "doc_id": "379642c4-6259-407d-8bc7-20486f37e723", "ref_doc_id": "379642c4-6259-407d-8bc7-20486f37e723", "chroma:document": "```python\ndef chunked_tokens(text, chunk_length, encoding_name='cl100k_base'):\n    # Get the encoding object for the specified encoding name. OpenAI's tiktoken library, which is used in this notebook, currently supports two encodings: 'bpe' and 'cl100k_base'. The 'bpe' encoding is used for GPT-3 and earlier models, while 'cl100k_base' is used for newer models like GPT-4.\n    encoding = tiktoken.get_encoding(encoding_name)\n    # Encode the input text into tokens\n    tokens = encoding.encode(text)\n    # Create an iterator that yields chunks of tokens of the specified length\n    chunks_iterator = batched(tokens, chunk_length)\n    # Yield each chunk from the iterator\n    yield from chunks_iterator\n```\n\nFinally, we can write a function that safely handles embedding requests, even when the input text is longer than the maximum context length, by chunking the input tokens and embedding each chunk individually. The average flag can be set to True to return the weighted average of the chunk embeddings, or False to simply return the unmodified list of chunk embeddings.\n\n> Note: there are other, more sophisticated techniques you can take here, including:\n> - using GPT-4o to capture images/chart descriptions for embedding.\n> - keeping text overlap between the chunks to minimize cutting off important context.\n> - chunking based on paragraphs or sections.\n> - adding more descriptive metadata about each article.\n\n\n```python\n## Change the below based on model. The below is for the latest embeddings models from OpenAI, so you can leave as is unless you are using a different embedding model..\nEMBEDDING_CTX_LENGTH = 8191\nEMBEDDING_ENCODING='cl100k_base'\n```\n\n\n```python\ndef generate_embeddings(text, model):\n    # Generate embeddings for the provided text using the specified model\n    embeddings_response = openai_client.embeddings.create(model=model, input=text)\n    # Extract the embedding data from the response\n    embedding = embeddings_response.data[0].embedding\n    return embedding\n\ndef len_safe_get_embedding(text, model=embeddings_model, max_tokens=EMBEDDING_CTX_LENGTH, encoding_name=EMBEDDING_ENCODING):\n    # Initialize lists to store embeddings and corresponding text chunks\n    chunk_embeddings = []\n    chunk_texts = []\n    # Iterate over chunks of tokens from the input text\n    for chunk in chunked_tokens(text, chunk_length=max_tokens, encoding_name=encoding_name):\n        # Generate embeddings for each chunk and append to the list\n        chunk_embeddings.append(generate_embeddings(chunk, model=model))\n        # Decode the chunk back to text and append to the list\n        chunk_texts.append(tiktoken.get_encoding(encoding_name).decode(chunk))\n    # Return the list of chunk embeddings and the corresponding text chunks\n    return chunk_embeddings, chunk_texts\n```\n\nNext, we can define a helper function that will capture additional metadata about the documents. This is useful to use as a metadata filter for search queries, and capturing richer data for search. \n\nIn this example, I'll choose from a list of categories to use later on in a metadata filter."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 419e1a72-9ef2-4e7a-81a3-50b356be494c
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"419e1a72-9ef2-4e7a-81a3-50b356be494c\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"379642c4-6259-407d-8bc7-20486f37e723\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"98759ab3f29100bf093ebd1d9dd7e4e2a5592f3db8faba9515e6b412b3ff62f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"edebc805-a2d9-41ce-8443-9b47ef3a9e2d\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"d95e5f30ed74662d7575553d97d4022de5c04f6bc36c1ec174c0fc2752d515ef\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"6d028dab-4951-4395-9da8-77886f57dd41\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"b31bb4a6fc5a48e1591deb96ac6db9d49beb87daf6417a8ea1bb02b6065157ab\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 12293, \"end_char_idx\": 15261, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "379642c4-6259-407d-8bc7-20486f37e723", "doc_id": "379642c4-6259-407d-8bc7-20486f37e723", "ref_doc_id": "379642c4-6259-407d-8bc7-20486f37e723", "chroma:document": "```python\n## Change the below based on model. The below is for the latest embeddings models from OpenAI, so you can leave as is unless you are using a different embedding model..\nEMBEDDING_CTX_LENGTH = 8191\nEMBEDDING_ENCODING='cl100k_base'\n```\n\n\n```python\ndef generate_embeddings(text, model):\n    # Generate embeddings for the provided text using the specified model\n    embeddings_response = openai_client.embeddings.create(model=model, input=text)\n    # Extract the embedding data from the response\n    embedding = embeddings_response.data[0].embedding\n    return embedding\n\ndef len_safe_get_embedding(text, model=embeddings_model, max_tokens=EMBEDDING_CTX_LENGTH, encoding_name=EMBEDDING_ENCODING):\n    # Initialize lists to store embeddings and corresponding text chunks\n    chunk_embeddings = []\n    chunk_texts = []\n    # Iterate over chunks of tokens from the input text\n    for chunk in chunked_tokens(text, chunk_length=max_tokens, encoding_name=encoding_name):\n        # Generate embeddings for each chunk and append to the list\n        chunk_embeddings.append(generate_embeddings(chunk, model=model))\n        # Decode the chunk back to text and append to the list\n        chunk_texts.append(tiktoken.get_encoding(encoding_name).decode(chunk))\n    # Return the list of chunk embeddings and the corresponding text chunks\n    return chunk_embeddings, chunk_texts\n```\n\nNext, we can define a helper function that will capture additional metadata about the documents. This is useful to use as a metadata filter for search queries, and capturing richer data for search. \n\nIn this example, I'll choose from a list of categories to use later on in a metadata filter.\n\n\n```python\n## These are the categories I will be using for the categorization task. You can change these as needed based on your use case.\ncategories = ['authentication','models','techniques','tools','setup','billing_limits','other']\n\ndef categorize_text(text, categories):\n    # Create a prompt for categorization\n    messages = [\n        {\"role\": \"system\", \"content\": f\"\"\"You are an expert in LLMs, and you will be given text that corresponds to an article in OpenAI's documentation.\n         Categorize the document into one of these categories: {', '.join(categories)}. Only respond with the category name and nothing else.\"\"\"},\n        {\"role\": \"user\", \"content\": text}\n    ]\n    try:\n        # Call the OpenAI API to categorize the text\n        response = openai_client.chat.completions.create(\n            model=\"gpt-4o\",\n            messages=messages\n        )\n        # Extract the category from the response\n        category = response.choices[0].message.content\n        return category\n    except Exception as e:\n        print(f\"Error categorizing text: {str(e)}\")\n        return None\n```\n\nNow, we can define some helper functions to process the .txt files in the oai_docs folder within the data folder. You can use this with your own data as well and supports both .txt and .pdf files."}


created_at: 2024-07-28 17:45:58
topic: persistent://default/default/308e7593-7e45-4080-bc68-ad4f88ce5b4b
id: 6d028dab-4951-4395-9da8-77886f57dd41
encoding: FLOAT32
metadata:
{"url": "https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb", "title": "Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT", "tokens": 8713, "retrieve_doc": false, "source": "OpenAI_Cookbooks", "_node_content": "{\"id_\": \"6d028dab-4951-4395-9da8-77886f57dd41\", \"embedding\": null, \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"excluded_embed_metadata_keys\": [\"url\", \"tokens\", \"retrieve_doc\", \"source\"], \"excluded_llm_metadata_keys\": [\"title\", \"tokens\", \"retrieve_doc\", \"source\"], \"relationships\": {\"1\": {\"node_id\": \"379642c4-6259-407d-8bc7-20486f37e723\", \"node_type\": \"4\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"98759ab3f29100bf093ebd1d9dd7e4e2a5592f3db8faba9515e6b412b3ff62f6\", \"class_name\": \"RelatedNodeInfo\"}, \"2\": {\"node_id\": \"419e1a72-9ef2-4e7a-81a3-50b356be494c\", \"node_type\": \"1\", \"metadata\": {\"url\": \"https://github.com/openai/openai-cookbook/blob/main/examples/chatgpt/rag-quickstart/azure/Azure_AI_Search_with_Azure_Functions_and_GPT_Actions_in_ChatGPT.ipynb\", \"title\": \"Azure AI Search as a vector database + Azure Functions for GPT integration in ChatGPT\", \"tokens\": 8713, \"retrieve_doc\": false, \"source\": \"OpenAI_Cookbooks\"}, \"hash\": \"d3520ea6231e1850e9077054756ace32b5d8f92928bffae3a79181e6322391f6\", \"class_name\": \"RelatedNodeInfo\"}, \"3\": {\"node_id\": \"982bb43a-2c94-47a0-aebd-11d8e603ac1e\", \"node_type\": \"1\", \"metadata\": {}, \"hash\": \"5ab9ebcfaaf610da64bfd282c681175b718da1a6aab53fad94b53ae79559cad4\", \"class_name\": \"RelatedNodeInfo\"}}, \"text\": \"\", \"mimetype\": \"text/plain\", \"start_char_idx\": 13965, \"end_char_idx\": 17109, \"text_template\": \"{metadata_str}\\n\\n{content}\", \"metadata_template\": \"{key}: {value}\", \"metadata_seperator\": \"\\n\", \"class_name\": \"TextNode\"}", "_node_type": "TextNode", "document_id": "379642c4-6259-407d-8bc7-20486f37e723", "doc_id": "379642c4-6259-407d-8bc7-20486f37e723", "ref_doc_id": "379642c4-6259-407d-8bc7-20486f37e723", "chroma:document": "```python\n## These are the categories I will be using for the categorization task. You can change these as needed based on your use case.\ncategories = ['authentication','models','techniques','tools','setup','billing_limits','other']\n\ndef categorize_text(text, categories):\n    # Create a prompt for categorization\n    messages = [\n        {\"role\": \"system\", \"content\": f\"\"\"You are an expert in LLMs, and you will be given text that corresponds to an article in OpenAI's documentation.\n         Categorize the document into one of these categories: {', '.join(categories)}. Only respond with the category name and nothing else.\"\"\"},\n        {\"role\": \"user\", \"content\": text}\n    ]\n    try:\n        # Call the OpenAI API to categorize the text\n        response = openai_client.chat.completions.create(\n            model=\"gpt-4o\",\n            messages=messages\n        )\n        # Extract the category from the response\n        category = response.choices[0].message.content\n        return category\n    except Exception as e:\n        print(f\"Error categorizing text: {str(e)}\")\n        return None\n```\n\nNow, we can define some helper functions to process the .txt files in the oai_docs folder within the data folder. You can use this with your own data as well and supports both .txt and .pdf files.\n\n\n```python\ndef extract_text_from_pdf(pdf_path):\n    # Initialize the PDF reader\n    reader = PdfReader(pdf_path)\n    text = \"\"\n    # Iterate through each page in the PDF and extract text\n    for page in reader.pages:\n        text += page.extract_text()\n    return text\n\ndef process_file(file_path, idx, categories, embeddings_model):\n    file_name = os.path.basename(file_path)\n    print(f\"Processing file {idx + 1}: {file_name}\")\n    \n    # Read text content from .txt files\n    if file_name.endswith('.txt'):\n        with open(file_path, 'r', encoding='utf-8') as file:\n            text = file.read()\n    # Extract text content from .pdf files\n    elif file_name.endswith('.pdf'):\n        text = extract_text_from_pdf(file_path)\n    \n    title = file_name\n    # Generate embeddings for the title\n    title_vectors, title_text = len_safe_get_embedding(title, embeddings_model)\n    print(f\"Generated title embeddings for {file_name}\")\n    \n    # Generate embeddings for the content\n    content_vectors, content_text = len_safe_get_embedding(text, embeddings_model)\n    print(f\"Generated content embeddings for {file_name}\")\n    \n    category = categorize_text(' '.join(content_text), categories)\n    print(f\"Categorized {file_name} as {category}\")\n    \n    # Prepare the data to be appended\n    data = []\n    for i, content_vector in enumerate(content_vectors):\n        data.append({\n            \"id\": f\"{idx}_{i}\",\n            \"vector_id\": f\"{idx}_{i}\",\n            \"title\": title_text[0],\n            \"text\": content_text[i],\n            \"title_vector\": json.dumps(title_vectors[0]),  # Assuming title is short and has only one chunk\n            \"content_vector\": json.dumps(content_vector),\n            \"category\": category\n        })\n        print(f\"Appended data for chunk {i + 1}/{len(content_vectors)} of {file_name}\")\n    \n    return data"}


